Unsupervised Clustering for Language Identification

The current state of the art in language identification comes from n-gram language models. While these can reach 99% accuracy (Hammarstrom, 2007), they have three major shortcomings. First, n-gram language models are supervised. They require substantial labeled training data in each language in order to be functional. For best results, this training data should also be in the same genre as the text ultimately to be identified. Second, n-grams are most reliable in, and the 99% accuracy cited above comes from, identification of longer texts. Other papers evaluating n-gram language models on shorter texts, specifically 6to 7-word documents, found accuracies closer to 60% (Vatanen, 2010). Finally, as access to the internet expands across the globe, members of more minority language groups are beginning to produce content. Languages are appearing on the internet for which training data is limited, often noisy, and not freely available in the form of a corpus. In order to get away from these issues with supervised language identification, for our project we will explore unsupervised language identification in a domain where all of the above problems for supervised identification become relevant. We chose to use Twitter as the source of data for our project because tweets are short, exist in a very large number of languages, and are very unlike any labeled corpus of training data. Twitter imposes a hard limit on each tweet of 140 characters, at which length word n-gram models will almost certainly have difficulties. The language used on Twitter is unlike most corpora in that it includes a large amount of netspeak This includes abbreviations such as u and lol, symbols like :), and intentional misspellings, for example when a user expresses that she is huuuuungry. In addition to these intentional differences from standard written language, users often write from phones or type quickly, leading to a higher rate of typos. All of these examples are drawn from English, but English is only one of many languages used on Twitter. And while English is unsurprisingly most common, and Portuguese in second place is also a major language with plenty of available data, the third most commonly used language on Twitter is Indonesian. Already at the third most-used language we find one that is not included in standard multilingual corpora, and of course the underrepresentation gets worse as we continue. While we could potentially solve all of these problems by paying annotators to label a new corpus of Twitter language and training classifiers from that new corpus, this approach requires that annotators re-label whenever a new language community makes its presence known and does not extend to other domains. Instead, we propose an unsupervised approach, which may not reach quite the same classification accuracy, but will instantly discover new languages, can be quickly adapted to new domains, and does not require human annotation. Previous work on unsupervised language identification has, like most language identification literature, focused on longer documents. The state of the art in this area is Chinese Whispers, an algorithm which clusters the word co-occurrence graph to identify languages. According to the paper in which it was introduced, Chinese Whispers achieved an F1 score over . 99 for separating a 7-lingual corpus. These results are comparable to supervised results without need for a labeled corpus. Much like supervised identification, however, the performance of the system decreases significantly when applied to the short, noisy data found on Twitter. For our project, we will attempt to improve the performance of Chinese Whispers on Twitter language identification.