langid.py for better language modelling

Large corpora are crucial resources for building many statistical language technology systems, and the Web is a readilyavailable source of vast amounts of linguistic data from which to construct such corpora. Nevertheless, little research has considered how to best build corpora from the Web. In this study we consider the importance of language identification in Web corpus construction. Beginning with a Web crawl consisting of documents identified as English using a standard language identification tool, we build corpora of varying sizes both with, and without, further filtering of non-English documents with a stateof-the-art language identifier. We show that the perplexity of a standard English corpus is lower under a language model trained from a Web corpus built with this extra language identification step, demonstrating the importance of state-of-the-art language identification in Web corpus construction. 1 The need for large corpora Corpora are essential resources for building language technology (LT) systems for a variety of applications. For example, frequency estimates for n-grams — which can be used to build a language model, a key component of many contemporary LT systems — are typically derived from corpora. Furthermore, bigger corpora are typically better. Banko and Brill (2001) show that for a classification task central to many LT problems, performance increases as a variety of models are trained on increasingly large corpora. The Web is a source of vast amounts of linguistic data, and the need for large corpora has motivated a wide range of research into techniques for building corpora of various types from the Web (e.g., Baroni and Bernardini, 2004; Ferraresi et al., 2008; Kilgarriff et al., 2010; Murphy and Stemle, 2011). In stark contrast to manual corpus construction, such automatic methods enable large corpora to be built quickly and inexpensively. Moreover, large Web crawls have recently been produced which are readily-available to the LT community (e.g., ClueWeb091 and CommonCrawl2) and can easily be exploited to build corpora much larger than those currently available (and indeed Pomikalek et al. (2012) have already done so); based on the findings of Banko and Brill, such corpora could be exploited to improve LT systems. Despite the importance of large Web corpora, the issue of how to best derive a corpus from a Web crawl remains an open question. Once a large collection of documents is obtained (from, e.g., either a Web crawl or the results of issuing queries to a commercial search engine) they must be post-processed to remove non-linguistic document portions, for example, boilerplate text such as menus; filter unwanted content such as documents in languages other than that intended for the corpus, and spam; and finally remove deduplicate or near-duplicate documents or document portions to produce a corpus. Furthermore, this document post-processing can potentially have a tremendous impact on corpus quality (Kilgarriff, 2007). For example, if texts in languages other than the target language(s) are not reliably identified and removed, n-gram frequency estimates for the target language will be less accurate than they would otherwise be, potentially having a negative http://lemurproject.org/clueweb09/ http://commoncrawl.org/ Paul Cook and Marco Lui. 2012. langid.py for better language modelling. In Proceedings of Australasian Language Technology Association Workshop, pages 107−112. impact on LT systems trained on such a corpus. Similar problems are encountered with the presence of boilerplate text, and duplicate or nearduplicate documents or text segments. Although document post-processing is clearly important to corpus construction, little work has studied it directly, with the notable exception of CleanEval (Baroni et al., 2008), a shared task on cleaning webpages by removing boilerplate and markup. Liu and Curran (2006) and Versley and Panchenko (2012) compare Web corpora with standard corpora in task-based evaluations, but do not specifically consider the impact of document post-processing. Web corpus construction projects have tended to rely on readily-available tools, or simple heuristics, to accomplish this post-processing. This is not a criticism of these projects — their goals were to build useful language resources, not specifically to study the impact of document post-processing on corpora. Nevertheless, because of the immediate opportunities for improving LT by building larger Web corpora, and the importance of post-processing on the quality of the resulting corpora, there appear to be potential opportunities to improve LT by improving Web corpus construction methods. In this paper we consider the importance of language identification — which has already been shown to benefit other LT tasks (e.g., Alex et al., 2007) — in Web corpus construction. We build corpora of varying sizes from a readily-available Web crawl (the English portion of ClueWeb09) using a standard corpus construction methodology. This dataset contains only documents classified as English according to a commonly-used language identification tool (TEXTCAT). 3 We then produce versions of these corpora from which non-English documents according to a state-ofthe-art language identification tool (langid.py, Lui and Baldwin, 2012) are filtered. In this preliminary work, we measure the impact of language identification in a task-based evaluation. Specifically, we train language models on the Web corpora, and demonstrate that, for corpora built from equal amounts of crawl data, the perplexity of a standard (manually-constructed) corpus is lower under a language model trained on a corpus filtered using langid.py, than a model trained on a corpus without this filtering. http://odur.let. rug.nl/vannoord/ TextCat/ 2 Materials and methods This section describes the language identification tools, corpus construction methods, and language modelling approach used in this study. 2.1 Language identification The initial language identification for ClueWeb09 was performed using TEXTCAT, an implementation of the language identification method of Cavnar and Trenkle (1994),4 which is based on the relative frequencies of byte n-grams. The reported language identification precision is over 99.7% across all 10 languages in ClueWeb09. However, the method of Cavnar and Trenkle has been shown to perform poorly when applied to test data outside the domain of the training data (Lui and Baldwin, 2011), as was the case for ClueWeb09 where the training data was drawn from newswire and European parliament corpora. langid.py is an implementation of the method described in Lui and Baldwin (2011), which improves on the method of Cavnar and Trenkle (1994) in a number of ways; both classifiers are based on relative frequencies of byte n-grams, but langid.py uses a Naive Bayes classifier and cross-domain feature selection, allowing it to ignore non-linguistic content such as HTML, without the need to explicitly model such content. Lui and Baldwin (2012) show that langid.py significantly and systematically outperforms TEXTCAT on a number of domains, and we therefore use it in this study.