Creating General-Purpose Corpora Using Automated Search Engine Queries

The Internet is a natural source of linguistic data, providing an abundance of texts of various types in a large number of languages. These texts are already in electronic form suitable for corpus studies, either as downloadable pages, or as a resource to be searched using search engines. On the other hand, large representative corpora of the size of the British National Corpus, BNC (Aston and Burnard, 1998) exist for very few languages, because they are expensive to build. They are absent even for major world languages, such as Chinese or French. Many ad-hoc text collections are available, but they are restricted in either their size or the variety of text types. Typically they are produced on the basis of out of copyright ction (such as Project Gutenberg) or newswire/newspaper texts that are available in large quantities and relatively easy to acquire from their publishers, e.g. the Reuters corpus for English (Rose and Stevenson, 2002), or the Gigaword corpora for Arabic, Chinese and English (Cieri and Liberman, 2002). News corpora are useful for many applications, such as development of gazeteers, parsing and word sense disambiguation algorithms, yet they cannot replace corpora representative for general language, such as the BNC, as the former re ect only the formal register of reporting news stories, while corpora that are claimed to be representative should include a variety of text types. Below we compare the language of news corpora against the language used in the BNC and the language derived from the Web. The comparison shows that the news corpora di er signi cantly from either representative or Internet corpora and cannot provide a window into modern language use in general. The usefulness of Web data is evidenced by numerous recent corpus studies based on the number of pages returned by Google for speci c queries, see many references to this research in Section 2 of Kilgarri and Grefenstette (2003). Some researchers in traditional linguistics also use data returned from Google as the basis for their research, cf. Robb (2003), Volk (2002). However, Google is a poor concordancer. It provides only limited context for results of queries, cannot be used for linguistically complex queries, such as searching for lemmas (as opposed to word forms), restricting the part of speech or specifying the distance between components in the query in less than crude ways. More importantly results are ordered according to their relevance to the topic of the query using page-rank considerations, not according to left or right context as it is often useful for corpus work. When two linguistic phenomena are compared on the basis on the number of results returned by Google, the counts cannot be trusted. For instance, Veronis (2005) analyses problems with the logic of Google output and shows (among other things) that a search like (Chirac OR Sarkozy) produces fewer results than a search for a single term in the OR expression. The problems with ordering the results and the amount of returned contexts have been addressed by several projects, such as KWICFinder (Fletcher, 2004) or WebCorp (Renouf, 2003), which rely on Altavista or Google queries, but present results in the form of traditional concordances. However, this does not solve the problems with counts, query language and richer linguistic information. The ideal solution for corpus linguists would be a Google-like engine adapted to linguistic criteria. Kilgarri (2001) discussed this idea under the name of D3CI (Distributed Data Distributed Collection Initiative),