Abstract: This paper evaluates large georeferenced corpora, taken from both web-crawled and social media sources, against ground-truth population and language-census datasets. The goal is to determine (i) which dataset best represents population demographics; (ii) in what parts of the world the datasets are most representative of actual populations; and (iii) how to weight the datasets to provide more accurate representations of underlying populations. The paper finds that the two datasets represent very different populations and that they correlate with actual populations with values of r=0.60 (social media) and r=0.49 (web-crawled). Further, Twitter data makes better predictions about the inventory of languages used in each country.

Introduction: In recent years there has been an increasing amount of research investigating the use of unstructured geographic information, such as user-generated social media content and textual web data, for geographical analysis. Such unstructured data sources can provide information about places that is difficult to measure through traditional sensor observations. Aggregated representations of places that are extracted from textual corpora, e.g., give us insight into what people feel and think about places, potentially providing a much richer understanding of human geography BIBREF0, BIBREF1. It can also give insight into the relationships between places and human behavior BIBREF2, BIBREF3. However, a recurring issue with this kind of big data, and user-generated content in general, is the question of how representative these data sets are compared to the populations that we wish to study BIBREF4. There exists little previous empirical work to establish how representative web corpora are with respect to different geographic regions of the world (c.f. BIBREF5, BIBREF6). In this paper we describe such a computational experiment using language identification models on two global-scale corpora. How well does language data represent both regional population densities and the social characteristics of regional populations? To answer this question, we collect and analyze two large global-scale datasets: web-crawled data from the Common Crawl (16.65 billion words) and social media data from Twitter (4.14 billion words). This paper evaluates demographic-type information that is derived from these datasets, information which traditionally has been collected using survey-instruments as part of a census. When labeled with a language identification model, this data provides a representation of both (i) how much language a particular country produces, a proxy for population density and (ii) the mix of languages used in a country, a proxy for population demographics. These corpus-based representations are compared against four ground-truth baselines. First, the UN country-level population estimates BIBREF7. Second, because not all populations have equal access to internet technologies, we use per capita GDP BIBREF8 and internet-usage statistics BIBREF9 to adjust raw populations. Third, the UN country-level census aggregations are used to represent what languages are used in each country BIBREF10 and, where these are not available, the World Factbook BIBREF11 estimations are used. The goal is to measure how well corpus-based representations correspond with each of these ground-truth, survey-based representations. Thus, we are not concerned at this point if the corpus-based representations are skewed or inaccurate in particular locations. Rather, the purpose is to measure how and where these datasets are skewed as a method for evaluating and improving future data collection methods. We can view this problem from two perspectives: 1) from a human geography perspective, is it possible to use global-scale corpora to understand characteristics of regional populations?, and 2) from the perspective of computational linguistics, is it possible to normalize corpora to proportionally represent diverse populations? For example, some countries (like the United States) and some languages (like English) dominate many datasets. Is it possible to systematically correct this imbalance? We begin by describing the corpora and how they were collected (Section 2) and the language identification model that is used to label them with language codes (Section 3). After looking at the frequency distribution of languages across the entire dataset (Section 4), we undertake a country-level evaluation of the datasets, first against population-density baselines (Section 5) and then against language-use baselines (Section 6).

Question: What countries and languages are represented in the datasets?
Answer: English, Spanish, Russian, Serbo-Croatian, Mandarin, German, French, Slovenian, Portuguese, Finnish, Bulgarian, Arabic, Indonesian, Latvian, Estonian, Slovak, Azerbaijani, Romanina, Icelandic, Italian, among others.