SlowGuess/ACL_2020_OCR huggingface dataset:
Pratik Joshi* Sebastin Santy* Amar Budhiraja* Kalika Bali Monjit Choudhury
Microsoft Research, India
{t-prjos, t-sesan, amar.budhiraja, kalikab, monojitc} @microsoft.com
Language technologies contribute to promoting multilingualism and linguistic diversity around the world. However, only a very small number of the over 7000 languages of the world are represented in the rapidly evolving language technologies and applications. In this paper we look at the relation between the types of languages, resources, and their representation in NLP conferences to understand the trajectory that different languages have followed over time. Our quantitative investigation underlines the disparity between languages, especially in terms of their resources, and calls into question the "language agnostic" status of current models and systems. Through this paper, we attempt to convince the ACL community to prioritise the resolution of the predicaments highlighted here, so that no language is left behind.
1 The Questions
Languages X and Y are the official languages of two different countries; they have around 29M and 18M native speakers, and 2M and 5.5K Wikipedia articles, respectively. X is syntactically quite similar to English, though uses dimunitives and has grammatical gender. Y, on the other hand, has a different word order from English, and has a rare typological feature - generally it is a head-final language, but noun phrases are head-initial. It also features full and partial reduplication. 69 items on LDC and ELRA contain data in X, whereas for Y there are only 2 items. X boasts of some of the best online machine translation systems, whereas Y is supported by very few online MT systems and that too with far inferior translation quality. Figure 1 shows the number of papers in conferences (ACL, NAACL, EACL, EMNLP, LREC, WS) that
(a)  $\mathrm{ACL} + \mathrm{NAACL} + \mathrm{EACL} + \mathrm{EMNLP}$
(b) LREC + WS
Figure 1: Number of papers with mentions of  $\mathbf{X}$  and  $\mathbf{Y}$  language for two sets of conferences.
mention  $\mathbf{X}$  and  $\mathbf{Y}$  in the paper, across the years. As you can see, while  $\mathbf{X}$  has a steady and growing trend of research, our community has been mostly oblivious to  $\mathbf{Y}$ , until recently when some of the zero-shot learning papers have started mentioning it. Can you guess what  $\mathbf{X}$  and  $\mathbf{Y}$  are?
Regardless of whether you can guess the exact answer, most NLP researchers surely know of (and might even speak) several languages which are in the same boat as  $\mathbf{X}$ ; languages which have a large amount of resources and therefore access to the benefits of the current NLP breakthroughs, and languages like  $\mathbf{Y}$ ; those which lack resources and consequently the attention of the NLP community, despite having similar speaker base sizes and typologically diverse features.
You probably have come across the issue of extremely skewed distribution of resources across the world's languages before. You might also be aware of the fact that most of our NLP systems, which are typically declared language agnostic, are not truly so (Bender, 2011). The handful of languages on which NLP systems are trained and tested are often related and from the same geography, drawn from a few dominant language families, leading to a typological echo-chamber. As a result, a vast majority of typologically diverse linguistic phenomena are never seen by our NLP systems (Ponti et al., 2019).
* Authors contributed equally to the work. https://microsoft.github.io/linguisticiversity
6282
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Nevertheless, it would be prudent to re-examine these issues in the light of recent advances in deep learning. Neural systems, on one hand, require a lot more data for training than rule-based or traditional ML systems, creating a bigger technological divide between the Xs and Ys; yet, some of the most recent techniques on zero-shot learning of massively multilingual systems (Devlin et al., 2019; Conneau and Lample, 2019; Aharoni et al., 2019; Artetxe and Schwenk, 2019) bridge this gap by obliterating the need for large labeled datasets in all languages. Instead, they need only large unlabeled corpora across languages and labeled data in only some languages. Assuming that this approach can be taken to its promising end, how does the fate of different languages change?
We break down this complex prescient question into the following more tractable and quantifiable questions on Linguistic Diversity and Inclusion:
1. How many resources, labeled and unlabeled, are available across the World's languages? How does this distribution correlate to their number of native speakers? What can we expect to achieve today and in the near future for these languages?
2. Which typological features have current NLP systems been exposed to, and which typological features mostly remain unexplored by systems because we have hardly created any resources and conducted data-driven research in those languages?
3. As a community, how inclusive has ACL been in conducting and publishing research on various languages? In 1980s and early 90s, when large scale datasets were not the prime drivers of research, was the linguistic diversity of ACL higher than what it has been in 2000s and 2010s? Or has ACL become more inclusive and diverse over the years?
4. Does the amount of resource available in a language influence the research questions and the venue of publication? If so, how?
5. What role does an individual researcher, or a research community have to play in bridging the linguistic-resource divide?
In this paper, we take a multi-pronged quantitative approach to study and answer the aforementioned questions, presented in order, in the following five sections. One of the key findings of our study, to spill the beans a bit, is that the languages of the World can be broadly classified into 6 classes based on how much and what kind of resources they have; the languages in each class have followed a
distinct and different trajectory in the history of ACL, and some of the hitherto neglected classes of languages have more hope of coming to the forefront of NLP technology with the promised potential of zero-shot learning.
2 The Six Kinds of Languages
In order to summarize the digital status and 'richness' of languages in the context of data availability, we propose a taxonomy based on the number of language resources which exist for different languages. We frame the rest of our analyses based on this taxonomy and use it to emphasize the existence of such resource disparities.
2.1 Features
We design this taxonomy using two feature axes: number of unlabeled resources vs. number of labeled resources. Previous methods have mostly relied on supervised learning techniques which require labeled corpora. However, the advent of transfer learning methods have boosted the importance of unlabeled data: massively multilingual models such as mBERT use Wikipedia for pre-training, and then fine-tune on downstream NLP tasks. These features are suitable because the current NLP research is predominantly data-driven, and language inclusion depends on how much labeled or unlabeled data is available. We believe these features are sufficient for the taxonomical design as the required metadata is consistently available across all languages, whereas features such as number of hours required to collect data aren't available.
We treat each data resource as a fundamental unit, based on the assumption that the collection of one unit is proportional to a certain extent of effort being invested towards the resource improvement of that language. Moreover, this feature discretization is unambiguous and concrete. Other units such as the total number of datapoints across datasets can be misleading because different NLP tasks have different data requirements. For example, while Machine Translation (MT) models require datapoints to the order of millions (Koehn and Knowles, 2017) to perform competitively, competent models in Question Answering require around 100 thousand datapoints (Rajpurkar et al., 2016). Moreover, the unit of datapoints vary across different technologies (e.g. Speech data measured in hours, MT data measured in number of parallel sentences).
6283
Figure 2: Language Resource Distribution: The size of the gradient circle represents the number of languages in the class. The color spectrum VIBGYOR, represents the total speaker population size from low to high. Bounding curves used to demonstrate covered points by that language class.
2.2 Repositories
We focus our attention on the LDC catalog<sup>1</sup> and the ELRA Map<sup>2</sup> for labeled datasets. Although there are other repositories of data available online, we found it practical to treat these organized collections as a representation of labeled dataset availability. This way, we look at standardized datasets that have established data quality and consistency, and which have been used in prior work. There are strong efforts such as PanLex (Kamholz et al., 2014), which is a large lexical database of a wide range of languages being used for a lexical translator, and OLAC (Simons and Bird, 2003), which contains a range of information for different languages (e.g. text collections, audio recordings, and dictionaries). However, keeping within the purview of NLP datasets used in *CL conferences, we decided to focus on popular repositories such as the above-mentioned.
We look at Wikipedia pages as a measure for unlabeled data resources. With regards to language technologies, Wikipedia pages represent a strong source of unsupervised training data which are freely and easily accessible. In the perspective of digital resource availability, they are a comprehensive source of factual information and are accessed by a large, diverse set of online users.
2.3 Language Classes
Figure 2 is a visualization of the taxonomy. We find a set of distinct partitions which can be used
to categorize languages into 6 unique positions in the language resource 'race':
0 - The Left-Behinds These languages have been and are still ignored in the aspect of language technologies. With exceptionally limited resources, it will be a monumental, probably impossible effort to lift them up in the digital space. Unsupervised pre-training methods only make the 'poor poorer', since there is virtually no unlabeled data to use.
1 - The Scraping-Bys With some amount of unlabeled data, there is a possibility that they could be in a better position in the 'race' in a matter of years. However, this task will take a solid, organized movement that increases awareness about these languages, and also sparks a strong effort to collect labelled datasets for them, seeing as they have almost none.
2 - The Hopefuls With light at the end of the tunnel, these languages still fight on with their gasping breath. A small set of labeled datasets has been collected for these languages, meaning that there are researchers and language support communities which strive to keep them alive in the digital world. Promising NLP tools can be created for these languages a few years down the line.
3 - The Rising Stars Unsupervised pre-training has been an energy boost for these languages. With a strong web presence, there is a thriving cultural community online for them. However, they have been let down by insufficient efforts in labeled data collection. With the right steps, these languages can be very well off if they continue to ride the 'pre-training' wave.
4 - The Underdogs Powerful and capable, these languages pack serious amounts of resource 'firepower'. They have a large amount of unlabeled data, comparable to those possessed by the winners, and are only challenged by lesser amount of labeled data. With dedicated NLP communities conducting research on these languages, they have the potential to become winners and enjoy the fruits of 'digital superiority'.
5 - The Winners Running strong and fast, these languages have been in the lead for quite a while now, some longer than others. With a dominant online presence, there have been massive industrial and government investments in the development of resources and technologies for these languages. They are the quintessential rich-resource
$^{1}$ https://catalog.ldc.upenn.edu/
<sup>2</sup> http://catalog.elra.info/en-us/
Class	5 Example Languages	#Langs	#Speakers	% of Total Langs
0	Dahalo, Warlpiri, Popoloca, Wallisian, Bora	2191	1.2B	88.38%
1	Cherokee, Fijian, Greenlandic, Bhojpuri, Navajo	222	30M	5.49%
2	Zulu, Konkani, Lao, Maltese, Irish	19	5.7M	0.36%
3	Indonesian, Ukrainian, Cebuano, Afrikaans, Hebrew	28	1.8B	4.42%
4	Russian, Hungarian, Vietnamese, Dutch, Korean	18	2.2B	1.07%
5	English, Spanish, German, Japanese, French	7	2.5B	0.28%
Table 1: Number of languages, number of speakers, and percentage of total languages for each language class.
(a) LDC
(b) LRE
(c) Wikipedia
(d) Web
Figure 3: Plots of different available resources for different languages. Languages to the far right do not have a representation in the resource category. Languages annotated are: Class 0-Dahalo (Dh), Wallisian(Wl); Class 1-Bhojpuri (Bh), Greenlandic (Gr); Class 2-Lao (La), Zulu (Zu); Class 3- Bengali (Bn), Indonesian (In); Class 4-Korean (Ko), Italian (It); Class 5- English (En), Spanish (Es).
languages, reaping benefit from each state-of-the-art NLP breakthrough.
Some more information about the taxonomy is shown in Table 1. We also take 10 languages, and annotate their positions in Figure 3.
2.4 Findings
On your marks As can be seen in Figure 3, the Winners take pole position in all rankings, and Class 0 languages remain 'out of the race' with no representation in any resource. The Wikipedia distribution seems to be more fair for classes 1, 2, and 3 when compared to classes 4 and 5, whereas the Web distribution has a clear disparity.
Talk ain't cheap Looking at Table 1, we see that Class 0 contains the largest section of languages and represents  $15\%$  of all speakers across classes. Although there is a large chunk of speakers which converse with Class 5 languages, the lack of technological inclusion for different languages could draw native speakers away from Class 0 languages and towards Class 5, exacerbating the disparity.
3 Typology
Linguistic typology is a field which involves the classification of languages based on their structural and semantic properties. Large-scale efforts have led to the creation of a database of typological
features (Dryer and Haspelmath, 2013). Such documentation becomes important as there are barely any other classifications of similar scale. In the context of NLP research, there has been work indicating the effectiveness of injecting typological information to guide the design of models (Ponti et al., 2019). Also, transfer learning of resource-rich to resource-poor languages have been shown to work better if the respective languages contain similar typological features (Pires et al., 2019). We look at how skewed language resource availability leads to an under-representation of certain typological features, which may in turn cause zero-shot inference models to fail on NLP tasks for certain languages.
We look at the WALS data (Dryer and Haspelmath, 2013), which contains typological features for 2679 languages. There are a total of 192 typological features, with an average of 5.93 categories per feature. We take the languages in classes 0, 1, 2, all of which have limited or no data resources as compared to 3, 4, 5 and look at how many categories, across all features, exist in classes 0, 1, 2 but not 3, 4, 5. This comes to a total of 549 out of 1139 unique categories, with an average of 2.86 categories per feature being ignored. Typological features with the most and least 'ignored' categories are shown in Table 2.
To get an idea of what these typological 'exclu
6285
Feature	#Cat	#Lang
144E	23	38
144M	23	45
144F	22	48
144O	21	30
Feature	#Cat	#Lang
83A	0	1321
82A	0	1302
97A	0	1146
86A	0	1083
Table 2: Most and least 'ignored' typological features, the number of categories in each feature which have been ignored, and the number of languages which contain this feature.
Language	Class	#Speakers	‘Ignored’	Error
Amharic	2	22M	9	60.71
Breton	1	210k	7	83.50
Swahili	2	18M	8	45.64
Kabyle	1	5.6M	8	39.10
Table 3: Relevant examples of typologically 'excluded' languages. The error rate is that of English  $\rightarrow$  Language from Artetxe and Schwenk (2019).
sions' mean in the context of modern multilingual methods, we look at the specific languages which contain these 'excluded' categories in the respective features, and compare their performances in similarity search, from the results of Artetxe and Schwenk (2019). Table 3 shows some examples of how 'ignored' features have been difficult to deal with even when jointly training of all languages.
3.1 Findings
Far-reaching repercussions The most 'ignored' feature in Table 2, 144E (Multiple Negative Constructions in SVO Languages), is a rare feature, existing in only 38 languages over the world. These languages, however, are from various regions (e.g. Wolof, Icelandic, and Kilivila). Language tools in all these areas can be adversely affected without sufficient typological representation. On the other hand, common features such as 83A (Order of Object and Verb) are well represented with definite feature values for 1321 languages, ranging from English to Mundari.
Does it run in the family? Amharic, in Table 3, which among the Semitic family of languages, is the second most spoken language after Arabic (which has 300M speakers). However, it has 9 'ignored' typological features, whereas Arabic has none. This reflects in the error rate of English to Amharic (60.71), which is significantly worse compared to 7.8 for English to Arabic.
4 Conference-Language Inclusion
NLP conferences have a huge impact on how language resources and technologies are constructed. Exciting research in venues such as ACL, EMNLP, LREC have the ability to turn heads in both industry and government and have the potential to attract funds to a particular technology. Has the usage of a small set of resource-rich languages in such conferences led to a disparity, pushing the less represented to the bottom of the ladder in terms of research? We analyze the involvement of various languages in NLP research conferences over the years.
The ACL Anthology Corpus (ACL-ARC) (Bird et al., 2008) is the most extensively used dataset for analyzing trends in NLP research. This dataset contains PDFs, and parsed XMLs of Anthology papers. However, the latest versioned copy of ACL-ARC is till 2015 which makes it insufficient for analyzing trends in the most recent years. Moreover, paper data for non-ACL conferences such as LREC, COLING are absent from this dataset. In order to create a consistent data model, we augment this dataset by using Semantic Scholar's API and scraping ACL Anthology itself. Thus, we gather a consolidated dataset for 11 conferences which are relevant in judging global trends in NLP research. These include ACL, NAACL, EMNLP, EACL, COLING, LREC, CONLL, Workshops (WS) (all since 1990), SEMEVAL, TACL and CL Journals. We have attached the statistics of the dataset in Appendix A.
4.2.1 Language Occurrence Entropy
The primary step of measuring the language diversity and inclusion of a conference and their progress is to measure the usage of language in that conference over multiple iterations. One of the ways to do it is by using frequency-based techniques where we can measure the occurrence of languages in that iteration. However, it is not a unified measure which represents the nature of language distribution with a single number. To this end, we use entropy as our metric to measure language inclusivity of each conference. It efficiently captures the skew in the distribution of languages,
6286
(a)  $c = \mathbf{ACL}$
(f)  $c = \mathbf{CL}$
Figure 4: Language occurrence entropy over the years for different conferences  $(\{S\}^{c,y})$
(b)  $c = \mathbf{N}\mathbf{A}\mathbf{A}\mathbf{C}\mathbf{L}$
(g)  $c = \mathbf{WS}$
(c)  $c = \mathbf{EMNLP}$
(h)  $c = \mathrm{CONLL}$
(d)  $c = \mathbf{EACL}$
(i)  $c = \mathbf{S}\mathbf{E}\mathbf{M}\mathbf{E}\mathbf{A}\mathbf{L}$
(e)  $c = \mathrm{COLING}$
(j)  $c = \mathbf{LREC}$
thereby making the disparity in language usage more clearer. The language occurrence entropy is calculated as follows:
For a conference  $c$  held in year  $y$  having  $P$  papers, there exists a binary matrix  $\{M_{P \times L}\}^{c,y}$  where  $M_{ij}$  is 1 if  $i^{th}$  paper  $(\in P)$  mentions the  $j^{th}$  language  $(\in L)$ . Then the entropy  $\{S\}^{c,y}$  is:
$$
\left\{S _ {j} \right\} ^ {c, y} = \frac {1}{P} \sum_ {i = 1} ^ {P} \left\{M _ {i j} \right\} ^ {c, y}
$$
$$
\left\{S _ {j} ^ {\prime} \right\} ^ {c, y} = \frac {\left\{S _ {j} \right\} ^ {c , y}}{\sum_ {j = 1} ^ {L} \left\{S _ {j} \right\} ^ {c , y}} \tag {1}
$$
$$
\left\{S \right\} ^ {c, y} = - \sum_ {j = 1} ^ {L} \left\{S _ {j} ^ {\prime} \right\} ^ {c, y} l o g _ {e} \left\{S _ {j} ^ {\prime} \right\} ^ {c, y}
$$
where  $\{S_j\}^{c,y}$  is a array of length  $L$  accounting for number of papers in a specific language,  $\{S_j'\}^{c,y}$  is normalization done in order to get probability distribution for calculating entropy. In short, the higher the entropy, the more spread out is the distribution over the languages. The more peaked or skewed the distribution is, the lower is the entropy.
In Figure 4, we can observe the entropy  $S$  plotted for each  $c$  as a function of  $y$ .
4.2.2 Class-wise Mean Reciprocal Rank
To quantify the extent of inclusion of language classes from our taxonomy in different conferences, we employ class-wise Mean Reciprocal Rank (MRR) as a metric. This helps in determining the standing of each class in a conference. If the rank of the language  $(\mathrm{rank}_i)$  is ordered by the frequency of being mentioned in papers of a particular conference, and  $Q$  is the total number of queries
aka number of languages in each class, then:
$$
\mathrm {M R R} = \frac {1}{| Q |} \sum_ {i = 1} ^ {| Q |} \frac {1}{\operatorname {r a n k} _ {i}} \tag {2}
$$
Table 4 shows inverse mean reciprocal ranks of each category for a conference. The smaller the inverse MRR value, the more inclusive that conference is to that language class.
Conf / Class	0	1	2	3	4	5
ACL	725	372	157	63	20	3
CL	647	401	175	76	27	3
COLING	670	462	185	74	21	2
CONLL	836	576	224	64	16	3
EACL	839	514	195	63	15	3
EMNLP	698	367	172	67	19	3
LREC	811	261	104	45	13	2
NAACL	754	365	136	63	18	3
SEMEVAL	730	983	296	121	19	3
TACL	974	400	180	50	15	3
WS	667	293	133	59	15	3
Table 4: Class-wise (1/MRR) for each conference.
4.3 Findings
All-Inclusive Looking at the combined trends, both the entropy plots and the MRR figures suggest that LREC and WS have been the most inclusive across all categories and have been continuing to do so over the years.
A ray of hope With regards to the proceedings of ACL, EMNLP, NAACL, LREC, we note a marked spike in entropy in the 2010s, which is absent in
6287
other conferences. This might be due to the increased buzz surrounding cross-lingual techniques.
The later the merrier An interesting point to note is that conferences which started later have taken lessons from past in matters of language inclusion. While the earlier established conferences have continued to maintain interest in a particular underlying theme of research which may or may not favour multilingual systems. This can be observed in : COLING, ACL, EACL, EMNLP (order of their start dates).
Falling off the radar The taxonomical hierarchy is fairly evident when looking at the MRR table (Table 4) with class 5 coming within rank 2/3 and class 0 being 'left-behind' with average ranks ranging from 600 to 1000. While the dip in ranks is more forgiving for conferences such as LREC, WS, it is more stark in CONLL, TACL, SEMEVAL.
5 Entity Embedding Analysis