SlowGuess/ACL_2020_OCR huggingface dataset:
The measures discussed in the previous section signal at variance in acceptance of different languages at different NLP venues across time. However, there are usually multiple subtle factors which vanilla statistics fail to capture. Embeddings, on the other hand, have been found extensively useful in NLP tasks as they are able to learn relevant signals directly from the data and uncover these rather complex nuances. To this end, we propose a novel approach to jointly learn the representations of conferences, authors and languages, which we collectively term as entities. The proposed embedding method allows us to project these entities in the same space enabling us to effectively reveal patterns revolving around them.
5.1 Model
We define the following model to jointly learn the embeddings of entities such that entities which have similar contextual distributions should co-occur together. For example, for an author  $A$ , who works more extensively on language  $L_{i}$  than  $L_{j}$  and publishes more at conference  $C_{m}$  than at conference  $C_{n}$ , the embeddings of  $A$  would be closer  $L_{i}$  than  $L_{j}$  and  $C_{m}$  than  $C_{n}$ .
Given an entity and a paper associated with the entity, the learning task of the model is to predict  $K$  randomly sampled words from the title and the abstract of the paper. We only select the title and abstract as compared to the entire paper text as
Figure 5: Model architecture to learn entity embeddings.  $W_{E \times N}$  is the weight matrix from input layer (entity layer) to the hidden layer, and  $W_{N \times V}$  is the weight matrix for the hidden layer to output layer computation. At the end of training,  $W_{E \times N}$  is the matrix containing embeddings of entities and  $W_{N \times V}$  is the matrix containing the embeddings of words.
they provide a concise signal with reduced noise. This model draws parallels to the Skipgram model of Word2Vec (Mikolov et al., 2013), where given an input word in Skipgram model, the task is to predict the context around the word. The input entity and  $K$  randomly sampled words in our case correspond to the input word and context in the Skipgram model. The goal of the model is to maximize probability of predicting the random  $K$  words, given the entity id as the input:
$$
\frac {1}{M} \frac {1}{K} \sum_ {m = 1} ^ {M} \sum_ {k = 1} ^ {K} \sum_ {i = 1} ^ {I} p \left(w _ {k} \mid E _ {<   i, P _ {j} >}\right) \tag {3}
$$
where  $E_{<i,P_j>}^{\text{th}}$  is the entity  $E_i$  which is associated with the  $P_j^{th}$  paper and  $p$  is the probability of predicting the word  $w_i$  out of the  $K$  words sampled from the paper and  $M$  is the total number of papers in the dataset. To optimize for the above distribution, we define the typical SGD based learning strategy similar to Word2Vec(Mikolov et al., 2013).
Figure 5 shows an outline of the model. The entity input layer has dimension equal to the total number of entities in the dataset  $(E)$ . Hidden layer size is set to the desired embedding dimension  $(N)$ . The output layer predicts words for the input entity and is of the same size as the vocabulary  $(V)$ . The entities we learn are: (1) authors of the paper, (2) languages mentioned in the paper, (3) conference where the paper was accepted (e.g. ACL), and (4) the conference iteration (e.g. ACL'19). We describe the model detail and hyperparameter tuning in Appendix A.
6288
Figure 6: t-SNE visualization of the learnt conference and language embeddings.
Class	MRR(5)	MRR(10)	MRR(15)	MRR(20)
0	0.72281	0.69146	0.63852	0.57441
1	0.57210	0.52585	0.45354	0.40904
2	0.47039	0.45265	0.41521	0.38157
3	0.59838	0.52670	0.45131	0.42899
4	0.56016	0.47795	0.51199	0.50681
5	0.56548	0.51471	0.54326	0.47619
Table 5: Language-Author-Language MRR on Taxonomy Classes. MRR(K) considers the closest K authors.
In order to better understand how languages are represented at different venues, we visualize the distribution of entity embeddings by projecting the generated embeddings into 2 dimensions using t-SNE (Maaten and Hinton, 2008) (as shown in Figure 6). For clarity, we only plot ACL, LREC, WS and CL among the conferences, and all languages from the taxonomy, except those in Class 0. We omit plotting Class 0 languages as their projections are noisy and scattered due to their infrequent occurrence in papers.
To understand the research contributions of individual authors or communities towards research in respective language classes, we leverage the distribution between author and language entities by computing a variation of the Mean Reciprocal Rank (MRR). We consider a language  $\mathbf{L}$ , and take the  $\mathbf{K}$  closest authors to  $\mathbf{L}$  using cosine distance, and then take the closest  $\mathbf{M}$  languages to each author. If  $\mathbf{L}$  is present in the closest languages of an author, then we take the rank of  $\mathbf{L}$  in that list, inverse it, and average it for the  $\mathbf{K}$  authors. To compute this
metric for a class of languages from the taxonomy, we take the mean of the MRR for all languages in that class. We fix  $\mathbf{M}$  to be 20, so as to understand the impact of the community when the number of languages remains unchanged. Table 5 shows the MRR of various class of languages. A higher value of this measure indicates a more focused community working on that particular language, rather than a diverse range of authors.
5.3 Findings
Time waits for no conference We can see a left to right trend in Figure 6 with ACL in 1983 in the left, and subsequent iterations laid out as we go right. We observe the same trend for EACL, NAACL, EMNLP, CONLL, TACL, and COLING. We can say that the axis represents the progression of time to a certain extent. Alternatively, it may even represent a shift in the focus of NLP research, moving from theoretical research focused on grammar and formalisms on the left to a data-driven, more ML-oriented approach on the right. This can be observed as most of the CL embeddings are positioned on the left given their theoretical research focus.
Long distance relationships? From Figure 6, we can note that the less-resourced language classes are farther away from the trend-line of ACL than the more resourced ones, with class 5 being closest, and class 1 being farthest. The visualization illustrates that languages are spreading out radially downwards from the ACL trendline with popular classes of taxonomy like class 5 and class 4 being closer while others spreading out farther. Again, as
6289
previous analyses have shown us, LREC and WS embeddings are closer to the language embeddings as compared to the other conferences as shown in Figure 6. In fact, LREC cluster is right in the middle of language clusters and so is the major part of the WS cluster, especially in recent iterations.
Not all heroes wear capes Table 5 shows the MRR for each class of languages in the taxonomy. From Table 5, it can be seen that class 0 has the highest MRR across different  $\mathbf{K}$  values. This shows that perhaps low resource languages have some research groups solely focused on the challenges related to them. There is a decreasing trend of MRR from class 0 to class 5, except for class 2, thereby indicating that more popular languages are addressed by more authors. We also observe that even though Japanese, Mandarin, Turkish and Hindi (MRR(10) > 0.75) are part of class 5 and class 4, their MRR is higher even compared to low resource languages in another classes, indicating that these languages have focused research communities working on them. On the other end of the spectrum, we observe a lot of low resource languages like Burmese (MRR(10) = 0.02), Javanese (MRR(10) = 0.23) and Igbo (MRR(10) = 0.13) which have millions of speakers but significantly low MRR values, potentially indicating that not a lot of attention is being given to them in the research community.
We set out to answer some critical questions about the state of language resource availability and research. We do so by conducting a series of quantitative analyses through the lens of a defined taxonomy. As a result, we uncover a set of interesting insights and also yield consistent findings about language disparity:
- The taxonomical hierarchy is repeatedly evident from individual resource availabilities (LDC, LRE, Wikipedia, Web), entropy calculations for conferences, and the embeddings analysis.
LREC and Workshops(WS) have been more inclusive across different classes of languages, seen through the inverse MRR statistics, entropy plots and the embeddings projection.
- There are typological features (such as 144E), existing in languages over spread out regions, represented in many resource-poor languages but not sufficiently in resource-rich languages. This could
potentially reduce the performance of language tools relying on transfer learning.
- Newer conferences have been more language-inclusive, whereas older ones have maintained interests in certain themes of research which don't necessarily favour multilingual systems.
- There is a possible indication of a time progression or even a technological shift in NLP, which can be visualized in the embeddings projection.
- There is hope for low-resource languages, with MRR figures indicating that there are focused communities working on these languages and publishing works on them, but there are still plenty of languages, such as Javanese and Igbo, which do not have any such support.
We believe these findings will play a strong role in making the community aware of the gap that needs to be filled before we can truly claim state-of-the-art technologies to be language agnostic. Pertinent questions should be posed to authors of future publications about whether their proposed language technologies extend to other languages.
There are ways to improve the inclusivity of ACL conferences. Special tracks could be initiated for low-resource, language-specific tasks, although we believe that in doing so, we risk further marginalization of those languages. Instead, a way to promote change could be the addition of D&I (Diversity and Inclusion) clauses involving language-related questions in the submission and reviewer forms: Do your methods and experiments apply (or scale) to a range of languages? Are your findings and contributions contributing to the inclusivity of various languages?
Finally, in case you're still itching to know, Language  $\mathbf{X}$  is Dutch, and  $\mathbf{Y}$  is Somali.
We would like to thank Anshul Bawa, Adithya Pratapa, Ashish Sharma for their valuable feedback during the final phase of work. We would also like to thank the anonymous reviewers for their many insightful comments and suggestions.
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo
gies, Volume 1 (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota. Association for Computational Linguistics.
Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Emily M. Bender. 2011. On achieving and evaluating language-independence in NLP. In Vol 6: Interaction of Linguistics and Computational Linguistics.
Steven Bird, Robert Dale, Bonnie J Dorr, Bryan Gibson, Mark Thomas Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R Radev, and Yee Fan Tan. 2008. The ACL anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. LREC'08.
Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 7059-7069. Curran Associates, Inc.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig.
David Kamholz, Jonathan Pool, and Susan Colowick. 2014. PanLex: Building a resource for pan-lingual lexical translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 3145-3150, Reykjavik, Iceland. European Language Resources Association (ELRA).
Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39, Vancouver. Association for Computational Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579-2605.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Ivan Vulic, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics, 45(3):559-601.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Gary Simons and Steven Bird. 2003. The open language archives community: An infrastructure for distributed archiving of language resources. Computing Research Repository - CORR, 18:117-128.
6291
A.1 Embedding Visualization
We have compiled a visualization of the embedding space of conferences and languages which can be run on a browser. This is available as an interactive visualization on https://microsoft.github.io/linguisticiversity, and can be used to play around with different combinations to see how NLP research has progressed over the years in terms of language inclusion. The legends are self-explanatory and are clickable to add or remove those points. The numbers in the legend represent the respective classes.
A.2 ACL Anthology Dataset Statistics
We have accounted for all the papers which have appeared in the main track proceedings of the conference. This includes all the long and short papers and excludes System Demonstrations, Tutorial Abstracts, Student Research Workshops, Special Issues, and other such tracks out of the scope of measuring language usage trends in general NLP research. We are in the process of releasing the dataset along with the documentation.
Conf / Class	#Papers	#Body	#NoProc	% Missing
LREC	5835	15	6	0.1%
WS	17844	337	332	1.86%
CONLL	1035	0	0	0.0%
EACL	1165	4	1	0.09%
ACL	5776	46	29	0.5%
TACL	280	7	0	0.0%
CL	2025	88	0	0.0%
NAACL	2188	2	1	0.05%
COLING	4233	5	2	0.05%
EMNLP	3865	16	16	0.41%
A.3 Hyperparameter Tuning
Our model has same hyperparameters as that of Word2Vec. To determine the optimal hyperparameters for the model, we take the entire dataset and split it into a 80-20 ratio, and given the embedding of a paper, the task is to predict the year in which the paper is published. Given this vector for a paper, we use a linear regression model such that given this vector, the model is supposed to predict the year in which the paper was published. We measured both  $R^2$  measure of variance in regression and mean absolute error (MAE).  $R^2$  is usually
in the range of 0 to 1.00 (or 0 to  $100\%$ ) where 1.00 is considered to be the best. MAE has no upper bound but the smaller it is the better, and 0 is its ideal value. We observed that our model does not show significant difference across any hyperparaeters except for the size of embeddings. The best dimension size for our embeddings is 75, and, we observed the corresponding  $R^2$  value of 0.6 and an MAE value of 4.04.
A.4 Cosine distance between conferences and languages
From Figure 6, we can see that languages are somewhat below the conferences are closer to some conferences while distant from others. To quantify this analysis, we compute the cosine distance between the conference vector and the mean of the vector each category of the taxonomy. Table 7 shows the cosine distance between the conferences and the each category of languages and we see a very similar trend that while ACL is an at average distance of 0.291 from category 5 languages, its almost more than double far away from category 2. There is also a very steep rise in distance of the ACL vector from category 4 to category 3. In fact, similar trends are visible for other ACL related conferences including EACL, NAACL, EMNLP and TACL. We can also see that in Table 7, WS and LREC are closest from category 2 to category 5 whereas almost all conferences are somewhat at the same distance from category, except the CL journal. The trend for category 0 languages seems somewhat different than the usual trend is this table, probably because of the large number of languages in this category as well as the sparsity in papers.
Table 6: Dataset Statistics.
Conf / Class	0	1	2	3	4	5
LREC	0.51	0.51	0.52	0.42	0.36	0.32
WS	0.50	0.55	0.53	0.40	0.28	0.21
CONLL	0.54	0.60	0.63	0.49	0.40	0.46
EACL	0.53	0.55	0.59	0.45	0.34	0.32
ACL	0.48	0.51	0.60	0.42	0.34	0.29
TACL	0.52	0.56	0.66	0.48	0.38	0.47
CL	0.67	0.78	0.80	0.75	0.65	0.59
NAACL	0.48	0.52	0.59	0.47	0.39	0.33
COLING	0.48	0.53	0.55	0.46	0.37	0.30
EMNLP	0.57	0.59	0.66	0.51	0.46	0.45
Table 7: Cosine Distance between conference vectors and mean class vectors of languages.
6292
A.5 Taxonomy classification
We release our full language taxonomy classification on the website: https://microsoft.github.io/linguisticiversity.
A.6 Class-wise log(MRR) over the years per conference
We plot MRR on a log scale for each conference to measure the progress of inclusion of the defined taxonomy classes over the years. It is very interesting to note how LREC has very smooth forward progression.
(a)  $c = \mathbf{LREC}$
(a)  $c = \mathbf{ACL}$
(b)  $c = \mathbf{WS}$
(b)  $c = \mathbf{CL}$
(c)  $c = \mathbf{EMNLP}$
(c)  $c = \mathbf{COLING}$
(d)  $c = \mathbf{CONLL}$
(d)  $c = \mathbf{N}\mathbf{A}\mathbf{A}\mathbf{C}\mathbf{L}$
(e)  $c =$  SEMEVAL
6293