SlowGuess/NAACL_2021_OCR huggingface dataset:
Leyang Cui, Sijie Cheng, Yu Wu, and Yue Zhang. 2020. Does bert solve commonsense task via commonsense knowledge? arXiv preprint arXiv:2008.03945.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Philip Edmonds and Scott Cotton. 2001. SENSEVAL-2: Overview. In Proceedings of SENSEVAL-2 Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1-5. Association for Computational Linguistics.
Darja Fiser, Jernej Novak, and Tomaz Erjavec. 2012. SloWNet 3.0: development, extension and cleaning. In Proc. of 6th International Global Wordnet Conference.
Edouard Grave, Armand Joulin, Moustapha Cisse, David Grangier, and Hervé Jégou. 2017. Efficient softmax approximation for gpus. In International Conference on Machine Learning (ICML), pages 1302-1310. PMLR.
Xavier Gómez Guinovart. 2011. Galnet: WordNet 3.0 do galego. Línguamática, 3(1).
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visualizing and understanding the effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4143-4152, Hong Kong, China. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. Proceedings of the 37th International Conference on Machine Learning.
Chu-Ren Huang, Shu-Kai Hsieh, Jia-Fei Hong, YunZhu Chen, I-Li Su, Yong-Xiang Chen, and ShengWei Huang. 2010. Chinese WordNet: Design, Implementation and Application of an Infrastructure for Cross-Lingual Knowledge Processing. Journal of Chinese Information Processing, 24(2).
Hitoshi Isahara, Francis Bond, Kiyotaka Uchimoto, Masao Utiyama, and Kyoko Kanzaki. 2008. Development of the Japanese WordNet. In Sixth International conference on Language Resources and Evaluation.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. International Conference on Learning Representations.
Patrick Lewis, Barlas Oguz, Rudy Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315-7330, Online. Association for Computational Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
Márton Miháltz, Csaba Hatvani, Judit Kuti, György Szarvas, János Csiirik, Gábor Prószéky, and Tamás Varadi. 2008. Methods and Results of the Hungarian WordNet Project. In Proc. of The Fourth Global WordNet Conference.
George A Miller, Claudia Leacock, Randee Tengi, and Ross T Bunker. 1993. A semantic concordance. In Proceedings of the workshop on Human Language Technology, pages 303-308.
Andrea Moro and Roberto Navigli. 2015. SemEval 2015 task 13: Multilingual all-words sense disambiguation and entity linking. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 288-297, Denver, Colorado.
Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 task 12: Multilingual word sense disambiguation. In Proc. of SemEval.
Roberto Navigli, Kenneth C. Litkowski, and Orin Hargraves. 2007. SemEval-2007 task 07: Coarse-grained English all-words task. In Proc. of SemEval.
Roberto Navigli and Simone Paolo Ponzetto. 2010. BabelNet: Building a very large multilingual semantic network. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 216-225, Uppsala, Sweden. Association for Computational Linguistics.
Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, et al. 2018. Universal dependencies 2.2.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Crosslingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946-1958, Vancouver, Canada. Association for Computational Linguistics.
Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. XL-WSD: An extra-large and crosslingual evaluation framework for word sense disambiguation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
Bolette S. Pedersen, Sanni Nimb, Jorg Asmussen, Nicolai Hartvig Sorensen, Lars Trap-Jensen, and Henrik Lorentzen. 2009. DanNet: the challenge of compiling a wordnet for Danish by reusing a monolingual dictionary. Language Resources and Evaluation, 43.
Matthew E Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 43-54.
Jason Phang, Iacer Calixto, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, Katharina Kann, and Samuel Bowman. 2020. English intermediate-task training improves zero-shot crosslingual transfer too. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 557-575.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computational Linguistics.
Eli Pociello, Antton Gurrutxaga, Eneko Agirre, Izaskun Aldezabal, and German Rigau. 2008. WNTERM: Enriching the MCR with a terminological
dictionary. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Association (ELRA).
Marten Postma, Emiel van Miltenburg, Roxane Segers, Anneleen Schoen, and Piek Vossen. 2016. Open Dutch WordNet. In Proc. of the Eight Global WordNet Conference.
Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. SemEval-2007 task-17: English lexical sample, SRL and all words. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 87-92, Prague, Czech Republic. Association for Computational Linguistics.
Yada Pruksachatkun, Phil Yeres, Haokun Liu, Jason Phang, Phu Mon Htut, Alex Wang, Ian Tenney, and Samuel R. Bowman. 2020. jiant: A software toolkit for research on general-purpose text understanding models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 109-117, Online.
Ida Raffaelli, Marko Tadic, Bozo Bekavac, and Željko Agić. 2008. Building Croatian WordNet. In Fourth global wordnet conference (gwc 2008).
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017. Word sense disambiguation: A unified evaluation framework and empirical comparison. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 99-110, Valencia, Spain. Association for Computational Linguistics.
Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2020. XL-WiC: A multilingual benchmark for evaluating semantic contextualization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7193-7206, Online. Association for Computational Linguistics.
Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287-297, Brussels, Belgium.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2021. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866.
Kiril Simov and Petya Osenova. 2010. Constructing of an Ontology-based Lexicon for Bulgarian. In Proc. of LREC.
Benjamin Snyder and Martha Palmer. 2004. The English all-words task. In Proceedings of SENSEVAL3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 41-43, Barcelona, Spain. Association for Computational Linguistics.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8968-8975.
Kadri Vider and Heili Orav. 2002. Estonian WordNet and Lexicography. In Proc. of the Eleventh International Symposium on Lexicography.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pretrained language representation. TACL.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In International Conference on Learning Representations.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, Online. Association for Computational Linguistics.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.
Ae-Sun Yoon, Soon-Hee Hwang, Eun-Ryoung Lee, and Hyuk-Chul Kwon. 2009. Construction of Korean WordNet. Journal of KIISE: Software and Applications, 36(1).
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora.
In Proceedings of 11th Workshop on Building and Using Comparable Corpora, pages 39-42.
A Wikipedia Data Details
We download the Wikipedia dump from January 11, 2020, and preprocess it using the WikiExtractor script (Attardi, 2015). We download Wikipedia articles for the following 100 languages as in Conneau and Lample (2019):  $^{6}$  af, als, am, ang, an, ar, arz, ast, az, bar, be, bg, bn, br, bs, ca, ceb, ckb, cs, cy, da, de, el, en, eo, es, et, eu, fa, fi, fr, fy, ga, gan, gl, gu, he, hi, hr, hu, hy, ia, id, is, it, ja, jv, ka, kk, kn, ko, ku, la, lb, lt, lv, mk, ml, mn, mr, ms, my, nds, ne, nl, nn, no, oc, pl, pt, ro, ru, scn, sco, sh, si, simple, sk, sl, sq, sr, sv, sw, ta, te, th, tl, tr, tt, uk, ur, uz, vi, war, wuu, yi, zh_classical, zh, zh_min_nan, zh_yue.
Language sets used for training We finetune MMLM models on the Wikipedia hyperlink prediction task using articles in different sets of languages to investigate the impact of multilingualism. Wiki EN includes only articles in English (en); Wiki 15 includes articles in bg, da, de, en, es, et, eu, fa, fr, hr, it, ja, ko, nl, zh; finally, Wiki 100 includes articles in all 100 languages listed above.
Rationale Wiki EN is a monolingual albeit resource-rich baseline. In Wiki 15, we explore the impact of including languages with different amounts of data and from a mixture of different language families. In Wiki 100, we wish to see if going massively multilingual has a noticeable impact on our models' crosslingual transferability.
Hyperlink extraction We use BabelNet (Navigli and Ponzetto, 2010) — a large multilingual knowledge base comprising WordNet, Wikipedia, and many other resources — to map Wikipedia articles in different languages about the same subject onto unique identifiers. For instance, all "computer science" articles (e.g., Ciencias de la computacion in Spanish, Computer science in English, Informatik in German, etc.) are mapped to the same identifier  $h_t$ , in this case bn:00021494n. After each article is mapped to a single identifier, we create prediction targets for every hyperlink by using the identifier of its referenced article. For example, in Figure 3 the text "algorithmic processes" ( $x_{n:k}$ )
Figure 3: We show Wikipedia articles in different languages, which topics include computer science, automata theory, and algorithm, being mapped to BabelNet IDs. Articles on the same topic, regardless of their language, are mapped to the same identifier. Bottom-right: we show a part of the English article on Computer science and show two example hyperlinks and their targets  $h_t$ .
refers to the article "Algorithm", which is mapped to the ID bn: 00002705n $^9$  ( $h_t$ ).
In Table 4 we show detailed per-language statistics for the Wikipedia data used in our experiments, including the size of the datasets and the number of hyperlinks appearing in the articles (this count already includes only the hyperlinks in our hyperlinks vocabulary of 250k types).
B Hyperparameters, Training Procedure and Model Architectures
We use XLM-R-large (Conneau et al., 2020), which has an encoder with 24 layers and a hidden state size 1024. We finetune XLM-R-large using AdamW (Kingma and Ba, 2015; Loshchilov and Hutter, 2018) with learning rate 0.00005, no weight decay, and batch size 16. We train on minibatches with maximum sequence length 256, gradient norm set to 1.0, and for 300k model updates. When finetuning XLM-R on Wikipedia hyperlink prediction, we only update the last two layers of the model.
Training data sampling We sample batches of training data from each of the languages available, i.e., depending on the experiment these can be English only, 15 languages, or 100 languages. We sample with probability  $r_{l} = \frac{\min(e_{l},K)}{\sum(\min(e_{l},K))}$ , where  $e_{l}$  is the number of examples per language  $l$  and the constant  $K = 2^{17}$  leads to sampling more often
$^{6}$ https://github.com/facebookresearch/XLM $^{7}$ https:// babelnet.org/synset?word=bn:00021494n&lang=EN
$^{8}$ https://en.wikipedia.org/wiki/ Algorithm  $\mathbf{\Psi}^{9}$ https://babelnet.org/synset?word=bn: 00002705n&lang=EN
from resource-poor languages (Raffel et al., 2020).
Adaptive softmax We collect hyperlink targets  $h_t$  from across Wikipedia articles in all the 100 languages available, sort these hyperlinks from most to least frequent, and keep only the top 250k hyperlink targets  $h_t$ . Since hyperlink frequencies follow a natural Zipfian distribution, we use the adaptive softmax activation (Grave et al., 2017) to predict hyperlinks. We bin hyperlink mentions from most to least frequent, i.e. the most frequent  $h_t$  is ranked 1st and the least frequent  $h_t$  is ranked 250k-th. We use five bins, which include hyperlinks with ranks in the following intervals: [1, 10k], (10k, 40k], (40k, 50k], (50k, 70k], (70k, 250k].
The adaptive softmax activation is efficient to compute because: (1) we use one matrix multiplication for each bin, drastically reducing the number of parameters; and (2) the latter bins are only computed in case there is at least one entry in the minibatch with a target in that bin. The five-weight matrices that parameterise each bin in our adaptive softmax layer have sizes:  $\text{hdim} \times 10,000$ ,  $\text{hdim} \times 30,000$ ,  $\text{hdim} \times 10,000$ ,  $\text{hdim} \times 20,000$ ,  $\text{hdim} \times 180,000$ , respectively. Since bins are constructed so that the least frequent hyperlinks are added to the latter bins, we rarely need to compute them. This is especially important in case of the last bin, which is the most costly to compute (and is rarely used).
B.1 Model Architectures
We refer the reader for the mathematical notation in Section 2 Approach. The Wikipedia hyperlink prediction head for a single hyperlink using each of our models is shown below. Token is computed in Equation 1.
$$
p \left(x _ {i} = k\right) \propto \text {A d a p t i v e S o f t m a x} _ {k} \left(W _ {t} \cdot \boldsymbol {x} _ {i} + b _ {t}\right), \tag {1}
$$
where AdaptiveSoftmax  $k$  computes the probability of the hyperlink target  $h_t = k$ ,  $\pmb{x}_{n:k}$  is a hyperlink consisting of words  $\{\pmb{x}_n,\dots ,\pmb{x}_k\}$ , and  $W_{t}$  and  $b_{t}$  are trained parameters.
Concat CLS is computed in Equation 2.
$$
p (x _ {i} = k) \propto
$$
$$
\text {A d a p t i v e S o f t m a x} _ {k} \left(W _ {c} \cdot \left[ \boldsymbol {x} _ {i}; \boldsymbol {x} _ {\mathrm {C L S}} \right] + b _ {c}\right), \tag {2}
$$
where AdaptiveSoftmax $_k$  computes the probability of the hyperlink target  $h_t = k$ ,  $x_{n:k}$  is a hyperlink consisting of words  $\{x_n, \dots, x_k\}$ , and  $W_c$  and  $b_c$  are trained parameters.
Replace CLS is computed in Equation 3.
$$
\boldsymbol {x} = \operatorname {s a m p l e} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {\mathrm {C L S}}\right), \tag {3}
$$
$$
p \left(x _ {i} = k\right) \propto \text {A d a p t i v e S o f t m a x} _ {k} \left(W _ {r} \cdot \boldsymbol {x} + b _ {r}\right),
$$
where sample  $(a,b)$  samples  $a$  or  $b$  with probability 0.9 and 0.1, respectively; AdaptiveSoftmax $k$  computes the probability of the hyperlink target  $h_t = k$ ,  $\pmb{x}_{n:k}$  is a hyperlink consisting of words  $\{\pmb{x}_n,\dots ,\pmb{x}_k\}$ , and  $W_{r}$  and  $b_{r}$  are trained parameters.
B.1.1 XL-WSD
We freeze the pretrained MMLM model weights and simply add a trained classification head on top of the pretrained MMLM. We compute representations for each subword as the sum of the last 4 layers of the model, and for each word as the average of its subword representations (Bevilacqua and Navigli, 2020).
B.1.2 XL-WiC
We follow Raganato et al. (2020) and add a binary classification head on top of the pretrained MMLM model, which takes as input the concatenation of the target words' embedding in the two contexts. We use the output of the 24-th layer as the target words' representation.
B.1.3 XTREME
We use the Jiant library (Pruksachatkun et al., 2020) to carry out the evaluation on XTREME. We use the output of the 24-th layer as the input token representations so as to better measure the impact of our intermediate training on the XTREME tasks.
B.2 XTREME Sentence Retrieval Tasks
BUCC (Zweigenbaum et al., 2018), and Tatoeba (Artetxe and Schwenk, 2019) are two unsupervised tasks requiring, given a sentence in a language  $L$  to retrieve its closest sentence in another language  $L'$ . XTREME baselines use the average of the 14-th layer outputs to represent the sentence. Since our intermediate training procedure only tunes the last two layers, the output of the 14-th layer would be the exact same of the plain XLM-R baseline. For this reason, we did not report the results in both tasks.
10https://github.com/nyu-mll/jiant/ blob/master/guides/tasks/taskspecific.md
Language Code	Language	# training links	size	Language Code	Language	# training links	size
AF	Afrikaans	524,682	37M	KO	Korean	2,955,253	191M
ALS	Tosk Albanian	203,333	13M	KU	Kurdish	91,047	4,6M
AM	Amharic	35,586	2,1M	LA	Latin	791,760	29M
ANG	Anglo-Saxon	315,250	530K	LB	Luxembourgish	224,604	13M
AN	Aragonese	8,376	12M	LT	Lithuanian	1,276,418	76M
AR	Arabic	6,342,628	343M	LV	Latvian	748,963	46M
ARZ	Egyptian Arabic	1,738,581	35M	MK	Macedonian	983,105	63M
AST	Asturian	971,410	77M	ML	Malayalam	332,530	37M
AZ	Azerbaijani	786,016	53M	MN	Mongolian	90,807	6,3M
BAR	Bavarian	78,614	4,4M	MR	Marathi	169,347	13M
BE	Belarusian	1,138,871	82M	MS	Malay	1,242,850	54M
BG	Bulgarian	2,340,267	158M	MY	Burmese	48,285	5,9M
BN	Bengali	549,982	56M	NDS	Low Saxon	168,053	12M
BR	Breton	318,303	15M	NE	Nepali	66,667	5,8M
BS	Bosnian	596,758	34M	NL	Dutch	10,647,696	551M
CA	Catalan	6,180,563	395M	NN	Norwegian Nynorsk	983,245	54M
CEB	Cebuano	15,029,079	178M	NO	Norwegian	4,095,644	227M
CKB	Central Kurdish	88,593	6,1M	OC	Occitan	562,718	22M
CS	Czech	4,697,945	341M	PL	Polish	10,753,690	685M
CY	Welsh	561,936	23M	PT	Portuguese	10,065,298	581M
DA	Danish	2,273,079	135M	RO	Romanian	2,376,428	129M
DE	German	28,064,840	2,1G	RU	Russian	15,691,268	1,4G
EL	Greek	1,611,904	166M	SCN	Sicilian	64,902	3,4M
EN	English	73,084,305	4,9G	SCO	Scots	174,304	9,7M
EO	Esperanto	2,330,837	110M	SH	Serbo-Croatian	3,076,574	113M
ES	Spanish	19,125,611	1,2G	SI	Sinhala	26,321	3,5M
ET	Estonian	1,417,295	82M	SIMPLE	Simple English	1,260,400	57M
EU	Basque	1,743,033	71M	SK	Slovak	1,455,414	84M
FA	Persian	3,429,725	185M	SL	Slovenian	1,343,091	78M
FI	Finnish	3,748,928	252M	SQ	Albanian	280,756	18M
FR	French	23,415,178	1,5G	SR	Serbian	3,218,656	194M
FY	Western Frisian	423,234	26M	SV	Swedish	21,025,833	475M
GA	Ga	183,946	11M	SW	Swahili	317,669	11M
GAN	Gan Chinese	8,742	425K	TA	Tamil	691,010	54M
GL	Galician	1,632,786	108M	TE	Telugu	307,488	26M
GU	Gujarati	256,284	6,6M	TH	Thai	862,265	82M
HE	Hebrew	6,256,536	396M	TL	Tagalog	218,323	13M
HI	Hindi	546,648	45M	TR	Turkish	2,336,668	150M
HR	Croatian	1,825,455	112M	TT	Tatar	499,022	15M
HU	Hungarian	3,785,965	275M	UK	Ukrainian	7,949,672	562M
HY	Armenian	1,639,954	124M	UR	Urdu	526,498	29M
IA	Interlingua	51,882	2,4M	UZ	Uzbek	308,536	9,3M
ID	Indonesian	3,504,017	159M	VI	Vietnamese	4,877,318	221M
IS	Icelandic	252,888	17M	WAR	Waray	4,738,778	46M
IT	Italian	15,407,079	1011M	WUU	Wu Chinese	47,388	4,4M
JA	Japanese	13,318,170	949M	YI	Yiddish	85,374	5,1M
JV	Javanese	213,822	11M	ZH CLASSICAL	Classical Chinese	7,654,040	2,9M
KA	Georgian	894,180	70M	ZH	Chinese	38,104	499M
KK	Kazakh	660,065	41M	ZH MIN NAN	Min Nan	1,101,622	12M
KN	Kannada	114,724	16M	ZH YUE	Cantonese	344,432	17M
Table 4: Data statistics: total number of hyperlinks appearing in articles in Wikipedia in a given language, and size of the dataset for each language. K stands for kilobyte, M for megabyte, and G for gigabyte.