SlowGuess/NAACL_2021_OCR huggingface dataset:
Iacer Calixto $^{1,2}$  Alessandro Raganato $^{3}$  Tommaso Pasini $^{4,*}$
$^{1}$ Center for Data Science, New York University  $^{2}$ ILLCC, University of Amsterdam
$^{3}$ Department of Digital Humanities, University of Helsinki, Finland
$^{4}$ Department of Computer Science, University of Copenhagen
iacer.calixto@nyu.edu, alessandro.raganato@helsinki.fi,
Masked language models have quickly become the de facto standard when processing text. Recently, several approaches have been proposed to further enrich word representations with external knowledge sources such as knowledge graphs. However, these models are devised and evaluated in a monolingual setting only. In this work, we propose a language-independent entity prediction task as an intermediate training procedure to ground word representations on entity semantics and bridge the gap across different languages by means of a shared vocabulary of entities. We show that our approach effectively injects new lexical-semantic knowledge into neural models, improving their performance on different semantic tasks in the zero-shot crosslingual setting. As an additional advantage, our intermediate training does not require any supplementary input, allowing our models to be applied to new datasets right away. In our experiments, we use Wikipedia articles in up to 100 languages and already observe consistent gains compared to strong baselines when predicting entities using only the English Wikipedia. Further adding extra languages lead to improvements in most tasks up to a certain point, but overall we found it non-trivial to scale improvements in model transferability by training on ever increasing amounts of Wikipedia languages.
Pretrained Multilingual Masked Language Models (MMLMs) such as mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020) and their variants have achieved state-of-the-art results across diverse natural language understanding tasks. Typically, a MMLM model is pretrained on very large amounts of raw text in different languages using the masked language modelling (MLM) objective and is further finetuned on (usually limited amounts of) task data.
In the zero-shot crosslingual setting, which is our focus in this paper, a MMLM is finetuned on the target task using data in a single language (e.g., English) and is evaluated on the same task but in different languages (e.g., non-English languages).
We introduce the multilingual Wikipedia hyperlink prediction objective to contextualise words in a text with entities and concepts from an external knowledge source by using Wikipedia articles in up to 100 languages. Hyperlink prediction is a knowledge-rich task designed to (1) inject semantic knowledge from Wikipedia entities and concepts into the MMLM token representations, and (2) with a similar motivation as the translated language modelling loss of Conneau and Lample (2019), i.e., to inject explicit language-independent knowledge into a model trained via self-supervised learning, but in our case without parallel data. We devise a training procedure where we mask out hyperlinks in Wikipedia articles and train the MMLM to predict the hyperlink identifier similarly to standard MLM but using a "hyperlink vocabulary" of 250k concepts shared across languages.
We use the state-of-the-art MMLM XLM-R-large (Conneau et al., 2020) and show that by adding an add-on training step using Wikipedia hyperlink prediction we consistently improve several zero-shot crosslingual natural language understanding tasks across a diverse array of languages: crosslingual Word Sense Disambiguation in 18 languages including English (XL-WSD; Pasini et al., 2021); the crosslingual Word-in-Context task (XL-WiC; Raganato et al., 2020) in 12 non-English languages; and in 7 tasks from the XTREME benchmark (Hu et al., 2020) in up to 40 languages.
Recently, Zhang et al. (2019, ERNIE) and Peters et al. (2019, KnowBERT) devised different methods to incorporate entities from external knowledge graphs into masked language model (LM) training. Since then, several works followed (Wang et al., 2021; Sun et al., 2020; Xiong et al., 2020; Yamada
*Work carried out while at the University of Rome "La Sapienza".
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3651-3661 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: We finetune a pretrained MMLM using multilingual Wikipedia hyperlink prediction, then further train a model on a set of target tasks in English and evaluate on non-English data (i.e., zero-shot crosslingual setting).
et al., 2020) showing increasingly better performance than masked LMs that rely on information from raw text only. Nevertheless, all these methods were proposed for a single language<sup>1</sup> and cannot be easily applied to transfer learning in a zero-shot crosslingual setting.
Notation Let  $\pmb{x}_{1:m} = \mathrm{MMLM}(x_{1:m})$  be contextualised word representations for some input text  $x_{1:m}$  with  $m$  words, and computed with a pretrained MMLM. Let  $\pmb{x}_{n:k}$  ( $n \geq 1, k \leq m$ ) be a subsequence of contextualised word representations of a single hyperlink  $x_{n:k}$  consisting of  $k - n$  words. In our working example we use a single hyperlink  $x_{n:k}$  for simplicity, but in practice there may be multiple hyperlinks in the input  $x_{1:m}$ .
Data We download and preprocess Wikipedia articles in 100 languages, and extract all hyperlinks in the text. We use BabelNet (Navigli and Ponzetto, 2010) — a large multilingual knowledge base comprising WordNet, Wikipedia, and many other resources — to map Wikipedia articles in different languages about the same subject onto unique identifiers. For instance, regardless of their language all "computer science" articles are mapped to the same identifier  $h_t$ , in this case bn: 00021494n. After each article is mapped to a single identifier, we create prediction targets for every hyperlink by using the identifier of its referenced article. See Appendix A for more details.
Wikipedia Hyperlink Prediction Our main goal is to use the rich semantic knowledge contained in the multilingual Wikipedia's structure to improve language model pretraining. Our approach can be seen as intermediate-task training (Phang et al., 2018, 2020) where we use Wikipedia's hyperlinks as labelled data to further finetune a pretrained MMLM model before training it one last time in the actual target task of interest. Motivated by recent studies on pretrained language encoders demonstrating that semantic features are highlighted in higher layers (Raganato and Tiedemann, 2018; Jawahar et al., 2019; Cui et al., 2020; Rogers et al., 2021), we further train only the last two layers of the MMLM. Moreover, similarly to the MLM procedure, we replace the hyperlink tokens  $x_{n:k}$  by the [MASK] token or by a random token  $80\%$  and  $10\%$  of the time, respectively (Devlin et al., 2019).
Since the number of Wikipedia articles is very large, we only consider the most frequent 250k referenced articles  $h_t$  as possible hyperlinks in our model and we use the adaptive softmax activation function to speed-up training (Grave et al., 2017). Our objective allows us to consider text-entity alignments during training only. At prediction time, instead, we simply feed the model with raw text with no need of precomputed alignments. This makes our model easy to use and to adapt to many different scenarios. For more details on the model architectures and objective, see Appendix B.
We use XLM-R-large (Conneau et al., 2020) as our MMLM, which is pretrained on a large volume
1Mostly English, except for Sun et al. (2020) where Mandarin is also used in a monolingual setting. 2https://babelnet.org/synset?word  $\equiv$  bn: 00021494n&lang  $\equiv$  EN
of raw multilingual corpora using MLM training.
We propose three different model architectures which differ in how the input to the hyperlink classification head is computed. In Token we use the vector representation of each token in the hyperlink text  $\boldsymbol{x}_i, i \in [n,k]$  as input to the prediction head. In Concat CLS we use the concatenation  $[\boldsymbol{x}_i; \boldsymbol{x}_{\mathrm{CLS}}]$  of the representation of each word in the hyperlink  $\boldsymbol{x}_i, i \in [n,k]$  with the [CLS] token representation as input to the prediction head. Finally, in Replace CLS the input to the prediction head is the representation of each word in the hyperlink  $\boldsymbol{x}_i, i \in [n,k]$  with probability  $p_r$  or the [CLS] token representation  $\boldsymbol{x}_{\mathrm{CLS}}$  with probability  $1 - p_r$ . More details on the architectures in Appendix B.1.
We follow a sequential, three steps approach to training and evaluating our models. We first finetune the pretrained MMLM on the Wikipedia hyperlink prediction task, then finetune again this time on the target-task training data in English, and finally evaluate the model on non-English target-task evaluation data in a zero-shot crosslingual setting (see Figure 1). We use Wikipedia articles in different sets of languages (Section 3.3) and experiment with many diverse target tasks (Section 3.4).
3.3 Wikipedia Languages
We experiment using only English (Wiki EN), 15 different languages (Wiki 15), or 100 Wikipedia languages (Wiki 100). By doing that, i) we include a monolingual albeit resource-rich baseline (Wiki EN), ii) we investigate the impact of including a varied mixture of languages from different families (Wiki 15), and iii) we also experiment if going massively multilingual has a noticeable impact on crosslingual transferability (Wiki 100).
3.4 Target Tasks
Word Sense Disambiguation We follow the zero-shot crosslingual setting of Pasini et al. (2021, XL-WSD), which includes 17 languages plus English, i.e., we train on the English SemCor (Miller et al., 1993) dataset merged with the Princeton WordNet Gloss corpus<sup>3</sup> and test on all available languages (Miller et al., 1993; Raganato et al., 2017; Edmonds and Cotton, 2001; Snyder and Palmer,
2004; Pradhan et al., 2007; Navigli et al., 2007; Agirre et al., 2010; Navigli et al., 2013; Moro and Navigli, 2015; Pociello et al., 2008; Simov and Osenova, 2010; Benitez et al., 1998; Huang et al., 2010; Raffaelli et al., 2008; Pedersen et al., 2009; Postma et al., 2016; Vider and Orav, 2002; Guinovart, 2011; Mihaltz et al., 2008; Isahara et al., 2008; Yoon et al., 2009; Fišer et al., 2012).
Word-in-Context We use the crosslingual Word-in-Context dataset (XL-WiC; Raganato et al., 2020) with data in 12 diverse languages. The task is to predict whether an ambiguous word that appears in two different sentences share the same meaning. We finetune the model on the English WiC (Pilehvar and Camacho-Collados, 2019) dataset and evaluate on the 12 XL-WiC languages.
XTREME The XTREME (Hu et al., 2020) evaluation suite contains diverse tasks in up to 40 different languages. We perform crosslingual evaluation on: question answering (XQuAD; MLQA; TyDiQA; Artetxe et al., 2020; Lewis et al., 2020; Clark et al., 2020), natural language inference (XNLI; Conneau et al., 2018), paraphrase detection (PAWS-X; Yang et al., 2019), part-of-speech tagging (POS; Nivre et al., 2018), and named entity recognition (NER; Pan et al., 2017). As standard in the two unsupervised sentence retrieval tasks, BUCC (Zweigenbaum et al., 2018), and Tatoeba (Artetxe and Schwenk, 2019), XLM-R is tested considering the output of its 14-th layer, which, however, is not tuned during our intermediate task. We therefore do not report results on these tasks.
Task Architectures Across all the tasks, we fine-tune transformer-based models by adding a classification head for each task.
Results on XL-WSD and XL-WiC tasks (Tables 1 and 2) suggest that our models have a better grasp of word-level semantics than XLM-R, which does not have explicit semantic signals during its pretraining. This is consistent across languages and hyperlink prediction architectures, also when compared to the baseline XLM-R additionally finetuned using MLM training on in-domain Wikipedia data. Our best models outperform the baselines in both tasks by several points. Interestingly, training on
4More details in Appendix B.2.
Details in Appendix B.1.1 (XL-WSD), B.1.2 (XL-WiC), and B.1.3 (XTREME).
<sup>3</sup>http://wordnetcode.princeton.edu/glosstag.shtml
WIKI EN	WIKI 15	WIKI 100
XLM-R	+MLM	+T	+C	+R	+MLM	+T	+C	+R	+MLM	+T	+C	+R
ENALL	77.7	77.4	76.8	78.4	78.5	77.6	78.5	78.7	78.5	77.4	78.4	78.6	78.3
BG	72.0	71.9	72.1	72.6	70.8	71.7	73.3	73.5	73.1	71.8	72.9	73.2	73.4
CA	50.0	49.5	52.7	52.9	50.8	49.9	54.0	53.7	54.3	50.1	54.6	52.7	54.7
DA	80.6	80.4	81.7	81.7	79.9	80.6	82.4	82.5	82.4	80.7	82.4	82.8	82.1
DE	83.2	83.6	83.6	84.1	83.9	83.3	83.6	85.2	83.1	83.4	84.1	83.1	83.3
ES	75.9	76.8	78.2	78.0	75.2	76.9	78.4	79.1	78.2	77.3	78.2	78.1	78.5
ET	66.1	66.2	66.6	67.2	65.9	66.6	67.7	68.4	68.3	66.7	68.3	68.2	68.0
EU	47.2	46.3	47.7	49.0	44.4	46.4	48.7	49.2	49.4	46.1	49.7	48.7	50.3
FR	83.9	83.9	84.2	84.4	83.4	83.9	84.7	84.1	84.6	83.6	83.4	84.1	84.1
GL	66.3	65.6	67.3	68.2	63.5	66.1	69.7	69.0	70.2	65.3	69.3	68.7	70.2
HR	72.3	72.7	73.9	74.0	72.2	72.8	74.3	74.2	74.5	72.9	74.5	74.1	74.8
HU	67.6	68.6	70.7	70.5	67.7	68.3	71.5	71.4	72.1	68.8	72.0	71.1	72.1
IT	77.7	78.9	78.7	78.8	77.1	78.8	79.3	79.4	79.1	78.5	79.7	79.5	79.5
JA	61.9	62.3	67.1	67.9	65.0	62.4	68.9	68.3	69.5	62.3	69.0	67.1	68.4
KO	64.2	63.6	64.8	64.5	64.9	63.6	65.5	65.7	65.9	63.4	64.8	65.6	65.1
NL	59.2	59.8	60.5	60.5	58.3	59.7	61.6	61.2	62.0	59.8	61.2	61.0	61.4
SL	68.4	67.2	68.9	68.6	67.0	67.4	69.1	67.9	69.0	67.8	68.4	69.5	69.6
ZH	51.6	52.0	55.9	56.2	56.2	52.2	56.6	56.8	56.5	52.5	56.4	56.0	55.9
Avg.	65.7	65.8	67.7	68.0	66.2	65.9	68.7	68.6	68.8	66.0	68.6	68.3	68.7
Table 1: We report F-1 performance on the XL-WSD dataset. Avg. is the micro-average across all languages but English. +MLM is the baseline model XLM-R which we continued training with the MLM objective only, whereas +T, +C, +R are the Token, Concat CLS and Replace CLS models, respectively.
BG	DA	ET	FA	HR	JA	KO	NL	ZH	DE	FR	IT	Avg.
XLM-R	61.8	65.2	62.6	65.8	66.9	61.7	65.6	69.2	68.3	61.1	58.8	62.2	64.1
+MLM (Wiki EN)	63.0	69.9	69.7	73.6	71.3	63.7	69.5	72.4	71.5	65.1	62.3	62.5	67.9
+MLM (Wiki 15)	64.1	67.8	68.5	73.0	70.8	62.7	66.8	72.9	69.8	64.1	61.5	64.5	67.2
+MLM (Wiki 100)	65.5	67.9	68.5	76.3	69.1	60.8	71.1	70.5	68.3	61.7	59.7	61.2	66.7
Wiki EN	+T	65.3	69.6	65.6	77.4	69.4	63.2	67.9	72.6	70.5	65.4	62.4	64.2	67.8
+C	66.6	69.0	68.7	74.9	74.3	65.9	69.5	72.9	70.8	67.1	63.4	66.6	69.1
+R	68.4	68.4	69.0	75.4	73.0	65.3	68.4	73.0	69.6	66.3	62.4	64.9	68.7
Wiki 15	+T	64.6	67.5	64.1	75.8	68.9	62.7	71.0	70.3	67.2	63.8	61.6	65.0	66.9
+C	65.0	69.5	68.7	75.3	69.6	64.3	69.9	73.4	70.1	65.6	61.9	62.5	68.0
+R	67.4	68.0	64.4	73.3	72.1	63.4	65.1	69.5	67.1	63.3	59.8	61.7	66.2
Wiki 100	+T	66.7	69.7	70.5	78.5	67.9	64.8	72.3	74.3	70.9	67.2	64.0	65.7	69.4
+C	61.1	64.1	65.6	71.3	66.9	60.1	68.0	67.9	66.9	59.5	57.9	59.1	64.0
+R	65.0	70.3	68.2	73.0	72.1	62.0	68.5	71.7	72.3	65.3	61.8	63.2	67.8
Table 2: Accuracy scores on the crosslingual Word-in-Context (XL-WiC) test set.
15 languages tends to slightly outperform training on all 100 languages on XL-WSD, but on XL-WiC results with our best models trained on 100 languages outperforms all other configurations most of the time by a reasonable margin. These results corroborate our hunch that the intermediate task injects semantic knowledge within the neural model.
In Table 3, we confirm that our models preserve the sentence-level comprehension capabilities of the underlying XLM-R architecture and that it performs either comparably or favourably to the baselines in the XTREME benchmark, across target tasks and languages.
Training on the English Wikipedia only can be surprisingly effective at times (Tables 2 and 3), and training on 100 languages shows more consistent improvements only on XL-WiC but fails to lead to similar improvements on other tasks. We note that performance on XL-WSD is similar when using 15 or 100 languages, while our evaluation using XTREME shows that performance is slightly worse when using 100 languages compared to using 15 languages only. We conjecture this could be due to the fact we finetune only the last two layers of XLM-R (see Appendix B), so the model retains most of the multilingual knowledge it learned dur
XNLI acc.	PAWS-X acc.	POS F1	NER F1	XQuAD F1 / EM	MLQA F1 / EM	TyDiQA F1 / EM	Avg.
Hu et al. (2020)	79.2	86.4	72.6	65.4	76.6 / 60.8	71.6 / 53.2	65.1 / 45.0	70.1
XLM-R (Ours)	78.6	87.9	76.1	64.0	71.7 / 56.3	70.3 / 50.0	72.6 / 57.0	70.8
+MLM (Wiki EN)	79.1	87.9	76.4	62.3	70.6 / 55.2	69.3 / 50.0	72.7 / 56.8	70.4
+MLM (Wiki 15)	79.3	88.7	75.7	64.4	71.8 / 56.8	70.2 / 50.6	72.6 / 56.7	71.1
+MLM (Wiki 100)	79.2	88.0	76.0	63.4	71.5 / 56.5	70.1 / 50.5	73.5 / 57.4	70.9
Wiki EN	+ T	78.7	88.3	77.3	63.6	70.8 / 55.6	69.8 / 50.4	73.2 / 57.0	70.9
+ C	79.0	87.9	76.9	63.7	71.5 / 55.7	70.3 / 50.0	73.0 / 57.2	70.9
+ R	78.7	88.6	76.9	64.4	71.1 / 55.8	69.6 / 50.1	72.7 / 57.0	71.0
Wiki 15	+ T	79.0	88.1	77.2	64.1	71.3 / 56.5	70.4 / 50.6	73.4 / 57.8	71.2
+ C	79.2	88.4	77.3	64.7	72.1 / 56.9	70.8 / 50.5	73.2 / 57.3	71.4
+ R	79.1	88.3	76.7	64.7	71.5 / 56.4	70.3 / 50.6	72.8 / 56.7	71.1
Wiki 100	+ T	78.6	88.8	76.9	64.8	71.7 / 56.0	70.1 / 50.0	72.7 / 56.9	71.1
+ C	78.6	88.6	77.6	62.1	71.2 / 56.4	69.9 / 50.0	73.2 / 57.4	70.9
+ R	78.8	87.6	76.7	64.2	71.1 / 56.1	69.9 / 50.3	73.1 / 57.4	70.9
Table 3: Results on different target tasks of the XTREME benchmark.
ing pretraining (Liu et al., 2019; Hao et al., 2019).
We also hypothesise that the English Wikipedia size (in number of words) and quality (in coverage of our hyperlink vocabulary) may also be a reason why training solely on English already brings large gains in transfer to other tasks. For comparison, the English Wikipedia is the one with the most data, i.e., about 73M hyperlinks, where the second highest resource language is German with only about 28M hyperlinks (see Table 4 in Appendix B). Regarding the coverage of our hyperlink vocabulary with 250k entries, the English Wikipedia covers over 249k hyperlink types at least 10 times, whereas the second highest coverage is for the French Wikipedia, which covers over 142k hyperlink types at least 10 times. We plan on investigating the effect of the size and coverage of hyperlinks further in future work.
Limitations Finally, we highlight that: (1) We report results using single model runs, therefore we have no estimates of the variance of these models; (2) We lack a more thorough hyperparameter search to further consolidate our results. In both cases, the reason we made such choices is because of the high cost of training large models such as XLM-R large.
We presented a multilingual Wikipedia hyperlink prediction intermediate task to improve the pretraining of contextualised word embedding models. We trained three model variants on different sets of languages, finding that injecting multilingual semantic knowledge consistently improves performance
on several zero-shot crosslingual tasks. As future work, we plan to devise a solution to allow crosslingual transferability to scale more efficiently with the number of languages. Finally, we will investigate the impact on resource-poor vs resource-rich languages, and the effect of the size and coverage of hyperlinks in model transferability.
We would like to thank Clara Vania and Sam Bowman for comments on early versions of this work, and our three anonymous reviewers for their helpful comments and feedback.
IC has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 838188. TP and AR gratefully acknowledge the support of the ERC Consolidator Grants MOUSSE No. 726487, and FoTran No. 771113 under the European Union's Horizon 2020 research and innovation programme. AR also thanks the CSC - IT Center for Science (Finland) for the computational resources.
Eneko Agirre, Oier Lopez de Lacalle, Christiane Fellbaum, Shu-Kai Hsieh, Maurizio Tesconi, Monica Monachini, Piek Vossen, and Roxanne Segers. 2010. SemEval-2010 task 17: All-words word sense dis
ambiguation on a specific domain. In Proc. of SemEval.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the ACL 2019.
Giuseppe Attardi. 2015. Wikiextractor. https://github.com/attardi/wikiextractor.
Laura Benitez, Sergi Cervell, Gerard Escudero, Mónica López, German Rigau, and Mariona Taule. 1998. Methods and tools for building the Catalan WordNet. Proc. of ELRA Workshop on Language Resources for European Minority Languages.
Michele Bevilacqua and Roberto Navigli. 2020. Breaking through the  $80\%$  glass ceiling: Raising the state of the art in word sense disambiguation by incorporating knowledge graph information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2854-2864, Online. Association for Computational Linguistics.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454-470.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishray Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 (NeurIPS), pages 7059-7069. Curran Associates, Inc.
Alexis Conneau, Rudy Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.