SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 4 includes the zero-shot cross-lingual WSD results for a collection of baseline approaches (Table 4a) from (Pasini et al., 2021), followed by our
models not utilizing the sparsification of the contextualized embeddings (Table 4b) and the ones that additionally benefit from sparsification as well (Table 4c). It is useful to note that the mono  $\rightarrow^{*}$  approaches are strictly more resource efficient during inference as they are based on 12-layer encoders instead of the 24 layers of the multilingual XLM-R model.
At this point, we separate the multi  $\rightarrow$  multi results into two, i.e., 1) those obtained when relying on the hidden representations from the same layer of XLM-R without mapping (or equivalently, with the identity mapping from the target to source representations); and 2) those obtained when the target and source language contextual representations could originate from different layers of the XLM-R encoder, and a non-identity (either isometric or RC-SLS) mapping was employed. We keep referring to the latter as multi  $\rightarrow$  multi, and denote the former type of experiments as multi (without the  $\rightarrow$  multi suffix as there were no real mappings performed in these cases). Inspecting the first two rows of Table 4b and Table 4c reveals that enhancing the multilingual encoder towards the treatment of a particular pair of languages by providing it a language pair specific mapping has a larger positive effect when using dense vectors. In fact, it increased the micro-averaged F-score over the 17 languages by 1.72 and 0.11 points for the dense and the sparse cases, respectively.
Overall, the micro-averaged F-score of our final approach managed to improve nearly 6.5 points (cf. the first row of Table 4b and the last row in Table 4c). A 5 point average improvement is due
bg	ca	da	de	es	et	eu	fr	gl	hr	hu	it	ja	ko	nl	sl	zh	Avg.
XLMR-Large	72.00	49.97	80.61	83.18	75.85	66.13	47.15	83.88	66.28	72.29	67.64	77.66	61.87	64.20	59.20	68.36	51.62	65.66
XLMR-Base	71.59	47.77	79.18	82.13	76.55	64.73	43.86	82.33	64.79	72.13	68.36	76.73	61.46	63.65	58.77	66.34	49.77	64.82
MBERT	68.78	47.35	76.04	80.63	74.66	64.33	42.41	81.64	68.07	70.65	65.24	76.16	60.34	63.37	56.64	62.16	48.99	62.84
EWISER (2020)	68.64	42.99	76.67	80.86	71.85	65.98	42.85	80.86	59.41	70.60	66.17	74.06	55.77	63.38	57.50	59.74	48.30	62.16
SyntagRank	61.10	43.98	72.93	75.99	68.58	56.31	42.91	69.57	67.56	68.35	57.98	69.57	57.46	50.29	56.00	52.25	41.23	57.68
Babelfy	60.39	36.52	71.33	77.84	64.07	49.62	36.65	67.41	64.17	63.75	51.99	64.22	51.91	51.95	44.27	35.38	34.94	52.85
MCS	58.16	27.17	64.33	75.99	55.65	46.87	32.72	59.31	60.85	62.88	47.29	52.77	48.71	52.48	44.61	36.71	29.62	49.13
(a) Baseline results (MCS stands for Most Common Sense) from (Pasini et al., 2021).
bg	ca	da	de	es	et	eu	fr	gl	hr	hu	it	ja	ko	nl	sl	zh	Avg.
multi	67.07	47.46	76.58	80.74	70.61	65.23	42.53	75.60	56.85	70.63	65.42	71.38	58.45	63.88	54.86	61.91	48.98	61.98
multi→multi	68.99	51.62	78.56	80.51	70.02	65.28	44.68	78.62	57.44	71.59	68.99	70.90	61.40	64.41	57.73	61.17	50.65	63.71
multi→mono	68.82	44.17	79.75	84.69	70.88	64.68	40.95	79.66	56.58	71.34	68.07	69.93	59.71	64.49	59.25	61.37	50.77	63.30
mono→multi	69.68	52.95	78.90	82.02	68.34	66.33	49.62	80.17	58.30	72.34	70.75	74.01	64.35	65.02	59.32	64.76	54.95	65.57
mono→mono	71.17	53.31	81.21	83.29	72.56	66.48	51.08	81.55	63.14	73.76	72.76	72.52	65.26	66.57	60.52	67.42	55.45	66.96
(b) Our results relying on dense sense vectors.
bg	ca	da	de	es	et	eu	fr	gl	hr	hu	it	ja	ko	nl	sl	zh	Avg.
multi	70.69	51.52	81.41	83.53	76.45	67.78	47.85	83.62	64.47	73.06	70.10	76.65	63.73	64.67	58.00	64.12	53.29	66.04
multi→multi	70.91	51.52	80.50	82.37	75.96	66.13	47.09	83.79	63.26	72.94	70.01	77.17	64.47	64.73	60.16	66.49	53.05	66.15
multi→mono	71.91	50.54	81.21	79.93	76.93	64.83	44.05	83.62	62.87	71.64	69.26	77.48	63.59	64.59	60.39	61.07	53.48	65.82
mono→multi	70.76	52.49	79.67	82.25	75.09	67.83	50.89	83.19	60.68	73.99	72.97	75.33	63.80	65.86	61.57	65.70	55.65	66.79
mono→mono	72.00	57.47	81.15	83.76	76.12	68.88	51.71	83.10	63.92	74.40	75.52	76.12	67.47	67.52	61.95	67.47	57.05	68.47
(c) Our results based on sparse sense vectors.
Table 4: Test set results on the XL-WSD benchmark. The hyperparameters of the individual approaches (e.g. which layer of the target language encoder to align with which layer of the source language encode) were determined based on the development set of each language.
to the replacement of the XLM-R encoder for both the source language during training and target languages for inference (cf. the first and last row of Table 4b) and an additional 1.5 points of improvement was an effect of our sparsification in the crosslingual setting. The inspection of the third and fourth rows in both Table 4b and Table 4c reveals that using a monolingual encoder during inference helps more compared to the application of a monolingual encoder for encoding the source language during training.
We conducted the McNemar test between our system outputs when a non-identity mapping was used between a pair of languages. Our investigation revealed that all such  $\binom{8}{2}$  pairs of system outputs from Table 4b and Table 4c differ significantly from each other with  $p < 0.0007$ , with only four exceptions, i.e., 1) multi  $\rightarrow$  multi and multi  $\rightarrow$  mono from Table 4b; 2) multi  $\rightarrow$  multi and multi  $\rightarrow$  mono from Table 4c; 3) mono  $\rightarrow$  multi from Table 4c and mono  $\rightarrow$  mono from Table 4b; 4) multi  $\rightarrow$  mono from Table 4c and mono  $\rightarrow$  multi from Table 4b.
Figure 2 summarizes the results of all the possible runs conducted. When using the multilingual encoder for both the target and source languages without a mapping step between the two (multi), we ran 4 different experiments per each language based on the hidden representations obtained from
Figure 2: Overall averaged results for all the experiments conducted for the different approaches.
one of the last 4 layers of the multilingual encoder. For the remaining experiments relying on the dense and sparse representations, there were 32 and 64 experiments for each language, respectively. The 32 experiments were a result of choosing any of the 16 possible combination of the final four layers on the target and source language encoder, coupled with the type of mapping utilized (isometric/RCSLS). For the experiments involving the sparse representations, there was an extra parameter, whether the normalization of the PMI scores for obtaining the sense representations to be performed, resulting
Figure 3: Comparison of the two best performing systems when the same hyperparameters were employed.
in  $2 \times 32$  experiments all together. Our ablation study in Table 5 illustrates that this extra factor of 2 for the sparse experiments did not provided us an unfair advantage, i.e., when fixing the value of normalization in any way, the overall results did not differ substantially.
The difference in the average performance of our approach transforming sparse contextualized representations obtained by monolingual models is significant (using unpaired t-test $^2$ ,  $p < 0.005$ ) compared to any other configuration. This suggests that the mono  $\rightarrow$  mono approach has a robust advantage over alternative variants, and the improvements seen in Table 4 are not an effect of careful hyperparameter selection, but they generalize over a wide range of choices.
This effect is further corroborated in Figure 3, which offers a comparison between the two systems with the best average performance, i.e., mono  $\rightarrow$  mono that operates with the dense vectors (results are along the x-axis) and the same model but with the enhancement of sparsification (results are along the y-axis). Each data point corresponds to a setting with the same hyperparameter choices, and points above the diagonal line with slope one demonstrate the benefits of sparsification.
We have demonstrated the improved utility of mapping language-specific sparse contextualized representations for conducting zero-shot WSD, requiring large pre-trained language-specific text encoders for the target languages. While such models are available for all languages in XL-WSD, a vari
ety of the existing languages lack their dedicated language-specific pre-trained language model.
As such, an important question emerges whether it is possible to enjoy the benefits of mapping sparse contextualized representations for zero-shot WSD in the absence of a large pre-trained language model dedicated to the target language. To this end, we shall inspect the results of our multi  $\rightarrow$  mono approach in Table 4, a series of mapping-based experiments in which we acted as if the monolingual language models (other than the one for English) did not exist. In these experiments, the sense embeddings were obtained with bert-large-cased (being specialized to English), and the mapping to the non-English target languages were performed towards their XLM-R representations during the evaluation. This way, we could simulate the effects of the absence of language-specific models.
The multi  $\rightarrow$  mono approach provided a substantially better average performance compared to the mere utilization of a multilingual encoder in the case of dense contextualized representations as it can be seen in Table 4b. The average results of multi  $\rightarrow$  mono are slightly inferior (albeit statistically insignificantly) to that of the multi approach for the application of sparse contextualized representations. However, when comparing the multi  $\rightarrow$  multi results with that of multi  $\rightarrow$  mono, we can see that by relying on a multilingual encoder alone, and allowing a mapping to be employed between its hidden representations pertaining to different languages, one can obtain the same (or even slightly better) performance as with the multi  $\rightarrow$  mono approach. This highlights the importance of monolingual encoders for the target language, which seems to be more important than having access to a monolingual encoder for the source language.
In this paper we provided a systematic investigation of the benefits of using large monolingual pretrained language models in place of multilingual language models, such as XLM-R. We have shown that since monolingual neural language models are specifically tailored for a single (or at most a few related) languages, they can effectively mitigate the curse of multilinguality typical of multilingual models, and their application can significantly improve the F-scores in zero-shot WSD. We additionally showed that the benefits of sparse con
2We used unpaired t-test as the number of experiments was not same in all cases, i.e., 4 experiments/language in the multi case, and either 32 or 64 experiments/language in the rest of the cases.
textualized word representations, obtained via a dictionary learning procedure, also convey to the cross-lingual setting, and that it provides complementary improvements to the usage of monolingual neural language models.
The research was supported by the Ministry of Innovation and Technology NRDI Office within the framework of the Artificial Intelligence National Laboratory Program. Additionally, we are thankful for the usage of ELKH Cloud (https://science-cloud.hu/) that helped us achieving the results published in this paper.
Judit Ács. 2019. Exploring BERT's Vocabulary. Blog Post.
Rodrigo Agerri, Iñaki San Vicente, Jon Ander Campos, Ander Barrena, Xabier Saralegi, Aitor Soroa, and Eneko Agirre. 2020. Give your text representation models some love: the case for Basque. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4781-4788, Marseille, France. European Language Resources Association.
Mikhail Arkhipov, Maria Trofimova, Yuri Kuratov, and Alexey Sorokin. 2019. Tuning multilingual transformers for language-specific named entity recognition. In Proceedings of the 7th Workshop on Balto-Slavic Natural Language Processing, pages 89-93, Florence, Italy. Association for Computational Linguistics.
Jordi Armengol-Estape, Casimiro Pio Carrino, Carlos Rodriguez-Penagos, Ona de Gibert Bonet, Carme Armentano-Oller, Aitor Gonzalez-Agirre, Maite Melero, and Marta Villegas. 2021. Are multilingual models the best choice for moderately under-resourced languages? A comprehensive assessment for Catalan. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4933-4946, Online. Association for Computational Linguistics.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.
Vanda Balogh, Gábor Berend, Dimitrios I. Diochnos, and György Turán. 2020. Understanding the semantic content of sparse word embeddings using a commonsense knowledge base. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of
Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7399-7406. AAAI Press.
Gábor Berend. 2020a. Sparsity makes sense: Word sense disambiguation using sparse contextualized word representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8498-8508, Online. Association for Computational Linguistics.
Gábor Berend. 2020b. Massively multilingual sparse word representations. In International Conference on Learning Representations.
Michele Bevilacqua and Roberto Navigli. 2020. Breaking through the  $80\%$  glass ceiling: Raising the state of the art in word sense disambiguation by incorporating knowledge graph information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2854-2864, Online. Association for Computational Linguistics.
Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4330-4338. International Joint Conferences on Artificial Intelligence Organization. Survey Track.
G. Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. In From Form to Meaning: Processing Texts Automatically, Proceedings of the Biennial GSCL Conference 2009, volume Normalized, pages 31-40, Tübingen.
Jose Cañete, Gabriel Chaperon, Rodrigo Fuentes, JouHui Ho, Hojin Kang, and Jorge Pérez. 2020. Spanish pre-trained bert model and evaluation data. In PML4DC at ICLR 2020.
Yo Joong Choe, Kyubyong Park, and Dongwoo Kim. 2020. word2word: A collection of bilingual lexicons for 3,564 language pairs. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3036-3045, Marseille, France. European Language Resources Association.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle-moyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of
the Association for Computational Linguistics, pages 6022-6034, Online. Association for Computational Linguistics.
Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and M. Nissim. 2019. Bertje: A dutch bert model. ArXiv, abs/1912.09582.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Christiane Fellbaum. 1998. WordNet: An Electronic Lexical Database. Bradford Books.
William A. Gale, Kenneth W. Church, and David Yarowsky. 1992. A method for disambiguating word senses in a large corpus. Computers and the Humanities, 26(5):415-439.
Armand Joulin, Piotr Bojanowski, Tomas Mikolov, Hervé Jégou, and Edouard Grave. 2018. Loss in translation: Learning bilingual word mapping with a retrieval criterion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2979-2984, Brussels, Belgium. Association for Computational Linguistics.
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumont, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierrick Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Daniel Loureiro and Alípio Jorge. 2019. Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5682-5691, Florence, Italy. Association for Computational Linguistics.
Daniel Loureiro, Alípio Mário Jorge, and Jose Camacho-Collados. 2021a. LMMS reloaded: Transformer-based sense embeddings for disambiguation and beyond.
Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021b. Analysis and Evaluation of Language Models for Word Sense Disambiguation. Computational Linguistics, 47(2):387-443.
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. 2009. Online dictionary learning for sparse coding. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 689-696, New York, NY, USA. ACM.
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoit Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203-7219, Online. Association for Computational Linguistics.
Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168.
George A. Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G. Thomas. 1994. Using a semantic concordance for sense identification. In HUMAN LANGUAGE TECHNOLOGY: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.
Roberto Navigli and Simone Paolo Ponzetto. 2012. Babelnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artif. Intell., 193:217-250.
Dávid Márk Nemeskey. 2021. Introducing huBERT. In XVII. Magyar Szamítogepes Nyelvészeti Konferencia (MSZNY 2021), pages 3-14, Szeged.
Tommaso Pasini. 2020. The knowledge acquisition bottleneck problem in multilingual word sense disambiguation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4936-4942. International Joint Conferences on Artificial Intelligence Organization. Survey track.
Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Xl-wsd: An extra-large and cross-lingual evaluation framework for word sense disambiguation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13648-13656.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for
2468
Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996-5001, Florence, Italy. Association for Computational Linguistics.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Kiamehr Rezaee, Daniel Loureiro, Jose Camacho-Collados, and Mohammad Taher Pilehvar. 2021. On the cross-lingual transferability of contextualized sense embeddings. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 107-115, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Phillip Rust, Jonas Pfeiffer, Ivan Vulic, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118-3135, Online. Association for Computational Linguistics.
Peter H. Schonemann. 1966. A generalized solution of the orthogonal Procrustes problem. Psychometrika, 31(1).
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1599–1613, Minneapolis, Minnesota. Association for Computational Linguistics.
Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017).
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eighth International Conference on Language Resources and