SlowGuess/NAACL_2022_OCR huggingface dataset:
Evaluation (LREC'12), pages 2214-2218, Istanbul, Turkey. European Language Resources Association (ELRA).
Matej Ulčar and Marko Robnik-Šikonja. 2020. Finest bert and crosloengual bert. In Text, Speech, and Dialogue, pages 104–111, Cham. Springer International Publishing.
David Vilares, Marcos Garcia, and Carlos Gomez-Rodriguez. 2021. Bertinho: Galician BERT representations. Proces. del Leng. Natural, 66:13-26.
Yuxuan Wang, Wanxiang Che, Jiang Guo, Yijia Liu, and Ting Liu. 2019. Cross-lingual BERT transformation for zero-shot dependency parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5721-5727, Hong Kong, China. Association for Computational Linguistics.
Warren Weaver. 1949/1955. Translation. In William N. Locke and A. Donald Boothe, editors, *Machine Translation of Languages*, pages 15-23. MIT Press, Cambridge, MA. Reprinted from a memorandum written by Weaver in 1949.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120-130, Online. Association for Computational Linguistics.
Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006-1011, Denver, Colorado. Association for Computational Linguistics.
Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. 2021. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 1-10, Online. Association for Computational Linguistics.
A Analysis of the language models
We compare some of the basic properties of the pretrained language models that we employed in Figure 4 and Figure 5. This can be useful as the monolingual quality of the language models we used could influence and account for their utility when used in conjunction with our mapping-based algorithm.
Figure 4 includes quantitative scores over the different languages related to the subword tokenizers employed by the various language models. Fertility in Figure 4a refers to the average number of subtokens a single token gets separated into by the tokenizer of the given language model. Multi-token ratio (MTR) in Figure 4b indicates the fraction of tokens that gets split into more than one piece upon tokenization (Ács, 2019; Rust et al., 2021). Smaller values of MTR mean a better adaptation of the tokenizer to the peculiarities of the given language. It can be seen that the monolingual models do a much better job compared to XLM-R, which can be part of the reason why mapping independently trained monolingual.
In Figure 5a, we refer to the last four layers of the investigated models as  $\{-4, -3, -2, -1\}$  as the English BERT is a 24-layer model, whereas the rest of the monolingual models consist of 12 layers. This means that layer -1 refers to layer 24 for English and layer 12 for some non-English model. Even though Figure 5a shows pathological masked language modeling (MLM) losses for certain monolingual models (e.g. Bulgarian or Basque) when measured on the XL-WSD database, their mapping-based utilization in zero-shot WSD was still possible as indicated by our main results (see Table 4). A further interesting phenomenon is that the performance of XLM-R exceeds that of the bert-large-cased model in terms of MLM for English. These results suggest that the masked language modeling performance of pretrained language models and their utility in WSD are not strongly related with each other.
B Analysis on using the normalization of PMI scores
Upon the calculation of the sense representation matrix  $\Phi$ , involving the calculation of PMI scores between the various senses from the sense inventory and the coordinates of a sparse contextual representation being non-zero, Berend (2020a) suggested the use of normalized PMI scores (Bouma, 2009). Our preliminary results suggested that the normalization of PMI scores can have a mixed effect over the different languages. Table 5 includes a detailed breakdown on this effect for the individual languages.
(a) Fertility
(b) Multi-token ratio (MTR)
Figure 4: Comparison of the tokenizers of the multilingual (XLM-R) and the monolingual language models.
(a) MLM loss of the monolingual models
Figure 5: The comparison of the multilingual (XLM-R) and the monolingual neural language models in terms of subword tokenization and their MLM objective.
(b) MLM loss of XLM-R
multi	70.69	49.46	81.38	83.53	76.45	67.23	47.22	83.62	64.47	72.16	68.29	76.65	62.79	64.70	59.02	67.96	53.29	65.82
multi→multi	70.91	49.31	80.81	82.37	75.96	67.28	44.30	83.79	62.63	71.89	69.24	77.17	63.26	64.52	60.16	66.49	52.82	65.71
multi→mono	71.91	48.79	81.64	79.93	76.93	64.83	42.72	83.62	62.87	71.34	69.29	77.48	62.58	64.59	60.68	63.09	52.44	65.54
mono→multi	70.76	50.49	79.93	83.41	75.09	66.13	49.37	83.19	60.68	73.11	71.66	75.33	63.01	64.44	60.70	66.63	55.14	66.23
mono→mono	72.00	54.90	81.27	83.76	76.12	67.28	49.87	83.10	63.92	73.33	74.12	76.12	65.57	66.31	61.55	67.47	56.32	67.69
(a) Our results based on sparse sense vectors when always using the normalization of PMI scores as done in (Berend, 2020a).
bg	ca	da	de	es	et	eu	fr	gl	hr	hu	it	ja	ko	nl	sl	zh	Avg.
multi	69.62	51.52	81.41	82.25	74.01	67.78	47.85	82.76	62.79	73.06	70.10	74.32	63.73	64.67	58.00	64.12	53.62	65.68
multi→multi	69.97	51.52	80.50	82.13	74.07	66.13	47.09	82.76	63.26	72.94	70.01	74.63	64.47	64.73	60.16	65.55	53.05	65.82
multi→mono	71.56	50.54	81.21	83.18	74.45	65.68	44.05	79.05	61.77	71.64	69.26	74.19	63.59	64.41	60.39	61.07	53.48	65.51
mono→multi	70.16	52.49	79.67	82.25	70.77	67.83	50.89	81.29	58.65	73.99	72.97	73.92	63.80	65.86	61.57	65.70	55.65	66.42
mono→mono	71.31	57.47	81.15	82.25	72.29	68.88	51.71	81.38	61.03	74.40	75.52	73.49	67.47	67.52	61.95	65.94	57.05	67.96
(b) Our results based on sparse sense vectors when not using the normalization of PMI scores as done in (Berend, 2020a).
bg	ca	da	de	es	et	eu	fr	gl	hr	hu	it	ja	ko	nl	sl	zh	Avg.
multi	70.69	51.52	81.41	83.53	76.45	67.78	47.85	83.62	64.47	73.06	70.10	76.65	63.73	64.67	58.00	64.12	53.29	66.04
multi→multi	70.91	51.52	80.50	82.37	75.96	66.13	47.09	83.79	63.26	72.94	70.01	77.17	64.47	64.73	60.16	66.49	53.05	66.15
multi→mono	71.91	50.54	81.21	79.93	76.93	64.83	44.05	83.62	62.87	71.64	69.26	77.48	63.59	64.59	60.39	61.07	53.48	65.82
mono→multi	70.76	52.49	79.67	82.25	75.09	67.83	50.89	83.19	60.68	73.99	72.97	75.33	63.80	65.86	61.57	65.70	55.65	66.79
mono→mono	72.00	57.47	81.15	83.76	76.12	68.88	51.71	83.10	63.92	74.40	75.52	76.12	67.47	67.52	61.95	67.47	57.05	68.47
(c) Our results based on sparse sense vectors when the normalization of PMI scores was optional and based on the development set for each language.
Table 5: The effects of making the normalization of PMI scores (Bouma, 2009) (a) mandatory, (b) prohibited, (c) optional to use (based on development set results) during the creation of the sparse sense representations.