SlowGuess/NAACL_2022_OCR huggingface dataset:
Cross-lingual vs. Monolingual Colexification. Cross-lingual colexification approaches COLEXcross and COLEXmaxsim outperform the monolingual model COLEXmono by a large margin for every test language, and overall (Table 1). More specifically, the COLEXcross heatmap (Fig.1) shows significantly improved agreement on most dissimilar word pairs (i.e., the bottom left yellow squares), while more clearly converging on semantically similar instances (upper right squares), and reducing mis-ranked instances (away from the diagonal). This brings supporting empirical evidence for our main claim: adding concepts (synsets) and edges in other languages captured as colexification patterns substantially contributes to the lexical similarity task. For instance, unrelated words like 'aggressive'-'curved', 'airport'-'piece' are penalized by COLEXcross, bringing the ranks closer to the gold standard. At the other end, the model better scores very similar pairs (like near synonyms): 'weird'-strange', 'amazingly'-fantastically', 'area'-region', 'capability'-competence'. Various cases of colexification also bring to surface interesting lexico-semantic differences across languages - like 'charcoal'-'coal' or 'understand'-know' which are not connected in English, but are colexified in other languages like Romanian -{'carbune', 'jar', taciune'} and {'a cunoaste', 'a pricepe', 'a intelege', 'a sti'}, respectively. Some languages have dedicated
words that differentiate special instances of concepts, and thus are ranked as more dissimilar – i.e., 'toe'-'finger' or 'orteil-doigt' (FR), while other languages (ES, RO: Romanian) colexify them, and perceive them as more similar: 'dedo del pie' – 'dedo' (ES) and 'deget de la pocion' – 'deget' (RO) (translation: 'finger of/from foot' - 'finger').<sup>13</sup>
Mean vs. Max Similarity Representation. While both cross-lingual colexification approaches perform best on the LSIM task, there is still noticeable improvement of COLEXcross over COLEXmaxsim for each language and overall across languages. In our experiments, comparing words by the average of their concepts proved more effective in modeling the semantic similarity for the evaluation of word pairs than making a comparison based on the most similar concepts from each word. A close comparison of the two models shows they differ in some specific cases, although these tendencies do not seem to generalize. On a few occasions, COLEXmaxsim is better in penalizing hypernymy relations ('metal'- 'aluminium', 'anatomy'- 'biology'), as well as some syntagmatic relations ('breakfast'- 'bacon', 'tsunami'- 'sea'). On the other hand, it seems to be over confident when it comes to nuanced concepts like 'stink'- 'smell', 'mind'- 'brain' whose interpretation varies more across languages, cultures, and philosophies.
Comparison to Baselines. All three baseline models are fundamentally limited by the quality of the contextual representations learned from raw text during training. While ARES makes use of external annotations and knowledge bases to decide which
<sup>13</sup>Note that the addition 'of/from foot' (RO: 'de la picior' and ES: 'del pie') is one of the many context-rich ways to point to the right finger, and is not part of the concept itself in these languages.
(a) COLEXmono (AR)
(b) COLEXcross (AR)
(c) C+F Fusion (AR)
(d) COLEXmono (EN)
(e) COLEXcross (EN)
(f) C+F Fusion (EN)
(g) COLEX<sub>mono</sub> (FI)
(h) COLEXcross (FI)
(i) C+F Fusion (FI)
(j) COLEXmono (HE)
(k) COLEXcross (HE)
(1) C+F Fusion (HE)
(m) COLEX<sub>mono</sub> (PL)
(n) COLEXcross (PL)
(o) C+F Fusion (PL)
(p) COLEX<sub>mono</sub> (RU)
Figure 2: Heatmaps representing correlation of gold standard (human) ranks and computed ranks for word pairs across six evaluation languages for COLEXmono, COLEXcross, and  $\mathrm{C + F}$  Fusion. Due to each language having slightly different numbers of word pairs in our evaluation, we normalize each plot such that the density of each square represents the percentage (\% out of 100) of the evaluation word pairs for the language. The density of word pairs per square is represented by the square's color. High rank indicates that the words in a given word pair are determined to be very similar, whereas a low rank indicates dissimilarity. Heatmaps for ES, FR, and ZH can be found in the Appendix (Fig. 4).
(q) COLEXcross (RU)
(r) C+F Fusion (RU)
text should be used to represent synsets/senses, all text is passed through a BERT model for the final representation. To the best of our knowledge, distributional models (fastText, BERT) achieve the best performance published so far for Multi-SimLex (Vulic et al., 2020a,b). COLEXcross outperforms all baselines for all languages with a mean score  $>0.1$  above the next-best baseline. This provides evidence that, for languages considered here, cross-lingual colexification-based word embeddings seem to capture word meaning more effectively compared to the baselines. The baseline scores correlate somewhat with one another, with lower scores for HE, PL, and RU, whereas this trend is not observed for cross-lingual colexification approaches. Additionally, cross-lingual colexification-based scores are more stable across evaluation languages with the lowest standard deviation of 0.03 for COLEXmaxsim, which is one big advantage of these approaches.
Embedding Fusion. We hypothesize that the baseline and cross-lingual colexification embeddings may contain rather different and possibly complementary semantic information due to the different
paradigms for their construction (distributional hypothesis vs. knowledge-based), so we fuse these representations and evaluate on the LSIM task. Previous work has shown the simple concatenation of each method's word vector is rather unstable (Liu et al., 2020), leading to possibly worse results than each individual approach alone. However, by performing PCA on the resulting concatenated word vectors in the LSIM evaluation set, we see improved performance from fused methods for all languages (Table 1). These results favor our hypothesis: these combined representations align better with human perception than when evaluated individually.
Comparison across Individual Languages. We also analyzed the results of the models in each and across individual languages (see Fig. 2). When comparing COLEX $_{\text{mono}}$  to COLEX $_{\text{cross}}$  to C+F Fusion, we notice a huge improvement in similarity rank correlations with human judgments across all individual languages from COLEX $_{\text{mono}}$  to COLEX $_{\text{cross}}$ . According to the individual language heatmaps (Fig. 2), the languages that seem to benefit most from the cross-lingual colexifica
(a)
Figure 3: (a) Performance on LSIM task for each evaluation language using 9, 20, 50, 100, 200, or 499 input languages to build the COLEX<sub>cross</sub> graph. Number of word pairs per language is given in parentheses in the legend. (b) Percentage of total colexified synset pairs in BabelNet collected for each language scenario.
#Langs	9	20	50	100	200	499
%	46.4	77.6	96.1	99.5	99.9	100.0
tion approach vs. the monolingual one are AR, HE, PL, RU (compare with Table 1), with HE, AR and PL having the smallest dedicated lexical resources (i.e., the smallest number of colexified synset pairs, see Table 2, A.2). From COLEXcross to C+F Fusion, however, we see significant boosts in number of instances ranked across the diagonal, especially in the lower-bottom squares, across all individual languages. An interesting case here is FI which improves consistently and uniformly across all three models: along the diagonal, but also in reducing the mis-ranked instances (i.e., away from the diagonal).
Effect of Language Inventory Size on Embedding Quality. As shown so far, the large raw number of colexified synset pairs has contributed, in part, to the boost in performance between monolingual and cross-lingual colexification methods. We also show here that, the more languages are added to the colexification graph, the more unique synset colexifications are gathered, leading to a richer semantic network, and thus, better correlation with human judgments. Due to the imbalance of resources per language in BabelNet, we first create an ordered list of languages and choose the first 9, 20, 50, 100, 200, or all 499 languages to build separate graphs. The language list is ordered as
(b)
EN	ES	FI	FR	HE	PL	RU	ZH
EN	-	0.56	0.63	0.72	0.56	0.54	0.55	0.58
ES	0.70	-	0.58	0.65	0.51	0.50	0.49	0.55
FI	0.71	0.69	-	0.66	0.53	0.56	0.53	0.61
FR	0.75	0.68	0.69	-	0.55	0.68	0.64	0.68
HE	0.66	0.64	0.66	0.64	-	0.49	0.47	0.56
PL	0.69	0.67	0.69	0.71	0.65	-	0.45	0.53
RU	0.64	0.61	0.65	0.66	0.62	0.61	-	0.50
ZH	0.70	0.66	0.69	0.69	0.64	0.66	0.60	-
Table 3: Comparison of COLEXcross and fastText on the CLSIM task for eight evaluation languages. The values reported below the diagonal are from COLEXcross while those above are from cross-lingual fastText embeddings created using the fully-supervised configuration of VECMAP (Artetxe et al., 2018). Best score for each language pair is bolded. (See Table 6 in the Appendix for information about number of cross-lingual word pairs per language pair.)
follows: (1) Put the nine evaluation languages at the front of the list; (2) Add remaining languages in decreasing order by number of colexified synset pairs. LSIM results for each graph are given in Figure 3 (a). We find noticeable improvements in performance as we move from 9 to 50 languages, after which it saturates. Figure 3 (b) provides the percentage of total colexified synset pairs available from concept synsets in BabelNet collected for the aforementioned numbers of languages. Performance correlates well with the percentage of colexified synsets collected, supporting the hypothesis that the number of synset-synset relationships acquired across languages is the main driver in performance for our colexification-based approach.
Cross-lingual Performance. We also compared COLEXcross to fastText (the best-performing baseline on the LSIM task), on CLSIM (Cross-lingual Semantic Similarity), a task identical to LSIM, except the words in each word pair are from different languages (Vulic et al., 2020a) (see Table 3). We rely on VECMAP (Artetxe et al., 2018) to map two monolingual fastText embedding spaces to a common bilingual space. Table 6 (see Appendix) shows the total number of word pairs used per language pair. For every language pair, COLEXcross outperforms fastText, often by a significant margin. We show these results to provide additional evidence of the universality of our synset embeddings. Words from different languages can be compared directly under our formulation with no language-specific adaptation while simultaneously outperforming a competitive baseline for this task.
7 Suggestions for Future Improvement
Future work can expand our large-scale study of constructing synset and word representations from cross-lingual colexification principles in a number of directions. First, our cross-lingual embedding models seem specifically useful at ranking highly similar words just by amassing a large number of colexified synset pairs from many of the world's languages. However, while some colexification patterns might show more universal tendencies, others are very specific to a geographic area or language family, while others are more unique, identifying isolated cases of homonymy or other non-similarity phenomena. One possible solution is to represent as edge weight the number of languages that have a colexification pattern between two given synsets. This might result in a stronger model to identify either generalizations or more specific areal patterns (like language contact) by zooming out or in various areas of the graph, depending on one's research interests. Our model of semantic similarity does not distinguish the degree of similarity captured by each colexified synset. Figuring out a way to remove semantic links between colexified synsets that are only weakly or historically related may lead to higher quality synset and word representations capturing universal semantic tendencies, and thus run less the risk of an ethnocentric bias in favour of a specific language/area. Since languages can be compared at various levels of linguistic organization, it would be interesting to empirically investigate how colexification patterns involving core vocabulary differ in their genealogical stability compared with patterns at the periphery of the lexicon (Gast and Koptjevskaja-Tamm, 2021).
Unlike fastText and BERT, which are fully unsupervised, our proposed approach relies on external resources (BabelNet) for lexeme and synset information. Moreover, BabelNet is rather skewed in geographical coverage, typological diversity, and size of vocabulary across languages. From a sociolinguistic perspective, most of the BabelNet coverage comes from socio-politically dominant modern languages, even heavily Anglocentric (i.e., very rich, fine-grained distinctions of English lexicalizations). It would, thus, be interesting to test the efficacy of our model on a more balanced set of languages (as well as number of lexemes and synsets) from a more diverse (sub)set of language families.
For our semantic similarity evaluation, we relied on Multi-SimLex whose perception ratings of
wide coverage lexical words were determined in an out-of-context fashion via human subject questionnaires, and through translation from English. Norm-generating studies involving large number of words have become increasingly popular across the cognitive sciences particularly due to their ability to provide greater statistical power, reduce experimenter bias in item selection, and increase study reliability (Lynott et al., 2020). Thus, correlation plots which intend to capture the relative strength of different colexification patterns are, in fact, an exploratory method and do not represent an attempt at rigorous hypothesis testing (Georgakopoulos et al., 2021). A comparison of out-of-context vs. incontext judgments and of differences between universal vs. more culturally-specific types of knowledge would advance research in lexical semantics.
This paper contributes to the investigation of lexico-semantic structures in the lexicon as manifested by colexification patterns captured in large synset graphs across BabelNet's diverse set of 499 world languages. We introduced several approaches – monolingual and cross-lingual colexification graphs, popular distributional vector space models, as well as a fusion of such systems. We evaluated the extent to which these models correlate with human judgments on a semantic similarity task covering 9 typologically diverse languages.
A deep analysis of the semantic similarity – relatedness/association continuum is not only important for research in lexical semantics and typology, but can also benefit a range of language understanding tasks in NLP. Our large scale cross-lingual colexification graph investigations highlight an important contribution: our word representation approach relies on synset embeddings across languages as captured in colexification graphs, and thus, no adaptation of such word embeddings is necessary for cross-linguistic comparisons (i.e., there is no need for mapping monolingual embeddings to a shared bilingual vector space). We have also tested and validated our cross-lingual colexification models (Tables 3 and 6) on the CLSIM task (Vulić et al., 2020a). The findings of our exploratory investigation of large-scale colexification graphs could inspire significant advances in NLP across languages, especially for tasks involving languages which lack dedicated lexical resources, and can benefit from language transfer from multilingual repositories.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 789–798, Melbourne, Australia. Association for Computational Linguistics.
Hongchang Bao, Bradley Hauer, and Grzegorz Kondrak. 2021. On universal colexifications. In Proceedings of the 11th Global Wordnet Conference, pages 1-7.
Francis Bond and Ryan Foster. 2013. Linking and extending an open multilingual Wordnet. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1352-1362, Sofia, Bulgaria. Association for Computational Linguistics.
Jose Camacho-Collados and Mohammad Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743-788.
Simone Conia and Roberto Navigli. 2020. *Conception: Multilingually-enhanced, human-readable concept vector representations.* In *Proceedings of the 28th International Conference on Computational Linguistics*, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In proceedings of the 52nd annual meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1370-1380.
Anna Di Natale, Max Pellert, and David Garcia. 2021. Colexification networks encode affective meaning. Affective Science, 2(2):99-111.
Alexandre François. 2008. Semantic maps and the typology of colexification. From polysemy to semantic change: Towards a typology of lexical semantic associations, 106:163.
Volker Gast and Maria Koptjevskaja-Tamm. 2021. Patterns of persistence and diffusibility in the European lexicon. Linguistic Typology.
Thanasis Georgakopoulos, Eitan Grossman, Dmitry Nikolaev, and Stéphane Polis. 2021. Universal and macro-areal patterns in the lexicon. Linguistic Typology.
Sahar Ghannay, Benoit Favre, Yannick Esteve, and Nathalie Camelin. 2016. Word embedding evaluation and combination. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 300-305.
Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864.
Martin Haspelmath. 2003. The geometry of grammatical meaning: Semantic maps and cross-linguistic comparison.
Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. Wikireading: A novel large-scale language understanding task over wikipedia. arXiv preprint arXiv:1608.03542.
Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for word sense disambiguation: An evaluation study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 897-907.
Joshua Conrad Jackson, Joseph Watts, Teague R Henry, Johann-Mattis List, Robert Forkel, Peter J Mucha, Simon J Greenhill, Russell D Gray, and Kristen A Lindquist. 2019. Emotion semantics show both cultural variation and universal structure. Science, 366(6472):1517-1522.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification.
Päivi Juvonen and Maria Koptjevskaja-Tamm. 2016. The lexical typology of semantic shifts, volume 58. Walter de Gruyter GmbH & Co KG.
Hang Le, Loic Vial, Jibril Frej, Vincent Segonne, Maxim Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbe, Laurent Besacier, and Didier Schwab. 2020. Flaubert: Unsupervised language model pre-training for french.
Johann-Mattis List, Simon J Greenhill, Cormac Anderson, Thomas Mayer, Tiago Tresoldi, and Robert Forkel. 2018. Clics2: An improved database of cross-linguistic colexifications assembling lexical data with the help of cross-linguistic data formats. Linguistic Typology, 22(2):277-306.
Qianchu Liu, Diana McCarthy, and Anna Korhonen. 2020. Towards better context-aware lexical semantics: adjusting contextualized representations through static anchors. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4066-4075, Online. Association for Computational Linguistics.
Dermot Lynott, Louise Connell, Marc Brysbaert, James Brand, and James Carney. 2020. The Lancaster Sensorimotor Norms: multidimensional measures of perceptual and action strength for 40,000 English words. Behavior Research Methods, 3:1271 - 1291.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
Roberto Navigli and Simone Paolo Ponzetto. 2010. Babelnet: Building a very large multilingual semantic network. In Proceedings of the 48th annual meeting of the association for computational linguistics, pages 216-225.
Vladimir Pericliev et al. 2015. On colexification among basic vocabulary. Journal of Universal Language, 16(2):63-93.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701-710.
Christoph Rzymski, Tiago Tresoldi, Simon J Greenhill, Mei-Shin Wu, Nathanael E Schweikhard, Maria Koptjevskaja-Tamm, Volker Gast, Timotheus A Bodt, Abbie Hantgan, Gereon A Kaiping, et al. 2020a. The database of cross-linguistic colexifications, reproducible analysis of cross-linguistic polysemies. Scientific data, 7(1):1-12.
Christoph Rzymski, Tiago Tresoldi, Simon J Greenhill, Mei-Shin Wu, Nathanael E Schweikhard, Maria Koptjevskaja-Tamm, Volker Gast, Timotheus A Bodt, Abbie Hantgan, Gereon A Kaiping, et al. 2020b. The database of cross-linguistic colexifications, reproducible analysis of cross-linguistic polysemies. Scientific data, 7(1):1-12.
Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020. With more contexts comes better performance: Contextualized sense embeddings for all-round word sense disambiguation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3528-3539.
Ivan Vulic, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, et al. 2020a. Multi-simlex: A large-scale evaluation of multilingual and crosslingual lexical semantic similarity. Computational Linguistics, 46(4):847-897.
Ivan Vulic, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020b. Probing pretrained language models for lexical semantics. arXiv preprint arXiv:2010.05731.
Yang Xu, Khang Duong, Barbara C. Malt, Serena Jiang, and Mahesh Srinivasan. 2020. Conceptual relations predict colexification across languages. Cognition, 201:104280.
Hyejin Youn, Logan Sutton, Eric Smith, Christopher Moore, Jon F Wilkins, Ian Maddieson, William Croft, and Tanmoy Bhattacharya. 2016. On the universal structure of human lexical semantics. Proceedings of the National Academy of Sciences, 113(7):1766-1771.
Jie Zhang, Yuxiao Dong, Yan Wang, Jie Tang, and Ming Ding. 2019. Prone: Fast and scalable network representation learning. In IJCAI, volume 19, pages 4278-4284.
Appendix includes additional statistical information on experiments performed in this paper. Tables and Figures included: Tables 4, 5, 6; Fig. 4.
A.1 Context Examples and Pretrained BERT Models