SlowGuess/NAACL_2022_OCR huggingface dataset:
John Harvill<sup>1</sup>, Roxana Girju<sup>2</sup>, Mark Hasegawa-Johnson<sup>1</sup>
$^{1}$ Department of Electrical and Computer Engineering,
$^{2}$ Departments of Linguistics and Computer Science,
harvill12, girju, jhasegaw}@illinois.edu
In this paper we focus on patterns of colexification (co-expressions of form-meaning mapping in the lexicon) as an aspect of lexical-semantic organization, and use them to build large scale synset graphs across BabelNet's typologically diverse set of 499 world languages. We introduce and compare several approaches: monolingual and cross-lingual colexification graphs, popular distributional models, and fusion approaches. The models are evaluated against human judgments on a semantic similarity task for nine languages. Our strong empirical findings also point to the importance of universality of our graph synset embedding representations with no need for any language-specific adaptation when evaluated on the lexical similarity task. The insights of our exploratory investigation of large-scale colexification graphs could inspire significant advances in NLP across languages, especially for tasks involving languages which lack dedicated lexical resources, and can benefit from language transfer from large shared cross-lingual semantic spaces.
Distributional models like word embeddings have been widely used in Natural Language Processing (NLP) (Iacobacci et al., 2016; Devlin et al., 2014; Hewlett et al., 2016). They operate under the assumption that words appearing in similar contexts have similar meanings, and thus close representations. However, as do other unsupervised learning models, they suffer from classic limitations – i.e., there is no guarantee that all context words contribute to the meaning of the target word, while, in fact, it is possible that some low frequency words, with poorly-trained embeddings, are highly semantically connected. Also, they don't distinguish between topically related words and near synonyms.
Dictionaries and thesauruses, on the other hand, have traditionally offered an alternative approach,
through their discrete lists of fine-grained senses, textual definitions, and relationships with other senses. Given a sufficiently large dictionary of many fine-grained sense representations in many of the world's languages, one could perform sophisticated semantic tasks on word senses (Conia and Navigli, 2020). In fact, investigating universal and areal cross-linguistic variations in the lexicon has been the focus of lexical typology. One increasingly popular empirical method of investigating senses based on cross-linguistic comparison in typological studies has been that of colexification patterns. "A given language is said to colexify two functionally distinct senses if, and only if, it can associate them with the same lexical form" (François, 2008), reflecting natural semantic connections (Haspelmath, 2003). For example, 'fire' and 'firewood' are colexified in Kamoro (New Guinea language) as 'uta' and in Wayuu (Arawakan language) as 'siki', but receive distinct lexemes in English and Romanian. Each polysemous lexeme as a whole is language-specific, yet a great number of lexical polysemies are each attested across multiple languages.
In this paper, we investigate semantic structures in the lexicon as manifested by colexification patterns in a large number of languages and assess, at a large scale, their usefulness to the Lexical Semantic Similarity (LSIM) task (Vulic et al., 2020a). We used one of the largest digital lexical resources to date, BabelNet 5.0 (953.4M (concept) lexicalizations in 499 languages $^1$ ) to build and process colexification graphs.
Specifically, we make the following contributions: 1) Propose a simple, yet effective algorithm to automatically construct large-scale synset similarity graphs based on the principle of colexification, and use the graphs to generate high-quality
<sup>1</sup>BabelNet 5.0's claim of supporting 500 languages seems to be a typo. There are only 499 (see "Languages and Coverage" tab) https://babelnet.org/statistics
Human Language Technologies, pages 5259 - 5270
synset and word representations. 2) Demonstrate the importance of universality of our graph synset embedding representations with no need for any language-specific adaptation when evaluated on LSIM. 3) Show that our proposed approach significantly outperforms state-of-the-art synset and word embedding techniques on the LSIM task. 4) When combined with knowledge-based approaches like our cross-lingual colexification patterns, purely unsupervised distributional models like fastText and BERT result in better alignment with human perception, as measured on the LSIM task.
Our findings and models contribute to advances in computational modeling of natural language understanding across languages, and offer new insights into linguistic typology.
The concept of colexification has been introduced by Haspelmath (2003) to distinguish senses in the grammatical domain, but has been formalized for the field of lexicon by François (2008) who, in a cross-linguistic study of the world's lexicons, investigated colexification patterns captured in a semantic map. Unlike Haspelmath who showed that 12 diverse languages are sufficient to build a stable semantic map, Francois posits that, in fact, the number of distinctions between senses increases with the number and variety of considered languages. Following these studies, List et al. (2018) built a weighted colexification graph using data from 195 languages in 44 language families, with subsequent improved language coverage versions (Rzymski et al., 2020a). Here, closely-related or similar concepts tend to be often densely connected (List et al., 2018; Georgakopoulos et al., 2021). Youn et al. (2016) constructed colexification graphs in the domain of natural objects (celestial and landscape) and investigated their polysemy distributions for the task of semantic similarity. We also take advantage of recurrent patterns in semantic structure across different language families. However, unlike them, we found some evidence that geographical and cultural differences matter in the human perception of our cross-linguistically connected concepts. Pericliev et al. (2015) distinguished between homonymy and polysemy by investigating the colexifications of 100 basic concepts. Georgakopoulos et al. (2021) discovered cross-linguistic similarities based on colexification patterns of verbs of vision and hearing in the do
main of perception-cognition. Jackson et al. (2019) relied on colexification patterns to test the universality of emotion perception. Like us, they show that, while there are shared structures of (affective) meanings across cultures, there are also some variations. Di Natale et al. (2021) tested whether colexification patterns in multilingual resources are correlated with affective meaning similarity between words. Bao et al. (2021) showed that no two concepts are colexified in every language by analyzing colexification data from three resources: BabelNet, Open Multilingual WordNet (Bond and Foster, 2013), and the Database of Cross-Linguistic Colexifications (CLICS3) (Rzymski et al., 2020b).
Although the scope of research on colexification varies across projects, most studies have assumed that colexification captures some degree of semantic similarity. Indeed, this is implied to some extent by the very definition of colexification, and supported by previous results in linguistics and NLP, suggesting that more commonly colexified meanings across languages require less cognitive effort to relate and recall (Xu et al., 2020). However, such a connection between cross-linguistic colexification patterns and semantic similarity has not been fully assessed at a large scale. Given that features of the lexicon are not easily identifiable across many languages, one solution is to rely on large data repositories to unveil cross-linguistic generalizations. However, since most languages lack dedicated lexical resources for semantic similarity (henceforth, low-resource languages), one option is to transfer lexico-semantic knowledge from large shared cross-lingual semantic spaces. In this paper, as part of a large scale empirical study, we show that lexico-semantic associations captured by cross-lingual colexification patterns in BabelNet contribute significantly in assessing if two words are semantically similar.
3 Synset Cross-Lingual Colexification
Under the colexification framework, the primary unit of observation for lexical typology is no longer the word, but the sense - a functionally-based criterion of concept definition (François, 2008). In this project, however, we use the synset as the primary unit. For us, this is also a technical consideration, since only synsets can be compared across languages, especially in BabelNet. Lexical concepts are grouped into sets of cognitive (near) synonyms, called synsets, each encoding a distinct
meaning. Synsets are connected through lexical relations and conceptual/semantic relations (i.e., hyperonymy, hyponymy, meronymy, etc.). In this paper, we use only lexical relations to capture co-lexifications between meanings. For instance, the lexical relation between the senses of 'fire' and 'firewood' of the word  $\underline{\text{uta}}$  (in Kamaro) is a case of strict colexification. Loose colexifications like derivationally-related forms can show interesting semantic associations, but are not considered here.
Two synset concepts that are colexified in at least one language are usually perceived as somehow semantically connected, either directly or indirectly. However, proving such connectedness is by no means an easy task. The accurate description of lexical data often requires taking into account the many functional properties of real-world referents as well as culture-specific aspects of a language or geographic area. Such cases might capture underlying linguistic phenomena such as metaphor, metonymy, hyperonymy, analogical extension, and a rich set of cases of semantic shifts unique to each language or language family (Juvonen and Koptjevskaja-Tamm, 2016) – whose analysis falls within the scope of semantic or etymological studies, and beyond our goal here. Instead, our purpose is to organize cross-linguistic sense information in a way that captures various semantic connections between senses, allowing one to zoom in and out on aspects of the lexicon in cross-linguistic comparative studies. We rely here on the powerful structure of BabelNet that maps concepts (i.e., synsets) across a large, typologically diverse set of languages. This allows us to empirically examine at a large scale the contribution of such a rich body of knowledge to the task of semantic similarity – such empirical evidence is still lacking in the field. Our model of semantic connection between synsets (we call Syn2Vec) is simple: Given the set of concepts (synsets) of a lexeme, we assign a semantic link between every synset-synset pair. We want to investigate the idea that, as more and more languages are explored, and more and more senses are amassed, the resulting graph of cross-linguistic inter-connected synset concepts will capture aspects of semantic knowledge that might be missing in one language alone.
Given this intuition, next we briefly introduce the lexical resource used and our proposed algorithm to construct colexification graphs which model the synset semantic connections. We hypothesize
Algorithm 1 Construction of Colexification Graph: Given a set of languages  $L$  and corresponding vocabularies  $V$ , create graph edges between all colexified synset pairs (nodes).
that the conceptual representations of lexical typology captured by our cross-lingual colexification patterns do match, to some extent, the language-internal perception of native speakers, and test this hypothesis empirically on the LSIM task.
A. The Lexical Resource, BabelNet. To collect synset information, we use BabelNet (Navigli and Ponzetto, 2010), to our knowledge, the largest cross-linguistic semantic network that extends the popular WordNet (Miller, 1995) by integrating other resources (Wikipedia, Wiktionary, etc). Every BabelNet synset (20.3M total) is identified as either a concept or named entity, and we only use concept synsets (7.2M) for our analysis. The maximum and minimum numbers of (concept²) lexemes in a language are 6.1M (English) and 1.8M (Parthian), respectively.
B. Building the Colexification Graphs. We denote the set of languages as  $L = \{l_1, l_2, \dots, l_N\}$ , and the language vocabulary lists as  $V = \{V_{l_1}, V_{l_2}, \dots, V_{l_N}\}$ . The vocabulary for each language is represented by  $V_{l_n} = \{x_1, x_2, \dots, x_{|V_{l_n}|}\}$  where the elements  $x_k$  are lexemes. For each lexeme  $x_k$  we have a corresponding set of synsets  $S_{x_k} = \{s_1, s_2, \dots, s_{|S_{x_k}|}\}$ . For a colexification graph  $G$ , nodes represent synsets, and edge weights the semantic connection between two synsets. We denote the weight for each edge  $\{s_1, s_2\}$  in the graph as  $G(s_1, s_2) = G(s_2, s_1)$  (undirected graph). Algorithm 1 details the graph construction.
In this study, we examine two types of colexification graphs: (1) Monolingual, and (2) Cross
2We filter out lexemes that have no concept synsets.
lingual. For monolingual graphs, we choose one language  $l$  and provide  $L = \{l\}$  and  $V = \{V_l\}$  to Algorithm 1. For the cross-lingual graph, we use all languages, i.e.  $L = \{l_1, l_2, \dots, l_N\}$ ,  $V = \{V_{l_1}, V_{l_2}, \dots, V_{l_N}\}$ . In BabelNet, the same concept in each language will be mapped to a common synset and thus be represented as a node in the colexification graph. $^3$
C. Creating Synset and Word Embeddings. As we like to capture lexico-semantic associations, given a colexification graph  $G$ , we assume that vector representations of the nodes (synsets) that are close to one another are similar as computed by cosine distance in the embedding space. We use a recently-developed node embedding approach, ProNE (Zhang et al., 2019), which, compared to other popular node embedding techniques like Deepwalk and Node2Vec (Perozzi et al., 2014; Grover and Leskovec, 2016), is much faster and demonstrates superior node representations on several classification datasets using a lib-linear classifier. We use the Python implementation from nodevectors<sup>4</sup> with all default hyperparameters.
In predicting the lexical similarity of two words, we assume that their perceptual similarity is determined by summing the synset embeddings of each word, then comparing the results. Thus, a word embedding  $w$  is computed as:
$$
w = \sum_ {s \in S _ {\text {B a b e l} (w)}} s _ {\text {e m b}} \tag {1}
$$
where  $s_{emb}$  is the embedding for synset  $s$  and  $S_{Babel(w)}$  is the set of synsets of word  $w$  in BabelNet. Prior to each semantic similarity task, we normalize each word embedding to have magnitude one. Next, we take all evaluation words and perform mean centering, then Principal Component Analysis<sup>5</sup> (PCA) following (Ghannay et al., 2016), as we empirically found this improves performance.
We evaluate the quality of our BabelNet dictionary approach by comparing it to high-quality and pop
ular word and synset embedding approaches. We want to test whether the structural regularities observed in distributed text representations provide a route past some of the limitations of dictionaries, whether these two representations are comparable, and whether their combination might benefit the task of lexical similarity. Specifically, we compare to the well-known static word embedding approach fastText (Joulin et al., 2016), and an approach that extracts contextualized representations of words from a pretrained BERT language model (Vulic et al., 2020b), which we call "BERT." We also compare to "ARES", a recent synset/sense embedding model (Scarlini et al., 2020) that builds representations for each synset by collecting relevant contexts and extracting contextual embeddings of lemmas belonging to each synset from a pretrained language model (BERT). Similar to (Scarlini et al., 2020), we rely on ARES synset embeddings for our multilingual analysis. To compute ARES word embeddings, we follow Equation (1), but use the pretrained ARES multilingual synset embeddings<sup>6</sup>. We re-implement the BERT baseline<sup>7</sup>, but use pretrained word embeddings for fastText<sup>8</sup>.
Lexical semantic similarity (LSIM) seeks to accurately measure the perceived similarity in meaning between two words and does so by the Spearman's rank correlation $^{9}$  between similarity scores of human judgments and those computed automatically (cosine similarity of the words' vector representations). We rely here on Multi-SimLex (Vulic et al., 2020a), arguably the most comprehensive semantic similarity evaluation resource to date, which contains monolingual lists of 1,888 word pairs, with aligned concepts in 13 typologically diverse languages $^{10}$ . Diverse criteria were used here to test whether two words are semantically similar, and not vaguely associated. Thus, the Multi-SimLex datasets could be used as "an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms" (Vulic et al., 2020a). Of the original 13 languages, we limit our
3We removed the three BabelNet noisy lexemes (with  $>1,000$  synsets): the empty string "" (common to all languages); and "asteroid list" in both Russian (RU) and Armenian (HY): "cINcOK_acTepeOnIOB" in (RU); "uunptnnhutnh-guful" in (HY).
4https://github.com/VHRanger/nodevectors
We keep the vector dimension the same.
http://sensembert.org/resources/ares embedding.tar.gz, ares_bert_baseMULTILING.txt
<sup>7</sup>We must extract embeddings ourselves using the pretrained models (see A.1).
<sup>8</sup>https://pypi.org/project/fasttext/
<sup>9</sup>We use "average" rank mode from scipy.stats数据分析()
<sup>10</sup>Since publication of the dataset, Arabic has been added.
AR(838)	EN(1822)	ES(1728)	FI(1717)	FR(1798)	HE(1085)	PL(1176)	RU(972)	ZH(1563)	Mean↑	Std.↓
fastText	0.50	0.54	0.53	0.64	0.59	0.41	0.45	0.45	0.53	0.52	0.07
BERT	0.44	0.57	0.52	0.62	0.41	0.37	0.38	0.37	0.62	0.48	0.10
ARES	0.49	0.50	0.51	0.58	0.50	0.43	0.47	0.43	0.55	0.49	0.05
COLEXcross	0.59	0.72	0.67	0.69	0.65	0.61	0.66	0.60	0.66	0.65	0.04
COLEXmaxsim	0.59	0.68	0.62	0.62	0.60	0.60	0.66	0.60	0.62	0.62	0.03
COLEXmono	0.25	0.36	0.47	0.42	0.44	0.13	0.24	0.22	0.38	0.32	0.11
C+F	0.65	0.76	0.71	0.76	0.73	0.64	0.68	0.64	0.70	0.70	0.05
C+B	0.63	0.75	0.70	0.75	0.68	0.62	0.66	0.61	0.70	0.68	0.05
B+F	0.54	0.58	0.55	0.66	0.58	0.41	0.46	0.46	0.61	0.54	0.08
C+F+B	0.66	0.75	0.70	0.78	0.72	0.61	0.65	0.62	0.72	0.69	0.05
Table 1: Comparison of each word embedding technique on the LSIM task (Spearman's rank correlation) for nine different languages. Number of word pairs per language is given in parentheses below each language code. The  $+$  ’ symbol indicates concatenation followed by PCA for fusion of two or more models ('C'=COLEXcross, 'F'=fastText, 'B'=BERT). Best score per language is bolded; best non-fusion model score per language is underlined.
AR	EN	ES	FI	FR	HE	PL	RU	ZH	All
Vocabulary size (# lexemes)	2.4M	6.1M	3.3M	2.6M	3.3M	2.1M	2.7M	3.2M	3.1M	953.4M
# Polysemous lexemes	41.2k	323k	163k	106k	159k	24.7k	106k	151k	86k	4.7M
# Synsets	2.1M	4.1M	2.8M	2.4M	2.9M	2.0M	2.5M	2.8M	2.6M	7.2M
# Colexified synset pairs	233k	1.3M	582k	474k	773k	54.6k	244k	1.0M	312k	8.5M
Mean # synsets per lexeme	1.03	1.08	1.08	1.07	1.08	1.02	1.05	1.08	1.05	1.007
Mean # syns. per polys. lexeme	2.97	2.56	2.58	2.67	2.66	2.46	2.3	2.61	2.65	2.43
Table 2: BabelNet statistics for evaluation languages given our experimental setup (using concept synsets only). "All" column indicates cumulative statistics over all 499 languages. Polysemous lexemes have two or more synsets.
study to 9 languages: Arabic (AR), Spanish (ES), English (EN), Finnish (FI), French (FR), Hebrew (HE), Polish (PL), Russian (RU), Mandarin Chinese (ZH). Thus, we can directly compare with other baselines that are only readily available for languages with large pretrained language models.
Cross-lingual lexical semantic similarity (CLSIM) is identical to LSIM, except the words in each word pair are from different languages. Multi-SimLex (Vulic et al., 2020a) also makes available these cross-lingual similarity scores, excluding AR.
Colecxification Evaluation Setups. We evaluate monolingual and cross-lingual colexification-based synset embeddings<sup>11</sup> in three variants:
COLEX<sub>mono</sub>: Construct the colexification graph using one language only. Build word embeddings from synset embeddings using Equation (1).
COLEXcross: Same procedure as COLEXmono except that we use all 499 languages to construct the colexification graph. The purpose is to capture the full complexity of the BabelNet data.
$\underline{\mathrm{COLEX}}_{\mathrm{maxsim}}$  : We use here the same synset em
beddings from  $\mathrm{COLEX}_{\mathrm{cross}}$ . Specifically, following (Camacho-Collados and Pilehvar, 2018), for each evaluation word pair in the test set, we determine their similarity score by computing the maximum similarity among all pairs of synset embeddings. We perform PCA on the entire set of  $\mathrm{COLEX}_{\mathrm{cross}}$  synset embeddings prior to similarity computation.
We introduce here various experiments to compare multiple methods. Since BERT and fastText can form representations from arbitrary string inputs, they have no out-of-vocabulary (OOV) words, while all methods relying on external resources (ARES and COLEX) require synset information for any word included in our experiments. For fair comparison, we limit our study to word pairs that include both words in the vocabulary of all approaches. Table 1 and Fig. 3 show the total number of word pairs (of the original 1,888) used for evaluation (relevant BabelNet stats in Table 2) $^{12}$ .
We also use heatmaps (correlation plots) to give a clearer visual overview of the correlation between human judgments (i.e., gold standard) and our models' outputs, by word pair similarity rank (an ordinal measure) for all pairs in the test sets. Figure 1 shows the overall correlation
11All experiments were performed on an x86-64 server with a 32-core Intel(R) Xeon(R) Silver 4215R CPU and 754GB RAM. The node embedding process for the largest graph  $\mathrm{(COLEX_{cross})}$  took 20 minutes and 30GB of RAM. Our code is publicly available at https://github.com/jharvill23/Syn2Vec.
$^{12}$ OOV words breakdown by method: Table 5 (A.2)
(a) COLEXmono
(b) COLEXcross
Figure 1: Heatmaps showing correlation of human rank judgments (X-coordinate) and overall computed ranks (Y-coordinate) for word pairs across all nine evaluation languages for four approaches: (a) COLEX<sub>mono</sub>; (b) COLEX<sub>cross</sub>, (c) fastText; (d) C+F Fusion (C: COLEX<sub>cross</sub> and F: fastText). The density of word pairs per square is represented by the square's color. Higher rank indicates that words in a given word pair are determined to be more similar, whereas low rank indicates dissimilarity.
(c) fastText
(d) C+F Fusion
plots for all the test sets combined comparing our dictionary models  $\mathrm{(COLEX_{mono}}$ ,  $\mathrm{COLEX}_{\mathrm{cross}})$ , the best baseline (fastText), and our best fusion model  $\mathrm{(COLEX_{cross} + fastText)}$ . The color intensity of a square region corresponds to the number of word pairs in that region. We analyze the results of these different experiments next.