Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphs
Authors: Yihong Liu
Haotian Ye
Leonie Weissweiler
Renhao Pei
Hinrich Schuetze
Venue: EMNLP 2023 Findings (2023)
Keywords: multilingual
colexification
transfer learning
TL;DR: We propose multilingual graphs based on colexification patterns for massively crosslingual transfer learning.

Abstract: In comparative linguistics, colexification refers to the phenomenon of a lexical form conveying two or more distinct meanings. Existing work on colexification patterns relies on annotated word lists, limiting scalability and usefulness in NLP. In contrast, we identify colexification patterns of more than 2,000 concepts across 1,335 languages directly from an unannotated parallel corpus. We then propose simple and effective methods to build multilingual graphs from the colexification patterns: \textbf{ColexNet} and \textbf{ColexNet+}. ColexNet's nodes are concepts and its edges are colexifications. In ColexNet+, concept nodes are additionally linked through intermediate nodes, each representing an ngram in one of 1,334 languages. We use ColexNet+ to train $\overrightarrow{\mbox{ColexNet+}}$, high-quality multilingual embeddings that are well-suited for transfer learning. In our experiments, we first show that ColexNet achieves high recall on CLICS, a dataset of crosslingual colexifications. We then evaluate $\overrightarrow{\mbox{ColexNet+}}$ on roundtrip translation, sentence retrieval and sentence classification and show that our embeddings surpass several transfer learning baselines. This demonstrates the benefits of using colexification as a source of information in multilingual NLP.

Crosslingual Transfer Learning for Low-Resource Languages Based on Multilingual Colexification Graphs
Yihong Liu", Haotian Ye" Leonie Weissweiler"s oo Renhao Pei", and Hinrich Schütze"
"Center for Information and Language Processing, LMU Munich
Munich Center for Machine Learning (MCML)
{yihong, yehao, weissweiler] Ocis. 1mu. de
Abstract
In comparative linguisties,, colexification refers the phenomenon of lexical form conveying two more distinct meanings. Existing work on colexification patterns relies on annotated word lists, limiting scalability and usefulness in NLP. In contrast, we identify colexification patterns more than 2,000 concepts across 1,335 languages directly from an unannotated parallel corpus. We then propose simple and effective methods to build multilingual graphs from the colexification patterns: ColexNet and ColexNet+. ColexNet's nodes are concepts and its edges are colexifications. In ColexNet+, concept nodes are additionally linked through intermediate nodes, each representing an ngram in one of 1,334 languages. We use ColexNet+ train ColexNett. high-quality multilingual embeddings that are well-suited for transfer learning. In our experiments, we first show that ColexNet achieves high recall on CLICS, dataset of crosslingual colexifications. We then evaluate ColexNet+ roundtrip translation, sentence retrieval and sentence classification and show that our embeddings surpass several transfer learning baselines. This demonstrates the benefits of using colexification as a source of information in multilingual NL.P.
Siaen
Hal
wind
MuSppeedd
<blow>
eviolent>
Figure 1: A subgraph of ColexNet+. Circles (<air>, <storm>, ...): concept nodes. Rectangles (nlqiSndiihis, Iwo:Syamos, ...): ngram nodes (each prefixed by its ISO 693-3 code). Ngram nodes realize colexifications, e.g.. <air> and <storm> linked, through translations in the parallel corpus, to the Querétaro Otomí ngram "ndöhi" (i.e. Sndöhhi)
languages (Zhou et al., 2023). Alternatively, multi- lingual word embeddings are widely used in NLP because of their simplicity and good performance (Ammar et al., 2016; Lample et al., 2018; Jawan- puria et al., 2019). However, most existing multilin- gual embeddings are learned through word-context information, without leveraging global cooccur- rence information individual languages or across languages, which can help distinguish distinct meanings conveyed by a lexical form. Therefore, we see pressing need in NLP for massively multi- lingual word embeddings that span a large number of languages (1,335 in case) and that specifi- cally account for global occurrence and are good basis for crosslingual transfer learning.
1 Introduction
Multilingual representations are beneficial in natu- ral language processing (NLP) due to their ability to transfer knowledge across languages (Artetxe and Schwenk, 2019; Conneau et al., 2020; Fan et al., 2021). Typically, such representations are learned through pre-training Large Language Mod- cls (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Touvron al., 2023) multilingual word embeddings (Ammar al., 2016; Lample al., 2018; Dufter et al., 2018). However, LLMSs require enormous amounts of data to train, limiting their use mostly to high-resource and medium-resource
Colexification has gained increasing attention in comparative linguistics and crosslingual NLP. According to François (2008), a language colex- ifies two distinct meanings if it expresses them with the same lexical form. Different languages have different colexification patterns. For exam- ple, while English has separate words for <hand>

---

and <arm>', Russian 'pyka' colexifies these two concepts. Most prior work explores colexification (Floyd et al., 2021; Brochhagen and Boleda, 2022; List, 2023) using manually curated crosslingual datasets that consist of multilingual word lists such CLICS (List, 2018; List et al., 2018; Rzymski et al., 2020). However, relying on these datasets has several limitations: extension to more lan- guages and more concepts be challenging; these datasets contain lists of lemmata and (in a corpus-based approach for low-resource languages without morphological resources) cannot casily be used for the processing of occurrences in context.
lingual graph that additionally contains ngrams in 1,334 languages that instantiate those patterns. (iii) We contribute to crosslingual transfer learning, by using ColexNet+ to generate multilingual embed- dings: ColexNet+. We show that ColexNet+ out- performs several baselines roundtrip translation, verse retrieval, and classification. (iv) We evaluate ColexNet on CLICS and show that we identify a large portion of the ground-trutt colexifications. (v) Going beyond many works on crosslingual trans- fer that focus on transfer from English, we sys- tematically investigate the effect of the source lan- guage on successful transfer with ColexNet+: we use 1,245 languages as sources and experiment on 1,245 x 1,245 transfer directions. (vi) We make our code, graphs, and embeddings publicly available.
To overcome these limitations and boost crosslin- gual transfer learning especially for low-resource languages, we use the Parallel Bible Corpus (PBC) (Mayer and Cysouw, 2014), which has verse-level aligned translations of the Bible in 1,335 languages, to identify colexification patterns (a verse in PBC roughly corresponds to sentence). With the iden- tified patterns between wide range of concepts, we propose novel algorithms that efficiently build large-scale multilingual graphs. To the best of our knowledge, this is the first work that constructs graphs of colexification and trains multilingual rep- resentations for crosslingual transfer learning di- rectly from a parallel corpus on a large scale. We show that the graphs capture the links between concepts across languages and that the derived multilingual representations considerably improve crosslingual transfer on downstream tasks. Previ- ous work on building monolingual graphs (Jauhar et al., 2015; Ustalov et al., 2017) or multilingual graphs (Harvill et al., 2022; Jafarinejad, 2023; Chen et al., 2023) is different as it (1) does not consider words in context and only uses lemmata, (2) is based on external sense inventories such as WordNet (Miller, 1995) and BabelNet (Navigli and Ponzetto, 2012; Navigli et al., 2021), which are not available for many low-resource languages, and (3) does not investigate the crosslingual transfer- ability of the multilingual representations on NLP downstream tasks such as sentence retrieval or clas- sification in a crosslingual scenario.
2 Related Work
There are many ways to learn multilingual word embeddings. One common way is to first learn monolingual embeddings on each language sep- arately through, e.g., Word2Vec (Mikolov et al., 2013), GloVe (Pennington al., 2014), or fast- Text (Bojanowski et al., 2017), and then map them into the same space (Artetxe et al., 2017; Lample et al., 2018; Artetxe et al., 2018). Another group of methods uses parallel corpora to directly learn bilin- gual embeddings (Hermann and Blunsom, 2014; Chandar et al., 2014; Levy et al.. 2017). Our work is related to that of Dufter et al. (2018), which also learns embeddings on the PBC, but does not take advantage of colexification, i.c., the explicit modeling of relations between colexified concept- s/ngrams. We use S-ID (Levy et al., 2017) and embeddings from Dufter et al. (2018) as baselines.
One of the best-known and widely used multilin- gual resources is BabelNet (Navigli and Ponzetto, 2012; Navigli et al., 2021). BabelNet has been used for learning or enhancing embeddings (Ia- cobacci et al., 2015;
Camacho-Colladde and Pile- hvar, 2018; Conia and Navigli, 2020; Levine et al., 2020; Harvill et al., 2022; Chen et al., 2023) for lexical-level tasks such as semantic word similarity and word sense disambiguation (Speer and Lowry- Duda, 2017; Conia and Navigli, 2020; Procopio et al., 2021; Navigli et al., 2022). Our focus is the coverage of many more languages (i.e., larger scale in terms of languages) for crosslingual trans- fer learning. While hand-curated lexica often have
The contributions of this work are as follows: (i) We present Net, a graph of concepts based on colexification patterns that are directly extracted from a parallel corpus. (ii) By extending ColexNet, we further present ColexNet+, a large-scale multi-
We represent a concept (from any language) hy surround- ing an English word that refers to the concept with "<>".
ccm/ct of sn lp/
Colex iiciationnee

---

better quality than automatically learned resources, they are not available for most of our languages. UI- timately, the two approaches should be combined. Colexification was introduced by Haspelmath (2003) in the context of grammatical semanties. François (2008) then used colexification as the foundation for studying semantic change crosslin- guistically. CLICS (List, 2018; List et al., 2018; Rzymski al., 2020) is crosslingual database that facilitates research on colexification. Lan- guages can differ in their colexification patterns, which are influenced by many factors such as hu- man cognition, language family, and geographic area (Jackson et al., 2019; Xu et al., 2020; Segerer and Vanhove, 2022). An empirical study by Bao et al. (2021) indicates that pair of concepts is colexified in every language. On the other hand, a recent investigation on conceptualization from the PBC shows some concepts are more likely to be involved in colexification than others (Liu et al., 2023). Such universal colexification patterns across languages reflect crosslinguistic similarities (Youn et al., 2016; Georgakopoulos et al., 2022). Therefore, by integrating colexification patterns of many languages possible, we can generate multilingual representations that are suitable for massively crosslingual transfer learning.
Linglish corpus and regard each lemma a concept. The set of all lemmata forms the concept pool F.
Conceptualizer. Conceptualizer (Liu et al., 2023) creates a bipartite directed alignment graph between source language concepts and target lan- guage strings. It consists of a forward pass (FP) and a backward pass (BP). This kind of two-step workflow is also used in extracting semantic rela- tions (Dyvik, 2004) and paraphrases (Bannard and Callison-Burdee 2005) from bilingual parallel cor- pora. A key difference compared with this prior work is that Conceptualizer works on the ngram level instead of the token level; this facilitates the extraction of any associations hidden inside words. In Conceptualizer, FP first searches for target lan- guage ngrams highly associated with given fo- cal concept; BP then searches for English ngrams highly correlated with the target ngrams identified in FP. The association is measured using x² score. The process can detect if the conceptualization of the focal concept diverges in any language. For ex- ample, starting from concept <hand>, FP finds the Russian ngram 'pyk', and BP then finds two En- glish ngrams 'hand' and 'arm'. This indicates that the conceptualization of these concepts diverges in English and Russian. The divergence of conceptual- ization in the lexical forms indicates a difference in their colexification patterns: Russian colexifies the concepts <hand> and <arm> (in the word 'pyk') ) while English does not.
3 Methodology
3.1 Data
We use 1,335 Bible translations from the PBC cor- pus (Mayer and Cysouw, 2014). Each translation is from different language (identified by its ISO 639-3 code). Prior work (Asgari and Schütze, 2017: Dufter et al., 2018; Weissweiler et al., 2022) has used subsets of the corpus. In contrast, we follow Conceptualizer (Liu et al., 2023) and use all par- allel verses between English and other languages. This gives us better coverage of concepts and the contexts in which they occur.
Let f be a focal concept in F and Vy the set of verses in which f occurs. FP identifies set of ngrams T in target-language where each ngram can refer to concept f, i.e., T FP I). We exhaustively search all ngrams within all to- kens in the parallel corpus in target language / for high correlation with Vy. This procedure is simi- lar to Östling and Kurfah (2023)'s subword-level alignment, but we align concepts in English and subwords in other target languages. E.g., start from <hand> and find that the Russian ngram 'pyk' has the highest correlation with Vehand>, which in- dicates 'pyk' can refer to <hand>. Like Conceptu- alizer, we use x² as a measure of correlation and iterate FP until the cumulative coverage of focal concept f exceeds a threshold a = 0.9, but for maximum of M = 3 iterations. See SA for a discussion of these hyperparameters.
Forward Pass.
3.2 Colexification pattern identification
Concept Pool. Conceptualizer (Liu et al., 2023) uses a small manually selected group of focal con- cepts, i.e., concepts of interest (83 in total) and constructs set of strings to represent each concept. For example, it uses ($bellys, $bellies$) to represent the focal concept <belly>, where the word boundary. Manually defining the sets is not feasible when a large number of concepts are to be explored. Thus, in this work, we lemmatize the
Ut
Simillrr to the setting in Conceptualizer (Liu et al., 2023). we use $ to denote token boundaries; e-g., kk āke, Skeš, k, ke, keš, e, e$ are all valid ngrams of token SkeS.

---

Backward Pass. BP is essentially the same as FP. but the search direction is reversed. Let Vr be the set of verses in which at least one ngram in T (identified in FP for target language I and concept f) from target language / occurs. We exhaustively search all concepts e from the concept pool F for high correlations with Vr. Let C' 1)) be the final set of identified concepts. If |C| and E C c f. this indicates the ngrams can only refer the concept f according to the bilingual context. Alternatively, |C| > 1, this indicates language colexifies concepts in C' through ngrams T. For example, by performing BP on ngram 'pyk', we get <hand> and <arm> as the result, which indicates Russian colexifies <hand> and <arm>. Notably, since we consider ngrams instead of tokens on the target language side, this allows to also identify partial colexification patterns in BP, i.e. patterns that do not involve an entire word, but rather part of it. We show such examples in $B.
Algorithm 1: ColexNet & ColexNet+
Input: of languages concept pool F, minimum number of languages threshold X;
Output: ColexNet Gt. ColexNet+ G2:
Gi t- graph with F nodes and edges E₁;
G2 + graph with no nodes V and no edges &2;
V
F;
w.(() )
and ue, (-) 0 by default;
for I. do
for f E F
7
T' T
FP(S.
C BP(T,
V + VUT:
for C do & & wc((J.)))+
13
wh (()(())) + w. w.((f.)))TT
14
end
end
end
E & do if (e) < then remove from &
19
20
end
end
for
E E, do (f).f)) f for ((fi do & + (o. end
3.3 ColexNet
We run FP and BP for all 1,806 focal concepts in the English concept pool F that have frequency between 5 and 2000 and for every language I in set of 1,334 target languages (excluding En- glish). This allows us to uncover the colexifica- tion patterns in 1,334 languages. We formalize the relations of the colexification patterns as an undirected graph, where each node is a concept represented by an English lemma and each edge in- dicates that at least one language colexifies the two connected concepts. Formally, let ( & We; wr) be weighted undirected graph on vertices F, i.e., the concept pool, where & is set of undirected edges; we an edge weighting (counting) function: F + Z4, which returns, for a pair of concepts, the number of languages colexifying them; Wn is an edge record function, which returns all ngrams that colexify a given pair of concepts. We show the graph construction in Algorithm 1.
end 28 remove nodes in Gi and G2 that have zero degree; 29 return Go. G2
the data. See Table 4 for the influence of different on graph statistics. In addition, we remove zero- degree nodes to filter out isolated concepts.
3.4 ColexNet+
ColexNet only contains concepts that are expressed in English lemmata and cannot be directly used to learn multilingual representations for the target lan- guages. Therefore, we propose ColexNet+, a large multilingual graph expanded from ColexNet by including target language ngrams that instantiate the colexification patterns identified in ColexNet. Specifically, we replace each edge (fi, fe) with a set of new pairs of edges: (1) find the set of ngrams Wn (f1, ( f2)) that colexify concepts fi and f2 (in any language) and (2) for each ngram t in the set, insert new edges v) and (v. f2). To define clean bipartite structure, we do not include the original (f1, f2); this guarantees that only concept- ngram edges and no concept-coneept edges occur. In addition, any two related concepts (i.e., there is an edge connecting the two concepts in ColexNet) are always implicitly connected through ngram nodes in ColexNet+ that associate the two con-
In this study, we use a threshold À to control the confidence of the colexification edges: we remove an edge e if we(e) if two concepts f1 and f2 are colexified in many languages, can be more certain that the edge between fi and f2 is correctly identified. Looking it the other way around, if two concepts are only colexified few languages, this might be a wrongly identified pattern because of verse-level misalignment, free translation, or other errors in
A. The intuition is that:

---

cepts. Figure shows subnetwork of ColexNet+ consisting of a few concepts and ngrams in dif- ferent languages that colexify them. The graph construction is shown in Algorithm I.
evaluates the colexification patterns we identify in ColexNet. Experiments (2), (3) and (4) evaluate
the learned multilingual embeddings ColexNet+.
As ColexNet+ is expanded from ColexNet, this allows us to only include pairs of edges expanded from reliable edges (wa(e) > X) in ColexNet. The number of nodes and edges included in ColexNet+ is thus influenced by A. The higher X, the fewer nodes and edges will be in ColexNet+, $A presents statistics and performance for different values of X.
4.1 Baselines
To evaluate the effectiveness of ColexNet+, our multilingual embeddings, we consider several pre- viously proposed strong multilingual embeddings as baselines for downstream tasks. The dimen- sion of all embeddings (ours and the baselines) is set to 200 for fair comparison. In addition, we consider three non-embedding baselines: bag-of- words (BOW), XLM-R (Conneau et al., 2020) and Glot500-m (ImaniGooghari al., 2023). The first random baseline and is expected to perform the worst because BOW model only trained on the English corpus, which does not directly transfer to other languages. The latter two are strong multi- lingual pretrained models. XLM-R is pretrained on 100 languages while Glot500-m is continued- pretrained version of XLM-R on the Glot500-c corpus (ImaniGooghari et al., 2023) that includes more than 500 languages. We choose the base ver- sion of these multilingual pretrained models. We introduce the embedding baselines below.
3.5 Multilingual Embedding learning
To capture the semantic relations among the nodes and the structure of ColexNet+, we use Node2 Vec (Grover and Leskovec, 2016) to generate node rep- resentations. Let e be the node that a random walk currently resides in, & the node that the walk has tra- versed in the last step, and z the node that the walk will visit in the next step. Vode2Vec calculates the unnormalized transition probability from to x as Tex Opy (t, x)- w((x,=)) for sampling the next node x in the graph, where w((n, æ)) is the weight the undirected edge (u, x),
}
if dtz = 0
app(1, =
if diz = 1
S-ID embedding. Levy et al. (2017) show that S-ID embeddings, which leverage the sentence ID feature. are effective in learning good multilingual embeddings from parallel corpora. We use pairs of verse ID identifier and token in this verse input to Word2Vec (Mikolov et al., 2013) to train S- ID embeddings. For example,, the pairs (01049027, Wolf) and (01049027, Rit data because ' (resp. 'Wolf") occurs in Chinese (resp. German) in verse number 01049027. This is strong baseline because the verse number is an abstract representation of the context. Therefore encourages words occurring across languages in the same verse to have similar embeddings.
if dez = 2
and diz is the shortest path distance between / and x. The transition probability determines if either a new node or an already-visited node (regardless a concept or ngram node) will be sampled. In ColexNett, diz + for any nodes t and x, because a concept (resp. ngram) node will not connect with other concept (resp. ngram) nodes. We set return parameter = -5 and in-out parameter q 2 in the hope of encoding more "local" information, this setting approximates breadth-first sampling according to Grover and Leskovec (2016).
will be presented in the
Below, we show that the multilingual representa- tions trained this way have some desirable proper- ties, e.g., representations of ngrams from different languages that refer to the same concept can be highly cosine-similar, which is important for zero- shot crosslingual transfer learning.
CLIQUE & N(I) embedding. CLIQUE embed- dings (Dufter et al., 2018) are learned on cliques extracted from PBC. Each clique is a set of tokens from different languages that refer to the same con- cept. The embeddings are then learned from token- clique pairs. Additionally, take the connections between concepts into account, Dufter al. (2018) consider the neighbors (tokens that are connected with the current node in the dictionary graph) of each token and train embeddings on those pairs neighbors, which we refer to as (tt) embedding.
4 Experiments
To evaluate our proposed methods, we conduct the following experiments: (1) colexification iden- tification; (2) roundtrip translation; (3) verse re- trieval; and (4) verse classification. Experiment (1)

---

Eftomal-aligned embedding. We construct an alignment graph of words by using Eflomal (Östling and Tiedemann, 2016) and learn embed- dings on the graph as another strong baseline. Specifically, we align the English Bible with Bibles in all other target languages. We define the edge set of the graph the set of all edges that connect an English word with its aligned target language word (if there least two such alignments). Finally, we use Node2Vec (same hyperparameters as for ColexNet+) to learn multilingual embeddings.
MicroRec.
MacroRec.
#aw_colex
1220
0.71
0.80
580.87
1056
0.63
0.77
84.34
10
1001
0.58
0.73
42.04
935
0.54
0.70
22.91
50
833
0.48
0.66
10.69
100
761
0.42
0.62
5.78
Table The number of common concepts (P) in ColexNet and CLICS, Micro and Macro average call, as well as the average number of colexification patterns per concept that are not in CLICS (#aw_cd sow_cllex for different thresholds À (the minimum number of lan- guages for keeping a colexification edge).
4.2 Colexification identification
We first evaluate how well ColexNet performs at identifying colexification patterns. We use CLICS (List, 2018; List et al., 2018; 8; Rzymski et al., 2020), a database of colexifications, as the gold standard. Each node in CLICS is a concept expressed in En- glish. In ColexNet, we use English lemmata ex- pressions of concepts whereas CLICS also includes short noun phrases. We only consider the common concepts, i.e., concepts that are expressed as En- glish words and occur in both CLICS and ColexNet. For each start concept s in the common concepts P, let (s)) be the neighbors in CLICS, i.e., a set of concepts that have colexification relation with and C(s) be the neighbors in ColexNet. Then we compute the recall for as (T(s) ((s))/[T(ss)
If the constraint is stricter, fewer concepts and fewer edges (both colexification edges contained and not contained in CLICS) will be included in ColexNet. Thus, we observe a consistent drop in both micro and macro recall. On the other hand, observe decrease in #aw_coles when we in- crease X, as CLICS edges are less likely be re- moved than edges missing from CLICS: many lan- guages can share the same colexification patterns for some concepts whereas edges not in CLICS will not be shared across many languages. This can be verified by the steepness of the decrease in #aw cclex. #aw_colex. From À to 5, around 500 edges not in CLICS are removed for each concept. When À >> 5, the speed decreases, suggesting the iden- tified colexification edges are more reliable. In summary, high recall indicates that we successfully identify many ground-truth colexifications directly from PBC. It is important to note that CLICS' cov- erage is far from complete for low-resource lan- guages: for many of them, fewer than 100 concepts included in CLICS. Therefore, #aw_colex gives some indication of performance discrepancy be- tween CLICS and ColexNet, but many of the edges not in CLICS are actually correct. On the other hand, ColexNet not immune to semantic errors (Peirsman and Padó, 2008), such as antonyms, hy- pernyms, or hyponyms, due to co-occurrence or free translation. See $B for a detailed analysis of the identified colexifications.
To have global view of the performance, re- port the micro average recall (MicroRec.):
MicroRec. = [Ts)
[TE)(T(s)
macro average recall (MacroRec.):
MacroRec. = Σ [(s))
PP
sEP
and the average number of incorrect (not present in CLICS) colexifications per concept (#aw_colex)):
[C(s)
-ss)
#aw_coles Σ
P
sEP
#aw_coles has a similar function to precision. We do not use precision as a measure because it can underestimate the performance, as many patterns included in ColexNet can actually be correct (sce $B for examples). The #aw_colex measure can bet- ter and more directly reflect how the exact number of "incorrect" patterns per concept changes with respect to the value of X. Results are shown in Table
4.3 Roundtrip translation
We additionally use roundtrip translation (Dufter et al., 2018) to assess the quality of multilingual representations. Let [lo, I1 12, ..., 1n] be sequence of languages where lo [ is the source language and * lo 1 S is R different intermedi- ate languages. Roundtrip translation starts with a word wy in lo and iteratively finds the word w, in

---

roundtrip translation
verse retrieval
verse classification
top-1
top-5
top-10
top-1
top-5
top-
BOW
0.02
0.05
0.06
0.09
S-ID (Levy et al., 2017)
0.10
0.21
0.25
0.17
0.29
0.35
0.32
CLIQUE (Dufter et al., 2018)
0.22
0.63
0.79
0.41
0.62
0.70
0.44
[() (Dufter et al., 2018)
0.22
0.53
0.65
0.28
0.46
0.55
0.47
XLM-R (Conneau et al., 2020)
0.04
0.07
0.09
0.15
Glot500-m (ImaniGooghari 2023)
0.11
0.17
0.21
0.22
Eftomal-aligned
0.24
0.58
0.70
0.61
0.76
0.81
0.48
ColexNet+
0.44
0.85
0.93
0.65
0.80
0.84
0.49
Table 2: Results of different multilingual embeddings roundtrip translation, verse retrieval, and verse classification tasks. Each number for roundtrip translation (top-k accuracy, & E (1,5, 10)) is the average of 10 runs with 3 randomly selected intermediate languages. Each number in verse retrieval (top-k accuracy, & {1,5, 10)) and verse classification (macro F)) is the average over all available languages (1,250 for verse retrieval, 1,245 for verse classification). We also report the performance of BOW, XLM-R, and Glot500-m. The first serves as a random baseline whereas the latter are strong multilingual model baselines. The performance reported for Glot500-m is different from the original paper as it was evaluated on a subset of languages the model supports, while we evaluate the model all languages that ColexNet+ supports, making comparison casier. Bold: best result per column.
language 4, (1 r < R) that is closest to word 1r-1 in language 1y-1 in the embedding space. If we = wn, this indicates that the R "intermedi- ate" words have representations similar to wo and represent the meaning of Wn- We compute the per- centage of roundtrips for wo that are successful, i.c.. we wr (top-1 accuracy). addition, we also report top-5 and top-10 accuracies (i.e., wp in the k (k: = or = 10) nearest neighbors). We set R= 1, 40 = English and take 1,654 English words that occur in all embedding spaces as the starting point wo. For each trial, we randomly select three intermediate languages and then compute results for each of the 1,654 top. We run this experiment ten times and report averages. We ensure that the intermediate languages are different in each trial.
verses for train/valid/test sets in more than 1,500 languages, Each verse is annotated with one of six classes: 'recommendation' 'faith', "description', 'sin', 'grace', and 'violence'. We use a subset of 1,245 languages, those covered by both Taxi1500 and ColexNet+. We perform zero-shot transfer by training a logistic classifier on English train and evaluating on the test set of the other 1,244 lan- guages. Similar to verse retrieval, we represent verse the average of its embeddings. We report macro F1, first averaged over verses (per language) and then averaged over languages.
4.6 Results & Discussion
Table 2 shows that ColexNet+ BOW, S-ID, CLIQUE, N(t., Glot500-m and Eflomal-aligned on all three tasks: roundtrip translation, verse re- trieval, and verse classification. ColexNet+ shows large improvement over the basclines, especially for roundtrip translation and verse retrieval. The bad performance BOW is expected, as previ- ously mentioned, because the English vocabulary does not necessarily transfer to other languages. ColexNet4's improvement over S-ID is probably due the fact that there is only verse ID informa- tion provided serve as verse-level context infor- mation S-ID. Token-level alignment information, however, is not available to S-ID. In other words, using abstract context identifiers alone cannot pro- vide enough information to learn good multilingual embeddings for crosslingual transfer. When com- paring ColexNet+ with XLM-R and Glot500-m we see a clear improvement either verse retrieval
4.4 Verse retrieval
Similarly to Glot500-m, we use 500 English- aligned verses from PBC for verse retrieval. 1,250 languages are used (we remove 85 languages that cover fewer than 400 out of the 500 verses). We rep- resent each verse as the average of the embeddings of its units. Given verse ve in English, we find the most cosine-similar verses iy in target language I. We then compute top-1, top-5 and top-10 accuracy for the returned ranking (i.e., the correct verse is in the t,p- top-5, top-10 10 nearest neighbors) and average first over verses and then languages.
4.5 Verse classification
We evaluate our multilingual embeddings on Taxi 500 (Ma et al., 2023). It provides 860/1 860/106// 106/1 11

---

verse classification. The major reason is that both XLM-R and Glot500-m are not trained on all languages that are supported by ColexNet+. Due to a lack of data in some low-resource languages, it is difficult to train a good language model in those languages. In contrast, ColexNet+ demon- strates the possibility of multilingual embeddings: with small multilingual corpus where we can ex- tract colexifications, it is already enough to support large-scale zero-shot transfer for the low-resource languages by training embeddings.
distribution
Figure 2: Degree centrality (left). betweenness centrality (middle) and degree distribution (right) of ColexNet.
Analysis
5.1 Analyses on ColexNet
Basic statistics. We find that ColexNet has one very large connected component along with few small connected components. See a visualization of the largest community in ColexNet (A 50) in Figure in the Appendix. Therefore, in the largest community, there is always a path in the colexification graph between two concepts even if they are less related. Figure 2 shows degree/ betweenness centrality and degree distribution of ColexNet. From the figure, can infer that the connectivity can be attributed to (1) small group of concepts that are involved in many colexification patterns and (2) a small group of edges serving as "bridges" to connect concepts that are rarely colex- ified in some languages. Therefore, ColexNet, a graph built by the identified colcxification patterns across many languages, approximately forms a small-world or scale-free (Barabási and Bonabeau, 2003) network. See $A.2 for graph-related statis- tics of ColexNet under different X.
CLIQUE, (t),, and Eflomal-aligned achieve similar performance on roundtrip translation (top- I). However, when becomes larger (A: = or 10), we see that CLIQUE performs better than (()) and Eflomal-aligned. This is not surprising, since CLIQUE specifically creates cliques of tokens that are translations of each other in different languages. Therefore the representations of translations should be similar. fflomal-alignee also achieves good performance on roundtrip translation when & is large (k = 10) and very close performance to ColexNet+ in verse retrieval classification. There are a few possible explanations. First, the word alignments are noisy in Eflomal-aligned because it operates the token level and any information hidden inside each token (i.e., ngrams inside each token) cannot be extracted and utilized (see the discussion also in Liu et al. (2023)). Therefore, by increasing & in roundtrip translation, the influ- ence of such alignment noise is offset, resulting in better results. Second, as we use the average of embeddings of tokens in a verse the verse representation in verse retrieval / classification; this can mitigate the impact of unimportant tokens.
Communities. We use the Louvain algorithm (Blondel et al., 2008) to find communities ColexNet. We identify 288 communities. Each community forms cluster of semantically related concepts. Figure 3 gives the example of commu- nity#29: it contains several concepts related to <wind>, <storm> and <wave>, We see that <wind> often colexified with <blow> (wind blows), with <wave> (waves are caused by wind) and with <vio- lent> (winds can be fierce). At the center of a com- munity, we often find a densely connected clique, indicating their connections are strong in many lan- guages. Some concepts, located at the fringe of the community and connected with one of the densely- connected concepts in the center, less related the semantic field of the community and serve "bridges" to connect with other communities. See &C for further details of the identified communitics.
For verse classification, we find that different em- beddings achieve similar performance except for S-ID. On the one hand, this phenomenon indicates that S-ID, though it learns from abstract context information, cannot align words from different lan- guages that refer to the same concepts well, thus preventing transfer from English low-resource languages. On the other hand, might indicate that classification a less difficult task: it does not require the model to have equally good alignment for all concepts as the model can achieve good results just by aligning important concepts. Never- theless, ColexNet+ still achieves better results than other baselines, suggesting it has better zero-shot transferability. See SE for complete results.
5.2 Transfer learning beyond English
NLP research in general and even typological studies are frequently conducted from an English-

---

verse retrieval
verse classification
top-1
top-5
top-10
arb
0.56
0.72
0.78
0.47 (0.07)
rus
0,55
0.72
0.78
0.48 (0.07)
Stipef
zho
0.60
0.77
0.82
0.49 (0.05)
Steppeets
apn 0.21
0.38
0.46
0.38 (0.07)
$howes
muh
0.22
0.39
0.48
0.38 (0.07)
sll 0.24
0.42
0.51
0.39 (0.08)
setrmng
trieience
avg.
0.46
0.64
0.71
0.44 (0.08)
Table 3: Verse retricvalcllassifictiin for three high- resource languages, the three worst performing lan- guages, and average over all languages (avg.). We also report BOW results for verse classification (in parenthe- ses), which serves as the random baseline. In contrast to the good performance for Arabic (arb), Russian (rus) and Chinese (zho), Apinayé (apn), Mündü (muh) and Salt- Yui (sIl) each pose specific difficulties for inducing reliable colexification patterns.
Stonip
Figure 3: Community #29. Line thickness indicates the number languages that instantiate a colexification.
centric perspective. To reduce this bias and further verify our multilingual embeddings' transfer capa- bility, additionally use all available languages (1,245 languages) as the query / train languages for retrieval and classification tasks. To this end, we conduct large-scale experiments that contain 1,245 1,245 transfer directions. The setup is the same as in $4, where each language takes the role of English as the query / train language. We again represent each verse as the average of the embed- dings of its units. For each language, we calculate the average top-k (I: 1, 5, 10) accuracy for verse retrieval and macro Fi for verse classification over all other languages except the language itself.
from most high-resource languages. Apinayé and Mündü seem to frequently use several consecutive whitespace-tokenized syllables to express single concept, which makes finding the correct align- ments much harder. Salt-Yui, on the other hand, seems to be highly ambiguous because the writing does not reflect its contrastive tones (Irwin, 1974). We hypothesize such ambiguity can negatively in- fluence performance. See $F for an analysis of the factors that can influence transfer performance.
6 Conclusion
In Table 3, we list the transfer performance of three major languages that are typologically differ- ent from English: Arabic (arb), Russian (rus), and Chinese (zho); and three languages that achieve the worst overall performance: Apinayé (apn), Mündü (muh), and Salt-Yui Yui (sll). See $F for complete results for all languages. For high-resource lan- guages, the performance is close to that achieved for English (see Table 2), indicating that the ngrams are well-aligned and ColexNet+ has good transfer ability. Chinese performs better than Arabic and Russian. The possible reasons are as follows: (1) Both Arabic and Russian are morphologically rich whereas Chinese is not. Morphological variation makes finding aligned ngrams in the forward pass harder, with negative impact on performance; (2) To prevent bad tokenization for Chinese, we al- low all ngrams (unlimited-Iength) combination of continuous characters) in verse to be candidates in the forward pass. This setting gives ngrams more freedom and thus better results are expected. For the three low-resource languages, we find that they diverge morphologically and typologically
In this work, we present the multilingual graphs ColexNet and ColexNet+, based on colexifications extracted from a highly parallel corpus. Compar- ing with CLICS, we show that we identify many cold-standard patterns in ColexNet. In addition, analyze the structure of ColexNet and show it nearly forms scale-free graph, with many com- munities of semantically related concepts. Most importantly, we contribute crosslingual trans- fer learning by inducing multilingual embeddings ColexNet+ that are learned on ColexNett. Our experiments indicate that ColexNet+ largely repre- sents concepts across languages in the same seman- tic space. We show that ColexNet+ outperforms several approaches, including multilingual embed- dings and pretrained models, on three downstream tasks. This indicates that embeddings learned from colexification graphs improve crosslingual transfer, especially for low-resource languages for which often infeasible to pretrain good models. Fi- nally, our embeddings exhibit robust transfer per- formance across many different source languages.

---

Limitations
cross cross-lingens lingual mappings of word embeddings. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 789-798, Melbourne, Australia. As- sociation for Computational Linguistics.
Theoretically, one could identify, explore, and ana- lyze colexification patterns from any parallel cor- pora and construct graphs of colexifications using the methods proposed in this paper. We use the PBC, genre-specific parallel corpus in this work. which can limit some of the concepts to religions. Nevertheless, the goal of this work is to explore colexification patterns in as many languages as pos- sible, including a lot of low-resource languages, without relying on any external resources. There- fore, the PBC corpus is good fit for us.
Mikel Artetxe and Holger Schwenk. 2019. sively multilingual sentence embeddings for zero- shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610.
Mas-
Ehsaneddin Asgari and Hinrich Schüitze 2017. Past, present, future: A computational investigation of the typology of tense in 1000 languages. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 113-124, Copenhagen, Denmark. Association for Computa- tional Linguisties.
We conduct extensive experiments verify the crosslingual transfer capability of the multilingual embeddings learned on ColexNet+. However, some experiments are in-domain (the evaluation tasks are still related to the Bible), e.g., verse retrieval and verse classification. The major reason is that we want to test the embedding's performance on all our supported languages. Unfortunately, far we know, evaluation datasets that cover such wide range of languages, including low-resource languages, are missing the community. Some datasets, for example, Tatoeba', support hundreds languages but contain many concepts, c.g., pizza, that do not occur in the Bible. Therefore, we do not evaluate our embeddings on those datasets.
Colin Bannard and Chris Callison -urrchh 2005. Para- phrasing with bilingual parallel corpora. In Proceed- ings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'0S), (O),, pages 597- 604, Ann Arbor, Michigan. Association for Compu- tational Linguistics.
Hongchang Bao, Bradley Hauer, and Grzegorz Kondrak. 2021. On universal colexifications. In Proceedings of the 11th Global Wordner Conference, pages 1-7, University of South Africa (UNISA). Global Wordnet Association.
Albert-László Barabási and Eric Bonabeau. 2003. Scale- free networks. Scientific americun, 288(5)660-69.
A Acknowledgement
Vincent D Blondel, Jean-Loup Guillaume, Renaud
We would like to thank Verena Blaschke for con- structive discussions and Yuxuan Zahed for the im- plementation of the online demo of ColexNet and ColexNet+. This work was funded by the European Research Council (grant #740516).
Lambiotte, and Etienne Lefebvre. 2008. Fast folding of communities in large networks. Jour- nal of statistical mechanics: theory and experiment, (008(10):P100))
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Associa- tion for Computational Linguistics, 5:135-146.
References
Waleed Ammar, George Muleaire, Yulia Tsvetkov, Guil- laume Lample, Chris Dyer, and Noah A Smith. 2016. Massively multilingual word embeddings. arXiv
Thomas Brochhagen and Gemma Boleda. 2022. When do languages use the same word for different mcan- ings? the goldilocks principle in colexification. Cog- nition, 226: 105179.
preprint arXiv: 1602.01925.
Mikel Artetxe, Gorka Labaka, and Eneko Agire, 2017. Learning bilingual word embeddings with (almost) no bilingual data. Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages 451-462. Vancouver, Canada. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 33: 1877-1911
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust seaf-igg method for fully unsupervised
José Camacho-Collados and Mohammad Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. Artif. Intell. Res., 63:743-788.
hhttps://tatoells

---

A. P. Sarath Chandar. Stanislas Lauly, Hugo Larochelle, Mitesh M. Khapra, Balaraman Ravindran, Vikas C. Raykar, and Amrita Saha. 2014. An autoencoder approach learning bilingual word representations. In Advances in Neural Information Processing Sys- tems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Mon- treal. Quebec, Canada, pages 1863-1861.
Alexandre François. 2008. Semantic maps and the ty- pology of colexification. From polysemy to semantic change: Towards typology of lexical semantic asso- ciations, 106:163.
Thanasis Georgakopoulos, Eitan Grossman, Dmmitr Nikolaev, and Stéphane Polis. 2022. Universal and macro-areal patterns in the lexicon. Linguistic Typol- egy; 26(2):39-487.
Yiyi Chen, Russa Biswas, and Johannes Bjerva. 2023. Language embeddings from semantic typology. In The 24rd Nordic Conference on Compu- tational Linguistics.
Colex2lang:
Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceed- ings of the 22nd ACM SIGKDD international con- ference on Knowledge discovery and data mining, pages 855-864.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
John Harvill, Roxana Girju, and Mark Hasegawa- Johnson. 2022. Syn2Vec: Synset colexification graphs for lexical semantic similarity. In Proceed- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2259-5270 Seattle, United States. Association for Computational Linguistics.
Simone Conia and Roberto Navigli. 2020. Conception: Halitingually--enanneeh human-readable concept vector representations. In Proceedings of the 28th International Conference on Computational Linguis- lics, pages 3268-3284, Barcelona, Spain (Online). International Committee on Computational Linguis- tics.
Martin Haspelmath. 2003. The geometry of grammat- ical meaning: Semantic maps and cross-linguistic comparison. In The new psychology of language, pages 217-248. Psychology Press.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cinssllingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 8440- 8451, Online. Association for Computational Lin- guistics.
Karl Moritz Hermann and Phil Blunsom. 2014, Multi- lingual models for compositional distributed seman- tics. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics ( (Vol- une I: Long Papers), pages 58-66 68, Baltimore, Mary- land. Association for Computational Linguistics.
Lawrence Hubert and Phipps Arabie. 1985. Comparing partitions. Journal of classification, 2:193-218
Philipp Dufter, Mengjie Zhao, Martin Schmitt, Alexan- der Fraser, and Hinrich Schütze. 2018 Embedding learning through multilingual concept induction. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume Long Papers), pages 1520-1530, Melbourne, Aus- tralia. Association for Computational Linguistics.
Ignacio lacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. SensEmbed: Learning sense embeddings for word and relational similarity. Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 95-105, Beijing, China. Association for Computational Lin- guistics.
Helge Dyvik. 2004. Translations as semantic mirrors: from parallel corpus to wordnet. Brill, Leiden, The Netherlands.
Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kar- garan, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Mar- tins, François Yvon, and Hinrich Schüitze. 2023, Glot500: Scaling multilingual corpora and language models to 500 languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 1082- 1117, Toronto, Canada. Association for Computa- tional Linguistics.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed Kiskky,, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multi- lingual machine translation. The Journal of Machine Learning Research, 22(1):8839)) 4886.
Sammy Floyd, Kavindya Dalawella, Adele Goldberg. Casey Lew-Williams, and Tom Griffiths. 2021. Mod- cling rules and similarity in colexification. In Pro- ceedings of the annual meeting of the cognitive sci- ence society, volume 43.
Barry Irwin. 1974. Salt-Yui Grammar. The Australian National University.

---

Joshua Conrad Jackson, Joseph Watts, Teague R Henry, Johann-Mattis List, Robert Forkel, Peter J Mucha, Simon J Greenhill, Russell D Gray, and Kristen A Lindquist. 2019. Emotion semantics show both cultural variation and universal structure. Science, 366666472)11117-1522
Yihong Liu, Haotian Ye, Leonie Weissweiler, Philipp Wicke, Renhao Pei, Robert Zangenfeind, and Hin- rich Schütze. 2023. A crosslingual investigation of conceptualization in 1335 languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 12969- 13000, Toronto, Canada, Association for Computational Linguistics.
Fatemeh Jafarinejad. 2023. Synset2node: A new synset embedding based upon graph embeddings. Intelli- gent Systems with Applications, 17:200159.
Chunlan Ma. Ayyoob ImaniGooghari, Haotian Ye, Ehsaneddin Asgari, and Hinrich Schütze. 2023. Taxi1500: A multilingual dataset for text clas- sification in 1500 languages. arXiv preprint arXiv:2305.08487.
Sujay Kumar Jauhar, Chris Dyer, and Eduard Hovy. 2015. Ontologically grounded multi-sense represen- tation learning for semantic vector space models. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computa- tional L Linguistics: Human Language Technologies, pages 683-693, Denver, Colorado. Association for Computational Linguistics.
Thomas Mayer and Michael Cysouw. 2014, Creating massively parallel Bible corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 14), pages 3158- 3163, Reykjavik, Iceland. European Language Re- sources Association (ELRA).
Pratik Jawanpuria, Arjun Balgovind, Anoop Kunchukut- tan, and Bamdev Mishra. 2019. Learning multilin- gual word embeddings in latent metric space: A geo- metric approach. Transactions of the Association for Computational Linguisties, 7:107-120.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013, Efficient estimation of word representa- tions vector space. In Ist International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In 6th In- ternational Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenRe- view.net.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):33 11.39-41.
Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Cecconi. 2021. Ten years of babelnet: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4559-45677 International Joint Conferences on Artificial Intelli- gence Organization. Survey Track.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driv- ing some sense into BERT. In Proceedings the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4656-4667, Online. Asso- ciation for Computational Linguistics.
Roberto Navigli, Rexhina Blloshmi, and Abelardo Car- los Martínez Lorenzo. 2022. Babelnet meaning rep- resentation: A fully semantic formalism overcome language barriers. Proceedings of the AAAI Confer- ence on Artificial Intelligence, 36(11):1224411277)
Omer Levy, Anders Søgaard, and Yoav Goldberg, 2017. strong baseline for learning cross-lingual word em- beddings from sentence alignments. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Vol- ume I, Long Papers, pages 765-774, Valencia, Spain. Association for Computational Linguistics.
Roberto Navigli and Simone Paolo Ponzetto. 2012. Ba- belnet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial intelligence, 193:217-22 22-0000..
Robert Östling and Murathan Kurfah. 2023. Language embeddings sometimes contain typological general- izations. arXiv preprint arXiv:2301.081
Johann- Mattis List. 2018. Data underlying clics version 1.0. Zenodo.
Johann-Matti List. 2023. Inference of partial colexifi- cations from multilingual wordlists. arXiv preprint ar2iv:2302.00739.
Robert Östling and Jörg Tiedemann. 2016. Efficient word alignment with markov chain monte carlo. The Prague Bulletin of Mathematical Linguistics.
fohann-Matiis List, Simon J Greenhill, Cormac An- derson, Thomas Mayer, Tiago Tresoldi, and Robert Forkel. 2018. Clics2: An improved database of cross- linguistic colexifications assembling lexical data with the help of cross-ling linguisic data formats. Linguistic Typology, 22(2):277-306
Yves Peirsman and Sebastian Pad6. 2008. Semantic relations in bilingual lexicons. ACM Transactions on Speech and Language Processing (TSLP), 8(2):1-21.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word

---

representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguisties.
Yang Xu, Khang Duong, Barbara Malt, Serena Jiang, and Mahesh Srinivasan. 2020. Conceptual relations predict colexification across languages. Cognition, 201:104280
Luigi Procopio, Edoardo Barba, Federico Martelli, and Roberto Navigli. 2021. Multimirror: Neural cross- lingual word alignment for multilingual word sense disambiguation. In Proceedings of the Thirtieth International Joint Conference on Artificial Intel- ligence, IJCAI-21, pages 3915-3921. International Joint Conferences on Artificial Intelligence Organi- zation. Main Track.
Hyejin Youn, Logan Sutton, Eric Smith, Cristopher Moore, Jon F Wilkins, Ian Maddieson, William Croft, and Tanmoy Bhattacharya. 2016. On the universal structure of human lexical semanties. Proceedings of the National Academy of Sciences, 113(7):1766 1771.
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang. Cheng Ji, Qiben Yan. Lifang He, et al. 2023. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. arXiv preprint arXiv.2302.09419
Christoph Rzymski, Tiago Tresoldi. Simon J Green- hill, Mei-Shin Wu, Nathanael Schweikhard,, Maria Koptjevskaja- Tamm, Volker Gast, Timotheus A Bodt, Abbie Hantgan, Gereon A Kaiping, et al. 2020. The database of cross-linguistic colexifications, repro- ducible analysis of cross-linguistic polysemies. Sci-
A Choice of hyperparameters and discussion
entific data, 7(1):1-12.
A.1 Forward/backward pass
Guillaume Segerer and Martine Vanhove. 2022, Areal patterns and colexifications of colour terms in the languages of africa. Linguistic Typology, 26(2):247- 281.
Two hyperparameters in the forward pass and back- ward pass when searching for the colexification patterns can influence the results, i.e., (1) the maxi- mum number of iterations M for given concept in each language and (2) the threshold a for the minimum cumulative coverage of the set of iden- tifid ngrams. We set M = 3 and a = .9 as default values for all involved computations. We are different with Conceptualizer (Liu al., 2023) in the setting of M. Conceptualizer sets M to 5 whereas we set 3. The major reasons are as fol- lows, We are searching for colexification patterns with high accuracy. This requires us to identify the larget-language ngrams that instantiate the colex- ifications with high certainty. Based on empirical explorations, we find that when M is large (e.g., 3), we can include less accurate even unre- lated ngrams (because those ngrams are rare and occur in the same verse where the concept occurs, which is also discussed by Liu al. (2023)). By setting M = 3 in the forward pass, we will be more confident that the identified target-language ngrams are highly correlated with the concept and this setting achieves the best performance for few examples in our manual inspection. As for the min- imum cumulative coverage threshold a, directly follow the setting in Conceptualizer, i.e., 0.9, to en- sure that the forward pass and backward pass find enough ngrams/ccncepts while guaranteeing the quality of the associations.
Robyn Speer and Joanna Lowry-Dudde 2017. Concept- task 2: Extending word em-
Net at SemEval-2011 beddings with multilingual relational knowledge. In Proceedings of the I 1thh International Workshop on Semantic Evaluation (SemEval-2007)) pages 85-89, Vancouver, Canada. Association for Computational Linguistics.
Douglas Steinley. 2004.
Properties of the hubert- arable adjusted rand index. Psychological methods, 933:3886.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. 2023. Ilama: Open and effi- cient foundation language models. arXiv preprint M.Xiv:3302.39711
Darrell T Tryon. 1987. Bislama: An introduction to the national language of Vanuatu. The Australian National University.
Dmitry Ustalov, Alexander Panchenko, and Chris Bie- mann. 2017. Watset: Automatic induction of synsets from a graph of synonyms. In Proceedings the 55th Annual Meeting of the Association for Com- putational Linguisties (Volume I: Long Papers). pages 1579-1590, Vancouver, Canada. Association for Computational Linguistics.
Leonie Weissweiler, Valentin Hofmann, Masoud Jalili Sabet, and Hinrich Schuetze. 2022. CaMEL: Case Marker Extraction without Labels. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 5506-5516, Dublin, Ireland. Association for Computational Linguistics.
A.2 ColexNet/ColexNet+ construction
In the construction of ColexNet and ColexNet+, we have an important hyperparameter X: the minimum

---

#nodes
#edges
degree
#components
5870
1000937
170.61
1
4028
122798
30.48
1
10
3562
58031
16.29
1
3133
30175
9.63
2
50
2591
13607
5.25
100
2221
7634
3.44
60
Table 4: Basic statistics of ColexNet under different thresholds A. We report the number of nodes (#nodes), the number of edges (#edges), the average degree per node (degree), and the number of connected components (itcomponents..
A
#nodes
#edges
degree
3,613,546
15,251.571
4.22
Figure 4: Visualization of the largest community which contains 2,581 nodes out 2,591 nodes ColexNet. Each node is concept and each edge indicates that two concepts are colexified in at least 50 languages.
3,611,704
12,197,241
3.37
3,611,238
11,227 402
3.11
3,610,809
10,369.418
2.87
50
3,610,267
9,235,395
2.56
100
3,609,897
8,314,760
2.30
Table 5: Basic statistics of ColexNet+ under different thresholds X. We report the number nodes (#nodes)) the number of edges (#edges), and the average degree per node (degree).
concepts, because most of the nodes ColexNet+ belongs to the ngram nodes. Each concept, how- ever, can be frequently associated with more than 3 concepts in ColexNet, we noticed the average degree of ColexNet (A =50) is around 5.
number of languages for a colexification edge to be included. As shown in Table 4, different À can in- fluence the number of nodes and edges in ColexNet well the number of connected components. It is clear that both #edges and degree decrease dra- matically from À (1) increasing decreases the number of incor- rectly identified colcxification patterns (c.g., due to verse-level misalignment); (2) some colexification patterns might be specific to very few languages. Because of many plausible incorrect edges between concepts, when À = 1, or 10, ColexNet forms large connected graph. When À is larger (e.g., g., 50 100), the graph is longer connected because many less reliable edges are removed from it.
The number of nodes and edges also influences the random walks which used for sampling, thus influencing the quality of multilingual embeddings trained on ColexNet+ using Node2Vec (Grover and Leskovec, 2016). Therefore, we conduct exper- iments using embeddings trained on ColexNet+ under different X. Same as $4, we conduct experi- ments on roundurip translation, verse retrieval, and verse classification tasks. For roundtrip translation, we again set lo - English and use 2,221 words that occur in all embeddings the start points. For verse retrieval (resp. classification), also use En- glish as the query (resp. train) language, and report top-k accuracy (resp. macro F1 score), averaged over all languages. Results are shown Table 6.
5, which might indicate:
The influence of À also apply ColexNet+, since edges being removed in ColexNet also im- pact ColexNet+: pairs of edges that are expanded from removed edges from ColexNet are then not included in ColexNet+. We show the number of nodes and edges as well as the average degree in ColexNet+ under different À in Table 5. The changes in degree with the increase of X are not prominent in ColexNet (shown in Table 4). This mainly because the number of nodes in ColexNet+ is far more than that in ColexNet. Most the nodes are only associated with around other nodes ColexNett, which indicates that many ngrams from target languages colexify about three
We there are different trends between the changes in À and changes in performance for differ- tasks: (1) the performance of roundtrip transla- tion is positively correlated with and the best re- sult is achieved when À = =100; (2) the performance of verse retrieval is also positively correlated with À (3) the performance of verse classification is gen- erally negatively correlated with À where the best result is achieved when À =1. Those trends be explained follows. Roundtrip translation and verse retrieval, compared with verse classification. require better alignment quality among concepts and ngrams. When is small, some incorrect edges are included in the graph. These edges induce

---

Roundtrip translation
verse retrieval
verse classification
top-1
top-5
top-10 10
top-1
top-5
top- 10
1
0.29
0.66
0.78
0.51
0.68
0.74
0.51
5
0.34
0.71
0.82
0.58
0.74
0.79
0.49
10
0.36
0.74
0,84
0.59
0.75
0.81
0.48
20
0.38
0.77
0.87
0.63
0.78
0.83
0,48
50
0.40
0.81
0.91
0.65
0.80
0.84
0.49
100
0.42
0.84
0.93
0.66
0.81
0.85
0.45
Table 6: Results of multilingual embeddings trained on ColexNet+ under different X. Each number for roundtrip
translation (top-k accuracy, & = [1, 5, 10) is the average of 10 runs with 3 randomly selected intermediate languages.
Each number verse retrieval (top-k accuracy, l = 5, 10)) and verse classification (macro Fi) is the average over all available languages. Bold (underlined); best (second-best) result per column.
noises for sampling, therefore slightly noisy em- beddings are obtained, negatively influencing the performance. As for verse classification, the results suggest that when have fewer out-of-vocabulayy ngrams in the embeddings (higher À induces fewer ngrams in ColexNet+), slightly better performance is achieved. Moreover, seems to have more obvious impact on roundtrip translation and verse retrieval than on verse classification. summary, the results verify our choice of =50, a relatively large number, in the main content of this paper, as it offers very competitive results compared to other choices while not losing many interesting patterns.
Kyrgyz (kir): $apren
Nogai (nog): Sэpтeн
Another interesting example is that <love> and <wish> are frequently colexified, e.g., in Min Nan Chinese (nan) through the word $åi (char- acter: )), as the character means both <love> and <wish>. Lastly, in Western Frisian (fry), <dragon> and <snake> are colexified through the word $dracks, for which we manually verify in PBC. worth noting that there is another word s slang which denotes <snake> in Western Frisian.
Partial colexification. Partial colexification de- notes the pattern that does not involve an entire word, but rather part of it. The part can be shared root or a shared element in compound. Since our algorithm works on the character ngrams within word, we find many partial colexification patterns. For example, <stand> and <build> are colexified in Kazakh (kaz) through $ryp. In Kazakh, <stand> is expressed by the word Typy while <build> by тургызy. турrызy is actually derived from the root TYP- plus causative suffix -Ти3, so that турrыи- means 'to make something stand', thus meaning 'build, erect'. Such partial colexifications through a root can even include more than two concepts, e.g., <morning>, <early>, and <next> many Turkic languages:
B Investigation of identified colexifications
In $4.2, we show that we identify many ground- truth colexification patterns compared with CLICS. However, there quite a few colexification pat- terns that are not present in CLICS. Therefore, conduct a qualitative investigation on those colexifi- cation patterns. We classify each pattern (an colex- ification edge in ColexNet between two concepts) as one of the following categories: (1) full colexi- fication (2) partial colexification and (3) incorrect colexification.
Full colexification. Full colexification indicates that a word in a language directly colexifies two concepts. We list 4 examples of colexifications not included in CLICS but verified by us. An ob- vious example is that <ground> and <land> are colexified in many languages, e.g., through Hb in Japanese (jpn) and Att in Chinese (zho). <early> and <tomorrow> are frequently colexified in Turkic languages, e.g..
Turkmen (luk):
$ertesi <next>
$erte + <morning>
Turkish (tur): $ertesi + <next>
$erte + <morning> <early>
Uyghur (uig): $атисиς -> <next>
Southern Altai (alt): $apreh
$етиген + <morning>
Bashkir (bak): $upron$
$aTH + <early>

---

types
incorrect colex.
languages ngrams
context
<four> <[wenty>
cat
$quatres, $vint
#66004004; vint-i-quatre setials més
co-occurrence
<les>> <right>
nog Somb., Scon8
#01048013:
Out RollI ... col
<want> <know>
nds SweeteS. $wellş
#4601 1003: Oba etj well, daut weete
free translation
<man> <answer>
cat $contes: t-jj
#40012048; Però ell va contestar...
<hundred> <thousand>
#13005021: A
Table 7:
Examples of incorrectly identified colexifications in ColexNet.
Some concepts may be expressed using multiple lexemes, forming compound, and part of the compound may also oceur in the expression of different concept. Note that, in some languages (such as German), a compound is written together without any space between, whereas in some other languages (such as English), there is space between each part. In either case, these are con- sidered as compounds, since all the separate ele- ments together constitute one concept. This can- not be confused with co-oceurrence, where the concepts themselves co-occur. For example, in Tatar (tat), two colors: <purple> and <sear- let> are partially colexified through SKbзь91, be- cause <purple> is kye KLE3ET (literally 'thick red'), which contains a part KI33AJ meaning 'red, scar- let'. Such partial relation also frequently exists in numbers. For example, empat belas (resp. + H) which means 14, and empat puluh (resp.
Incorrect colexification. As an automatic sta- tistical method, the results not immune rors. Typically, find the incorrectly identified colexifications are mainly due to two reasons: (1) co-oceurrence and (2) free translation. We list some incorrectly identified colexifications Ta- ble 7. Co-occurrence denotes that two particular concepts tend to co-occur very often so that the al- gorithm wrongly establishes connections between the concept. For example, we found <four> and <twenty> associated in Catalan (cat) because the ngrams $quat re $vint which refer to the two concepts respectively co-occur very frequently in PBC. Similarly, <left> and <right> for Nogai (nog), and <want> and <know> for Low German (nds) also belong to this type of error. Free trans- lation means that the translation is not done word by word so that the corresponding word for spe- cific concept does not occur in the same sentence. In this case, the algorithm has chance of find- ing the corresponding ngram, which ideally would align with the intended concept. Free translation is very common in the Bible because of its religious textbook nature. For example, in Catalan (cat), the English verse #40012048 starts with "But to the man who told him" but the Catalan translation starts with "Però ell va contestar al qui deia això", which means "But he answered the one who said this", where the concept <man> does not occur Catalan and the concept <answer> does not occur in English verse. Similarly, Chinese word +J means one hundred thousand, i.e., 100,000 (with + being 10 and JJ being 10,000). As the formation of the number expression in Chinese is different from its English counterpart, the algorithm wrongly associates <hundred> and <thousand>.
m H)), which means 40, are partially colexified in Indonesian (ind) (resp. Chinese (zho)), as empat
(resp. [4 means 4. Some languages, e.g., Chi- nese and German, construct compounds without inserting blanks between each lexeme, so we also observe many partial colexifications in Chinese and Germanic languages, e.g..
Chinese (cmn): <tremble>
Hb P2 <carthquake>
Bavarian (bar): bibn + <tremble>
$ercbibn + <earthquake>
German (deu): $beben - <tremble>
Serdbebeng + <earthquake>
C Communities of ColexNet
In summary, many identified colexification patterns in ColexNet belong to this category, which is the reason why we found many patterns that do not cxist in CLICS, since CLICS only includes full colexification patterns.
There are 288 communities in total detected in ColexNet (A 50) by Louvain community detec- tion algorithm (Blondel al., 2008). Two impor- tant hyperparameters, i.c., resolution and random seed are set to 0.1 and I 1145 14 respectively. As

---

mentioned in $5, each community is a cluster of concepts that are semantically related to each other. We create a demonstration website to show the subnetworks each concept and the community figures.' For illustration purposes, we randomly se- lect 15 communities that have more than 10 nodes for illustration in this paper. See Visualizations of those communities in Figure 5.
E English-centric transfer learning
We have shown the English-centric transfer per- formance of verse retrieval and verse classification averaged over languages in Table 2. We believe it is also important to have fine-grained view of the re- sults for individual languages, to better understand the crosslingual transfer capability of ColexNet+. Therefore, we show the transfer performance (sen- tence retrieval and sentence classification) of cach individual language clustered by its corresponding language family in Figure 8. Globally, we see that results not only vary across language families but also vary within each language family. English
D Influence of language families & areas
We create subnetworks specific to each language family and to each area. We consider six lan- guage families that have more than 50 languages in PBC: Austronesian (aust), Atlantic-Conge (atla), Indo-European (indo), Nuclear Trans New Guinea (nucl), Otomanguean (otom) and Sino-Tibetan (sino). We consider five areas: South America (SA), North America (NA), Eurasia, Africa and Papunesia. We only keep the edges in CelexNet that oceur in each language family (resp. area) for the subnetwork of each language family (resp. area). To quantify agreement of community struc- ture, we use adjusted rand index (ARI) (Hubert and Arabie, 1985; Steinley, 2004), similar to (Jack- et al., 2019). We also compute ART between ColexNet and each subnetwork. Figures 6 and 7 show pairwise ARI for language families and ar- eas. It is clear that any language family subnetwork cannot represent the global colexification patterns encoded in ColexNet, since no family's ARI with ColexNet is high. In addition, no two language fam- ilies have a similar community structure according to ARI: for the pair with the highest ARI, atla-aust, ARI 0.5. In comparison, area-specific subnet- works generally have larger pairwise ARIs. The two areas Africa and Papunesia have a very high ARI of 0.76 and also high ARIS with ColexNet (0.78 and 0.80). This can be explained by (1) there are many languages in those two areas so there are more possible colexifications included in the subnetworks and (2) the diversity (in terms of colex- ification) of languages spoken in these two areas is high. In summary, relatively low ARIs between families and areas also suggest many colexification patterns are only specific to a small group lan- guages (either in a specific language family or in an area).
We find in Figure (top) that, though a top- 10 ac- curacy higher than 0.5 is achieved for all languages, the average retrieval accuracy in the Indo-Europe European language family is slightly better than in other fami- lies which have many languages (e.g., Tibetan or Otomanguean language family). We speculate this is probably because other Indo-Europed lan- guages can learn more accurate alignments source language is English which also belongs the same family. Better alignments influence the quality of embeddings in that language, therefore having an impact on the transfer performance.
The trend in classification as shown in Figure 8 (bottom), however, is slightly different: average Fi remains stable at around 0.5 for almost all lan- guage families, with less variance in each language family. This is evidence for our conjecture that clas- sification is a less difficult task: apparently, good performance can be obtained if only words refer- ring to the most important concepts that are highly associated with specific classes are aligned well. In summary, good performance indicates that ColexNet+ assigns similar representations ngrams that refer to the same concept, thus im- proving crosslingual transfer.
"httees//cooneetesploreet cis. 1mu.

---

powee
(a) community #6
(b) community #26
(c) community #29
Apanne
Hemb
Audeed
(d) community #38
(c) community #40
(f) community #46
Ruelk
(g) community #60
(h) community #73
ii) community #79
Heppirs ailing
i community #125
kk) community #143
community #194
(m) community #214
(n) community #221
(0) community #263
Figure 5: Visualizations of 15 randomly selected communitics that have more than 10 nodes from 288 communities detected in ColexNet. Each community forms cluster of concepts that semantically related to each other. E.g., community #60 is related to the concept <hunger>; and community #73 is related to the concept <money>.

---

60
64
04
0:50
6.24
115
69m
6.20
<bm
640
400
9338
0,20
Figure 6: Pairwise ARIs between language family- subnetworks. Each subfigure contains pairwise ARIs between one family (indicated by the color: atla, aust, indo, nuel, otom, sino, base) and all other families (indicated edges). The ARIs are computed by averaging the results 50 runs using Louvain algorithm random states. Pairs of the same family, e.g., indo-indo, are not shown because the ARI will always be in such cases. base is the graph including all edges, i.e., ColexNet. Note that the scale is adjusted for each family individually.
different
47
6.74
0.4
M4
5M49 M
260 M.
14009
Pax AVAA
Figure 7: Pairwise ARIs between area-specific subnetworks. Each subfigure contains pairwise ARIs between one (indicated by the color: Africa, Eurasia, Papunesia, NA, SA, base) and all other areas (indicated the edges). The ARIs are computed by averaging the results of 50 runs using the Louvain algorithm with different random states. Pairs the same area, e.g., Africa-Africa, are not shown because ARI always be such base is the graph including all edges, i... ColexNet. Note that the scale is adjusted for each area individually.

---

Figure 8: Top- 10 accuracy of verse retrieval (top) and Fj for verse classificate fication (boltom) by language family for ColexNete Each small dot represents a lamguage, each large dot average per family. Families are color-coded for better readability and sorted by accuracy/ Asterisk (*) before name: language isolate. Most results are good (average accuracy higher than 0.8 and average Fj higher than 0.5 for half of the language families), but transfer performance varies across lang aaguage familics.
Saa Macre-k 30
South Oelice
gang
Lith Pay
Sod Tartod
Eal Brsha "rard
Continn Thy
East 24 rohlnd
Congo
family
Moustan Crimunl
Nort lismaners
Mat Poliis
Paas
Le 20 Larguage
Starrry
Tlng
Lurg Nantyy
anllnce Margoe
praadd
Lince
Ranamad
522
3.0
538
530
accuracy

---

F Beyond English-centric transfer
language
#ngrams
avg_ avg_colex
We show the complete transfer performance by using any language as train/query language (1,245 languages in total, as we filtered some languages which have a very small size of train or test set). The results are shown in Table 10, 11, 12, 13, 14.
arb
4,107
1.84
rus
3,574
2.21
zho
3,659
2.07
apn
2,119
2.91
muh 1,408
4.00
We hypothesize that the quality of identified colexification can influence the transfer learning performance. Some languages, because their mor- phology, typology, or conceptualization are very different from other languages, might pose diffi culties in finding reliable colexification patterns, thus being detrimental to crosslingual transfer. To this end, we compute the average colexification patterns per ngram (avg_colex) for each language. That is, for language 1, we compute the average number of neighbors of ngram in CetexNet+. The neighbors of an ngram node are concept nodes, which indicates the concepts that this ngram can re- fer to. The higher the avg_colex is for a language, the more polysemous or ambiguous the ngrams tend to be. Of course, the extracted colexification patterns are not always correct due to verse-level misalignment, free translation, some language- specific properties like morphology. Therefore the metric avg_coles can, to some degree, indicate the level of difficulty to find correct alignments.
sll 2,118
2.93
avg.
2,702
2.64
Table 8: ColexNet+ (#ngrams)) and the average number of colex- ified concepts per ngram (avg colex) for Arabic (arb), Russian (rus), Chinese (zho), Apinayé (apn), Mündü (muh), Salt- Yui (sIl) as well the average guages. We see that the lower three worst performing languages have fewer #ngrams but larger avg _ colex than the average statistics over all languages.
Number target-language ngrams
all lan-
rl
r5
r10
#tgrams 0.20
0.28
0.25
0.24
tvg_colex -0.18
-0.25
-0.21
-0.19
Table 9:
Pearson correlations between #ngrams/ ovg_colex and the transfer performance (c: classifica- tion Fi score, rl: retrieval top-1 accuracy, r5: retrieval top-5 accuracy, r10: retrieval top-10 accuracy), All val- are statistically significant under p 0.01.
related with the performance while avg_ vg_collex is negatively correlated. However, it is important to note that the correlation is not high: there are quite few languages that have small #ngrams but large avg__vgggolooooa colex perform quite well when they are used as the source languages for large-scale trans- fer. For example, Bislama (bis), whose #ngrams is only 1,202 but avg_colex is 4.81, achieves good performance: 0.41, 0.46, 0.66, 0.73 for classifica- tion, retrieval top- top-5, and top- respectively. We speculate this is because Bislama is highly in- fluenced by English (Tryon, 1987), therefore the patterns extracted are reliable since the concepts are represented in English lemmata. We leave the further exploration of finding reliable colexifica- tions from a parallel corpus for future research.
We list the number of target-language ngrams in ColexNet+ (#ngrams) as well as eyg_colex for the languages we show in $5.2: Arabic (arb), Rus- sian (rus), Chinese (zho), Apinayé (apn), Mündü (muh), Salt- Yui (sll) well the average over all languages in Table Three high-resource lan- guages, which are typologically and morphologi- cally different from each other, show similar trends in their statistics: more ngrams are included in ColexNet+ while avg avg_colex is less than the aver- age. This might indicate that the languages are less ambiguous and the colexifications extracted are mostly reliable, which explains good crosslin- gual performance when they are used as the train/ query languages. On the contrary, the three worst- performing languages have exactly the inverse trend, which indicates is harder to identify re- liable colexifications, thus the performance is bad when they are served as the source languages.
To sum up, the quality of the colexification pat- terns extracted for a language is closely related to the transfer performance when it is served as the train/query language. Due to various language- specific properties, the model can have difficulties in inducing reliable colexification patterns.
To further test our hypothesis, we compute the Pearson correlation between the performance (clas- sification F1 score and retrieval accuracy) and avg_colex. The results are shown in Table 9. is evident that #ngrams is weakly positively cor-

---

language
classificaion
retrieval
language
chassifsatiion
retrioval
language
classiliction
retrieval
lop-1
top-5
top-10
top-1
top-5
top-
top-1
top.5
top-10
081
0.46
0.47
0.65
0.72
aak
0.42
0.39
0:57
0.64
0.45
0.47
0.67
0.74
A82
0.58
0.65
abc
0.42
0.32
0.50
0.57
0.44
0.52
0.70
0.76
aby
0.40
0.30
0.49
0.57
acd
0.44
0.39
0.58
0.66
ace
0.44
0.51
0.69
0.75
acf
0.44
0.52
0.71
0.77
ach
0.48
0:53
0.71
0.77
ncn
0.44
0.51
0.70
0.76
ser
0.72
SCU
0.43
0.39
0.58
0.66
0.64
0.71
adh
0.47
0.53
0.71
0.77
adi
0.45
0:48
0.66
0.72
adj
0.43
0.42
0.61
0.68
adl
0.44
0.24
0.63
0.70
sch
0.46
D:54
0.72
0.78
a20
0.44
0.44
0.63
0.71
0.46
0,47
0.65
0.72
sft
0.47
0:58
0.74
0.79
0.43
0.38
0.58
0.66
agg
0.44
0.69
kgm
0.44
0.39
0.57
agn
0,50
0.68
agr
0.41
0.4
0.62
0.69
R2T
0.43
0.38
0.56
0.63
agu
0.42
0,40
0.59
0.66
agw
0.46
0.41
0.60
0.68
ahk
0.44
0.38
0.57
0.65
aia
0.42
0.44
0.63
0.71
uii
0.43
0.55
0.71
0.77
0.42
0.41
0.61
0.68
uji
0.44
0.48
0.67
0.73
ajz
0.46
0.77
ak
0,43
0.51
0.70
0.76
0,48
0,66
0.73
akh
0.46
0.31
0.49
0.58
ald
0.41
0.49
0.68
0.75
a]j
0.45
0.46
0.55
0.73
aln
0.47
0.56
0.72
0.78
alp
0.41
0.41
0.60
0.68
alg
0.44
0.51
0.59
0.76
alt
0.47
0.58
0.75
0.81
aiz
0.45
0.52
0.69
0.75
0.40
0.40
0.58
0.66
0.47
0.78
amh
0,44
0.55
0.73
0.78
amk
0.42
0.48
0,66
0.73
amm
0.43
0.39
0.59
0.66
amn
0,44
0.40
0.59
0.67
amp
0.40
0.34
0.52
0.60
amr
0.44
0.33
0.52
0.59
anm
1142
D-46
0.64
0.71
ann
0.44
0.44
0.63
0.70
0.44
0,42
0.68
aoj
0.45
0.41
0.60
0.67
apm
0.38
0.57
0.64
aon
0.42
0.39
0.58
0.66
800
0.51
0.52
0.70
0.76
apb
0.43
0,47
0.68
0.72
ape
0.39
0.40
0.60
0.67
apm
0338
0.21
0.38
0.46
apr
0.44
0.40
0.60
0.68
upt
0.45
0.52
0.69
0.76
0.45
0:44
0.62
0.69
spw
0.45
0.47
0.65
0.72
apy
045
0.35
0.54
0,56
0.72
0.78
ane
0.41
041
0.59
0.67
arl
0:44
0.36
0.55
0.63
arn
0,01
0.48
0.67
0.73
urz
0.46
0.53
0.70
0.76
0.112
0.11
0.60
0.67
use
0.45
0.38
0.57
0.64
ala
0.46
0.44
0.64
0.71
stb
0,44
0:53
0.71
0.77
aid
0.45
0.44
0.64
0.71
alg
0.71
0,43
0.47
0.66
auc
0,38
0,57
auy
0.42
0.37
0.56
0.64
ava
0,46
0.52
0.70
0.76
avi
0.44
0.27
0.45
0.53
avu
0.42
0.29
0.48
0.56
awa
0.43
0.50
0.68
0.75
awch
0.41
0.42
0.60
0.62
0.44
0.34
0.53
0.61
ayo
0.42
0.36
0.56
0.63
0.42
0.47
0.65
0.72
aze
0.76
a2g
0.42
0.41
0.59
0.45
0.45
0.64
0.71
bak
0.45
0.58
0.72
0.78
ham
0.47
0.57
0:74
0.80
ban
0.46
0.2
0.70
0.76
bao
0.47
0.39
0.58
0.66
har
0.48
0.47
0.64
0.71
buv
0.47
0.42
0.62
0.69
bba
0.45
0.69
0.76
bhh
0.42
0.34
0.53
0.60
0.47
0.55
0.64
0.71
0.44
0.36
0.55
0.62
hch
0.44
0.44
0.64
0.71
0.48
0,2
0.61
0.68
bel
0.49
0.58
0.75
0.81
how
0.43
0:42
0.60
0.68
bdd
0.43
0.44
0.63
0.70
bdih
0.36
0.34
0.53
0.61
hef
0.46
0.34
0:53
0.60
bel
0.43
0.6
0.77
0.82
0.76
bon
0.49
0:52
0.76
0,56
0.73
0.79
bex
0.41
0.38
0.58
0.65
hid
0.47
0:47
0.65
0.72
bic
0.40
0,49
0.67
0.74
bgr
0.42
0.52
0.70
0.77
bgs
0.44
0.49
0.68
0.75
bgg
0.45
0.51
0.69
0.75
bbl
0.45
0.50
0.49
0.57
hhp
0.45
0:49
0.67
0.74
bib
0.41
0.46
0.65
0.72
big
0.64
him
0,46
0.49
0,45
0,66
0.73
biu
0.47
0.56
0.73
0.79
hiv
0.42
0.46
0.64
0.71
bir
0.45
0.33
0.52
0.60
bjv
0.40
0.37
0.57
0.65
bkd
0.47
0.49
0.68
0.75
bkq
0.44
0.33
0.51
0.59
bku
0.44
0.45
0.64
0.71
bky
0.40
0.47
0.66
0.73
0.40
0.43
0.63
0.70
0.71
0.45
0.54
0.72
0.78
benh
0.46
0,56
0.73
0.79
bmuh
0.41
0.32
0.51
0.59
hmg
0.43
0.45
0.65
0.71
bear
0.46
0.46
0.65
0.72
bmu
0.46
0.44
0.63
0.70
hnj
0.4.
0.47
0.67
0.74
bep
0.47
0.41
0.60
0.67
boa
0.42
0.53
0.61
0.41
0.41
0.60
0.68
bom
0.52
0.70
0.77
bon
0.45
0,38
0.58
0.66
box
0.43
0.46
0.66
0.73
byr
0.42
0.50
0.69
0.75
bgs
0.43
0.48
0.66
0.73
hqe
0.42
0.47
0.65
0.72
bg
0.47
0.5
0.71
0.77
beg
0.46
0.50
0.68
0.74
hru
0.43
0:40
0.60
0.68
Isc
0.44
0.54
0.72
0.78
0.54
0.44
0.68
0.50
0.68
0.50
0.57
0.80
hto
0.50
0.57
0.74
0.80
Ixt
0.45
0
0.60
0.67
box
0.47
0.57
0.74
0.80
bud
0.44
0.52
0.70
0.76
bug
0.47
0.52
0.70
0.76
buk
0.45
0.39
0.59
0.66
bul
0.46
0.57
0.74
0.79
bum
0.46
0.44
0.62
0.69
0.75
but
0.41
0.38
0.57
0.65
0.32
0.51
bwg
0:46
0488
0.67
0.74
bwu
0,46
0:42
0.61
0.69
bxr
0.45
0.54
0.71
0.77
byz
0.45
0.45
0.54
0.71
byx
0.43
0.29
0.48
0.56
bzd
0.44
0.45
0.64
0.71
bah
0.39
0.39
0.59
0.66
hai
0.42
0.46
0.65
0.72
0.46
0.48
0.66
0.73
csa
0.68
cab
0,42
0.49
cac
0.44
0.37
0,56
caf
0.44
0.42
0.61
0.68
cag
0.45
0.45
0.64
0.71
cak
0.43
0.41
0.50
0.68
coo
0.47
0.40
0.59
0.66
cup
0.42
0.44
0.63
0.70
0.45
0.51
0.69
0.75
0.43
0.68
0.75
0.45
0.44
0.64
0.71
0.54
0.71
0.77
cav
0.37
0.60
cax
0,40
0.42
0.42
0.39
0.58
0.65
chi
0.44
0,55
0.63
0.70
0.48
0.66
0.73
chr
0.40
0.34
0.5
0.60
cbs
0.42
0.34
0.52
0.59
dii
0146
0.31
0.49
0.57
cbn
0.44
0.27
0.44
0.52
cbv
0.43
0.58
oce
0.4%
0.56
0.73
0.79
0,37
0.56
0.64
eah
0.47
0.57
0.74
0.79
oog
040
0.40
0.60
0.67
008
0.47
0.5
0.73
0.79
cfi
0.48
0.44
0.63
0.70
0.47
0.46
0.65
0.72
cha
0.46
0.5k
0.75
0.80
chd
0.42
0.42
0.60
0.67
cbe
0.65
0.44
0.63
0.70
chf
0.44
0.43
0.62
0.70
0.76
<bq
0.43
0.38
0.57
0.65
0.69
chu
0.46
0.61
0.77
0.82
dbv
0,48
0.53
0.71
0.77
chz
0.42
0.43
0.62
0.70
gjo
0.43
0.35
0.54
0.62
cip
0.45
0,50
0.68
0.75
cjv
0.44
0.29
0.47
0.55
ckb
0.51
0.59
0.75
0.80
0.44
0:46
0.65
0.72
clc
0.43
0.44
0.63
0.70
clu
0.75
cly
0.42
0.33
0.52
eme
0.41
0.42
0.68
cmn
0.49
0.60
0.76
0.82
cmo
0.41
0.46
0.65
0.73
cnh
0.47
0.45
0.63
0.70
eni
0.40
0.34
0.53
0.61
cal
0.46
D.41
0.59
0.67
cnr
0.41
0.44
0.62
0.69
0.45
0.45
0.63
0.70
0.43
0.34
0.53
0.61
0.42
0.38
0.57
0.65
cok
0.45
0.66
con
0.40
0.43
0.69
cap
0.46
0.56
0.73
0.79
car
0.50
0.60
0.76
0.81
0.44
0.43
661
0.68
qpa
0.38
0.38
0.57
0.65
qph
0.42
0.44
0.63
0.70
cpc
0.44
0.44
0.64
0.71
0.44
0.45
0.64
0.71
epy
0.45
0.43
0.70
0.44
0.54
0.71
0.77
0.39
0.58
0.66
0.42
0.40
0.59
0.66
erS
0.48
0.52
0.70
0.76
on
0.43
0.41
0.60
0.67
Table 10: Transfer performance using other languages as the train/query language (Part I).

---

language
classificaton
retrieval
language
chassifcationn
retrioval
language
classiliction
retrieval
lop-1
top-5
top-10
top-1
top-5
top-
top-1
top.5
top-10
0.44
0.41
0.59
0.67
csk
0.45
0:54
0.72
0.78
0.41
0.41
0.60
0.68
cty
0.47
0.65
0.72
cta
0.43
0.29
0.48
0.56
ctd
0.47
0.46
0.64
0.71
ctp
0.42
0.25
0.43
0.51
ctu
0.41
0.41
0.60
0.67
cub
0.44
0.40
0.59
0.66
cuc
0.47
0.47
0.65
0.72
0.43
D.41
0.60
0.68
cok
0.43
0.47
0.66
0.72
cul
0.67
0.37
0.32
0.51
0.39
0.58
ewe
0.49
0.54
0.71
0.77
CWt
0.45
0.54
0.72
0.77
eya
0.40
0.34
0.54
0.62
eym
0.49
0.52
0.70
0.76
29T
0.43
0.49
0.68
0.75
dan
0.43
0.45
0.64
0.72
dad
0.44
0,48
0.67
0.74
duh
0.43
0.38
0.57
0.65
dan
0,48
0.57
0.74
0.80
0.73
das
0.45
0.57
0.65
0.56
0.73
dge
0.45
0.44
0.63
0.71
dgi
0.47
0.49
0.67
0.74
dgr
0.41
0,44
0.63
0.70
dga
0.42
0.38
0.57
0.65
Jhm
0.46
0.56
0.73
0.79
dig
0.45
0.50
0.69
0.75
dik
0.46
0.38
0.57
0.65
dip
0.44
0.47
0.66
0.72
0.46
0.52
0.70
0.76
dge
0.47
0.80
djk
0.42
0.53
djr
0.44
0,42
0.61
0.68
dnj
0.45
0.35
0.53
0.61
dub
0.45
0.47
0.66
0.73
dop
0.43
0.42
0.51
0.68
dow
0.45
0.50
0.69
0.75
dip
0.44
0.51
0.70
0.76
dts
0.46
0.52
0.70
0.77
due
0.45
0.48
0.67
0.74
dug
0.47
0.50
0.69
0.25
0.49
0.64
0.20
0.45
0.69
dwr
0,48
0.57
0.73
dww
0,49
0,68
0.75
dyi
0.44
0.50
0.68
0.25
dyn
0.49
0.52
0.71
0.77
dyu
0.46
0.49
0.68
0.24
chk
0.47
0.54
0.72
0.78
eh
0.48
D:48
0.65
0.72
ekn
0.48
0.48
0.67
0.74
ell
0.48
0.59
0.75
0.81
emp
0.41
0.47
0.66
0.73
enb
0.47
0.65
0.72
eng
0.49
0.80
0.84
cnl
0,44
0.47
0.73
com
0.46
0.59
0.75
0.81
epo
0.47
0.61
0.77
0.82
ori
0.42
0:42
0.61
0.68
esc
0.39
0.29
0.47
0.55
esi
0.44
0.52
0.70
0.77
ext
0.50
0.52
0.70
0.76
0.47
0.57
0.73
0.79
0.54
0.78
0.43
D:45
0.64
0.50
0.59
0.75
0.81
ewe
0.50
0.50
0.68
0.74
eza
0,46
0.44
0.63
0.70
fea
0.44
0.36
0.54
0.61
füi
0.46
0.37
0.55
0.63
Fal
0.43
0.52
0.70
0.77
fao
0.45
0.55
0.72
0.78
fTm
0.47
0.55
0,73
0.79
0.44
0.50
0.70
0.76
fil
0,44
0.60
0.76
0.81
6in
0.46
0.81
Fom
0,43
0.62
0.42
0.43
0.62
0.69
fra.
0.47
0.56
0.73
0.78
Iry
0.47
0.56
0.72
0.78
fub
0.47
0.54
0.71
0.78
fuf
0.46
0.56
0.73
0.79
Fuhh
0.50
0.55
0.73
0.79
fug
0.47
0.56
0.73
0.79
fuv
0.46
0.54
0.71
0.78
gba
0.46
0.53
0.70
0,76
gug
0.45
0,60
0.76
0.81
guh
0.46
0.66
gam
0.40
0.33
gaw
0.44
0,38
0.57
0.64
ghi
0.43
045
0.63
0.70
gho
0.47
0.43
0.62
0.09
g/v
0.41
0.32
0.50
0.58
gde
0.43
0.50
0.68
0.75
gdg
0.45
0.45
0.64
0.71
gdn
0.44
0.40
0.59
0.67
gdr
0.48
0,48
0.67
0.24
geh
0.43
0.40
0.59
0.67
29)
0.46
0.51
0.68
0.75
gfk
0.45
0.46
0.65
0.72
ghe
0.49
0.54
0.72
0.78
gls
0.43
0,33
0.51
0.59
g3l
0.44
0.46
0.65
0.73
gil
0.48
0,51
0.69
0.75
0.47
0.45
0.64
0.71
gin
0.42
0.52
0.70
0.76
gkn
0142
D:45
0.64
0.72
8kp
0.49
0.44
0.63
0.70
gle
0.72
SmIv
0,46
0.55
gab
0,46
0.49
0.67
gnd
0.37
0.37
0.57
0.65
gng
0,11
0.52
0.70
0.77
pmn
0.46
0.35
0.54
0.62
gaw
0.46
0.47
0.63
0.72
gof
0.46
0.57
0.73
0.79
p0g
0.48
0.56
0.78
0.78
gor
0.47
0.52
0.69
0.75
0.41
0.38
0.58
0.66
0.46
0.55
0.72
0.78
gso
0.73
gub
0.37
0.29
guc
0,40
0,38
0,56
gud
0.48
0.52
0.70
0.77
guh
0.43
0.40
0.58
0.66
gui
0.44
0.46
0.66
0.73
guj
0.44
0.46
0.64
0.71
guk
0.44
0.52
0.70
0.76
gul
0.45
0.48
0.56
0.73
gum
0.45
0.48
0.66
0.72
gun
0.41
0.49
0.67
0.74
guo
0.40
0.37
0.56
0.63
gug
0.43
0.67
gur
0.46
0.41
0.68
guw
0.49
0.55
0,71
0.77
gux
0.47
0.52
0,70
0.76
guz
0.46
0.53
0.70
0.76
gvc
0.42
0.41
0.50
0.67
gvf
0.43
0.30
0.49
0.57
gvl
0.49
0.44
0.63
0.71
0.43
0.37
0.55
0.63
gui
0.42
0.58
0.63
вya
0.48
0.42
0.61
0.68
23m
0.40
0.39
0.58
0.66
eyy
0.46
0,44
0.63
0.70
hnc
0.47
0.55
0.73
0.79
hag
0.43
0,22
0.62
0.70
hak
0.44
0.58
0.75
0.81
hat
0.47
0.41
0.61
0.68
han
0.49
0.56
0.72
0.78
haw
0.42
0.50
0.69
0.75
hay
0.46
0:52
0.70
0.76
hih
0.44
0.1
0.69
0.75
0.78
hog
0.45
0.63
0,54
0.71
hif
0.46
0.45
0.63
0.70
hig
0.43
B.47
0.67
0.74
hil
0.47
0.60
0.77
0.82
hin
0.47
0.56
0.73
0.79
hix
0.41
0:40
0.59
0.66
his
0.44
0.40
0.59
0.67
hl:
0.44
0.54
0.73
0.79
hmo
0.46
0.53
0.71
0.77
hnc
0.48
0.55
0.72
0.78
haj
0.50
0.72
hnn
0.42
0.46
0.43
0.62
0.20
hop
0:45
0:49
0.67
0.74
hot
0.45
0.37
0.56
0.64
hra
0.44
0.49
0.67
0.74
tav
0.46
0.58
0.75
0.80
htc
0.42
0.43
0.62
0.69
bub
0.44
0.37
0.56
0.64
hui
0.46
0.39
0.58
0.66
hun
0.49
0.57
0.73
0.79
0.50
0.68
0.75
0.41
0.62
huv
0.44
0.38
0.57
0.65
hvn
0.42
0.44
0.64
0.71
hwe
0.38
041
0.60
0.68
hye
0.46
0.56
0.73
0.79
San
0.37
0.27
0.44
0.52
iha
0.46
0.58
0.75
0.81
ibo
0.47
D.46
0.65
0.72
icr
0.45
0.45
0.64
0.71
ifa
0.44
0.60
0.67
0.44
0.43
0.62
0.69
0.40
0.58
0.65
ifu
0.72
0.48
0.32
0.58
ign
0.40
0.41
0.60
0.67
ike
0.47
0.55
0.72
0.78
ikk
0.44
0.54
0.72
0.78
kow
0.44
0.5
0.71
0.77
ilb
0.48
0.57
0.74
0.80
ilo
0.47
0.56
0.73
0.78
imo
0.47
0.34
0.53
0.61
inh
0.47
0.45
0.64
0.71
ind
0.46
0.57
0.73
0.79
0.39
0.58
0.66
ion
0.43
0.35
0.55
0.63
ipi
0.41
0.38
0.57
0.65
kqW
0.45
0.45
0.64
0.71
ini
0.40
0-46
0.65
0.72
irk
0.43
0.54
0.72
0.78
iry
0.45
0.53
0.71
0.78
isd
0.45
0.47
0.66
0.73
0.46
0.58
0.75
0.50
0.45
0.59
0.75
0.81
itv
0.79
ium
0.44
0.61
0.65
ivv
0.42
0.43
0.62
0.70
iws
0.39
0.35
0.54
0.62
ial
0.11
0.41
0.59
0.67
izr
0.42
0.46
0.66
0.73
izz
0.45
0.47
0.66
0.73
jac
0.44
0.44
0.64
0.71
iae
0:45
0.49
0.68
0.75
jam
0.43
0:41
0.60
0.68
0.47
0.48
0.67
0.73
jbu
0.45
0.64
jic
0.36
0.28
0.54
Jiv
0.40
0.40
0.60
0.68
imc
0.49
0.49
0.67
0.73
jpn
0.45
0.61
0.77
0.82
0.43
0.53
0.72
0.78
jun
01.66
0.45
0.63
0.70
kea
0.45
0.54
D.71
0.77
kab
0.45
0.51
0.69
0.75
0.41
0.41
0.60
0.68
kal
0.46
0.58
0.75
0.50
0.44
0.60
0.77
0.82
0.78
kag
0.44
0.37
0.56
0.64
0.46
0.59
0.76
0.81
kaz
0.45
0.55
0.72
0.78
Khc
0.40
0.47
0,66
0.73
kbhh
0.46
0147
0.65
0.72
kbm
0.42
0.33
0.52
0.60
khp
0.43
0.51
0.69
0.76
kbq
0.43
0.44
0.62
0.70
khr
0.43
0.53
0.71
0.76
kok
0.52
0.56
0.73
0.79
0.46
0.56
0.73
0.79
kde
0.48
0.59
0.75
0.81
kdi
0.45
0.54
0.73
0.78
kdj
0.45
0.57
0.74
0.80
Table 11: Transfer perfermancceeec using other languages the train/query language (Part II).

---

language
classificatin
retrieval
language
chassification
retrioval
language
classiliction
retrieval
lop-1
top-5
top-10
top-1
top-5
top-
top-1
top-5
top-10
kdl
0.41
0.42
0.61
0.69
kek
0.47
0.39
0.58
0.66
ken
0.47
0.46
0.65
0.72
kew
0.35
0.62
kes
0.43
B.46
0.65
0.72
0.48
0.66
0.72
kgf
0.45
0.45
0.65
0.72
kgk
0.46
0.37
0.56
0.64
kgp
0.41
0.30
0.49
0.58
khi
0.47
0.57
0.74
0.80
khm
0.45
D.56
0.73
0.79
kni
0.44
0.32
0.51
0.59
khy
0.77
khz
0.46
0.50
0.69
0.75
0.61
k3k
0.48
0.54
0.71
0.77
kin
0.46
0.53
0.72
0.78
kir
0.46
0.53
0.70
0.76
kix
0.43
0.53
0.71
0.77
kib
0.40
0.39
0.59
0.66
kgo
0.42
0.38
0.58
0.66
kih
0.46
0.52
0.70
0.76
kjs
0.43
0.37
0.56
0.64
Kki
0.45
0.54
0.71
0.77
kkj
0.72
0.43
0.49
kma
0.48
0.67
kmg
0.42
0.44
0.63
0.70
kmh
0.40
0.27
0:45
0.55
kmk
0.45
0.50
0.68
0.75
kmm
0.47
0.48
0.67
0.73
kmo
0.43
0.35
0.54
0.62
kmn
0.47
0.59
0.75
0.80
kms
0.41
0.32
0.51
0.59
kmu
0.40
0.39
0.59
0.66
0.44
0.50
0.68
0.75
0.77
kng
0.45
0.52
0.76
knj
0.40
0.44
0.6
0.71
knk
0.45
0.50
0.58
0.75
kno
0.37
0.34
0.53
0.61
knv
0.42
0.39
0.58
0.65
kog
0.40
0.41
0.60
0.67
kar
0.46
0.57
0.74
0.79
kpf
0.43
0.39
0.58
0.65
kpg
0.41
0.39
0.58
0.66
kgj
0.45
0.36
0.54
0.62
0.45
0.35
0.53
0.61
kpp
0.78
kpw
0.41
0.27
kpx
0,35
0.54
kgu
0.44
0.53
0.71
0.77
kqe
0.48
0.52
0.90
0.77
lago
0.44
0.45
0.64
0.71
keg
01.41
0.38
0.57
0.65
Eqs
0.43
0:53
0.71
0.77
kuy
0.42
0.53
0.70
0.77
kre
0.47
0.57
0.74
0.79
0.42
0.43
0.62
0.20
0.43
0.57
0.74
0.80
kse
0..45
0,45
0.71
kid
0.46
0.50
0.75
0.48
0.1
0.69
0.76
kar
0.43
0.44
0.63
0.70
Kss
0.43
0.45
0.64
0.71
ksw
0.48
0.51
0.69
0.75
kib
0.45
0.56
0.73
0.79
0.43
0:40
0.60
0.68
0.42
0.41
0.61
0.68
0.47
0.78
Kna
0.46
0:54
0.71
kub
0.40
0.60
0.67
kud
0.46
045
0.63
0.70
kuc
0.47
0:42
0.62
0.69
kum
0.42
0.52
0,70
0.76
kup
0.41
0.31
0.49
0.57
kus
0.45
0.52
0.70
0.77
kvj
0.44
0.50
0.69
0.76
kva
0.44
0,42
0.60
0.68
Iwd
0.42
0:47
0.66
0.73
kwf
0.41
0.48
0.66
0.73
0.64
kwi
0.42
0.36
0.49
0.53
0,71
0.77
kxm
0.46
0.45
0.56
0.73
kxw
0.43
0.37
0.57
0.65
kyc
0.43
0.36
0.55
0.63
kyf
0.445
0.49
0.67
0.73
kyg
0.42
0.43
0.61
0.69
kyg
0.43
0.43
0.63
0.70
kyu
0.41
0.53
0.71
0.77
1ys
0.40
0.33
0.52
0.60
0.47
0,39
0.58
0.65
lac
0.62
0,46
0.57
0.75
laj
0.46
0,53
0,71
0.77
lam
0.43
0.56
0.73
0.78
Ino
0.47
0:56
0.73
0.79
las
0.43
0.5
0.64
0.21
lat
0.46
0.57
0.74
0.80
lav
0.45
0.58
0.75
0.50
1sk
0.46
0.53
0.71
0.77
0.47
0.39
0.58
0.66
0.47
0.49
0.68
0.74
0.48
0.44
0.63
0.20
lef
0.44
0.40
0.60
0.67
Ich
0.46
0.51
0.69
0.75
lesn
0.45
0,46
0.64
0.71
lou
0.47
0.45
0.65
0.72
low
0.46
0,50
0.69
0.76
lex
0.46
0.41
0.60
0.68
lgm
0.44
0.55
0.72
0.78
thi
0.44
0.39
0:59
0.67
Thm
0.43
0.49
0.68
0.74
0.70
0.47
0:51
0.35
0.54
lff
0.49
0448
0.67
0.73
lin
0,47
0.53
0.71
0.77
lip
0.39
0.42
0.60
0.67
lit
0.49
0.58
0.75
0.81
lip
0.48
0.54
0.71
0.78
1mkk
0.45
0.52
0.69
0.76
0.46
0.47
0.66
0.74
lob
0.47
0:47
0.66
0.72
3ol
0.48
0,50
0.68
0.74
0.71
luz
0.48
0.52
0.70
0.76
bsi
0.42
0.40
0,59
0.67
Ism
0.47
0.52
0.20
0.76
lug
0.45
0.57
0.74
0.79
hao
0.46
0.50
0.68
0.74
lus
0.41
0.54
0.71
0.77
Iwo
0.41
0.37
0.56
0.64
bww
0.45
0.31
0.50
0.58
Izh
0.47
0.62
0.77
0.82
0.44
0.47
0.66
0.73
mad
0.46
0.52
0.70
0.76
0.75
mah
0.46
0.50
0.69
0.35
0.44
0.57
0,74
0.80
maj
0.45
0.50
0.59
0.75
mak
0.44
0.52
0.70
0.77
mal
0.45
0.56
0.72
0.78
mam
0.44
0.48
0.67
0.74
mag
0.40
0.41
0.59
0.67
mar
0.44
0.50
0.68
0.75
0.45
0.59
0.67
0.40
0.28
0.46
0.55
maw
0.45
0.77
0.66
maz
0.43
0,38
0.58
0.65
mib
0.44
0.48
0.67
0.74
mbc
9,45
0,37
0.55
0.62
midd
0.45
0.22
0.61
0.69
mhd
0147
0.59
0.75
0.80
mhh
0.46
0.45
0.64
0.71
miii
0.43
0.45
0.64
0.71
mby
0.41
D:41
D.61
0.68
mih
0.45
0.29
0.48
0.57
0.71
mbt
0.45
0.53
0.71
mxa
0,48
0.67
meb
0.39
0.39
0.58
0.65
med
0.42
0.33
0.51
0.59
1,00
0.22
0.39
0.47
mnk
0.47
0.51
0.69
0.76
mon
0.45
0.50
0.69
0.75
mk0
0,40
0.42
0.60
0.68
map
0.45
0.50
0.68
0.75
0.45
0:42
0.62
0.69
mcu
0,44
0.59
0.58
0.66
mda
0.61
mdy
0.46
0.51
0.69
med
0.44
0.26
0.46
mee
0:43
0488
0.67
0.73
mcj
0.42
0.34
0.54
0.62
mok
0:12
0.43
0.63
0.70
men
0.45
0.48
0.57
0.73
mey
0.43
0.40
0.61
0.68
mzu
0.46
0.48
0.57
0.74
mfe
0.47
0.54
0.71
0.77
mth
0.46
0.40
0.59
0.66
0.45
0.43
0.62
0.69
0.70
m4y
0.45
mfy
0.45
0.44
0.62
mfz
0.43
0.43
0.62
0.69
mhi
0,44
0,40
0.59
0.66
mhl
0.44
0.38
0.57
0.64
mhr
0.45
0.57
0.74
0.79
mhx
0.45
0.39
0.59
0.66
mby
0.43
0.52
0.70
0.76
mih
0.41
0.44
0.70
mic
0.46
0.44
0.63
0.70
mic
0,48
0.45
0.64
0.71
0.71
mig
0.44
0.49
0.67
0.74
mih
0.40
0,35
0.55
0.63
mil
0.40
0.33
0.51
0.59
min
1145
0.52
0.70
0.76
mio
0.89
0.32
0.52
0.60
miq
0.45
0.42
0.62
0.69
mir
0.38
0.34
0.51
0.50
mit
0.43
0.40
0.58
0.66
miy
0.42
0.42
0.62
0.70
0.40
0.35
0.55
0.63
mjo
0,36
0.55
0.63
njw
0.49
0.53
0.71
0.78
mkdd
0.46
0.59
0.75
0.81
mkd
0.44
048
0.67
0.73
mkn
0.43
0.38
0.57
0.65
mks
0.44
0.38
0.58
0.66
mih
0.48
0.43
0.63
0.70
0.45
0.40
0.59
0.67
mlt
0.49
0.57
0.74
0.79
0.44
0.43
0.62
0.70
mmm
0.67
mixx
0.41
0.44
0.63
0.70
0,40
0.36
0.56
mmb
0.44
0.56
0.74
0.79
mnf
0.45
0.42
0.6]
0.69
mmh
0.44
0.44
0.63
0.70
mnk
0.46
0.57
0.74
0.80
mnx
0.41
0.35
0.54
0.62
mXC
0.44
0.48
0.65
0.72
mog
0.41
0.48
0.66
0.72
mop
0.41
0.40
0.59
0.67
0.46
0.50
0.69
0.75
mos
0.46
0.70
mux
0.46
0.47
0.72
mpg
0.49
0.68
mpm
0.44
0.31
0.50
0.58
mps
0.41
0.26
0.43
0.52
mpi
0.41
0.35
0.53
0.61
migh
01.77
0.38
0.58
0.66
maj
0.45
0:53
0.71
0.77
magy
0.46
0.49
0.68
0.74
0.45
0.45
0.64
0.71
0.47
0.55
0.72
0.78
0.47
0.58
0.75
0.81
0.45
0.78
mie
0.48
0.44
0.63
0.20
mik
0.45
0,1
0.61
0.68
mam
0.43
0.49
0.67
0.74
may
0.44
0.44
666
0.70
mna
0.44
0144
0.63
0.70
0.44
0.36
0.55
0.63
mn
0.44
0.46
0.65
0.72
0.43
0.34
0.53
0.62
0.40
0.73
mip
0.47
0.50
0.68
0.74
0.38
0.46
0.65
0.72
mah
0.38
0.22
0.39
0.48
mur
0.44
0.46
0.65
0.72
mix
0.42
0.37
0.55
0.63
Table 12: Transfer performance using other languages as the train/query language (Part III).

---

language
classificatin
retrieval
language
chassifsatiion
retrioval
language
classiliction
retrieval
lop-1
top-5
top-10
top-1
top-5
top-
top-1
top.5
top-10
may
0.43
0.43
0.62
0.69
0.45
0.47
0.66
0.72
mwn
0.45
0.44
0.63
0.70
mvp
0.45
0.68
0.74
mww
0.44
0.40
0.59
0.67
m00
0.45
0.45
0.65
0.72
mwv
0.45
0.52
0.20
0.76
mww
0.40
0.45
0.65
0.72
msh
0.39
0.42
0.61
0.69
mxx
0.44
0.57
0.56
0.63
mxg
0.43
D:45
0.64
0.71
1951
0.43
0.57
0.56
0.64
mya
0.81
mghh
0.43
0.48
0.67
0.73
myy
0.45
0.63
myu
0.41
0.38
0.58
0.65
myy
0.50
0.54
0.71
0.77
myw
0.40
0.41
0.60
0.68
mys
0.49
0.54
0.71
0.77
myy
0.44
0.33
0.51
0.50
1973
0.42
0.38
0.57
0.65
muh
0.46
0.44
0.63
0.70
m2k
0.44
0.39
0.58
0.66
m2
0.43
0.45
0.64
0.71
mum
0.40
0.63
mZW
0.45
0.43
0.62
0.34
0.53
nar
0.44
0.34
0.52
0.60
ma
0.42
0.32
0.51
0.59
nan
0.48
0.57
0.74
0.80
nay
0.49
0.57
0.74
0.80
mas
0.46
0:46
0.64
0.71
DaV
0.46
0.50
0.69
0.76
nbe
0.45
0.54
0.72
0.78
nhe
0.46
0.55
0.72
0.78
nbl
0.50
0.57
0.73
0.79
nea
0.44
0.70
nch
0.45
0.47
0.65
0.72
nC]
0.42
0.63
0.70
nel
0.44
0.40
0.58
0.56
net
0.47
0.44
0.63
0.70
DCU
0.45
0.41
0.50
0.67
ndc
0.48
0.59
0.76
0.81
nde
0.49
0.56
0.73
0.79
mdi
0.43
0.46
0.65
0.72
ndj
0.45
0.72
0.78
ndo
0.46
0:55
0.72
0.78
0,52
0.70
0.76
0.47
0.77
ndz
0,41
0.34
0.53
0.42
0,43
0.62
nep
0.48
0.61
0.78
0.83
nfa
0.42
0.37
0.57
0.65
nfr
0.43
0.43
0.62
0.69
0.46
0.55
0.72
0.78
0149
D:51
0.69
0.76
0.44
0.49
0.67
0.74
nbd
0.43
0.49
0.67
0.74
nhe
0.44
0.47
0.66
0.73
nbg
0,52
0.70
0.76
0.45
0.53
0.70
0.77
nho
0,47
0.39
0.59
0.66
0.47
0.54
0.72
0.78
oba
0.45
0.46
0.65
0.72
nhw
0.42
0.47
0.66
0.72
nhx
0.41
0.47
0.66
0.73
aby
0.46
0.51
0.69
0.75
nii
0.40
0.24
0:41
0.49
nij
0.48
0,50
0.68
0.75
0,46
0.79
nin
0.44
0.36
0.55
0.63
0.46
0.60
0,76
niy
0.43
0.51
0.69
0.75
njb
0,43
0.50
0.68
0.75
nim
0.43
0.50
0.67
0.74
nin
0.41
0.53
0.70
0.76
njo
0.45
0.56
0.73
0.78
0.46
0.51
0.69
0.76
nko
0.45
0.50
0.69
0.75
nlc
0.43
0.38
0.57
0.64
nkl
0.47
0.55
0.72
0.78
uma
0.44
0.76
nmf
0,44
0.51
0.69
mmo
0.18
0,50
0,68
0.75
amz
0.40
0.48
0.67
0.73
mnb
0.46
0.59
0.76
0.81
mng
0.45
0.52
0.70
0.76
nah
0.42
0.38
0.58
0.66
nno
0.46
0.55
0.72
0.78
nap
0.48
0.48
0.66
0.73
0.46
0.53
0.71
0.77
0.42
0.43
0.63
0,70
0.42
0,34
0.52
0.60
nob
0.48
0.80
aog
0.49
0.53
0.76
nop
0.42
0,41
0.60
0.68
not
0.42
0.37
0.56
0.64
nom
0.44
0.34
0:53
0.61
nph
0.46
0.56
0.74
0.79
upi
0.45
0.58
0.74
0.80
mpl
0.46
0.49
0.67
0.73
0.46
0.50
0.69
0.76
ngy
0.44
0.74
DSn
0.46
0.46
0.65
0.72
0.48
0.54
0.71
0.77
ntp
0.47
0,38
0.56
0.64
nh
0,44
0.40
0.60
0.67
mus
0.47
0.50
0.68
0.74
muy
0.48
0.44
0.63
0.70
mvm
0.45
0.34
0.52
0.59
mwb
0.42
0.39
0.58
0.65
mai
0.44
0.39
0.58
0.66
nya
1147
0.56
0.73
0.79
nyf
0.43
0.54
0.72
0.78
oyn
0.79
nya
0.44
0:55
nyy
0.56
0.73
0.79
ebo
0:17
0.50
0.68
0.75
oji
0,46
0:45
0.64
0.7]
ojs
0.48
0.53
0.72
0.78
okv
0.43
0.32
0.51
0.59
old
0.45
0.50
0.68
0.74
omn
0.43
0.35
0.54
0.62
0.42
0.51
0.50
0.58
ons
0:41
0.39
0.59
0.67
ood
0.44
0,56
0.55
0.63
opm
0.59
ary
0,44
0.52
0.69
0.76
0.47
0,51
0.69
0.75
ote
0.47
0.43
0.62
0.69
otm
0,44
0.37
0.56
0.64
ocn
0.40
0,42
0.51
0.68
osq
0.45
0.45
0.64
0.71
dbs
0.43
0.31
0.49
0.57
ozm
0.47
0.46
0.65
0.72
pab
0.46
0.43
0.62
0.69
pad
0.42
0.41
0.61
0.68
pag
0.50
0.61
0.77
0.82
pah
0.41
0.66
pam
0,46
0.56
0.73
pan
0.47
0.54
0,72
0.78
pue
0.43
0.23
0.41
0.49
pap
0.47
0.58
0.75
0.80
pbb
0.42
0,39
0.58
0.65
pbc
0.46
0.47
0.66
0.73
phi
0.41
0.44
0.63
0,70
pbl
0.48
0.52
0.70
0.76
pem
0.48
0.73
pdc
0.47
0.53
0.71
0.77
0.48
0.52
0.70
0.77
pes
0.45
0.57
0.73
0.79
pib
0,45
0.57
0:75
0.80
pio
1/00
0,40
0.59
0.67
pir
0.46
0.39
0.58
0.65
pis
0144
0.52
0.71
0.77
pkc
0.47
0.5
0.71
0.77
plg
0.42
0.47
0.66
0.73
pls
(141
D:42
0.60
0.68
0.44
0.22
0.61
0.68
plw
0.73
pmf
0.45
0.47
0.66
0.72
psc
0.45
0.65
poe
0.41
0.46
0.65
0.72
poh
0.47
0.39
0.59
0.66
poi
0.43
0.88
0.66
0.73
pol
0.47
0.57
0.73
0.79
pon
0.49
0.49
0.68
0.74
0.51
0.59
0.75
0.81
0.45
0.56
0.73
0.79
ppk
0.48
0.50
0.69
0.75
ppo
0.46
0.38
0.57
0.65
pef
0.47
0.80
pri
0.46
0.72
pek
0.47
0.49
0.67
0.74
pes
0.44
0.55
0.73
0.78
pso
0,42
0.55
0.73
0.79
ptp
0,01
0.35
0.55
0.63
ptu
0.45
0.53
0.71
0.77
pua
0.45
0.45
0.63
0.71
pw2
0.44
0.46
0.66
0.73
pww
0.41
0.46
0.65
0.72
quh
0.43
0.37
0.56
0.64
que
0.45
0.40
0.59
0.67
quf
0.66
qug
0,44
0.51
0.69
0.75
quh
0.47
0.53
0,70
0.77
qui
0.45
0.56
0.73
0.79
qup
0.48
0.38
0.57
0.65
QUw
0.43
0.55
0.73
0.79
0.45
0.50
0.69
0.75
quz
(145
D.55
0.73
0.79
0.41
0.42
0.51
0.68
que
0.45
0.74
quh
(.41
0.34
0:52
0.60
qui
0.51
0.70
qym
0.60
qun
044
0.41
0.59
0.66
QVO
0.44
0,46
0.64
0.20
qua
0.40
0.1
0.59
0.66
qVW
D.45
0.64
0.71
0.43
0.22
0.60
0.68
qwb
0.47
0,48
0.66
0.72
qsh
0440
0.36
0.55
0.62
0.44
0.47
0.67
0.73
guo
0.42
0.32
0.50
0.58
gur
0.42
0.50
0.69
0.76
0.46
0.66
0.73
rim
0.47
0.51
0.69
0.76
ribh
040
0.30
0.4%
0.56
mBO
0.42
0.40
0.59
0.67
my
0.46
0.52
0.69
0.75
ron
0.47
0,56
0.72
0.78
100
0.46
0.42
0.61
0.68
rop
0.42
0.35
0.54
0.62
0.43
0.49
0.67
0.74
0.44
0.55
0.73
0.79
run
0.50
0.78
FUS
0.48
0.55
0.78
[WO
0.39
0.57
sab
0.37
0288
0.47
0.55
sug
0.46
0.48
0.67
0.73
kah
0.46
0.53
0.71
0.77
sus
0.50
0.55
0.72
0.78
sbu
0.45
0.40
0.59
0.66
sbd
0.44
0.41
0.60
0.68
sbl
0.43
0.51
0.69
0.76
sda
0.46
0.56
0.73
0.79
0.43
0.43
0.63
0.70
agb
0.69
sgw
0.45
0.51
0.70
5g2
0.41
0.38
0.57
0.65
shi
0.45
0.51
0.69
0.75
sbk
0.44
0.48
0.67
0.73
shp
0.46
0.44
0.62
0.69
sha
01.55
0.50
0.69
0.75
dgg
0.41
D.44
0.63
0.71
sil
0.43
0.44
0.64
0.71
0.44
0.37
0.56
0.64
0.48
0.49
0.66
0.73
5ja
0.43
0.37
0.55
0.63
sid
0.46
0.73
slk
0.48
0.58
0.74
0.80
sil
0.39
0.24
0.42
0.51
stv
0.49
0.5
0.73
0.79
sne
0.47
0.62
0.7%
0.83
smkk
0.39
0147
0.65
0.72
snl
0.45
0.50
0.69
0.76
smo
0.46
0.52
0.70
0.76
0.49
0.56
0.73
0.79
0.47
0.76
snd
0.44
0.55
0.73
0.78
0.42
0.30
0.49
0.57
sap
0.43
0.42
0.61
0.68
SBW
0.45
0.46
0.64
0.71
say
0.38
0.30
0.49
0.57
Table 13: Transfer performance using other languages as the train/query language (Part IV).

---

language classification
retrieval
language
classification
rerrieval
language
classification
retrieval
top-1
top-5
top-10
top-1
top-5
top-10
rop-1
top-5
top-10
50m
01:48
0.57
0.74
0.80
50g
0.47
0:43
0.62
0.69
501
0.47
0.52
0.69
0.75
60y
0.44
0.48
0.67
0.74
4p8
0.47
0.58
0.74
0.80
501
0.38
0.25
0.42
0.50
809
0.47
0,48
0.66
0.73
1ps
0.44
0:42
0.60
0.67
0.47
0.48
0.67
0.73
sqi
0.51
0.42
0.43
0.62
0.53
sm
0.44
0.50
0.68
0.74
stp
0.46
0.59
0.75
0.80
0.41
0.31
0.50
0.59
ssd
0.47
0.38
0.58
0.65
55g
0.41
0.37
0.56
0.64
NSW
0.47
0.57
0.74
0.80
0.45
0.39
0.58
0.65
0.45
0:43
0.62
0.69
0.39
0.27
0.45
0.53
0.64
sue
0.42
0.64
0.71
0.27
sun
0.43
0.48
0.56
0.73
sur
0.40
0.49
0.68
0.74
5555
0.39
0.51
0.70
0.77
swe
0.45
0.55
0.72
0.78
Swg
0.48
0.57
0.73
0.79
swb
0.47
0.55
0.72
0.78
0.45
0.75
0.80
swp
0.44
0:51
0.69
0.75
0.44
0.51
0.69
0.76
son
0.75
syc
0.49
0.53
0.40
0,34
0,52
tab
0.45
0.51
0.69
0.25
Inc
0.42
0.28
0.45
0.53
[2
0.47
0.46
0.65
0.72
tam
0.43
0.60
0.77
0.82
th]
0.44
D:48
0.66
0.73
tar
0.42
0.58
0.58
0.65
0.46
0.56
0.74
0.79
0.47
0.32
0.51
0.60
0.39
0.33
0.52
0.60
the
0.47
0.58
0.65
thg
0.41
0.35
0.54
0.62
0.5
0.64
0.72
tho
0.45
0.46
0.66
0.73
thy
0.44
0.51
0.69
0.75
thu
0.44
0.40
0.60
0.67
0.44
0.41
0.60
0.67
loc
0.49
0.52
0.69
0,76
0.40
0.43
0.63
0.70
1c7
0.76
0.45
0:53
0.71
0.42
0.60
tel
0:46
0.54
0.72
0.78
Icm
0,46
0.56
0.73
0.79
tco
0.49
0.57
0.74
0.79
tor
0.45
0.43
0.63
0.70
Ifr
0.40
0.40
0.59
0.67
Lsk
0.46
0.59
0.75
0.80
0.46
0.55
0.72
0.78
0.47
0:44
0.64
0.7]
0.49
0,54
0.72
0.78
0.16
0.76
0.47
0.51
0.69
0,60
tim
0.44
0.30
0.49
0.57
fir
0.46
0.59
0.76
0.81
thu
0.42
0.44
0.63
0.70
tib
0.47
0.51
0.69
0.75
IIf
0.44
0.30
0.48
0.56
thh
0.50
0.58
0.75
0.80
0.45
0.40
0.59
0.67
Inn
0.42
0.38
0.58
0,66
0.43
0,39
0.59
0.67
toc
0.66
loh
0.49
0.71
0.77
soj
0.41
0,35
0,53
0.61
100
0.42
0.46
0.65
0.72
top
0.41
0.36
0:53
0.61
Tos
0.40
0.39
0.58
0.65
qpi
0.42
0.48
0.88
0.75
Ipm
0.41
0.31
0.51
0.59
0.45
0.50
0.68
0.74
tp6
0.44
0.45
0.63
0.70
1pz
0.46
0.38
0.58
0.66
0.39
0.29
0.47
0.55
trc
0.40
0.32
0.51
0.60
Iin
0,43
B.47
0.73
tng
0.44
0,38
0.57
0.65
Isn
0.46
0.57
0.74
0.80
182
0.44
0:44
0.63
0.70
0.43
0.44
0.63
0.70
ne
0.45
0.46
0.65
0.72
Tie
0142
0.39
0.58
0.66
0.44
0.44
0.62
0.70
tof
0.60
0.43
0.38
0.46
0.52
0.70
tum
0.17
0.59
0.76
0.81
tuo
0,45
0.39
0.58
0.66
tur
0.45
0.55
0.72
0.78
twi
0.48
0.45
0.63
0.70
twu
0.44
D.47
0.66
0.75
DX
0.43
0.31
0.50
0.58
ty
0.43
0.54
0.72
0.78
Izh
0.45
0.52
0.70
0.77
0.44
0.44
0.63
0.70
too
0.75
ubr
0.48
0.52
0.44
0,39
0,57
udu
0.46
0.44
0.62
0.70
uig
0.46
0.52
0.70
0.76
ukr
0.48
0.57
0.74
0.79
upv
0.44
0.42
0.51
0.68
ura
0.43
0.35
0.53
0.61
uzb
0.39
0.33
0.53
0.61
und
0.47
0.57
0.74
0.80
urk
0.45
0.49
0.69
0.76
0.43
0.40
0.59
0.66
usp
0.38
0.72
nvl
0.42
0.39
0.44
0.57
0.74
0.80
vag
0.42
0.47
0.56
0.73
vem
0.47
0.55
0.72
0.78
0.49
0.50
0.68
0.75
vi
0.44
0.47
0.66
0.72
vmy
0.44
0.49
0.68
0.74
vun
0.46
0.54
0.71
0.77
0.66
0.73
waj
0.43
0.41
0.60
0.67
0.48
0.5%
0.71
0.77
wap
0.44
0.44
0.63
0.70
0,48
0.60
0.77
0.82
way
2440
0,3
0.62
0.69
wbm
0.47
0.45
0.64
0.71
whp
0.45
0.29
0.47
0.55
Wea
0.41
0.33
0.52
0.60
wer
0.43
0,38
0.57
0.65
whk
0.46
D:49
0.67
0.74
wib
0.44
0,36
0.55
0.63
0.78
wne
0.40
0.53
WIU
0.29
0.48
wob
0.49
0.37
0.56
0.64
wol
0.45
0.52
0.69
0.76
W04
0.40
0,34
0.53
0.60
w15
0.43
0.39
0.58
0.65
wak
0.43
0.35
0.54
0.62
WUY
0.43
0.51
0.70
0.76
0.44
0.46
0.65
0.72
xal
0.46
0.49
0.67
0.74
0.45
0.33
0.53
0.61
0.78
xod
0.43
0.49
0.75
sho
0.47
0.55
0.72
xla
0.12
0.37
0.56
0.64
xon
0,44
0.49
0.68
0.74
xxb
0.42
0.36
0.56
0.64
xsi
0.43
0.37
0.56
0.65
25m
0.44
0.43
0.63
0.71
0.43
0.33
0.52
0.60
xid
0.43
0.35
0.54
0.62
0.40
0.39
0.59
0.66
0.42
0.41
0.60
0.68
yaa
0.58
yad
0,45
yul
0.47
0.55
0.72
yam
0.42
0.39
0.59
0.67
yam
0.45
0.39
0.59
0.66
yaq
0.44
0.40
0.59
0.66
yby
0.43
0.34
0.53
0.61
yen
0.44
0.32
0.51
0.59
ykc
0.40
0.25
0.43
0.51
yli
0.54
0.62
yml
0.42
D.41
0.60
0.67
yon
0.44
0.63
0.70
yor
0.45
0.74
yrb
0.44
0.31
0.58
yTE
0.45
0,36
0,56
0.41
0.33
0.52
0.60
yua
0.54
0.72
0.78
y1u
0.38
0.39
0.57
0.65
yut
0.43
0.43
0.63
0.70
yoW
0.43
0.41
D.61
0.68
0.40
0.40
0.58
0.65
ynn
0.47
0.45
0.65
0.72
Zan
0.41
0.40
0.61
0.68
0.44
0.46
0.64
0.71
73c
0.40
0.42
0.62
0.69
2dd
0.40
B.45
0.64
0.71
0.45
0.5
0.64
0.71
zai
0.45
0.42
0.62
0.69
zam
0.43
0.35
0.55
0.63
280
0.43
0.44
0.63
0.70
0.43
0.47
0.66
0.73
0.40
0.44
0.62
0.69
0.44
0.46
0.65
0.72
0.65
zaW
0.43
0.46
0.65
0.55
zhe
0.49
0.60
0.77
0.82
Zia
0.43
0.35
0.54
0.62
ziw
0.50
0.56
0.73
0.79
zom
0.48
0.48
0.56
0.73
206
0.44
0.44
0.62
0.69
0.41
0.39
0.58
0.65
xpi
0.45
0.47
0.66
0.73
zp)
0:45
0:41
0.60
0.68
xpm
0.42
0.34
0.53
0.60
xpu
0.46
0.68
429
0.37
0.39
zpt
0.43
0.63
zpu
0.43
0.38
0.57
0.65
0.43
0.40
0.59
0.67
2pz
0.46
0.38
0.57
0.65
74m
0.47
0.56
0.73
0.79
757
0.46
0.48
0.67
0.73
0.39
0.43
0.63
0.71
zty
0.45
0.45
0.64
0.71
zul
0.46
0.55
0.72
0.78
xyp
0.45
0.49
0.68
0.74
Table 14: Transfer performance using other languages as the train/query language (Part V).


Reviews:
[{"review_id": "lJVIAxJzsN", "reviewer": "Reviewer_YCED", "summary": "", "questions": "A) Is there any other work on the automatic construction of colexification graphs? Is it possible to use them to train word embeddings?\nB) Since your work is about multilingual word embeddings, is it possible to learn word embeddings based on the gold colexification patterns (e.g. CLICS)? Is there a reference or comparison to your results?\nC) Why have you decided to compute only the Recall metric on CLICS and to propose the new metric #aw_colex instead of Precision?\nD) What are the advantages and disadvantages of using lemmas for English concepts while using n-grams for other languages?\nE) Is there a correlation between performance on CLICS in low/high resource languages and the quality of transfer learning?\nF) The paper on Conceptualizer [1] also proposes a new metric of similarity between languages based on colexification patterns. Is there a correlation between this metric and the quality of transfer learning?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thorough review and valuable comments. We would like to address your concerns and questions as follows:\n\n- ***\"The title of the paper is about \"cross-lingual transfer learning\" ... number of concepts.\"***\n\nOur focus in this paper is large-scale **crosslingual transfer learning**, especially for low-resource languages. We achieve this by leveraging multilingual colexification graphs, which are constructed from the colexification patterns extracted from a parallel corpus. Thus, the multilingual graph is a very important part of our paper and we tried to be as detailed as possible when introducing the graph construction. Of course, the crosslingual transfer-learning evaluation is vital as well. Therefore we also conducted extensive experiments including two evaluations (verse retrieval and classification), which really tested the zero-shot crosslingual transferability. In addition to the evaluation in the main content, we also include a detailed analysis in Appendix E and F (English and beyond English-centric transfer), which shows the performance by using **any** language as train/query language. We will move some analysis of the transfer learning to the main content in the camera-ready version.\n\n- ***\"The authors propose two methods for constructing a colexification ... to one method.\"***\n\nThough, as you said, the evaluation on CLICS can also be done on ColexNet+ (then we have to filter out the ngram nodes), ColexNet serves as a graph where we can **easily and efficiently** analyze its structure because it is lightweight (5870 nodes at most) and homogeneous in nodes (just concept nodes). Furthermore, we explored the centralities and communities in ColexNet and obtained some interesting findings (please refer to Section 5.1, Appendix C and D), which cannot be done directly on ColexNet+. In addition, the aims of the two graphs are different: ColexNet is used to explore the colexification patterns across languages; ColexNet+ is used to train multilingual embeddings for crosslingual transfer. \n\n- ***\"A) Is there any other work on ... train word embeddings?\"***\n\nThere are a few works on constructing colexification graphs, such as [1] and [2] (mentioned in Introduction and Related Works). However, they are not \"automatic\". These methods rely on colexification patterns from external hand-curated lexica, while our method extracts the patterns directly from a parallel corpus. These two works also train embeddings, but they focus on lexical-level tasks such as semantic word similarity and word sense disambiguation, while we focus on crosslingual transfer learning in NLP. Their embeddings cannot easily be used in tasks such as sentence retrieval or classification, because the morphological changes are very frequent in many languages, while the tokens included in these embeddings are mostly in dictionary form.\n\n- ***\"B) Since your work is about multilingual word embeddings ... to your results?\"***\n\nThis is a good idea and we did some preliminary experiments earlier. The major problem is the coverage: for many low-resource languages that we support, CLICS only supports less than a hundred concepts, which will simply lead to very bad performance. But it would be a good future direction to combine the gold colexification patterns and automatically extracted patterns.\n\n- ***\"C) Why have you decided to compute only the Recall ... instead of Precision?\"***\n\nThe recall metric is more interesting to us because we want to know how many ground-truth colexification patterns in CLICS are included in ColexNet, e.g., if there is an edge between \"hand\" and \"arm\" in CLICS, we want an edge between these two concepts presenting in ColexNet as well. Precision might not be a good metric because patterns in CLICS are not comprehensive and many patterns included in ColexNet can actually be correct (please refer to Appendix B). Directly using precision then underestimates the performance. Therefore we use #aw_colex, which has a similar function to the precision, but can better and more directly reflect how the exact number of \"wrong\" patterns per concept changes with respect to $\\lambda$. \n\n- ***\"D) What are the advantages and disadvantages ... for other languages?\"***\n\nThe major advantages are that: (1) lemmatizer tools in English perform pretty well so we could use them to eliminate redundancy in concept representations, e.g., we will have \"belly\" instead of both \"belly\" and \"bellies\" to represent <belly> concept. (2) there are no available tokenization/lemmatization tools for most low-resource languages, therefore using ngrams gives **more freedom** when selecting best-matching character combinations. One disadvantage is that we observe some identified ngrams can sometimes include some redundant characters, e.g., \"鸽子的\" (pigeon in Chinese) instead of \"鸽子\" (pigeon), mainly for languages without separators between words.\n\n- ***\"E) Is there a correlation between performance ... quality of transfer learning?\"***\n\nIn Appendix F, we closely looked at how low/high resource languages when serving as train/query language influence the transfer performance. Here are some takeaways: (1) high-resource languages often mean higher quality of the extracted colexification patterns. And the quality is positively correlated with the transfer performance; (2) the model can have difficulties in inducing reliable colexification patterns for some low-resource languages due to various language-specific properties, and therefore sometimes can achieve bad performance. \n\n- ***\"F) The paper on Conceptualizer also proposes ... of transfer learning?\"***\n\nThis is a very insightful question. We guess there will be a positive correlation between the conceptual similarity and the transfer performance. We will do an additional test on this and include it in the camera-ready version.\n\n- ***\"What is for “d_tx = 1” in line 333 if it is not possible?***\n\nThe answer is: $d_{tx}$ can only be 0 or 2 in ColexNet+. The equation shown in the paper is the sampling strategy of Node2Vec. We explain how it is applied in our sampling in lines 335-344.\n\n\nWe will fix the typos and further improve the presentation of the paper, e.g., in more detail introducing some concepts from previous works, to improve the readability. \n\n\n\n[1] https://aclanthology.org/2022.naacl-main.386/ \n\n[2] https://openreview.net/pdf?id=Hn10RDGms3p", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper extends previous results [1], where Conceptualizer was applied and evaluated on a small set of manually annotated concepts and languages.\n2) The authors propose a new method for training multilingual word representations that are competitive in downstream tasks.\n3) This work could improve our understanding of the mechanism of cross-lingual transfer learning.", "reasons_to_reject": "My main concerns are about the focus of the paper and the presentation of the results (see section \"Typos, Grammar Style and Presentation Improvements\").\n1) The title of the paper is about \"cross-lingual transfer learning\", but big part of the paper is about graph construction. Basically it's a conversion of Conceptualizer output into a new format. I think the paper should focus more on multilingual embeddings and transfer learning or be renamed to something like \"evaluation of Conceptualizer on the large number of concepts\".\n2) The authors propose two methods for constructing a colexification graph: ColexNet and ColexNet+. It seems that the only purpose of the first method (ColexNet) is the evaluation on CLICS, which can also be done on ColexNet+. I suppose that the graph construction part can be reduced to one method.", "questions_to_authors": "A) Is there any other work on the automatic construction of colexification graphs? Is it possible to use them to train word embeddings?\nB) Since your work is about multilingual word embeddings, is it possible to learn word embeddings based on the gold colexification patterns (e.g. CLICS)? Is there a reference or comparison to your results?\nC) Why have you decided to compute only the Recall metric on CLICS and to propose the new metric #aw_colex instead of Precision?\nD) What are the advantages and disadvantages of using lemmas for English concepts while using n-grams for other languages?\nE) Is there a correlation between performance on CLICS in low/high resource languages and the quality of transfer learning?\nF) The paper on Conceptualizer [1] also proposes a new metric of similarity between languages based on colexification patterns. Is there a correlation between this metric and the quality of transfer learning?", "paper_topic_and_main_contributions": "This paper deals with the automatic construction of a colexification graph and its application to the training of multilingual word representations. The authors obtained the following results:\n1) Using Conceptualizer [1] and the Parallel Bible Corpus, the authors automatically constructed a colexification graph for 1,335 languages.\n2) The resulting graph was evaluated on the CLICS dataset.\n3) The authors propose a modified version of a colexification graph with nodes of target language n-grams between nodes of English concepts.\n4) Using a modified version of the colexification graph, the authors trained aligned multilingual word representations.\n5) The resulting multilingual representations were evaluated in three different tasks in a cross-lingual scenario.\n\n[1] [A Crosslingual Investigation of Conceptualization in 1335 Languages](https://aclanthology.org/2023.acl-long.726) (Liu et al., ACL 2023)", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "During reviewing I had to read a paper about Conceptualizer [1] because this paper uses some concepts from the previous work and poorly describes the main tool (Conceptualizer). E.g. it was not clear precisely what are colexification patterns, focal concepts, ngrams, dollar sign, input and output of the Conceptualizer. Ideally it would be nice to see an image how graph from Conceptualizer converts to the ColexNet+ graph.\n\nThere are some inaccuracies in the description of the algorithm:\n1) Conceptualizer returns not only n-gram but also a verse id and language id, which are not mentioned in the algorithm or its description\n2) Initially there are no edges in the graph. So what is “e” for which you initialize edges on the line 4 of the algorithm?\n3) Somewhere you write “w_c(e)” with one pair brackets, somewhere with two pairs ““w_c((e))”.\n4) Line 13 — missed closing bracket “wn((f, c)”\n\nLine 240 of the paper — use “l” as the maximum number of the iterations. I suppose it should be “M”.\n\nWhat is for “d_tx = 1” in the line 333 if it is not possible?\n\nIn Section 5.2 It would be nice to describe the results of transfer learning not only from English but also from the other languages."}}, {"review_id": "3vGb1qDXqw", "reviewer": "Reviewer_7BGY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and thoughtful feedback. We would like to address your concerns as follows:\n\n- ***\"i) The tasks are either not very representative of important downstream ... neither comprehensive nor mutually exclusive topics for Bible verses.\"***\n\nAs we acknowledged in the Limitation section, there are few evaluation datasets that cover such a wide range of languages including low-resource languages in the community. Therefore we have to resort to some in-domain tasks to evaluate the quality of the embeddings and compare with other embedding counterparts. The evaluation shows that our embeddings outperform all other embeddings on the same tasks, which can verify the effectiveness of the introduction of colexification into NLP. Indeed, good performance on word-level translation tasks does not guarantee good machine translation performance, since machine translation additionally requires the ability of language generation. Nevertheless, embeddings are an essential part of modern NLP applications and there are already some works showing how well-aligned multilingual embeddings, combined with Transformer-based language models, can boost the performance on various downstream tasks, including low-resource machine translation [1]. Therefore we would expect our work can similarly provide some insights for future works involving low-resource languages in the era of LLMs. Although the verse classification benchmark (the set of classes as well as the verses belonging to each class are carefully selected according to [2]) is also limited to the Bible, it is the only dataset that supports most of the languages that we support, making it possible to do a fair comparison between multiple methods on a large-scale set of languages.\n\n- ***\"ii) The baselines for multilingual embeddings do not include Transformer-based ... applicability of the proposed multilingual embeddings on downstream tasks.\"***\n\nThis is a very good point. Except for the BOW baseline, all other baselines are embeddings learned from the same data and therefore they are fully comparable. Transformer-based models like XLM-R are pre-trained on unannotated corpus and do not explicitly learn the lexical-level alignment knowledge. As a result, the transfer performance, especially for the low-resource languages, can be very bad in these tasks. A recent paper [3] confirms this. They reported XLM-R (base and large) and Glot500 (a continue-trained model of XLM-R on more than 500 languages) on several tasks including verse retrieval. For many extremely low-resourced languages, the results are far less than our multilingual embeddings. We will include this comparison in the camera-ready version. As an insight, training an LLM considerably consumes resources and it only works if enough data is available. On the other hand, training embeddings is much more efficient and we can get good performance if we use small-size parallel data to explicitly extract some lexical-level knowledge (like what we did). Furthermore, it is worth exploring if such lexical knowledge encoded in the multilingual embeddings could be used to improve the crosslinguality of an LLM, therefore boosting its transfer learning.\n\n\n[1] https://aclanthology.org/2021.naacl-main.16.pdf\n\n[2] https://arxiv.org/pdf/2305.08487.pdf\n\n[3] https://aclanthology.org/2023.acl-long.61.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "ColexNet and ColexNet+ provide valuable assets to the study of multilingual colexification and linguistic typology, and the proposed data-driven algorithm is shown to be well-aligned with the manually constructed gold standard CLICS for English-stemming concepts. The algorithm proposed is replicable and can be extended to any parallel corpus. The embeddings produced from ColexNet+ potentially hold value for downstream NLP tasks in very low-resourced languages. The table of experimental results between every pair of languages in the appendix is valuable for ranking the potential of the approach for many individual extremely low-resourced languages.", "reasons_to_reject": "The experiments section, while thorough, has the following weaknesses:\n\ni) The tasks are either not very representative of important downstream NLP applications of word embeddings, or are only evaluated within the context of PBC. For example, the success of the embeddings on roundtrip word translation does not provide insight into to their success on a task like low-resource machine translation. The verse classification task, while based on a recent multilingual text classification benchmark, uses a rather arbitrary set of classes which seem neither comprehensive nor mutually exclusive topics for Bible verses.\n\nii) The baselines for multilingual embeddings do not include Transformer-based models. This is understandable for extremely low-resourced languages, but it would be useful to see a comparison of ColexNet+ embeddings with multilingual transformer encoders such as XLM-R (Conneau et al. 2020). A task such as verse retrieval seems like a good place to do this, at least for the subset of language pairs that are represented in the 88 languages on which XLM-R was trained. Can the ColexNet+ embeddings be shown to outperform multilingual Transformer encoders for extremely low-resourced languages? An answer to this question is consequential to the applicability of the proposed multilingual embeddings on downstream tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an approach to build a multilingual colexification graph from the Parallel Bible Corpus. The resulting English-concept-centered colexification graph ColexNet is extended into the multilingual ColexNet+, which contains concept-realizing ngrams as intermediate nodes between abstract concepts. ColexNet is shown to robustly identify common colexification patterns to the gold standard CLICS. ColexNet+ is used to train multilingual embeddings which outperform multilingual baselines on 3 diverse downstream tasks of roundtrip translation, verse retrieval and verse classification.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5KnvEpcMNO", "reviewer": "Reviewer_knrQ", "summary": "", "questions": "- A. The working of the Conceptualizer reminds me of Bannard and Callison-Burch's pivot method to discover paraphrases (https://aclanthology.org/P05-1074.pdf). I guess that the lambda threshold limits, too, the presence of paraphrases, but I would have expected to see some comments on this. Could you briefly compare the methods or add some words on this?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review and positive comments. We would like to answer your question as follows:\n\n- ***\"Perhaps it would be nice if some precision-like measure ... not be straightforward.\"***\n\nYes, directly using precision might not be the best option as it can underestimate the performance. This is because the colexification patterns in CLICS itself are not comprehensive and many patterns included in ColexNet can actually be correct (please refer to Appendix B). Therefore we use #aw_colex, which has a similar function to the precision. This metric can better and more directly reflect how the exact number of \"wrong\" patterns per concept changes with respect to \\lambda. \n\n- ***\"A. The working of the Conceptualizer reminds me of Bannard and Callison-Burch's pivot method ... some words on this?\"***\n\n[1] also has a two-step working flow as Conceptualizer: (1) look at what foreign language phrases the English translates to and find all occurrences of those foreign phrases (this roughly corresponds to the forward pass in Conceptualizer) (2) then look back at what other English phrases they translate to (this roughly corresponds to the backward pass). \n\nThe major differences are that: [1] works on a token level instead of ngrams; [1] needs a token-level word/phrase aligner to get the initial alignments, while Conceptualizer itself integrates a $\\chi^2$-based concept-ngram alignment method. Thanks for mentioning the paper and we will cite it in our camera-ready version. \n\nThe $\\lambda$ threshold in our method does not directly influence the presence of paraphrases, as we have a pre-defined concept pool (a set of English lemmata) as the candidates on the English side for the two-step alignment. We agree that there might be some paraphrases e.g., <prison> and <jail>, in the concept pool. We simply regard them as different concepts under such cases. The lambda threshold mainly influences the quality of the identified colexification patterns. As we discussed in our paper, if the threshold \\lambda is set larger, fewer concepts and fewer edges will be included in ColexNet. If two concepts are colexified in many languages, we can be more certain of the colexification relation between the two concepts. \n\nWe will fix the typos and further improve the presentation of the paper. \n\n[1] https://aclanthology.org/P05-1074.pdf", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is very well-written, the topic is relevant, it addresses a very large number of languages, the proposed approach is simple and sound and the experiments are interesting. Additionally, there are interesting and extensive analyses in the Appendices. The limitations are appropriately addressed and acknowledged in the corresponding section.", "reasons_to_reject": "I don't see any big weaknesses.", "questions_to_authors": "- A. The working of the Conceptualizer reminds me of Bannard and Callison-Burch's pivot method to discover paraphrases (https://aclanthology.org/P05-1074.pdf). I guess that the lambda threshold limits, too, the presence of paraphrases, but I would have expected to see some comments on this. Could you briefly compare the methods or add some words on this?", "paper_topic_and_main_contributions": "UPDATE: I thank the authors for their clarifications.\n\nThis paper proposes to build multilingual graphs (ColexNet and ColexNet+) from colexification patterns that have been automatically identified on a corpus of 1,334 languages (the Bible). These graphs are shown to achieve high recall at capturing crosslingual colexifications. The authors propose to obtain multilingual embeddings from one of these graphs and evaluate them on roundtrip translation, verse retrieval and verse classification. Results show that the proposed embeddings perform better than a number of existing alternatives. In one experiment, transfer learning is performed from every language to every language.\n\nI do not have any big criticism for this paper, I think it should be accepted. \n\nPerhaps it would be nice if some precision-like measure could be calculated for the colexification identification task, but I understand this would not be straightforward.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- l240 \"l = 3 iterations\". -> \"l\" is already used for language. From the Appendix I'd assume it should be \"M\".\n\n- I found the CLIQUE embeddings description a bit confusing.\n\n- Could you make Figure 2's font larger?"}}, {"review_id": "OTihaQDPqi", "reviewer": "Reviewer_A1sh", "summary": "", "questions": "A. Can you comment on how your work is similar to (and also different from) the pivoting approach used by e.g. [1]? It seems to me that there is some overlap in the idea of colexification and the idea that paraphrases in a single language can be identfied by pivoting in a different language.\nB. In the very interesting analysis that you have in Appendix B, did you observe more semantically organized errors in colexification? For instance, [2] shows that automatically identified bilingual lexicons have a variety of relations exhibited in them. I wonder if colexification graphs also have such semantic errors.\n\n\n\n[1] - https://aclanthology.org/P05-1074.pdf\n[2] - https://dl.acm.org/doi/10.1145/2050100.2050102\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our paper. We would like to address your concern as follows:\n\n- ***\"I believe the paper could have been enhanced ... with its experiments.\"***\n\nThanks for mentioning this. The major problem is that most models only support a small portion of the languages that our model supports. As we wanted to compare on as many languages as possible, we did not include other non-control baselines in the current version. We are considering including the non-control baseline Glot500 [1] (a continue-trained model of XLM-R on more than 500 languages) for comparison in the camera-ready version. According to their results, for extremely low-resourced languages (very likely unseen by the transformer-based model), XLM-R or Glot500 perform far behind multilingual embeddings.\n\n\n- ***\"A. Can you comment on how your work is similar to ... a different language.\"***\n\nBoth our method and [2] have a two-step working flow: \n\n- [2] looks at what foreign language phrases the English translates to, and finds all occurrences of those foreign phrases (this roughly corresponds to the forward pass in our method to find target-language ngrams)\n- [2] then looks back at what other English phrases they (those foreign phrases) translate to (this roughly corresponds to the backward pass to find English concepts). \n\nThe major differences are that: [2] works on a token-level instead of ngrams; [2] needs a token-level word/phrase aligner to get the initial alignments, while Conceptualizer (used in our method) itself integrates a $\\chi^2$-based concept-ngram alignment method. Actually, this kind of idea is also similar to the semantic mirror [3], a method to explore semantic relations using translational data. Thanks for mentioning the paper and we will cite the paper in our camera-ready version. \n\n- ***\"B. In the very interesting analysis that you have in Appendix B ... such semantic errors.\"***\n\nThis is a very good question. In our setting, the extraction of colexification patterns is formed similarly to bilingual lexicon identification, where we first identify target-language ngrams that correspond to an English concept, and then English concepts that correspond to those identified ngrams. Therefore, our method is not immune to semantic errors, such as antonyms, hypernyms, or hyponyms. In our manual inspection, we found **free-translation** and **co-occurrence** are the two major reasons for many types of semantic errors, as the examples shown in Appendix B. This finding matches the statistics in [4], which shows that \"Related\" makes up the major part of the semantic errors. Thanks for mentioning the paper and will cite this in our camera-ready version.\n\n\n[1] https://aclanthology.org/2023.acl-long.61.pdf\n\n[2] https://aclanthology.org/P05-1074.pdf\n\n[3] https://clara.w.uib.no/files/2011/06/semmirrors.pdf\n\n[4] https://dl.acm.org/doi/10.1145/2050100.2050102", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Through experimentation: The paper does a very controlled and careful set of experiments to support its claim. For instance, all baselines are trained on the same corpus, with the same dimensionality, etc. to ensure there is no arbitrary variance. \n* Colexicifcation for language transfer is an interesting strategy that has not been explored by prior work on multilingual embeddings. This work does a very through investigation on this phenomenon, proposes a technique to automatically build a graph that identifies colexfication for > 1000 languages, and effectively uses it for cross-lingual transfer. That, in my opinion, is a series of contributions that are unique to this field. ", "reasons_to_reject": "No strong reason to reject. I believe the paper could have been enhanced had it been compared with non-control baselines also (e.g. models that have access to more data) to identify how far away this approach is from more powerful lexical representations, trained on more data. However, that is not required and the paper tells a complete and coherent story with its experiments. ", "questions_to_authors": "A. Can you comment on how your work is similar to (and also different from) the pivoting approach used by e.g. [1]? It seems to me that there is some overlap in the idea of colexification and the idea that paraphrases in a single language can be identfied by pivoting in a different language.\nB. In the very interesting analysis that you have in Appendix B, did you observe more semantically organized errors in colexification? For instance, [2] shows that automatically identified bilingual lexicons have a variety of relations exhibited in them. I wonder if colexification graphs also have such semantic errors.\n\n\n\n[1] - https://aclanthology.org/P05-1074.pdf\n[2] - https://dl.acm.org/doi/10.1145/2050100.2050102\n", "paper_topic_and_main_contributions": "This paper attempts to build multilingual word embeddings for a large number of languages (1335 to be precise) by exploiting colexification patterns i.e. the phenomenon where a language expresses two different meaning with the same lexical form. The central contribution of this paper is that it builds a large-scale undirected colexification graph where each node is a concept (i.e. a lemma in English, for the purposes of this work), and the edge between two nodes indicates how many languages co-lexify the two concepts. Such a graph is automatically constructed from a large multi-way parallel corpus using a two step alignment process on the corpus (using the process from Liu et al., ACL 2023). After construction, this graph is modified to represent colexification between concepts and n-grams, by replacing each edge in the graph with a new set of edges that capture the ngrams via which the concepts are co-lexified. \n\nThis graph is then used to train a multilingual embedding model using node2vec, with the hypothesis that colexification is a critical signal for cross-lingual transfer. Evaluation is then carried out first in an intrinsic way, by identifying whether automatically constructed graph captures known set of colexification patterns. Extrinsic evaluation focuses on comparing different embedding models (trained on the same corpus) on 3 different tasks---round trip translation, passage retrieval, document similarity---and the proposed embeddings are shown to be more accurate across the board.\n\nThere is additional analysis to identify the nature of the graph created as well an the ability to use a language other than English to seed the cross-lingual transfer process.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Summary (adapted from Reviewer 7BGY): This paper proposes an approach to build a multilingual colexification graph from the Parallel Bible Corpus (PBC). The resulting English-concept-centered colexification graph ColexNet is extended into the multilingual ColexNet+, which contains concept-realizing n-grams as intermediate nodes between abstract concepts. ColexNet is shown to robustly identify common colexification patterns to the gold standard CLICS. ColexNet+ is used to train multilingual embeddings which outperform multilingual baselines on 3 diverse downstream tasks of roundtrip translation, verse retrieval and verse classification.

Overall, the reviews were positive, with the “Reasons to Reject” mostly focusing on improvements in clarity in the paper and explanation of the results. Reviewer 7BGY lays out concerns about whether the colexification graph and embeddings will be of use to other tasks, and whether the Bible data is sufficient for evaluation.

These concerns are valid, but as the authors’ response makes clear regarding these comments and others, there is a tradeoff between trying to evaluate in as many languages as possible in terms of limitations of both data and models available (i.e. pretrained Transformers do not cover the number of languages needed). The authors have had to make compromises, but the paper seems sound within the scope it attempts to address.

The response period was extremely productive, and the authors provided clear responses to the reviews, all of which the reviewers acknowledged.

As came up in the discussion with Reviewer 7BGY, this paper might benefit from further discussion of its limitations with regards to the use of a Bible corpus as the primary dataset. While this is mentioned in the limitations, it receives little attention and it may be worthwhile to discuss it more explicitly and earlier in the paper.
Decision: Accept-Findings