prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the 12 languages covered?
Context 1: Introduction
The lack of annotated training and evaluation data for many tasks and domains hinders the development of computational models for the majority of the world's languages BIBREF0, BIBREF1, BIBREF2  The necessity to guide and advance multilingual and cross-lingual NLP through annotation efforts that follow cross-lingually consistent guidelines has been recently recognized by collaborative initiatives such as the Universal Dependency (UD) project BIBREF3  The latest version of UD (as of March 2020) covers more than 70 languages  Crucially, this resource continues to steadily grow and evolve through the contributions of annotators from across the world, extending the UD's reach to a wide array of typologically diverse languages  Besides steering research in multilingual parsing BIBREF4, BIBREF5, BIBREF6 and cross-lingual parser transfer BIBREF7, BIBREF8, BIBREF9, the consistent annotations and guidelines have also enabled a range of insightful comparative studies focused on the languages' syntactic (dis)similarities BIBREF10, BIBREF11, BIBREF12 
Inspired by the UD work and its substantial impact on research in (multilingual) syntax, in this article we introduce Multi-SimLex, a suite of manually and consistently annotated semantic datasets for 12 different languages, focused on the fundamental lexical relation of semantic similarity BIBREF13, BIBREF14  For any pair of words, this relation measures whether their referents share the same (functional) features, as opposed to general cognitive association captured by co-occurrence patterns in texts (i e , the distributional information)  Datasets that quantify the strength of true semantic similarity between concept pairs such as SimLex-999 BIBREF14 or SimVerb-3500 BIBREF15 have been instrumental in improving models for distributional semantics and representation learning  Discerning between semantic similarity and relatedness/association is not only crucial for theoretical studies on lexical semantics (see §SECREF2), but has also been shown to benefit a range of language understanding tasks in NLP  Examples include dialog state tracking BIBREF16, BIBREF17, spoken language understanding BIBREF18, BIBREF19, text simplification BIBREF20, BIBREF21, BIBREF22, dictionary and thesaurus construction BIBREF23, BIBREF24 
Despite the proven usefulness of semantic similarity datasets, they are available only for a small and typologically narrow sample of resource-rich languages such as German, Italian, and Russian BIBREF25, whereas some language types and low-resource languages typically lack similar evaluation data  Even if some resources do exist, they are limited in their size (e g , 500 pairs in Turkish BIBREF26, 500 in Farsi BIBREF27, or 300 in Finnish BIBREF28) and coverage (e g , all datasets which originated from the original English SimLex-999 contain only high-frequent concepts, and are dominated by nouns)  This is why, as our departure point, we introduce a larger and more comprehensive English word similarity dataset spanning 1,888 concept pairs (see §SECREF4) 
Most importantly, semantic similarity datasets in different languages have been created using heterogeneous construction procedures with different guidelines for translation and annotation, as well as different rating scales  For instance, some datasets were obtained by directly translating the English SimLex-999 in its entirety BIBREF25, BIBREF16 or in part BIBREF28  Other datasets were created from scratch BIBREF26 and yet others sampled English concept pairs differently from SimLex-999 and then translated and reannotated them in target languages BIBREF27  This heterogeneity makes these datasets incomparable and precludes systematic cross-linguistic analyses  In this article, consolidating the lessons learned from previous dataset construction paradigms, we propose a carefully designed translation and annotation protocol for developing monolingual Multi-SimLex datasets with aligned concept pairs for typologically diverse languages  We apply this protocol to a set of 12 languages, including a mixture of major languages (e g , Mandarin, Russian, and French) as well as several low-resource ones (e g , Kiswahili, Welsh, and Yue Chinese)  We demonstrate that our proposed dataset creation procedure yields data with high inter-annotator agreement rates (e g , the average mean inter-annotator agreement for Welsh is 0 742) 
The unified construction protocol and alignment between concept pairs enables a series of quantitative analyses  Preliminary studies on the influence that polysemy and cross-lingual variation in lexical categories (see §SECREF6) have on similarity judgments are provided in §SECREF5  Data created according to Multi-SimLex protocol also allow for probing into whether similarity judgments are universal across languages, or rather depend on linguistic affinity (in terms of linguistic features, phylogeny, and geographical location)  We investigate this question in §SECREF25  Naturally, Multi-SimLex datasets can be used as an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms  We conduct a systematic evaluation of several state-of-the-art representation models in §SECREF7, showing that there are large gaps between human and system performance in all languages  The proposed construction paradigm also supports the automatic creation of 66 cross-lingual Multi-SimLex datasets by interleaving the monolingual ones  We outline the construction of the cross-lingual datasets in §SECREF6, and then present a quantitative evaluation of a series of cutting-edge cross-lingual representation models on this benchmark in §SECREF8 
Contributions  We now summarize the main contributions of this work:
1) Building on lessons learned from prior work, we create a more comprehensive lexical semantic similarity dataset for the English language spanning a total of 1,888 concept pairs balanced with respect to similarity, frequency, and concreteness, and covering four word classes: nouns, verbs, adjectives and, for the first time, adverbs  This dataset serves as the main source for the creation of equivalent datasets in several other languages 
2) We present a carefully designed and rigorous language-agnostic translation and annotation protocol  These well-defined guidelines will facilitate the development of future Multi-SimLex datasets for other languages  The proposed protocol eliminates some crucial issues with prior efforts focused on the creation of multi-lingual semantic resources, namely: i) limited coverage; ii) heterogeneous annotation guidelines; and iii) concept pairs which are semantically incomparable across different languages 
3) We offer to the community manually annotated evaluation sets of 1,888 concept pairs across 12 typologically diverse languages, and 66 large cross-lingual evaluation sets  To the best of our knowledge, Multi-SimLex is the most comprehensive evaluation resource to date focused on the relation of semantic similarity 
4) We benchmark a wide array of recent state-of-the-art monolingual and cross-lingual word representation models across our sample of languages  The results can serve as strong baselines that lay the foundation for future improvements 
5) We present a first large-scale evaluation study on the ability of encoders pretrained on language modeling (such as bert BIBREF29 and xlm BIBREF30) to reason over word-level semantic similarity in different languages  To our own surprise, the results show that monolingual pretrained encoders, even when presented with word types out of context, are sometimes competitive with static word embedding models such as fastText BIBREF31 or word2vec BIBREF32  The results also reveal a huge gap in performance between massively multilingual pretrained encoders and language-specific encoders in favor of the latter: our findings support other recent empirical evidence related to the “curse of multilinguality” BIBREF33, BIBREF34 in representation learning 
6) We make all of these resources available on a website which facilitates easy creation, submission and sharing of Multi-Simlex-style datasets for a larger number of languages  We hope that this will yield an even larger repository of semantic resources that inspire future advances in NLP within and across languages 
In light of the success of Universal Dependencies BIBREF3, we hope that our initiative will instigate a collaborative public effort with established and clear-cut guidelines that will result in additional Multi-SimLex datasets in a large number of languages in the near future  Moreover, we hope that it will provide means to advance our understanding of distributional and lexical semantics across a large number of languages  All monolingual and cross-lingual Multi-SimLex datasets–along with detailed translation and annotation guidelines–are available online at: https://multisimlex com/ 
Lexical Semantic Similarity ::: Similarity and Association
The focus of the Multi-SimLex initiative is on the lexical relation of pure semantic similarity  For any pair of words, this relation measures whether their referents share the same features  For instance, graffiti and frescos are similar to the extent that they are both forms of painting and appear on walls  This relation can be contrasted with the cognitive association between two words, which often depends on how much their referents interact in the real world, or are found in the same situations  For instance, a painter is easily associated with frescos, although they lack any physical commonalities  Association is also known in the literature under other names: relatedness BIBREF13, topical similarity BIBREF35, and domain similarity BIBREF36 
Semantic similarity and association overlap to some degree, but do not coincide BIBREF37, BIBREF38  In fact, there exist plenty of pairs that are intuitively associated but not similar  Pairs where the converse is true can also be encountered, although more rarely  An example are synonyms where a word is common and the other infrequent, such as to seize and to commandeer  BIBREF14 revealed that while similarity measures based on the WordNet graph BIBREF39 and human judgments of association in the University of South Florida Free Association Database BIBREF40 do correlate, a number of pairs follow opposite trends  Several studies on human cognition also point in the same direction  For instance, semantic priming can be triggered by similar words without association BIBREF41  On the other hand, a connection with cue words is established more quickly for topically related words rather than for similar words in free association tasks BIBREF42 
A key property of semantic similarity is its gradience: pairs of words can be similar to a different degree  On the other hand, the relation of synonymy is binary: pairs of words are synonyms if they can be substituted in all contexts (or most contexts, in a looser sense), otherwise they are not  While synonyms can be conceived as lying on one extreme of the semantic similarity continuum, it is crucial to note that their definition is stated in purely relational terms, rather than invoking their referential properties BIBREF43, BIBREF44, BIBREF45  This makes behavioral studies on semantic similarity fundamentally different from lexical resources like WordNet BIBREF46, which include paradigmatic relations (such as synonymy) 
Lexical Semantic Similarity ::: Similarity for NLP: Intrinsic Evaluation and Semantic Specialization
The ramifications of the distinction between similarity and association are profound for distributional semantics  This paradigm of lexical semantics is grounded in the distributional hypothesis, formulated by BIBREF47 and BIBREF48  According to this hypothesis, the meaning of a word can be recovered empirically from the contexts in which it occurs within a collection of texts  Since both pairs of topically related words and pairs of purely similar words tend to appear in the same contexts, their associated meaning confounds the two distinct relations BIBREF14, BIBREF49, BIBREF50  As a result, distributional methods obscure a crucial facet of lexical meaning  This limitation also reflects onto word embeddings (WEs), representations of words as low-dimensional vectors that have become indispensable for a wide range of NLP applications BIBREF51, BIBREF52, BIBREF53  In particular, it involves both static WEs learned from co-occurrence patterns BIBREF32, BIBREF54, BIBREF31 and contextualized WEs learned from modeling word sequences BIBREF55, BIBREF29  As a result, in the induced representations, geometrical closeness (measured e g  through cosine distance) conflates genuine similarity with broad relatedness  For instance, the vectors for antonyms such as sober and drunk, by definition dissimilar, might be neighbors in the semantic space under the distributional hypothesis  BIBREF36, BIBREF56, and BIBREF53 demonstrated that different choices of hyper-parameters in WE algorithms (such as context window) emphasize different relations in the resulting representations  Likewise, BIBREF57 and BIBREF54 discovered that WEs learned from texts annotated with syntactic information mirror similarity better than simple local bag-of-words neighborhoods 
The failure of WEs to capture semantic similarity, in turn, affects model performance in several NLP applications where such knowledge is crucial  In particular, Natural Language Understanding tasks such as statistical dialog modeling, text simplification, or semantic text similarity BIBREF58, BIBREF18, BIBREF59, among others, suffer the most  As a consequence, resources providing clean information on semantic similarity are key in mitigating the side effects of the distributional signal  In particular, such databases can be employed for the intrinsic evaluations of specific WE models as a proxy of their reliability for downstream applications BIBREF60, BIBREF61, BIBREF14; intuitively, the more WEs are misaligned with human judgments of similarity, the more their performance on actual tasks is expected to be degraded  Moreover, word representations can be specialized (a k a  retrofitted) by disentangling word relations of similarity and association  In particular, linguistic constraints sourced from external databases (such as synonyms from WordNet) can be injected into WEs BIBREF62, BIBREF63, BIBREF16, BIBREF22, BIBREF64 in order to enforce a particular relation in a distributional semantic space while preserving the original adjacency properties 
Lexical Semantic Similarity ::: Similarity and Language Variation: Semantic Typology
In this work, we tackle the concept of (true) semantic similarity from a multilingual perspective  While the same meaning representations may be shared by all human speakers at a deep cognitive level, there is no one-to-one mapping between the words in the lexicons of different languages  This makes the comparison of similarity judgments across languages difficult, since the meaning overlap of translationally equivalent words is sometimes far less than exact  This results from the fact that the way languages `partition' semantic fields is partially arbitrary BIBREF65, although constrained cross-lingually by common cognitive biases BIBREF66  For instance, consider the field of colors: English distinguishes between green and blue, whereas Murle (South Sudan) has a single word for both BIBREF67 
In general, semantic typology studies the variation in lexical semantics across the world's languages  According to BIBREF68, the ways languages categorize concepts into the lexicon follow three main axes: 1) granularity: what is the number of categories in a specific domain?; 2) boundary location: where do the lines marking different categories lie?; 3) grouping and dissection: what are the membership criteria of a category; which instances are considered to be more prototypical? Different choices with respect to these axes lead to different lexicalization patterns  For instance, distinct senses in a polysemous word in English, such as skin (referring to both the body and fruit), may be assigned separate words in other languages such as Italian pelle and buccia, respectively BIBREF70  We later analyze whether similarity scores obtained from native speakers also loosely follow the patterns described by semantic typology 
Previous Work and Evaluation Data
Word Pair Datasets  Rich expert-created resources such as WordNet BIBREF46, BIBREF71, VerbNet BIBREF72, BIBREF73, or FrameNet BIBREF74 encode a wealth of semantic and syntactic information, but are expensive and time-consuming to create  The scale of this problem gets multiplied by the number of languages in consideration  Therefore, crowd-sourcing with non-expert annotators has been adopted as a quicker alternative to produce smaller and more focused semantic resources and evaluation benchmarks  This alternative practice has had a profound impact on distributional semantics and representation learning BIBREF14  While some prominent English word pair datasets such as WordSim-353 BIBREF75, MEN BIBREF76, or Stanford Rare Words BIBREF77 did not discriminate between similarity and relatedness, the importance of this distinction was established by BIBREF14 through the creation of SimLex-999  This inspired other similar datasets which focused on different lexical properties  For instance, SimVerb-3500 BIBREF15 provided similarity ratings for 3,500 English verbs, whereas CARD-660 BIBREF78 aimed at measuring the semantic similarity of infrequent concepts 
Semantic Similarity Datasets in Other Languages  Motivated by the impact of datasets such as SimLex-999 and SimVerb-3500 on representation learning in English, a line of related work focused on creating similar resources in other languages  The dominant approach is translating and reannotating the entire original English SimLex-999 dataset, as done previously for German, Italian, and Russian BIBREF25, Hebrew and Croatian BIBREF16, and Polish BIBREF79  Venekoski:2017nodalida apply this process only to a subset of 300 concept pairs from the English SimLex-999  On the other hand, BIBREF27 sampled a new set of 500 English concept pairs to ensure wider topical coverage and balance across similarity spectra, and then translated those pairs to German, Italian, Spanish, and Farsi (SEMEVAL-500)  A similar approach was followed by BIBREF26 for Turkish, by BIBREF80 for Mandarin Chinese, and by BIBREF81 for Japanese  BIBREF82 translated the concatenation of SimLex-999, WordSim-353, and the English SEMEVAL-500 into Thai and then reannotated it  Finally, BIBREF83 translated English SimLex-999 and WordSim-353 to 11 resource-rich target languages (German, French, Russian, Italian, Dutch, Chinese, Portuguese, Swedish, Spanish, Arabic, Farsi), but they did not provide details concerning the translation process and the resolution of translation disagreements  More importantly, they also did not reannotate the translated pairs in the target languages  As we discussed in § SECREF6 and reiterate later in §SECREF5, semantic differences among languages can have a profound impact on the annotation scores; particulary, we show in §SECREF25 that these differences even roughly define language clusters based on language affinity 
A core issue with the current datasets concerns a lack of one unified procedure that ensures the comparability of resources in different languages  Further, concept pairs for different languages are sourced from different corpora (e g , direct translation of the English data versus sampling from scratch in the target language)  Moreover, the previous SimLex-based multilingual datasets inherit the main deficiencies of the English original version, such as the focus on nouns and highly frequent concepts  Finally, prior work mostly focused on languages that are widely spoken and do not account for the variety of the world's languages  Our long-term goal is devising a standardized methodology to extend the coverage also to languages that are resource-lean and/or typologically diverse (e g , Welsh, Kiswahili as in this work) 
Multilingual Datasets for Natural Language Understanding  The Multi-SimLex initiative and corresponding datasets are also aligned with the recent efforts on procuring multilingual benchmarks that can help advance computational modeling of natural language understanding across different languages  For instance, pretrained multilingual language models such as multilingual bert BIBREF29 or xlm BIBREF30 are typically probed on XNLI test data BIBREF84 for cross-lingual natural language inference  XNLI was created by translating examples from the English MultiNLI dataset, and projecting its sentence labels BIBREF85  Other recent multilingual datasets target the task of question answering based on reading comprehension: i) MLQA BIBREF86 includes 7 languages ii) XQuAD BIBREF87 10 languages; iii) TyDiQA BIBREF88 9 widely spoken typologically diverse languages  While MLQA and XQuAD result from the translation from an English dataset, TyDiQA was built independently in each language  Another multilingual dataset, PAWS-X BIBREF89, focused on the paraphrase identification task and was created translating the original English PAWS BIBREF90 into 6 languages  We believe that Multi-SimLex can substantially contribute to this endeavor by offering a comprehensive multilingual benchmark for the fundamental lexical level relation of semantic similarity  In future work, Multi-SimLex also offers an opportunity to investigate the correlations between word-level semantic similarity and performance in downstream tasks such as QA and NLI across different languages 
The Base for Multi-SimLex: Extending English SimLex-999
In this section, we discuss the design principles behind the English (eng) Multi-SimLex dataset, which is the basis for all the Multi-SimLex datasets in other languages, as detailed in §SECREF5  We first argue that a new, more balanced, and more comprehensive evaluation resource for lexical semantic similarity in English is necessary  We then describe how the 1,888 word pairs contained in the eng Multi-SimLex were selected in such a way as to represent various linguistic phenomena within a single integrated resource 
Construction Criteria  The following criteria have to be satisfied by any high-quality semantic evaluation resource, as argued by previous studies focused on the creation of such resources BIBREF14, BIBREF15, BIBREF91, BIBREF27:
(C1) Representative and diverse  The resource must cover the full range of diverse concepts occurring in natural language, including different word classes (e g , nouns, verbs, adjectives, adverbs), concrete and abstract concepts, a variety of lexical fields, and different frequency ranges 
(C2) Clearly defined  The resource must provide a clear understanding of which semantic relation exactly is annotated and measured, possibly contrasting it with other relations  For instance, the original SimLex-999 and SimVerb-3500 explicitly focus on true semantic similarity and distinguish it from broader relatedness captured by datasets such as MEN BIBREF76 or WordSim-353 BIBREF75 
(C3) Consistent and reliable  The resource must ensure consistent annotations obtained from non-expert native speakers following simple and precise annotation guidelines 
In choosing the word pairs and constructing eng Multi-SimLex, we adhere to these requirements  Moreover, we follow good practices established by the research on related resources  In particular, since the introduction of the original SimLex-999 dataset BIBREF14, follow-up works have improved its construction protocol across several aspects, including: 1) coverage of more lexical fields, e g , by relying on a diverse set of Wikipedia categories BIBREF27, 2) infrequent/rare words BIBREF78, 3) focus on particular word classes, e g , verbs BIBREF15, 4) annotation quality control BIBREF78  Our goal is to make use of these improvements towards a larger, more representative, and more reliable lexical similarity dataset in English and, consequently, in all other languages 
The Final Output: English Multi-SimLex  In order to ensure that the criterion C1 is satisfied, we consolidate and integrate the data already carefully sampled in prior work into a single, comprehensive, and representative dataset  This way, we can control for diversity, frequency, and other properties while avoiding to perform this time-consuming selection process from scratch  Note that, on the other hand, the word pairs chosen for English are scored from scratch as part of the entire Multi-SimLex annotation process, introduced later in §SECREF5  We now describe the external data sources for the final set of word pairs:
1) Source: SimLex-999  BIBREF14  The English Multi-SimLex has been initially conceived as an extension of the original SimLex-999 dataset  Therefore, we include all 999 word pairs from SimLex, which span 666 noun pairs, 222 verb pairs, and 111 adjective pairs  While SimLex-999 already provides examples representing different POS classes, it does not have a sufficient coverage of different linguistic phenomena: for instance, it contains only very frequent concepts, and it does not provide a representative set of verbs BIBREF15 
2) Source: SemEval-17: Task 2 BIBREF92  We start from the full dataset of 500 concept pairs to extract a total of 334 concept pairs for English Multi-SimLex a) which contain only single-word concepts, b) which are not named entities, c) where POS tags of the two concepts are the same, d) where both concepts occur in the top 250K most frequent word types in the English Wikipedia, and e) do not already occur in SimLex-999  The original concepts were sampled as to span all the 34 domains available as part of BabelDomains BIBREF93, which roughly correspond to the main high-level Wikipedia categories  This ensures topical diversity in our sub-sample 
3) Source: CARD-660 BIBREF78  67 word pairs are taken from this dataset focused on rare word similarity, applying the same selection criteria a) to e) employed for SEMEVAL-500  Words are controlled for frequency based on their occurrence counts from the Google News data and the ukWaC corpus BIBREF94  CARD-660 contains some words that are very rare (logboat), domain-specific (erythroleukemia) and slang (2mrw), which might be difficult to translate and annotate across a wide array of languages  Hence, we opt for retaining only the concept pairs above the threshold of top 250K most frequent Wikipedia concepts, as above 
4) Source: SimVerb-3500 BIBREF15 Since both CARD-660 and SEMEVAL-500 are heavily skewed towards noun pairs, and nouns also dominate the original SimLex-999, we also extract additional verb pairs from the verb-specific similarity dataset SimVerb-3500  We randomly sample 244 verb pairs from SimVerb-3500 that represent all similarity spectra  In particular, we add 61 verb pairs for each of the similarity intervals: $[0,1 5), [1 5,3), [3,4 5), [4 5, 6]$  Since verbs in SimVerb-3500 were originally chosen from VerbNet BIBREF95, BIBREF73, they cover a wide range of verb classes and their related linguistic phenomena 
5) Source: University of South Florida BIBREF96 norms, the largest database of free association for English  In order to improve the representation of different POS classes, we sample additional adjectives and adverbs from the USF norms following the procedure established by BIBREF14, BIBREF15  This yields additional 122 adjective pairs, but only a limited number of adverb pairs (e g , later – never, now – here, once – twice)  Therefore, we also create a set of adverb pairs semi-automatically by sampling adjectives that can be derivationally transformed into adverbs (e g  adding the suffix -ly) from the USF, and assessing the correctness of such derivation in WordNet  The resulting pairs include, for instance, primarily – mainly, softly – firmly, roughly – reliably, etc  We include a total of 123 adverb pairs into the final English Multi-SimLex  Note that this is the first time adverbs are included into any semantic similarity dataset 
Fulfillment of Construction Criteria  The final eng Multi-SimLex dataset spans 1,051 noun pairs, 469 verb pairs, 245 adjective pairs, and 123 adverb pairs  As mentioned above, the criterion C1 has been fulfilled by relying only on word pairs that already underwent meticulous sampling processes in prior work, integrating them into a single resource  As a consequence, Multi-SimLex allows for fine-grained analyses over different POS classes, concreteness levels, similarity spectra, frequency intervals, relation types, morphology, lexical fields, and it also includes some challenging orthographically similar examples (e g , infection – inflection)  We ensure that the criteria C2 and C3 are satisfied by using similar annotation guidelines as Simlex-999, SimVerb-3500, and SEMEVAL-500 that explicitly target semantic similarity  In what follows, we outline the carefully tailored process of translating and annotating Multi-SimLex datasets in all target languages 
Multi-SimLex: Translation and Annotation
We now detail the development of the final Multi-SimLex resource, describing our language selection process, as well as translation and annotation of the resource, including the steps taken to ensure and measure the quality of this resource  We also provide key data statistics and preliminary cross-lingual comparative analyses 
Language Selection  Multi-SimLex comprises eleven languages in addition to English  The main objective for our inclusion criteria has been to balance language prominence (by number of speakers of the language) for maximum impact of the resource, while simultaneously having a diverse suite of languages based on their typological features (such as morphological type and language family)  Table TABREF10 summarizes key information about the languages currently included in Multi-SimLex  We have included a mixture of fusional, agglutinative, isolating, and introflexive languages that come from eight different language families  This includes languages that are very widely used such as Chinese Mandarin and Spanish, and low-resource languages such as Welsh and Kiswahili  We hope to further include additional languages and inspire other researchers to contribute to the effort over the lifetime of this project 
The work on data collection can be divided into two crucial phases: 1) a translation phase where the extended English language dataset with 1,888 pairs (described in §SECREF4) is translated into eleven target languages, and 2) an annotation phase where human raters scored each pair in the translated set as well as the English set  Detailed guidelines for both phases are available online at: https://multisimlex com 
Multi-SimLex: Translation and Annotation ::: Word Pair Translation
Translators for each target language were instructed to find direct or approximate translations for the 1,888 word pairs that satisfy the following rules  (1) All pairs in the translated set must be unique (i e , no duplicate pairs); (2) Translating two words from the same English pair into the same word in the target language is not allowed (e g , it is not allowed to translate car and automobile to the same Spanish word coche)  (3) The translated pairs must preserve the semantic relations between the two words when possible  This means that, when multiple translations are possible, the translation that best conveys the semantic relation between the two words found in the original English pair is selected  (4) If it is not possible to use a single-word translation in the target language, then a multi-word expression (MWE) can be used to convey the nearest possible semantics given the above points (e g , the English word homework is translated into the Polish MWE praca domowa) 
Satisfying the above rules when finding appropriate translations for each pair–while keeping to the spirit of the intended semantic relation in the English version–is not always straightforward  For instance, kinship terminology in Sinitic languages (Mandarin and Yue) uses different terms depending on whether the family member is older or younger, and whether the family member comes from the mother’s side or the father’s side  UTF8gbsn In Mandarin, brother has no direct translation and can be translated as either: 哥哥 (older brother) or 弟弟 (younger brother)  Therefore, in such cases, the translators are asked to choose the best option given the semantic context (relation) expressed by the pair in English, otherwise select one of the translations arbitrarily  This is also used to remove duplicate pairs in the translated set, by differentiating the duplicates using a variant at each instance  Further, many translation instances were resolved using near-synonymous terms in the translation  For example, the words in the pair: wood – timber can only be directly translated in Estonian to puit, and are not distinguishable  Therefore, the translators approximated the translation for timber to the compound noun puitmaterjal (literally: wood material) in order to produce a valid pair in the target language  In some cases, a direct transliteration from English is used  For example, the pair: physician and doctor both translate to the same word in Estonian (arst); the less formal word doktor is used as a translation of doctor to generate a valid pair 
We measure the quality of the translated pairs by using a random sample set of 100 pairs (from the 1,888 pairs) to be translated by an independent translator for each target language  The sample is proportionally stratified according to the part-of-speech categories  The independent translator is given identical instructions to the main translator; we then measure the percentage of matched translated words between the two translations of the sample set  Table TABREF12 summarizes the inter-translator agreement results for all languages and by part-of-speech subsets  Overall across all languages, the agreement is 84 8%, which is similar to prior work BIBREF27, BIBREF97 
Multi-SimLex: Translation and Annotation ::: Guidelines and Word Pair Scoring
Across all languages, 145 human annotators were asked to score all 1,888 pairs (in their given language)  We finally collect at least ten valid annotations for each word pair in each language  All annotators were required to abide by the following instructions:
1  Each annotator must assign an integer score between 0 and 6 (inclusive) indicating how semantically similar the two words in a given pair are  A score of 6 indicates very high similarity (i e , perfect synonymy), while zero indicates no similarity 
2  Each annotator must score the entire set of 1,888 pairs in the dataset  The pairs must not be shared between different annotators 
3  Annotators are able to break the workload over a period of approximately 2-3 weeks, and are able to use external sources (e g  dictionaries, thesauri, WordNet) if required 
4  Annotators are kept anonymous, and are not able to communicate with each other during the annotation process 
The selection criteria for the annotators required that all annotators must be native speakers of the target language  Preference to annotators with university education was given, but not required  Annotators were asked to complete a spreadsheet containing the translated pairs of words, as well as the part-of-speech, and a column to enter the score  The annotators did not have access to the original pairs in English 
To ensure the quality of the collected ratings, we have employed an adjudication protocol similar to the one proposed and validated by Pilehvar:2018emnlp  It consists of the following three rounds:
Round 1: All annotators are asked to follow the instructions outlined above, and to rate all 1,888 pairs with integer scores between 0 and 6 
Round 2: We compare the scores of all annotators and identify the pairs for each annotator that have shown the most disagreement  We ask the annotators to reconsider the assigned scores for those pairs only  The annotators may chose to either change or keep the scores  As in the case with Round 1, the annotators have no access to the scores of the other annotators, and the process is anonymous  This process gives a chance for annotators to correct errors or reconsider their judgments, and has been shown to be very effective in reaching consensus, as reported by BIBREF78  We used a very similar procedure as BIBREF78 to identify the pairs with the most disagreement; for each annotator, we marked the $i$th pair if the rated score $s_i$ falls within: $s_i \ge \mu _i + 1 5$ or $s_i \le \mu _i - 1 5$, where $\mu _i$ is the mean of the other annotators' scores 
Round 3: We compute the average agreement for each annotator (with the other annotators), by measuring the average Spearman's correlation against all other annotators  We discard the scores of annotators that have shown the least average agreement with all other annotators, while we maintain at least ten annotators per language by the end of this round  The actual process is done in multiple iterations: (S1) we measure the average agreement for each annotator with every other annotator (this corresponds to the APIAA measure, see later); (S2) if we still have more than 10 valid annotators and the lowest average score is higher than in the previous iteration, we remove the lowest one, and rerun S1  Table TABREF14 shows the number of annotators at both the start (Round 1) and end (Round 3) of our process for each language 
We measure the agreement between annotators using two metrics, average pairwise inter-annotator agreement (APIAA), and average mean inter-annotator agreement (AMIAA)  Both of these use Spearman's correlation ($\rho $) between annotators scores, the only difference is how they are averaged  They are computed as follows:
where $\rho (s_i,s_j)$ is the Spearman's correlation between annotators $i$ and $j$'s scores ($s_i$,$s_j$) for all pairs in the dataset, and $N$ is the number of annotators  APIAA has been used widely as the standard measure for inter-annotator agreement, including in the original SimLex paper BIBREF14  It simply averages the pairwise Spearman's correlation between all annotators  On the other hand, AMIAA compares the average Spearman's correlation of one held-out annotator with the average of all the other $N-1$ annotators, and then averages across all $N$ `held-out' annotators  It smooths individual annotator effects and arguably serves as a better upper bound than APIAA BIBREF15, BIBREF91, BIBREF78 
We present the respective APIAA and AMIAA scores in Table TABREF16 and Table TABREF17 for all part-of-speech subsets, as well as the agreement for the full datasets  As reported in prior work BIBREF15, BIBREF91, AMIAA scores are typically higher than APIAA scores  Crucially, the results indicate `strong agreement' (across all languages) using both measurements  The languages with the highest annotator agreement were French (fra) and Yue Chinese (yue), while Russian (rus) had the lowest overall IAA scores  These scores, however, are still considered to be `moderately strong agreement' 
Multi-SimLex: Translation and Annotation ::: Data Analysis
Similarity Score Distributions  Across all languages, the average score (mean $=1 61$, median$=1 1$) is on the lower side of the similarity scale  However, looking closer at the scores of each language in Table TABREF19, we indicate notable differences in both the averages and the spread of scores  Notably, French has the highest average of similarity scores (mean$=2 61$, median$=2 5$), while Kiswahili has the lowest average (mean$=1 28$, median$=0 5$)  Russian has the lowest spread ($\sigma =1 37$), while Polish has the largest ($\sigma =1 62$)  All of the languages are strongly correlated with each other, as shown in Figure FIGREF20, where all of the Spearman's correlation coefficients are greater than 0 6 for all language pairs  Languages that share the same language family are highly correlated (e g, cmn-yue, rus-pol, est-fin)  In addition, we observe high correlations between English and most other languages, as expected  This is due to the effect of using English as the base/anchor language to create the dataset  In simple words, if one translates to two languages $L_1$ and $L_2$ starting from the same set of pairs in English, it is higly likely that $L_1$ and $L_2$ will diverge from English in different ways  Therefore, the similarity between $L_1$-eng and $L_2$-eng is expected to be higher than between $L_1$-$L_2$, especially if $L_1$ and $L_2$ are typologically dissimilar languages (e g , heb-cmn, see Figure FIGREF20)  This phenomenon is well documented in related prior work BIBREF25, BIBREF27, BIBREF16, BIBREF97  While we acknowledge this as a slight artifact of the dataset design, it would otherwise be impossible to construct a semantically aligned and comprehensive dataset across a large number of languages 
We also report differences in the distribution of the frequency of words among the languages in Multi-SimLex  Figure FIGREF22 shows six example languages, where each bar segment shows the proportion of words in each language that occur in the given frequency range  For example, the 10K-20K segment of the bars represents the proportion of words in the dataset that occur in the list of most frequent words between the frequency rank of 10,000 and 20,000 in that language; likewise with other intervals  Frequency lists for the presented languages are derived from Wikipedia and Common Crawl corpora  While many concept pairs are direct or approximate translations of English pairs, we can see that the frequency distribution does vary across different languages, and is also related to inherent language properties  For instance, in Finnish and Russian, while we use infinitive forms of all verbs, conjugated verb inflections are often more frequent in raw corpora than the corresponding infinitive forms  The variance can also be partially explained by the difference in monolingual corpora size used to derive the frequency rankings in the first place: absolute vocabulary sizes are expected to fluctuate across different languages  However, it is also important to note that the datasets also contain subsets of lower-frequency and rare words, which can be used for rare word evaluations in multiple languages, in the spirit of Pilehvar:2018emnlp's English rare word dataset 
Cross-Linguistic Differences  Table TABREF23 shows some examples of average similarity scores of English, Spanish, Kiswahili and Welsh concept pairs  Remember that the scores range from 0 to 6: the higher the score, the more similar the participants found the concepts in the pair  The examples from Table TABREF23 show evidence of both the stability of average similarity scores across languages (unlikely – friendly, book – literature, and vanish – disappear), as well as language-specific differences (care – caution)  Some differences in similarity scores seem to group languages into clusters  For example, the word pair regular – average has an average similarity score of 4 0 and 4 1 in English and Spanish, respectively, whereas in Kiswahili and Welsh the average similarity score of this pair is 0 5 and 0 8  We analyze this phenomenon in more detail in §SECREF25 
There are also examples for each of the four languages having a notably higher or lower similarity score for the same concept pair than the three other languages  For example, large – big in English has an average similarity score of 5 9, whereas Spanish, Kiswahili and Welsh speakers rate the closest concept pair in their native language to have a similarity score between 2 7 and 3 8  What is more, woman – wife receives an average similarity of 0 9 in English, 2 9 in Spanish, and greater than 4 0 in Kiswahili and Welsh  The examples from Spanish include banco – asiento (bank – seat) which receives an average similarity score 5 1, while in the other three languages the similarity score for this word pair does not exceed 0 1  At the same time, the average similarity score of espantosamente – fantásticamente (amazingly – fantastically) is much lower in Spanish (0 4) than in other languages (4 1 – 5 1)  In Kiswahili, an example of a word pair with a higher similarity score than the rest would be machweo – jioni (sunset – evening), having an average score of 5 5, while the other languages receive 2 8 or less, and a notably lower similarity score is given to wa ajabu - mkubwa sana (wonderful – terrific), getting 0 9, while the other languages receive 5 3 or more  Welsh examples include yn llwyr - yn gyfan gwbl (purely – completely), which scores 5 4 among Welsh speakers but 2 3 or less in other languages, while addo – tyngu (promise – swear) is rated as 0 by all Welsh annotators, but in the other three languages 4 3 or more on average 
There can be several explanations for the differences in similarity scores across languages, including but not limited to cultural context, polysemy, metonymy, translation, regional and generational differences, and most commonly, the fact that words and meanings do not exactly map onto each other across languages  For example, it is likely that the other three languages do not have two separate words for describing the concepts in the concept pair: big – large, and the translators had to opt for similar lexical items that were more distant in meaning, explaining why in English the concept pair received a much higher average similarity score than in other languages  A similar issue related to the mapping problem across languages arose in the Welsh concept pair yn llwye – yn gyfan gwbl, where Welsh speakers agreed that the two concepts are very similar  When asked, bilingual speakers considered the two Welsh concepts more similar than English equivalents purely – completely, potentially explaining why a higher average similarity score was reached in Welsh  The example of woman – wife can illustrate cultural differences or another translation-related issue where the word `wife' did not exist in some languages (for example, Estonian), and therefore had to be described using other words, affecting the comparability of the similarity scores  This was also the case with the football – soccer concept pair  The pair bank – seat demonstrates the effect of the polysemy mismatch across languages: while `bank' has two different meanings in English, neither of them is similar to the word `seat', but in Spanish, `banco' can mean `bank', but it can also mean `bench'  Quite naturally, Spanish speakers gave the pair banco – asiento a higher similarity score than the speakers of languages where this polysemy did not occur 
An example of metonymy affecting the average similarity score can be seen in the Kiswahili version of the word pair: sunset – evening (machweo – jioni)  The average similarity score for this pair is much higher in Kiswahili, likely because the word `sunset' can act as a metonym of `evening'  The low similarity score of wonderful – terrific in Kiswahili (wa ajabu - mkubwa sana) can be explained by the fact that while `mkubwa sana' can be used as `terrific' in Kiswahili, it technically means `very big', adding to the examples of translation- and mapping-related effects  The word pair amazingly – fantastically (espantosamente – fantásticamente) brings out another translation-related problem: the accuracy of the translation  While `espantosamente' could arguably be translated to `amazingly', more common meanings include: `frightfully', `terrifyingly', and `shockingly', explaining why the average similarity score differs from the rest of the languages  Another problem was brought out by addo – tyngu (promise – swear) in Welsh, where the `tyngu' may not have been a commonly used or even a known word choice for annotators, pointing out potential regional or generational differences in language use 
Table TABREF24 presents examples of concept pairs from English, Spanish, Kiswahili, and Welsh on which the participants agreed the most  For example, in English all participants rated the similarity of trial – test to be 4 or 5  In Spanish and Welsh, all participants rated start – begin to correspond to a score of 5 or 6  In Kiswahili, money – cash received a similarity rating of 6 from every participant  While there are numerous examples of concept pairs in these languages where the participants agreed on a similarity score of 4 or higher, it is worth noting that none of these languages had a single pair where all participants agreed on either 1-2, 2-3, or 3-4 similarity rating  Interestingly, in English all pairs where all the participants agreed on a 5-6 similarity score were adjectives 
Multi-SimLex: Translation and Annotation ::: Effect of Language Affinity on Similarity Scores
Based on the analysis in Figure FIGREF20 and inspecting the anecdotal examples in the previous section, it is evident that the correlation between similarity scores across languages is not random  To corroborate this intuition, we visualize the vectors of similarity scores for each single language by reducing their dimensionality to 2 via Principal Component Analysis BIBREF98  The resulting scatter plot in Figure FIGREF26 reveals that languages from the same family or branch have similar patterns in the scores  In particular, Russian and Polish (both Slavic), Finnish and Estonian (both Uralic), Cantonese and Mandarin Chinese (both Sinitic), and Spanish and French (both Romance) are all neighbors 
In order to quantify exactly the effect of language affinity on the similarity scores, we run correlation analyses between these and language features  In particular, we extract feature vectors from URIEL BIBREF99, a massively multilingual typological database that collects and normalizes information compiled by grammarians and field linguists about the world's languages  In particular, we focus on information about geography (the areas where the language speakers are concentrated), family (the phylogenetic tree each language belongs to), and typology (including syntax, phonological inventory, and phonology)  Moreover, we consider typological representations of languages that are not manually crafted by experts, but rather learned from texts  BIBREF100 proposed to construct such representations by training language-identifying vectors end-to-end as part of neural machine translation models 
The vector for similarity judgments and the vector of linguistic features for a given language have different dimensionality  Hence, we first construct a distance matrix for each vector space, such that both columns and rows are language indices, and each cell value is the cosine distance between the vectors of the corresponding language pair  Given a set of L languages, each resulting matrix $S$ has dimensionality of $\mathbb {R}^{|L| \times |L|}$ and is symmetrical  To estimate the correlation between the matrix for similarity judgments and each of the matrices for linguistic features, we run a Mantel test BIBREF101, a non-parametric statistical test based on matrix permutations that takes into account inter-dependencies among pairwise distances 
The results of the Mantel test reported in Table FIGREF26 show that there exist statistically significant correlations between similarity judgments and geography, family, and syntax, given that $p < 0 05$ and $z > 1 96$  The correlation coefficient is particularly strong for geography ($r = 0 647$) and syntax ($r = 0 649$)  The former result is intuitive, because languages in contact easily borrow and loan lexical units, and cultural interactions may result in similar cognitive categorizations  The result for syntax, instead, cannot be explained so easily, as formal properties of language do not affect lexical semantics  Instead, we conjecture that, while no causal relation is present, both syntactic features and similarity judgments might be linked to a common explanatory variable (such as geography)  In fact, several syntactic properties are not uniformly spread across the globe  For instance, verbs with Verb–Object–Subject word order are mostly concentrated in Oceania BIBREF102  In turn, geographical proximity leads to similar judgment patterns, as mentioned above  On the other hand, we find no correlation with phonology and inventory, as expected, nor with the bottom-up typological features from BIBREF100 
Cross-Lingual Multi-SimLex Datasets
A crucial advantage of having semantically aligned monolingual datasets across different languages is the potential to create cross-lingual semantic similarity datasets  Such datasets allow for probing the quality of cross-lingual representation learning algorithms BIBREF27, BIBREF103, BIBREF104, BIBREF105, BIBREF106, BIBREF30, BIBREF107 as an intrinsic evaluation task  However, the cross-lingual datasets previous work relied upon BIBREF27 were limited to a homogeneous set of high-resource languages (e g , English, German, Italian, Spanish) and a small number of concept pairs (all less than 1K pairs)  We address both problems by 1) using a typologically more diverse language sample, and 2) relying on a substantially larger English dataset as a source for the cross-lingual datasets: 1,888 pairs in this work versus 500 pairs in the work of Camacho:2017semeval  As a result, each of our cross-lingual datasets contains a substantially larger number of concept pairs, as shown in Table TABREF30  The cross-lingual Multi-Simlex datasets are constructed automatically, leveraging word pair translations and annotations collected in all 12 languages  This yields a total of 66 cross-lingual datasets, one for each possible combination of languages  Table TABREF30 provides the final number of concept pairs, which lie between 2,031 and 3,480 pairs for each cross-lingual dataset, whereas Table TABREF29 shows some sample pairs with their corresponding similarity scores 
The automatic creation and verification of cross-lingual datasets closely follows the procedure first outlined by Camacho:2015acl and later adopted by Camacho:2017semeval (for semantic similarity) and Vulic:2019acl (for graded lexical entailment)  First, given two languages, we intersect their aligned concept pairs obtained through translation  For instance, starting from the aligned pairs attroupement – foule in French and rahvasumm – rahvahulk in Estonian, we construct two cross-lingual pairs attroupement – rahvaluk and rahvasumm – foule  The scores of cross-lingual pairs are then computed as averages of the two corresponding monolingual scores  Finally, in order to filter out concept pairs whose semantic meaning was not preserved during this operation, we retain only cross-lingual pairs for which the corresponding monolingual scores $(s_s, s_t)$ differ at most by one fifth of the full scale (i e , $\mid s_s - s_t \mid \le 1 2$)  This heuristic mitigates the noise due to cross-lingual semantic shifts BIBREF27, BIBREF97  We refer the reader to the work of Camacho:2015acl for a detailed technical description of the procedure  UTF8gbsn
To assess the quality of the resulting cross-lingual datasets, we have conducted a verification experiment similar to Vulic:2019acl  We randomly sampled 300 concept pairs in the English-Spanish, English-French, and English-Mandarin cross-lingual datasets  Subsequently, we asked bilingual native speakers to provide similarity judgments of each pair  The Spearman's correlation score $\rho $ between automatically induced and manually collected ratings achieves $\rho \ge 0 90$ on all samples, which confirms the viability of the automatic construction procedure 
Score and Class Distributions  The summary of score and class distributions across all 66 cross-lingual datasets are provided in Figure FIGREF31 and Figure FIGREF31, respectively  First, it is obvious that the distribution over the four POS classes largely adheres to that of the original monolingual Multi-SimLex datasets, and that the variance is quite low: e g , the eng-fra dataset contains the lowest proportion of nouns (49 21%) and the highest proportion of verbs (27 1%), adjectives (15 28%), and adverbs (8 41%)  On the other hand, the distribution over similarity intervals in Figure FIGREF31 shows a much greater variance  This is again expected as this pattern resurfaces in monolingual datasets (see Table TABREF19)  It is also evident that the data are skewed towards lower-similarity concept pairs  However, due to the joint size of all cross-lingual datasets (see Table TABREF30), even the least represented intervals contain a substantial number of concept pairs  For instance, the rus-yue dataset contains the least highly similar concept pairs (in the interval $[4,6]$) of all 66 cross-lingual datasets  Nonetheless, the absolute number of pairs (138) in that interval for rus-yue is still substantial  If needed, this makes it possible to create smaller datasets which are balanced across the similarity spectra through sub-sampling 
Monolingual Evaluation of Representation Learning Models
After the numerical and qualitative analyses of the Multi-SimLex datasets provided in §§ SECREF18–SECREF25, we now benchmark a series of representation learning models on the new evaluation data  We evaluate standard static word embedding algorithms such as fastText BIBREF31, as well as a range of more recent text encoders pretrained on language modeling such as multilingual BERT BIBREF29  These experiments provide strong baseline scores on the new Multi-SimLex datasets and offer a first large-scale analysis of pretrained encoders on word-level semantic similarity across diverse languages  In addition, the experiments now enabled by Multi-SimLex aim to answer several important questions  (Q1) Is it viable to extract high-quality word-level representations from pretrained encoders receiving subword-level tokens as input? Are such representations competitive with standard static word-level embeddings? (Q2) What are the implications of monolingual pretraining versus (massively) multilingual pretraining for performance? (Q3) Do lightweight unsupervised post-processing techniques improve word representations consistently across different languages? (Q4) Can we effectively transfer available external lexical knowledge from resource-rich languages to resource-lean languages in order to learn word representations that distinguish between true similarity and conceptual relatedness (see the discussion in §SECREF6)?
Monolingual Evaluation of Representation Learning Models ::: Models in Comparison
Static Word Embeddings in Different Languages  First, we evaluate a standard method for inducing non-contextualized (i e , static) word embeddings across a plethora of different languages: fastText (ft) vectors BIBREF31 are currently the most popular and robust choice given 1) the availability of pretrained vectors in a large number of languages BIBREF108 trained on large Common Crawl (CC) plus Wikipedia (Wiki) data, and 2) their superior performance across a range of NLP tasks BIBREF109  In fact, fastText is an extension of the standard word-level CBOW and skip-gram word2vec models BIBREF32 that takes into account subword-level information, i e  the contituent character n-grams of each word BIBREF110  For this reason, fastText is also more suited for modeling rare words and morphologically rich languages 
We rely on 300-dimensional ft word vectors trained on CC+Wiki and available online for 157 languages  The word vectors for all languages are obtained by CBOW with position-weights, with character n-grams of length 5, a window of size 5, 10 negative examples, and 10 training epochs  We also probe another (older) collection of ft vectors, pretrained on full Wikipedia dumps of each language   The vectors are 300-dimensional, trained with the skip-gram objective for 5 epochs, with 5 negative examples, a window size set to 5, and relying on all character n-grams from length 3 to 6  Following prior work, we trim the vocabularies for all languages to the 200K most frequent words and compute representations for multi-word expressions by averaging the vectors of their constituent words 
Unsupervised Post-Processing  Further, we consider a variety of unsupervised post-processing steps that can be applied post-training on top of any pretrained input word embedding space without any external lexical semantic resource  So far, the usefulness of such methods has been verified only on the English language through benchmarks for lexical semantics and sentence-level tasks BIBREF113  In this paper, we assess if unsupervised post-processing is beneficial also in other languages  To this end, we apply the following post-hoc transformations on the initial word embeddings:
1) Mean centering (mc) is applied after unit length normalization to ensure that all vectors have a zero mean, and is commonly applied in data mining and analysis BIBREF114, BIBREF115 
2) All-but-the-top (abtt) BIBREF113, BIBREF116 eliminates the common mean vector and a few top dominating directions (according to PCA) from the input distributional word vectors, since they do not contribute towards distinguishing the actual semantic meaning of different words  The method contains a single (tunable) hyper-parameter $dd_{A}$ which denotes the number of the dominating directions to remove from the initial representations  Previous work has verified the usefulness of abtt in several English lexical semantic tasks such as semantic similarity, word analogies, and concept categorization, as well as in sentence-level text classification tasks BIBREF113 
3) uncovec BIBREF117 adjusts the similarity order of an arbitrary input word embedding space, and can emphasize either syntactic or semantic information in the transformed vectors  In short, it transforms the input space $\mathbf {X}$ into an adjusted space $\mathbf {X}\mathbf {W}_{\alpha }$ through a linear map $\mathbf {W}_{\alpha }$ controlled by a single hyper-parameter $\alpha $  The $n^{\text{th}}$-order similarity transformation of the input word vector space $\mathbf {X}$ (for which $n=1$) can be obtained as $\mathbf {M}_{n}(\mathbf {X}) = \mathbf {M}_1(\mathbf {X}\mathbf {W}_{(n-1)/2})$, with $\mathbf {W}_{\alpha }=\mathbf {Q}\mathbf {\Gamma }^{\alpha }$, where $\mathbf {Q}$ and $\mathbf {\Gamma }$ are the matrices obtained via eigendecomposition of $\mathbf {X}^T\mathbf {X}=\mathbf {Q}\mathbf {\Gamma }\mathbf {Q}^T$  $\mathbf {\Gamma }$ is a diagonal matrix containing eigenvalues of $\mathbf {X}^T\mathbf {X}$; $\mathbf {Q}$ is an orthogonal matrix with eigenvectors of $\mathbf {X}^T\mathbf {X}$ as columns  While the motivation for the uncovec methods does originate from adjusting discrete similarity orders, note that $\alpha $ is in fact a continuous real-valued hyper-parameter which can be carefully tuned  For more technical details we refer the reader to the original work of BIBREF117 
As mentioned, all post-processing methods can be seen as unsupervised retrofitting methods that, given an arbitrary input vector space $\mathbf {X}$, produce a perturbed/transformed output vector space $\mathbf {X}^{\prime }$, but unlike common retrofitting methods BIBREF62, BIBREF16, the perturbation is completely unsupervised (i e , self-contained) and does not inject any external (semantic similarity oriented) knowledge into the vector space  Note that different perturbations can also be stacked: e g , we can apply uncovec and then use abtt on top the output uncovec vectors  When using uncovec and abtt we always length-normalize and mean-center the data first (i e , we apply the simple mc normalization)  Finally, we tune the two hyper-parameters $d_A$ (for abtt) and $\alpha $ (uncovec) on the English Multi-SimLex and use the same values on the datasets of all other languages; we report results with $dd_A = 3$ or $dd_A = 10$, and $\alpha =-0 3$ 
Contextualized Word Embeddings  We also evaluate the capacity of unsupervised pretraining architectures based on language modeling objectives to reason over lexical semantic similarity  To the best of our knowledge, our article is the first study performing such analyses  State-of-the-art models such as bert BIBREF29, xlm BIBREF30, or roberta BIBREF118 are typically very deep neural networks based on the Transformer architecture BIBREF119  They receive subword-level tokens as inputs (such as WordPieces BIBREF120) to tackle data sparsity  In output, they return contextualized embeddings, dynamic representations for words in context 
To represent words or multi-word expressions through a pretrained model, we follow prior work BIBREF121 and compute an input item's representation by 1) feeding it to a pretrained model in isolation; then 2) averaging the $H$ last hidden representations for each of the item’s constituent subwords; and then finally 3) averaging the resulting subword representations to produce the final $d$-dimensional representation, where $d$ is the embedding and hidden-layer dimensionality (e g , $d=768$ with bert)  We opt for this approach due to its proven viability and simplicity BIBREF121, as it does not require any additional corpora to condition the induction of contextualized embeddings  Other ways to extract the representations from pretrained models BIBREF122, BIBREF123, BIBREF124 are beyond the scope of this work, and we will experiment with them in the future 
In other words, we treat each pretrained encoder enc as a black-box function to encode a single word or a multi-word expression $x$ in each language into a $d$-dimensional contextualized representation $\mathbf {x}_{\textsc {enc}} \in \mathbb {R}^d = \textsc {enc}(x)$ (e g , $d=768$ with bert)  As multilingual pretrained encoders, we experiment with the multilingual bert model (m-bert) BIBREF29 and xlm BIBREF30  m-bert is pretrained on monolingual Wikipedia corpora of 102 languages (comprising all Multi-SimLex languages) with a 12-layer Transformer network, and yields 768-dimensional representations  Since the concept pairs in Multi-SimLex are lowercased, we use the uncased version of m-bert  m-bert comprises all Multi-SimLex languages, and its evident ability to perform cross-lingual transfer BIBREF12, BIBREF125, BIBREF126 also makes it a convenient baseline model for cross-lingual experiments later in §SECREF8  The second multilingual model we consider, xlm-100, is pretrained on Wikipedia dumps of 100 languages, and encodes each concept into a $1,280$-dimensional representation  In contrast to m-bert, xlm-100 drops the next-sentence prediction objective and adds a cross-lingual masked language modeling objective  For both encoders, the representations of each concept are computed as averages over the last $H=4$ hidden layers in all experiments, as suggested by Wu:2019arxiv 
Besides m-bert and xlm, covering multiple languages, we also analyze the performance of “language-specific” bert and xlm models for the languages where they are available: Finnish, Spanish, English, Mandarin Chinese, and French  The main goal of this comparison is to study the differences in performance between multilingual “one-size-fits-all” encoders and language-specific encoders  For all experiments, we rely on the pretrained models released in the Transformers repository BIBREF127 
Unsupervised post-processing steps devised for static word embeddings (i e , mean-centering, abtt, uncovec) can also be applied on top of contextualized embeddings if we predefine a vocabulary of word types $V$ that will be represented in a word vector space $\mathbf {X}$  We construct such $V$ for each language as the intersection of word types covered by the corresponding CC+Wiki fastText vectors and the (single-word or multi-word) expressions appearing in the corresponding Multi-SimLex dataset 
Finally, note that it is not feasible to evaluate a full range of available pretrained encoders within the scope of this work  Our main intention is to provide the first set of baseline results on Multi-SimLex by benchmarking a sample of most popular encoders, at the same time also investigating other important questions such as performance of static versus contextualized word embeddings, or multilingual versus language-specific pretraining  Another purpose of the experiments is to outline the wide potential and applicability of the Multi-SimLex datasets for multilingual and cross-lingual representation learning evaluation 
Monolingual Evaluation of Representation Learning Models ::: Results and Discussion
The results we report are Spearman's $\rho $ coefficients of the correlation between the ranks derived from the scores of the evaluated models and the human scores provided in each Multi-SimLex dataset  The main results with static and contextualized word vectors for all test languages are summarized in Table TABREF43  The scores reveal several interesting patterns, and also pinpoint the main challenges for future work 
State-of-the-Art Representation Models  The absolute scores of CC+Wiki ft, Wiki ft, and m-bert are not directly comparable, because these models have different coverage  In particular, Multi-SimLex contains some out-of-vocabulary (OOV) words whose static ft embeddings are not available  On the other hand, m-bert has perfect coverage  A general comparison between CC+Wiki and Wiki ft vectors, however, supports the intuition that larger corpora (such as CC+Wiki) yield higher correlations  Another finding is that a single massively multilingual model such as m-bert cannot produce semantically rich word-level representations  Whether this actually happens because the training objective is different—or because the need to represent 100+ languages reduces its language-specific capacity—is investigated further below 
The overall results also clearly indicate that (i) there are differences in performance across different monolingual Multi-SimLex datasets, and (ii) unsupervised post-processing is universally useful, and can lead to huge improvements in correlation scores for many languages  In what follows, we also delve deeper into these analyses 
Impact of Unsupervised Post-Processing  First, the results in Table TABREF43 suggest that applying dimension-wise mean centering to the initial vector spaces has positive impact on word similarity scores in all test languages and for all models, both static and contextualized (see the +mc rows in Table TABREF43)  BIBREF128 show that distributional word vectors have a tendency towards narrow clusters in the vector space (i e , they occupy a narrow cone in the vector space and are therefore anisotropic BIBREF113, BIBREF129), and are prone to the undesired effect of hubness BIBREF130, BIBREF131  Applying dimension-wise mean centering has the effect of spreading the vectors across the hyper-plane and mitigating the hubness issue, which consequently improves word-level similarity, as it emerges from the reported results  Previous work has already validated the importance of mean centering for clustering-based tasks BIBREF132, bilingual lexicon induction with cross-lingual word embeddings BIBREF133, BIBREF134, BIBREF135, and for modeling lexical semantic change BIBREF136  However, to the best of our knowledge, the results summarized in Table TABREF43 are the first evidence that also confirms its importance for semantic similarity in a wide array of languages  In sum, as a general rule of thumb, we suggest to always mean-center representations for semantic tasks 
The results further indicate that additional post-processing methods such as abtt and uncovec on top of mean-centered vector spaces can lead to further gains in most languages  The gains are even visible for languages which start from high correlation scores: for instance , cmn with CC+Wiki ft increases from 0 534 to 0 583, from 0 315 to 0 526 with Wiki ft, and from 0 408 to 0 487 with m-bert  Similarly, for rus with CC+Wiki ft we can improve from 0 422 to 0 500, and for fra the scores improve from 0 578 to 0 613  There are additional similar cases reported in Table TABREF43 
Overall, the unsupervised post-processing techniques seem universally useful across languages, but their efficacy and relative performance does vary across different languages  Note that we have not carefully fine-tuned the hyper-parameters of the evaluated post-processing methods, so additional small improvements can be expected for some languages  The main finding, however, is that these post-processing techniques are robust to semantic similarity computations beyond English, and are truly language independent  For instance, removing dominant latent (PCA-based) components from word vectors emphasizes semantic differences between different concepts, as only shared non-informative latent semantic knowledge is removed from the representations 
In summary, pretrained word embeddings do contain more information pertaining to semantic similarity than revealed in the initial vectors  This way, we have corroborated the hypotheses from prior work BIBREF113, BIBREF117 which were not previously empirically verified on other languages due to a shortage of evaluation data; this gap has now been filled with the introduction of the Multi-SimLex datasets  In all follow-up experiments, we always explicitly denote which post-processing configuration is used in evaluation 
POS-Specific Subsets  We present the results for subsets of word pairs grouped by POS class in Table TABREF46  Prior work based on English data showed that representations for nouns are typically of higher quality than those for the other POS classes BIBREF49, BIBREF137, BIBREF50  We observe a similar trend in other languages as well  This pattern is consistent across different representation models and can be attributed to several reasons  First, verb representations need to express a rich range of syntactic and semantic behaviors rather than purely referential features BIBREF138, BIBREF139, BIBREF73  Second, low correlation scores on the adjective and adverb subsets in some languages (e g , pol, cym, swa) might be due to their low frequency in monolingual texts, which yields unreliable representations  In general, the variance in performance across different word classes warrants further research in class-specific representation learning BIBREF140, BIBREF50  The scores further attest the usefulness of unsupervised post-processing as almost all class-specific correlation scores are improved by applying mean-centering and abtt  Finally, the results for m-bert and xlm-100 in Table TABREF46 further confirm that massively multilingual pretraining cannot yield reasonable semantic representations for many languages: in fact, for some classes they display no correlation with human ratings at all 
Differences across Languages  Naturally, the results from Tables TABREF43 and TABREF46 also reveal that there is variation in performance of both static word embeddings and pretrained encoders across different languages  Among other causes, the lowest absolute scores with ft are reported for languages with least resources available to train monolingual word embeddings, such as Kiswahili, Welsh, and Estonian  The low performance on Welsh is especially indicative: Figure FIGREF20 shows that the ratings in the Welsh dataset match up very well with the English ratings, but we cannot achieve the same level of correlation in Welsh with Welsh ft word embeddings  Difference in performance between two closely related languages, est (low-resource) and fin (high-resource), provides additional evidence in this respect 
The highest reported scores with m-bert and xlm-100 are obtained for Mandarin Chinese and Yue Chinese: this effectively points to the weaknesses of massively multilingual training with a joint subword vocabulary spanning 102 and 100 languages  Due to the difference in scripts, “language-specific” subwords for yue and cmn do not need to be shared across a vast amount of languages and the quality of their representation remains unscathed  This effectively means that m-bert's subword vocabulary contains plenty of cmn-specific and yue-specific subwords which are exploited by the encoder when producing m-bert-based representations  Simultaneously, higher scores with m-bert (and xlm in Table TABREF46) are reported for resource-rich languages such as French, Spanish, and English, which are better represented in m-bert's training data  We also observe lower absolute scores (and a larger number of OOVs) for languages with very rich and productive morphological systems such as the two Slavic languages (Polish and Russian) and Finnish  Since Polish and Russian are known to have large Wikipedias and Common Crawl data BIBREF33 (e g , their Wikipedias are in the top 10 largest Wikipedias worldwide), the problem with coverage can be attributed exactly to the proliferation of morphological forms in those languages 
Finally, while Table TABREF43 does reveal that unsupervised post-processing is useful for all languages, it also demonstrates that peak scores are achieved with different post-processing configurations  This finding suggests that a more careful language-specific fine-tuning is indeed needed to refine word embeddings towards semantic similarity  We plan to inspect the relationship between post-processing techniques and linguistic properties in more depth in future work 
Multilingual vs  Language-Specific Contextualized Embeddings  Recent work has shown that—despite the usefulness of massively multilingual models such as m-bert and xlm-100 for zero-shot cross-lingual transfer BIBREF12, BIBREF125—stronger results in downstream tasks for a particular language can be achieved by pretraining language-specific models on language-specific data  In this experiment, motivated by the low results of m-bert and xlm-100 (see again Table TABREF46), we assess if monolingual pretrained encoders can produce higher-quality word-level representations than multilingual models  Therefore, we evaluate language-specific bert and xlm models for a subset of the Multi-SimLex languages for which such models are currently available: Finnish BIBREF141 (bert-base architecture, uncased), French BIBREF142 (the FlauBERT model based on xlm), English (bert-base, uncased), Mandarin Chinese (bert-base) BIBREF29 and Spanish (bert-base, uncased)  In addition, we also evaluate a series of pretrained encoders available for English: (i) bert-base, bert-large, and bert-large with whole word masking (wwm) from the original work on BERT BIBREF29, (ii) monolingual “English-specific” xlm BIBREF30, and (iii) two models which employ parameter reduction techniques to build more compact encoders: albert-b uses a configuration similar to bert-base, while albert-l is similar to bert-large, but with an $18\times $ reduction in the number of parameters BIBREF143 
From the results in Table FIGREF49, it is clear that monolingual pretrained encoders yield much more reliable word-level representations  The gains are visible even for languages such as cmn which showed reasonable performance with m-bert and are substantial on all test languages  This further confirms the validity of language-specific pretraining in lieu of multilingual training, if sufficient monolingual data are available  Moreover, a comparison of pretrained English encoders in Figure FIGREF49 largely follows the intuition: the larger bert-large model yields slight improvements over bert-base, and we can improve a bit more by relying on word-level (i e , lexical-level) masking Finally, light-weight albert model variants are quite competitive with the original bert models, with only modest drops reported, and albert-l again outperforms albert-b  Overall, it is interesting to note that the scores obtained with monolingual pretrained encoders are on a par with or even outperform static ft word embeddings: this is a very intriguing finding per se as it shows that such subword-level models trained on large corpora can implicitly capture rich lexical semantic knowledge 
Similarity-Specialized Word Embeddings  Conflating distinct lexico-semantic relations is a well-known property of distributional representations BIBREF144, BIBREF53  Semantic specialization fine-tunes distributional spaces to emphasize a particular lexico-semantic relation in the transformed space by injecting external lexical knowledge BIBREF145  Explicitly discerning between true semantic similarity (as captured in Multi-SimLex) and broad conceptual relatedness benefits a number of tasks, as discussed in §SECREF4  Since most languages lack dedicated lexical resources, however, one viable strategy to steer monolingual word vector spaces to emphasize semantic similarity is through cross-lingual transfer of lexical knowledge, usually through a shared cross-lingual word vector space BIBREF106  Therefore, we evaluate the effectiveness of specialization transfer methods using Multi-SimLex as our multilingual test bed 
We evaluate a current state-of-the-art cross-lingual specialization transfer method with minimal requirements, put forth recently by Ponti:2019emnlp  In a nutshell, their li-postspec method is a multi-step procedure that operates as follows  First, the knowledge about semantic similarity is extracted from WordNet in the form of triplets, that is, linguistic constraints $(w_1, w_2, r)$, where $w_1$ and $w_2$ are two concepts, and $r$ is a relation between them obtained from WordNet (e g , synonymy or antonymy)  The goal is to “attract” synonyms closer to each other in the transformed vector space as they reflect true semantic similarity, and “repel” antonyms further apart  In the second step, the linguistic constraints are translated from English to the target language via a shared cross-lingual word vector space  To this end, following Ponti:2019emnlp we rely on cross-lingual word embeddings (CLWEs) BIBREF146 available online, which are based on Wiki ft vectors  Following that, a constraint refinement step is applied in the target language which aims to eliminate the noise inserted during the translation process  This is done by training a relation classification tool: it is trained again on the English linguistic constraints and then used on the translated target language constraints, where the transfer is again enabled via a shared cross-lingual word vector space  Finally, a state-of-the-art monolingual specialization procedure from Ponti:2018emnlp injects the (now target language) linguistic constraints into the target language distributional space 
The scores are summarized in Table TABREF56  Semantic specialization with li-postspec leads to substantial improvements in correlation scores for the majority of the target languages, demonstrating the importance of external semantic similarity knowledge for semantic similarity reasoning  However, we also observe deteriorated performance for the three target languages which can be considered the lowest-resource ones in our set: cym, swa, yue  We hypothesize that this occurs due to the inferior quality of the underlying monolingual Wikipedia word embeddings, which generates a chain of error accumulations  In particular, poor distributional word estimates compromise the alignment of the embedding spaces, which in turn results in increased translation noise, and reduced refinement ability of the relation classifier  On a high level, this “poor get poorer” observation again points to the fact that one of the primary causes of low performance of resource-low languages in semantic tasks is the sheer lack of even unlabeled data for distributional training  On the other hand, as we see from Table TABREF46, typological dissimilarity between the source and the target does not deteriorate the effectiveness of semantic specialization  In fact, li-postspec does yield substantial gains also for the typologically distant targets such as heb, cmn, and est  The critical problem indeed seems to be insufficient raw data for monolingual distributional training 
Cross-Lingual Evaluation
Similar to monolingual evaluation in §SECREF7, we now evaluate several state-of-the-art cross-lingual representation models on the suite of 66 automatically constructed cross-lingual Multi-SimLex datasets  Again, note that evaluating a full range of cross-lingual models available in the rich prior work on cross-lingual representation learning is well beyond the scope of this article  We therefore focus our cross-lingual analyses on several well-established and indicative state-of-the-art cross-lingual models, again spanning both static and contextualized cross-lingual word embeddings 
Cross-Lingual Evaluation ::: Models in Comparison
Static Word Embeddings  We rely on a state-of-the-art mapping-based method for the induction of cross-lingual word embeddings (CLWEs): vecmap BIBREF148  The core idea behind such mapping-based or projection-based approaches is to learn a post-hoc alignment of independently trained monolingual word embeddings BIBREF106  Such methods have gained popularity due to their conceptual simplicity and competitive performance coupled with reduced bilingual supervision requirements: they support CLWE induction with only as much as a few thousand word translation pairs as the bilingual supervision BIBREF149, BIBREF150, BIBREF151, BIBREF107  More recent work has shown that CLWEs can be induced with even weaker supervision from small dictionaries spanning several hundred pairs BIBREF152, BIBREF135, identical strings BIBREF153, or even only shared numerals BIBREF154  In the extreme, fully unsupervised projection-based CLWEs extract such seed bilingual lexicons from scratch on the basis of monolingual data only BIBREF103, BIBREF148, BIBREF155, BIBREF156, BIBREF104, BIBREF157 
Recent empirical studies BIBREF158, BIBREF135, BIBREF159 have compared a variety of unsupervised and weakly supervised mapping-based CLWE methods, and vecmap emerged as the most robust and very competitive choice  Therefore, we focus on 1) its fully unsupervised variant (unsuper) in our comparisons  For several language pairs, we also report scores with two other vecmap model variants: 2) a supervised variant which learns a mapping based on an available seed lexicon (super), and 3) a supervised variant with self-learning (super+sl) which iteratively increases the seed lexicon and improves the mapping gradually  For a detailed description of these variants, we refer the reader to recent work BIBREF148, BIBREF135  We again use CC+Wiki ft vectors as initial monolingual word vectors, except for yue where Wiki ft is used  The seed dictionaries of two different sizes (1k and 5k translation pairs) are based on PanLex BIBREF160, and are taken directly from prior work BIBREF135, or extracted from PanLex following the same procedure as in the prior work 
Contextualized Cross-Lingual Word Embeddings  We again evaluate the capacity of (massively) multilingual pretrained language models, m-bert and xlm-100, to reason over cross-lingual lexical similarity  Implicitly, such an evaluation also evaluates “the intrinsic quality” of shared cross-lingual word-level vector spaces induced by these methods, and their ability to boost cross-lingual transfer between different language pairs  We rely on the same procedure of aggregating the models' subword-level parameters into word-level representations, already described in §SECREF34 
As in monolingual settings, we can apply unsupervised post-processing steps such as abtt to both static and contextualized cross-lingual word embeddings 
Cross-Lingual Evaluation ::: Results and Discussion
Main Results and Differences across Language Pairs  A summary of the results on the 66 cross-lingual Multi-SimLex datasets are provided in Table TABREF59 and Figure FIGREF60  The findings confirm several interesting findings from our previous monolingual experiments (§SECREF44), and also corroborate several hypotheses and findings from prior work, now on a large sample of language pairs and for the task of cross-lingual semantic similarity 
First, we observe that the fully unsupervised vecmap model, despite being the most robust fully unsupervised method at present, fails to produce a meaningful cross-lingual word vector space for a large number of language pairs (see the bottom triangle of Table TABREF59): many correlation scores are in fact no-correlation results, accentuating the problem of fully unsupervised cross-lingual learning for typologically diverse languages and with fewer amounts of monolingual data BIBREF135  The scores are particularly low across the board for lower-resource languages such as Welsh and Kiswahili  It also seems that the lack of monolingual data is a larger problem than typological dissimilarity between language pairs, as we do observe reasonably high correlation scores with vecmap for language pairs such as cmn-spa, heb-est, and rus-fin  However, typological differences (e g , morphological richness) still play an important role as we observe very low scores when pairing cmn with morphologically rich languages such fin, est, pol, and rus  Similar to prior work of Vulic:2019we and doval2019onthe, given the fact that unsupervised vecmap is the most robust unsupervised CLWE method at present BIBREF158, our results again question the usefulness of fully unsupervised approaches for a large number of languages, and call for further developments in the area of unsupervised and weakly supervised cross-lingual representation learning 
The scores of m-bert and xlm-100 lead to similar conclusions as in the monolingual settings  Reasonable correlation scores are achieved only for a small subset of resource-rich language pairs (e g , eng, fra, spa, cmn) which dominate the multilingual m-bert training  Interestingly, the scores indicate a much higher performance of language pairs where yue is one of the languages when we use m-bert instead of vecmap  This boils down again to the fact that yue, due to its specific language script, has a good representation of its words and subwords in the shared m-bert vocabulary  At the same time, a reliable vecmap mapping between yue and other languages cannot be found due to a small monolingual yue corpus  In cases when vecmap does not yield a degenerate cross-lingual vector space starting from two monolingual ones, the final correlation scores seem substantially higher than the ones obtained by the single massively multilingual m-bert model 
Finally, the results in Figure FIGREF60 again verify the usefulness of unsupervised post-processing also in cross-lingual settings  We observe improved performance with both m-bert and xlm-100 when mean centering (+mc) is applied, and further gains can be achieved by using abtt on the mean-centered vector spaces  A similar finding also holds for static cross-lingual word embeddings, where applying abbt (-10) yields higher scores on 61/66 language pairs 
Fully Unsupervised vs  Weakly Supervised Cross-Lingual Embeddings  The results in Table TABREF59 indicate that fully unsupervised cross-lingual learning fails for a large number of language pairs  However, recent work BIBREF135 has noted that these sub-optimal non-alignment solutions with the unsuper model can be avoided by relying on (weak) cross-lingual supervision spanning only several thousands or even hundreds of word translation pairs  Therefore, we examine 1) if we can further improve the results on cross-lingual Multi-SimLex resorting to (at least some) cross-lingual supervision for resource-rich language pairs; and 2) if such available word-level supervision can also be useful for a range of languages which displayed near-zero performance in Table TABREF59  In other words, we test if recent “tricks of the trade” used in the rich literature on CLWE learning reflect in gains on cross-lingual Multi-SimLex datasets 
First, we reassess the findings established on the bilingual lexicon induction task BIBREF161, BIBREF135: using at least some cross-lingual supervision is always beneficial compared to using no supervision at all  We report improvements over the unsuper model for all 10 language pairs in Table TABREF66, even though the unsuper method initially produced strong correlation scores  The importance of self-learning increases with decreasing available seed dictionary size, and the +sl model always outperforms unsuper with 1k seed pairs; we observe the same patterns also with even smaller dictionary sizes than reported in Table TABREF66 (250 and 500 seed pairs)  Along the same line, the results in Table TABREF67 indicate that at least some supervision is crucial for the success of static CLWEs on resource-leaner language pairs  We note substantial improvements on all language pairs; in fact, the vecmap model is able to learn a more reliable mapping starting from clean supervision  We again note large gains with self-learning 
Multilingual vs  Bilingual Contextualized Embeddings  Similar to the monolingual settings, we also inspect if massively multilingual training in fact dilutes the knowledge necessary for cross-lingual reasoning on a particular language pair  Therefore, we compare the 100-language xlm-100 model with i) a variant of the same model trained on a smaller set of 17 languages (xlm-17); ii) a variant of the same model trained specifically for the particular language pair (xlm-2); and iii) a variant of the bilingual xlm-2 model that also leverages bilingual knowledge from parallel data during joint training (xlm-2++)  We again use the pretrained models made available by Conneau:2019nips, and we refer to the original work for further technical details 
The results are summarized in Figure FIGREF60, and they confirm the intuition that massively multilingual pretraining can damage performance even on resource-rich languages and language pairs  We observe a steep rise in performance when the multilingual model is trained on a much smaller set of languages (17 versus 100), and further improvements can be achieved by training a dedicated bilingual model  Finally, leveraging bilingual parallel data seems to offer additional slight gains, but a tiny difference between xlm-2 and xlm-2++ also suggests that this rich bilingual information is not used in the optimal way within the xlm architecture for semantic similarity 
In summary, these results indicate that, in order to improve performance in cross-lingual transfer tasks, more work should be invested into 1) pretraining dedicated language pair-specific models, and 2) creative ways of leveraging available cross-lingual supervision (e g , word translation pairs, parallel or comparable corpora) BIBREF121, BIBREF123, BIBREF124 with pretraining paradigms such as bert and xlm  Using such cross-lingual supervision could lead to similar benefits as indicated by the results obtained with static cross-lingual word embeddings (see Table TABREF66 and Table TABREF67)  We believe that Multi-SimLex can serve as a valuable means to track and guide future progress in this research area 
Conclusion and Future Work
We have presented Multi-SimLex, a resource containing human judgments on the semantic similarity of word pairs for 12 monolingual and 66 cross-lingual datasets  The languages covered are typologically diverse and include also under-resourced ones, such as Welsh and Kiswahili  The resource covers an unprecedented amount of 1,888 word pairs, carefully balanced according to their similarity score, frequency, concreteness, part-of-speech class, and lexical field  In addition to Multi-Simlex, we release the detailed protocol we followed to create this resource  We hope that our consistent guidelines will encourage researchers to translate and annotate Multi-Simlex -style datasets for additional languages  This can help and create a hugely valuable, large-scale semantic resource for multilingual NLP research 
The core Multi-SimLex we release with this paper already enables researchers to carry out novel linguistic analysis as well as establishes a benchmark for evaluating representation learning models  Based on our preliminary analyses, we found that speakers of closely related languages tend to express equivalent similarity judgments  In particular, geographical proximity seems to play a greater role than family membership in determining the similarity of judgments across languages  Moreover, we tested several state-of-the-art word embedding models, both static and contextualized representations, as well as several (supervised and unsupervised) post-processing techniques, on the newly released Multi-SimLex  This enables future endeavors to improve multilingual representation learning with challenging baselines  In addition, our results provide several important insights for research on both monolingual and cross-lingual word representations:
1) Unsupervised post-processing techniques (mean centering, elimination of top principal components, adjusting similarity orders) are always beneficial independently of the language, although the combination leading to the best scores is language-specific and hence needs to be tuned 
2) Similarity rankings obtained from word embeddings for nouns are better aligned with human judgments than all the other part-of-speech classes considered here (verbs, adjectives, and, for the first time, adverbs)  This confirms previous generalizations based on experiments on English 
3) The factor having the greatest impact on the quality of word representations is the availability of raw texts to train them in the first place, rather than language properties (such as family, geographical area, typological features) 
4) Massively multilingual pretrained encoders such as m-bert BIBREF29 and xlm-100 BIBREF30 fare quite poorly on our benchmark, whereas pretrained encoders dedicated to a single language are more competitive with static word embeddings such as fastText BIBREF31  Moreover, for language-specific encoders, parameter reduction techniques reduce performance only marginally 
5) Techniques to inject clean lexical semantic knowledge from external resources into distributional word representations were proven to be effective in emphasizing the relation of semantic similarity  In particular, methods capable of transferring such knowledge from resource-rich to resource-lean languages BIBREF59 increased the correlation with human judgments for most languages, except for those with limited unlabelled data 
Future work can expand our preliminary, yet large-scale study on the ability of pretrained encoders to reason over word-level semantic similarity in different languages  For instance, we have highlighted how sharing the same encoder parameters across multiple languages may harm performance  However, it remains unclear if, and to what extent, the input language embeddings present in xlm-100 but absent in m-bert help mitigate this issue  In addition, pretrained language embeddings can be obtained both from typological databases BIBREF99 and from neural architectures BIBREF100  Plugging these embeddings into the encoders in lieu of embeddings trained end-to-end as suggested by prior work BIBREF162, BIBREF163, BIBREF164 might extend the coverage to more resource-lean languages 
Another important follow-up analysis might involve the comparison of the performance of representation learning models on multilingual datasets for both word-level semantic similarity and sentence-level Natural Language Understanding  In particular, Multi-SimLex fills a gap in available resources for multilingual NLP and might help understand how lexical and compositional semantics interact if put alongside existing resources such as XNLI BIBREF84 for natural language inference or PAWS-X BIBREF89 for cross-lingual paraphrase identification  Finally, the Multi-SimLex annotation could turn out to be a unique source of evidence to study the effects of polysemy in human judgments on semantic similarity: for equivalent word pairs in multiple languages, are the similarity scores affected by how many senses the two words (or multi-word expressions) incorporate?
In light of the success of initiatives like Universal Dependencies for multilingual treebanks, we hope that making Multi-SimLex and its guidelines available will encourage other researchers to expand our current sample of languages  We particularly encourage creation and submission of comparable Multi-SimLex datasets for under-resourced and typologically diverse languages in future work  In particular, we have made a Multi-Simlex community website available to facilitate easy creation, gathering, dissemination, and use of annotated datasets: https://multisimlex com/ 
This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909)  Thierry Poibeau is partly supported by a PRAIRIE 3IA Institute fellowship ("Investissements d'avenir" program, reference ANR-19-P3IA-0001) 
Context 2: Introduction
Accurate grapheme-to-phoneme conversion (g2p) is important for any application that depends on the sometimes inconsistent relationship between spoken and written language  Most prominently, this includes text-to-speech and automatic speech recognition  Most work on g2p has focused on a few languages for which extensive pronunciation data is available BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3   Most languages lack these resources  However, a low resource language's writing system is likely to be similar to the writing systems of languages that do have sufficient pronunciation data  Therefore g2p may be possible for low resource languages if this high resource data can be properly utilized 
We attempt to leverage high resource data by treating g2p as a multisource neural machine translation (NMT) problem  The source sequences for our system are words in the standard orthography in any language  The target sequences are the corresponding representation in the International Phonetic Alphabet (IPA)  Our results show that the parameters learned by the shared encoder–decoder are able to exploit the orthographic and phonemic similarities between the various languages in our data 
Low Resource g2p
Our approach is similar in goal to deri2016grapheme's model for adapting high resource g2p models for low resource languages  They trained weighted finite state transducer (wFST) models on a variety of high resource languages, then transferred those models to low resource languages, using a language distance metric to choose which high resource models to use and a phoneme distance metric to map the high resource language's phonemes to the low resource language's phoneme inventory  These distance metrics are computed based on data from Phoible BIBREF4 and URIEL BIBREF5  
Other low resource g2p systems have used a strategy of combining multiple models  schlippe2014combining trained several data-driven g2p systems on varying quantities of monolingual data and combined their outputs with a phoneme-level voting scheme  This led to improvements over the best-performing single system for small quantities of data in some languages  jyothilow trained recurrent neural networks for small data sets and found that a version of their system that combined the neural network output with the output of the wFST-based Phonetisaurus system BIBREF1 did better than either system alone 
A different approach came from kim2012universal, who used supervised learning with an undirected graphical model to induce the grapheme–phoneme mappings for languages written in the Latin alphabet  Given a short text in a language, the model predicts the language's orthographic rules  To create phonemic context features from the short text, the model naïvely maps graphemes to IPA symbols written with the same character, and uses the features of these symbols to learn an approximation of the phonotactic constraints of the language  In their experiments, these phonotactic features proved to be more valuable than geographical and genetic features drawn from WALS BIBREF6  
Multilingual Neural NLP
In recent years, neural networks have emerged as a common way to use data from several languages in a single system  Google's zero-shot neural machine translation system BIBREF7 shares an encoder and decoder across all language pairs  In order to facilitate this multi-way translation, they prepend an artificial token to the beginning of each source sentence at both training and translation time  The token identifies what language the sentence should be translated to  This approach has three benefits: it is far more efficient than building a separate model for each language pair; it allows for translation between languages that share no parallel data; and it improves results on low-resource languages by allowing them to implicitly share parameters with high-resource languages  Our g2p system is inspired by this approach, although it differs in that there is only one target “language”, IPA, and the artificial tokens identify the language of the source instead of the language of the target 
Other work has also made use of multilingually-trained neural networks  Phoneme-level polyglot language models BIBREF8 train a single model on multiple languages and additionally condition on externally constructed typological data about the language  ostling2017continuous used a similar approach, in which a character-level neural language model is trained on a massively multilingual corpus  A language embedding vector is concatenated to the input at each time step  The language embeddings their system learned correlate closely to the genetic relationships between languages  However, neither of these models was applied to g2p 
Grapheme-to-Phoneme
g2p is the problem of converting the orthographic representation of a word into a phonemic representation  A phoneme is an abstract unit of sound which may have different realizations in different contexts  For example, the English phoneme has two phonetic realizations (or allophones):
English speakers without linguistic training often struggle to perceive any difference between these sounds  Writing systems usually do not distinguish between allophones: and are both written as INLINEFORM0 p INLINEFORM1 in English  The sounds are written differently in languages where they contrast, such as Hindi and Eastern Armenian 
Most writing systems in use today are glottographic, meaning that their symbols encode solely phonological information  But despite being glottographic, in few writing systems do graphemes correspond one-to-one with phonemes  There are cases in which multiple graphemes represent a single phoneme, as in the word the in English:
There are cases in which a single grapheme represents multiple phonemes, such as syllabaries, in which each symbol represents a syllable 
In many languages, there are silent letters, as in the word hora in Spanish:
There are more complicated correspondences, such as the silent e in English that affects the pronunciation of the previous vowel, as seen in the pair of words cape and cap 
It is possible for an orthographic system to have any or all of the above phenomena while remaining unambiguous  However, some orthographic systems contain ambiguities  English is well-known for its spelling ambiguities  Abjads, used for Arabic and Hebrew, do not give full representation to vowels 
Consequently, g2p is harder than simply replacing each grapheme symbol with a corresponding phoneme symbol  It is the problem of replacing a grapheme sequence INLINEFORM0
with a phoneme sequence INLINEFORM0
where the sequences are not necessarily of the same length  Data-driven g2p is therefore the problem of finding the phoneme sequence that maximizes the likelihood of the grapheme sequence: INLINEFORM0
Data-driven approaches are especially useful for problems in which the rules that govern them are complex and difficult to engineer by hand  g2p for languages with ambiguous orthographies is such a problem  Multilingual g2p, in which the various languages have similar but different and possibly contradictory spelling rules, can be seen as an extreme case of that  Therefore, a data-driven sequence-to-sequence model is a natural choice 
Encoder–Decoder Models
In order to find the best phoneme sequence, we use a neural encoder–decoder model with attention BIBREF9   The model consists of two main parts: the encoder compresses each source grapheme sequence INLINEFORM0 into a fixed-length vector  The decoder, conditioned on this fixed-length vector, generates the output phoneme sequence INLINEFORM1  
The encoder and decoder are both implemented as recurrent neural networks, which have the advantage of being able to process sequences of arbitrary length and use long histories efficiently  They are trained jointly to minimize cross-entropy on the training data  We had our best results when using a bidirectional encoder, which consists of two separate encoders which process the input in forward and reverse directions  We used long short-term memory units BIBREF10 for both the encoder and decoder  For the attention mechanism, we used the general global attention architecture described by luong2015effective 
We implemented all models with OpenNMT BIBREF11   Our hyperparameters, which we determined by experimentation, are listed in Table TABREF8  
Training Multilingual Models
Presenting pronunciation data in several languages to the network might create problems because different languages have different pronunciation patterns  For example, the string `real' is pronounced differently in English, German, Spanish, and Portuguese  We solve this problem by prepending each grapheme sequence with an artificial token consisting of the language's ISO 639-3 code enclosed in angle brackets  The English word `real', for example, would be presented to the system as
INLINEFORM0 eng INLINEFORM1 r e a l
The artificial token is treated simply as an element of the grapheme sequence  This is similar to the approach taken by johnson2016google in their zero-shot NMT system  However, their source-side artificial tokens identify the target language, whereas ours identify the source language  An alternative approach, used by ostling2017continuous, would be to concatenate a language embedding to the input at each time step  They do not evaluate their approach on grapheme-to-phoneme conversion 
Data
In order to train a neural g2p system, one needs a large quantity of pronunciation data  A standard dataset for g2p is the Carnegie Mellon Pronouncing Dictionary BIBREF12   However, that is a monolingual English resource, so it is unsuitable for our multilingual task  Instead, we use the multilingual pronunciation corpus collected by deri2016grapheme for all experiments  This corpus consists of spelling–pronunciation pairs extracted from Wiktionary  It is already partitioned into training and test sets  Corpus statistics are presented in Table TABREF10  
In addition to the raw IPA transcriptions extracted from Wiktionary, the corpus provides an automatically cleaned version of transcriptions  Cleaning is a necessary step because web-scraped data is often noisy and may be transcribed at an inconsistent level of detail  The data cleaning used here attempts to make the transcriptions consistent with the phonemic inventories used in Phoible BIBREF4   When a transcription contains a phoneme that is not in its language's inventory in Phoible, that phoneme is replaced by the phoneme with the most similar articulatory features that is in the language's inventory
Context 3: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International Licence  Licence details: http://creativecommons org/licenses/by/4 0/
In the spirit of the brevity of social media's messages and reactions, people have got used to express feelings minimally and symbolically, as with hashtags on Twitter and Instagram  On Facebook, people tend to be more wordy, but posts normally receive more simple “likes” than longer comments  Since February 2016, Facebook users can express specific emotions in response to a post thanks to the newly introduced reaction feature (see Section SECREF2 ), so that now a post can be wordlessly marked with an expression of say “joy" or “surprise" rather than a generic “like” 
It has been observed that this new feature helps Facebook to know much more about their users and exploit this information for targeted advertising BIBREF0 , but interest in people's opinions and how they feel isn't limited to commercial reasons, as it invests social monitoring, too, including health care and education BIBREF1   However, emotions and opinions are not always expressed this explicitly, so that there is high interest in developing systems towards their automatic detection  Creating manually annotated datasets large enough to train supervised models is not only costly, but also—especially in the case of opinions and emotions—difficult, due to the intrinsic subjectivity of the task BIBREF2 , BIBREF3   Therefore, research has focused on unsupervised methods enriched with information derived from lexica, which are manually created BIBREF3 , BIBREF4   Since go2009twitter have shown that happy and sad emoticons can be successfully used as signals for sentiment labels, distant supervision, i e  using some reasonably safe signals as proxies for automatically labelling training data BIBREF5 , has been used also for emotion recognition, for example exploiting both emoticons and Twitter hashtags BIBREF6 , but mainly towards creating emotion lexica  mohammad2015using use hashtags, experimenting also with highly fine-grained emotion sets (up to almost 600 emotion labels), to create the large Hashtag Emotion Lexicon  Emoticons are used as proxies also by hallsmarmulti, who use distributed vector representations to find which words are interchangeable with emoticons but also which emoticons are used in a similar context 
We take advantage of distant supervision by using Facebook reactions as proxies for emotion labels, which to the best of our knowledge hasn't been done yet, and we train a set of Support Vector Machine models for emotion recognition  Our models, differently from existing ones, exploit information which is acquired entirely automatically, and achieve competitive or even state-of-the-art results for some of the emotion labels on existing, standard evaluation datasets  For explanatory purposes, related work is discussed further and more in detail when we describe the benchmarks for evaluation (Section SECREF3 ) and when we compare our models to existing ones (Section SECREF5 )  We also explore and discuss how choosing different sets of Facebook pages as training data provides an intrinsic domain-adaptation method 
Facebook reactions as labels
For years, on Facebook people could leave comments to posts, and also “like” them, by using a thumbs-up feature to explicitly express a generic, rather underspecified, approval  A “like” could thus mean “I like what you said", but also “I like that you bring up such topic (though I find the content of the article you linked annoying)" 
In February 2016, after a short trial, Facebook made a more explicit reaction feature available world-wide  Rather than allowing for the underspecified “like” as the only wordless response to a post, a set of six more specific reactions was introduced, as shown in Figure FIGREF1 : Like, Love, Haha, Wow, Sad and Angry  We use such reactions as proxies for emotion labels associated to posts 
We collected Facebook posts and their corresponding reactions from public pages using the Facebook API, which we accessed via the Facebook-sdk python library  We chose different pages (and therefore domains and stances), aiming at a balanced and varied dataset, but we did so mainly based on intuition (see Section SECREF4 ) and with an eye to the nature of the datasets available for evaluation (see Section SECREF5 )  The choice of which pages to select posts from is far from trivial, and we believe this is actually an interesting aspect of our approach, as by using different Facebook pages one can intrinsically tackle the domain-adaptation problem (See Section SECREF6 for further discussion on this)  The final collection of Facebook pages for the experiments described in this paper is as follows: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney 
Note that thankful was only available during specific time spans related to certain events, as Mother's Day in May 2016 
For each page, we downloaded the latest 1000 posts, or the maximum available if there are fewer, from February 2016, retrieving the counts of reactions for each post  The output is a JSON file containing a list of dictionaries with a timestamp, the post and a reaction vector with frequency values, which indicate how many users used that reaction in response to the post (Figure FIGREF3 )  The resulting emotion vectors must then be turned into an emotion label 
In the context of this experiment, we made the simple decision of associating to each post the emotion with the highest count, ignoring like as it is the default and most generic reaction people tend to use  Therefore, for example, to the first post in Figure FIGREF3 , we would associate the label sad, as it has the highest score (284) among the meaningful emotions we consider, though it also has non-zero scores for other emotions  At this stage, we didn't perform any other entropy-based selection of posts, to be investigated in future work 
Emotion datasets
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset  In order to compare our performance to state-of-the-art results, we have used them as well  In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7   A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation 
Affective Text dataset
Task 14 at SemEval 2007 BIBREF7 was concerned with the classification of emotions and valence in news headlines  The headlines where collected from several news websites including Google news, The New York Times, BBC News and CNN  The used emotion labels were Anger, Disgust, Fear, Joy, Sadness, Surprise, in line with the six basic emotions of Ekman's standard model BIBREF8   Valence was to be determined as positive or negative  Classification of emotion and valence were treated as separate tasks  Emotion labels were not considered as mututally exclusive, and each emotion was assigned a score from 0 to 100  Training/developing data amounted to 250 annotated headlines (Affective development), while systems were evaluated on another 1000 (Affective test)  Evaluation was done using two different methods: a fine-grained evaluation using Pearson's r to measure the correlation between the system scores and the gold standard; and a coarse-grained method where each emotion score was converted to a binary label, and precision, recall, and f-score were computed to assess performance  As it is done in most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , we also treat this as a classification problem (coarse-grained)  This dataset has been extensively used for the evaluation of various unsupervised methods BIBREF2 , but also for testing different supervised learning techniques and feature portability BIBREF10  
Fairy Tales dataset
This is a dataset collected by alm2008affect, where about 1,000 sentences from fairy tales (by B  Potter, H C  Andersen and Grimm) were annotated with the same six emotions of the Affective Text dataset, though with different names: Angry, Disgusted, Fearful, Happy, Sad, and Surprised  In most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , only sentences where all annotators agreed are used, and the labels angry and disgusted are merged  We adopt the same choices 
ISEAR
The ISEAR (International Survey on Emotion Antecedents and Reactions BIBREF11 , BIBREF12 ) is a dataset created in the context of a psychology project of the 1990s, by collecting questionnaires answered by people with different cultural backgrounds  The main aim of this project was to gather insights in cross-cultural aspects of emotional reactions  Student respondents, both psychologists and non-psychologists, were asked to report situations in which they had experienced all of seven major emotions (joy, fear, anger, sadness, disgust, shame and guilt)  In each case, the questions covered the way they had appraised a given situation and how they reacted  The final dataset contains reports by approximately 3000 respondents from all over the world, for a total of 7665 sentences labelled with an emotion, making this the largest dataset out of the three we use 
Overview of datasets and emotions
We summarise datasets and emotion distribution from two viewpoints  First, because there are different sets of emotions labels in the datasets and Facebook data, we need to provide a mapping and derive a subset of emotions that we are going to use for the experiments  This is shown in Table TABREF8 , where in the “Mapped” column we report the final emotions we use in this paper: anger, joy, sadness, surprise  All labels in each dataset are mapped to these final emotions, which are therefore the labels we use for training and testing our models 
Second, the distribution of the emotions for each dataset is different, as can be seen in Figure FIGREF9  
In Figure FIGREF9 we also provide the distribution of the emotions anger, joy, sadness, surprise per Facebook page, in terms of number of posts (recall that we assign to a post the label corresponding to the majority emotion associated to it, see Section SECREF2 )  We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts  We will use this information to find the best set of pages for a given target domain (see Section SECREF5 ) 
Model
There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below  Specifically, we first set on a subset of pages and then experiment with features  Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6   For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 )  All results reported in this section are on this dataset  The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 ) 
Selecting Facebook pages
Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach  First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set 
For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages  Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data  Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class 
Features
In selecting appropriate features, we mainly relied on previous work and intuition  We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data  Results are in Table TABREF20   Future work will further explore the simultaneous selection of features and page combinations 
We use a set of basic text-based features to capture the emotion class  These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation 
This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model 
We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative  For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion  For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature 
As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks  We use three different embeddings:
Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14   This model contains 300-dimensional vectors for 3 million words and phrases 
Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences  Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0 01 and dimensionality of 100  We filtered out words with frequency lower than 2 occurrences 
Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings  In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space  Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token  Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource 
Results on development set
We report precision, recall, and f-score on the development set  The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19  
From Table TABREF20 we draw three main observations  First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0 368), although there is a rather substantial variation of scores per class  Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training  Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most  Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results  Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features 
Results
In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3  
Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4   The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon  This makes our approach completely independent of any manual annotation or handcrafted resource  Our model's performance is compared to the following systems, for which results are reported in the referred literature  Please note that no other existing model was re-implemented, and results are those reported in the respective papers 
Discussion, conclusions and future work
We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification  The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource  An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data 
We believe that this approach has a lot of potential, and we see the following directions for improvement  Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed  Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon 
The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages  For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23   Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label  For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24   In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for  Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue  This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience  Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution  Another criterion to be exploited would be vocabulary overlap between the pages and the datasets 
Lastly, we could develop single models for each emotion, treating the problem as a multi-label task  This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader 
Acknowledgements
In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper 

ground_truth: Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese

Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. what datasets did they use?
Context 2:  Sometimes this cleaning algorithm works well: in the German examples in Table TABREF11 , the raw German symbols and are both converted to   This is useful because the in Ansbach and the in Kaninchen are instances of the same phoneme, so their phonemic representations should use the same symbol  However, the cleaning algorithm can also have negative effects on the data quality  For example, the phoneme is not present in the Phoible inventory for German, but it is used in several German transcriptions in the corpus  The cleaning algorithm converts to in all German transcriptions, whereas would be a more reasonable guess  The cleaning algorithm also removes most suprasegmentals, even though these are often an important part of a language's phonology  Developing a more sophisticated procedure for cleaning pronunciation data is a direction for future work, but in this paper we use the corpus's provided cleaned transcriptions in order to ease comparison to previous results 
Experiments
We present experiments with two versions of our sequence-to-sequence model  LangID prepends each training, validation, and test sample with an artificial token identifying the language of the sample  NoLangID omits this token  LangID and NoLangID have identical structure otherwise  To translate the test corpus, we used a beam width of 100  Although this is an unusually wide beam and had negligible performance effects, it was necessary to compute our error metrics 
Evaluation
We use the following three evaluation metrics:
Phoneme Error Rate (PER) is the Levenshtein distance between the predicted phoneme sequences and the gold standard phoneme sequences, divided by the length of the gold standard phoneme sequences 
Word Error Rate (WER) is the percentage of words in which the predicted phoneme sequence does not exactly match the gold standard phoneme sequence 
Word Error Rate 100 (WER 100) is the percentage of words in the test set for which the correct guess is not in the first 100 guesses of the system 
In system evaluations, WER, WER 100, and PER numbers presented for multiple languages are averaged, weighting each language equally BIBREF13  
It would be interesting to compute error metrics that incorporate phoneme similarity, such as those proposed by hixon2011phonemic  PER weights all phoneme errors the same, even though some errors are more harmful than others: and are usually contrastive, whereas and almost never are  Such statistics would be especially interesting for evaluating a multilingual system, because different languages often map the same grapheme to phonemes that are only subtly different from each other  However, these statistics have not been widely reported for other g2p systems, so we omit them here 
Baseline
Results on LangID and NoLangID are compared to the system presented by deri2016grapheme, which is identified in our results as wFST  Their results can be divided into two parts:
High resource results, computed with wFSTs trained on a combination of Wiktionary pronunciation data and g2p rules extracted from Wikipedia IPA Help pages  They report high resource results for 85 languages 
Adapted results, where they apply various mapping strategies in order to adapt high resource models to other languages  The final adapted results they reported include most of the 85 languages with high resource results, as well as the various languages they were able to adapt them for, for a total of 229 languages  This test set omits 23 of the high resource languages that are written in unique scripts or for which language distance metrics could not be computed 
Training
We train the LangID and NoLangID versions of our model each on three subsets of the Wiktionary data:
LangID-High and NoLangID-High: Trained on data from the 85 languages for which BIBREF13 used non-adapted wFST models 
LangID-Adapted and NoLangID-Adapted: Trained on data from any of the 229 languages for which they built adapted models  Because many of these languages had no training data at all, the model is actually only trained on data in 157 languages  As is noted above, the Adapted set omits 23 languages which are in the High test set 
LangID-All and NoLangID-All: Trained on data in all 311 languages in the Wiktionary training corpus 
In order to ease comparison to Deri and Knight's system, we limited our use of the training corpus to 10,000 words per language  We set aside 10 percent of the data in each language for validation, so the maximum number of training words for any language is 9000 for our systems 
Adapted Results
On the 229 languages for which deri2016grapheme presented their final results, the LangID version of our system outperforms the baseline by a wide margin  The best performance came with the version of our model that was trained on data in all available languages, not just the languages it was tested on  Using a language ID token improves results considerably, but even NoLangID beats the baseline in WER and WER 100  Full results are presented in Table TABREF24  
High Resource Results
Having shown that our model exceeds the performance of the wFST-adaptation approach, we next compare it to the baseline models for just high resource languages  The wFST models here are purely monolingual – they do not use data adaptation because there is sufficient training data for each of them  Full results are presented in Table TABREF26   We omit models trained on the Adapted languages because they were not trained on high resource languages with unique writing systems, such as Georgian and Greek, and consequently performed very poorly on them 
In contrast to the larger-scale Adapted results, in the High Resource experiments none of the sequence-to-sequence approaches equal the performance of the wFST model in WER and PER, although LangID-High does come close  The LangID models do beat wFST in WER 100  A possible explanation is that a monolingual wFST model will never generate phonemes that are not part of the language's inventory  A multilingual model, on the other hand, could potentially generate phonemes from the inventories of any language it has been trained on 
Even if LangID-High does not present a more accurate result, it does present a more compact one: LangID-High is 15 4 MB, while the combined wFST high resource models are 197 5 MB 
Results on Unseen Languages
Finally, we report our models' results on unseen languages in Table TABREF28   The unseen languages are any that are present in the test corpus but absent from the training data  Deri and Knight did not report results specifically on these languages  Although the NoLangID models sometimes do better on WER 100, even here the LangID models have a slight advantage in WER and PER  This is somewhat surprising because the LangID models have not learned embeddings for the language ID tokens of unseen languages  Perhaps negative associations are also being learned, driving the model towards predicting more common pronunciations for unseen languages 
Language ID Tokens
Adding a language ID token always improves results in cases where an embedding has been learned for that token  The power of these embeddings is demonstrated by what happens when one feeds the same input word to the model with different language tokens, as is seen in Table TABREF30   Impressively, this even works when the source sequence is in the wrong script for the language, as is seen in the entry for Arabic 
Language Embeddings
Because these language ID tokens are so useful, it would be good if they could be effectively estimated for unseen languages  ostling2017continuous found that the language vectors their models learned correlated well to genetic relationships, so it would be interesting to see if the embeddings our source encoder learned for the language ID tokens showed anything similar  In a few cases they do (the languages closest to German in the vector space are Luxembourgish, Bavarian, and Yiddish, all close relatives)  However, for the most part the structure of these vectors is not interpretable  Therefore, it would be difficult to estimate the embedding for an unseen language, or to “borrow” the language ID token of a similar language  A more promising way forward is to find a model that uses an externally constructed typological representation of the language 
Phoneme Embeddings
In contrast to the language embeddings, the phoneme embeddings appear to show many regularities (see Table TABREF33 )  This is a sign that our multilingual model learns similar embeddings for phonemes that are written with the same grapheme in different languages  These phonemes tend to be phonetically similar to each other 
Perhaps the structure of the phoneme embedding space is what leads to our models' very good performance on WER 100  Even when the model's first predicted pronunciation is not correct, it tends to assign more probability mass to guesses that are more similar to the correct one  Applying some sort of filtering or reranking of the system output might therefore lead to better performance 
Future Work
Because the language ID token is so beneficial to performance, it would be very interesting to find ways to extend a similar benefit to unseen languages  One possible way to do so is with tokens that identify something other than the language, such as typological features about the language's phonemic inventory  This could enable better sharing of resources among languages  Such typological knowledge is readily available in databases like Phoible and WALS for a wide variety of languages  It would be interesting to explore if any of these features is a good predictor of a language's orthographic rules 
It would also be interesting to apply the artificial token approach to other problems besides multilingual g2p  One closely related application is monolingual English g2p  Some of the ambiguity of English spelling is due to the wide variety of loanwords in the language, many of which have unassimilated spellings  Knowing the origins of these loanwords could provide a useful hint for figuring out their pronunciations  The etymology of a word could be tagged in an analogous way to how language ID is tagged in multilingual g2p 

ground_truth: the Carnegie Mellon Pronouncing Dictionary BIBREF12, the multilingual pronunciation corpus collected by deri2016grapheme , ranscriptions extracted from Wiktionary

multilingual pronunciation corpus collected by deri2016grapheme

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which existing benchmarks did they compare to?
Context 4:  We used an implementation of sentiment analyzer from TextBlob to get a polarity score for each sentence  Table TABREF13 shows the distribution of positive, negative and neutral sentiments across the 3 class labels STRENGTH, WEAKNESS and SUGGESTION  It can be observed that distribution of positive and negative sentiments is almost similar in STRENGTH as well as SUGGESTION sentences, hence we can conclude that the information about sentiments is not much useful for our classification problem 
Discovering Clusters within Sentence Classes
After identifying sentences in each class, we can now answer question (1) in Section SECREF1   From 12742 sentences predicted to have label STRENGTH, we extract nouns that indicate the actual strength, and cluster them using a simple clustering algorithm which uses the cosine similarity between word embeddings of these nouns  We repeat this for the 9160 sentences with predicted label WEAKNESS or SUGGESTION as a single class  Tables TABREF15 and TABREF16 show a few representative clusters in strengths and in weaknesses, respectively  We also explored clustering 12742 STRENGTH sentences directly using CLUTO BIBREF19 and Carrot2 Lingo BIBREF20 clustering algorithms  Carrot2 Lingo discovered 167 clusters and also assigned labels to these clusters  We then generated 167 clusters using CLUTO as well  CLUTO does not generate cluster labels automatically, hence we used 5 most frequent words within the cluster as its labels  Table TABREF19 shows the largest 5 clusters by both the algorithms  It was observed that the clusters created by CLUTO were more meaningful and informative as compared to those by Carrot2 Lingo  Also, it was observed that there is some correspondence between noun clusters and sentence clusters  E g  the nouns cluster motivation expertise knowledge talent skill (Table TABREF15 ) corresponds to the CLUTO sentence cluster skill customer management knowledge team (Table TABREF19 )  But overall, users found the nouns clusters to be more meaningful than the sentence clusters 
PA along Attributes
In many organizations, PA is done from a predefined set of perspectives, which we call attributes  Each attribute covers one specific aspect of the work done by the employees  This has the advantage that we can easily compare the performance of any two employees (or groups of employees) along any given attribute  We can correlate various performance attributes and find dependencies among them  We can also cluster employees in the workforce using their supervisor ratings for each attribute to discover interesting insights into the workforce  The HR managers in the organization considered in this paper have defined 15 attributes (Table TABREF20 )  Each attribute is essentially a work item or work category described at an abstract level  For example, FUNCTIONAL_EXCELLENCE covers any tasks, goals or activities related to the software engineering life-cycle (e g , requirements analysis, design, coding, testing etc ) as well as technologies such as databases, web services and GUI 
In the example in Section SECREF4 , the first sentence (which has class STRENGTH) can be mapped to two attributes: FUNCTIONAL_EXCELLENCE and BUILDING_EFFECTIVE_TEAMS  Similarly, the third sentence (which has class WEAKNESS) can be mapped to the attribute INTERPERSONAL_EFFECTIVENESS and so forth  Thus, in order to answer the second question in Section SECREF1 , we need to map each sentence in each of the 3 classes to zero, one, two or more attributes, which is a multi-class multi-label classification problem 
We manually tagged the same 2000 sentences in Dataset D1 with attributes, where each sentence may get 0, 1, 2, etc  up to 15 class labels (this is dataset D2)  This labelled dataset contained 749, 206, 289, 207, 91, 223, 191, 144, 103, 80, 82, 42, 29, 15, 24 sentences having the class labels listed in Table TABREF20 in the same order  The number of sentences having 0, 1, 2, or more than 2 attributes are: 321, 1070, 470 and 139 respectively  We trained several multi-class multi-label classifiers on this dataset  Table TABREF21 shows the results of 5-fold cross-validation experiments on dataset D2 
Precision, Recall and F-measure for this multi-label classification are computed using a strategy similar to the one described in BIBREF21   Let INLINEFORM0 be the set of predicted labels and INLINEFORM1 be the set of actual labels for the INLINEFORM2 instance  Precision and recall for this instance are computed as follows: INLINEFORM3
It can be observed that INLINEFORM0 would be undefined if INLINEFORM1 is empty and similarly INLINEFORM2 would be undefined when INLINEFORM3 is empty  Hence, overall precision and recall are computed by averaging over all the instances except where they are undefined  Instance-level F-measure can not be computed for instances where either precision or recall are undefined  Therefore, overall F-measure is computed using the overall precision and recall 
Summarization of Peer Feedback using ILP
The PA system includes a set of peer feedback comments for each employee  To answer the third question in Section SECREF1 , we need to create a summary of all the peer feedback comments about a given employee  As an example, following are the feedback comments from 5 peers of an employee 
The individual sentences in the comments written by each peer are first identified and then POS tags are assigned to each sentence  We hypothesize that a good summary of these multiple comments can be constructed by identifying a set of important text fragments or phrases  Initially, a set of candidate phrases is extracted from these comments and a subset of these candidate phrases is chosen as the final summary, using Integer Linear Programming (ILP)  The details of the ILP formulation are shown in Table TABREF36   As an example, following is the summary generated for the above 5 peer comments 
humble nature, effective communication, technical expertise, always supportive, vast knowledge
Following rules are used to identify candidate phrases:
Various parameters are used to evaluate a candidate phrase for its importance  A candidate phrase is more important:
A complete list of parameters is described in detail in Table TABREF36  
There is a trivial constraint INLINEFORM0 which makes sure that only INLINEFORM1 out of INLINEFORM2 candidate phrases are chosen  A suitable value of INLINEFORM3 is used for each employee depending on number of candidate phrases identified across all peers (see Algorithm SECREF6 )  Another set of constraints ( INLINEFORM4 to INLINEFORM5 ) make sure that at least one phrase is selected for each of the leadership attributes  The constraint INLINEFORM6 makes sure that multiple phrases sharing the same headword are not chosen at a time  Also, single word candidate phrases are chosen only if they are adjectives or nouns with lexical category noun attribute  This is imposed by the constraint INLINEFORM7   It is important to note that all the constraints except INLINEFORM8 are soft constraints, i e  there may be feasible solutions which do not satisfy some of these constraints  But each constraint which is not satisfied, results in a penalty through the use of slack variables  These constraints are described in detail in Table TABREF36  
The objective function maximizes the total importance score of the selected candidate phrases  At the same time, it also minimizes the sum of all slack variables so that the minimum number of constraints are broken 
INLINEFORM0 : No  of candidate phrases INLINEFORM1 : No  of phrases to select as part of summary
INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6 INLINEFORM7 INLINEFORM8
INLINEFORM0 and INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 INLINEFORM5 INLINEFORM6
INLINEFORM0 (For determining number of phrases to select to include in summary)
Evaluation of auto-generated summaries
We considered a dataset of 100 employees, where for each employee multiple peer comments were recorded  Also, for each employee, a manual summary was generated by an HR personnel  The summaries generated by our ILP-based approach were compared with the corresponding manual summaries using the ROUGE BIBREF22 unigram score  For comparing performance of our ILP-based summarization algorithm, we explored a few summarization algorithms provided by the Sumy package  A common parameter which is required by all these algorithms is number of sentences keep in the final summary  ILP-based summarization requires a similar parameter K, which is automatically decided based on number of total candidate phrases  Assuming a sentence is equivalent to roughly 3 phrases, for Sumy algorithms, we set number of sentences parameter to the ceiling of K/3  Table TABREF51 shows average and standard deviation of ROUGE unigram f1 scores for each algorithm, over the 100 summaries  The performance of ILP-based summarization is comparable with the other algorithms, as the two sample t-test does not show statistically significant difference  Also, human evaluators preferred phrase-based summary generated by our approach to the other sentence-based summaries 
Conclusions and Further Work
In this paper, we presented an analysis of the text generated in Performance Appraisal (PA) process in a large multi-national IT company  We performed sentence classification to identify strengths, weaknesses and suggestions for improvements found in the supervisor assessments and then used clustering to discover broad categories among them  As this is non-topical classification, we found that SVM with ADWS kernel BIBREF18 produced the best results  We also used multi-class multi-label classification techniques to match supervisor assessments to predefined broad perspectives on performance  Logistic Regression classifier was observed to produce the best results for this topical classification  Finally, we proposed an ILP-based summarization technique to produce a summary of peer feedback comments for a given employee and compared it with manual summaries 
The PA process also generates much structured data, such as supervisor ratings  It is an interesting problem to compare and combine the insights from discovered from structured data and unstructured text  Also, we are planning to automatically discover any additional performance attributes to the list of 15 attributes currently used by HR 

ground_truth: Affective Text, Fairy Tales, ISEAR

 Affective Text dataset, Fairy Tales dataset, ISEAR dataset