prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the 12 languages covered?
Context 1: Introduction
The lack of annotated training and evaluation data for many tasks and domains hinders the development of computational models for the majority of the world's languages BIBREF0, BIBREF1, BIBREF2  The necessity to guide and advance multilingual and cross-lingual NLP through annotation efforts that follow cross-lingually consistent guidelines has been recently recognized by collaborative initiatives such as the Universal Dependency (UD) project BIBREF3  The latest version of UD (as of March 2020) covers more than 70 languages  Crucially, this resource continues to steadily grow and evolve through the contributions of annotators from across the world, extending the UD's reach to a wide array of typologically diverse languages  Besides steering research in multilingual parsing BIBREF4, BIBREF5, BIBREF6 and cross-lingual parser transfer BIBREF7, BIBREF8, BIBREF9, the consistent annotations and guidelines have also enabled a range of insightful comparative studies focused on the languages' syntactic (dis)similarities BIBREF10, BIBREF11, BIBREF12 
Inspired by the UD work and its substantial impact on research in (multilingual) syntax, in this article we introduce Multi-SimLex, a suite of manually and consistently annotated semantic datasets for 12 different languages, focused on the fundamental lexical relation of semantic similarity BIBREF13, BIBREF14  For any pair of words, this relation measures whether their referents share the same (functional) features, as opposed to general cognitive association captured by co-occurrence patterns in texts (i e , the distributional information)  Datasets that quantify the strength of true semantic similarity between concept pairs such as SimLex-999 BIBREF14 or SimVerb-3500 BIBREF15 have been instrumental in improving models for distributional semantics and representation learning  Discerning between semantic similarity and relatedness/association is not only crucial for theoretical studies on lexical semantics (see §SECREF2), but has also been shown to benefit a range of language understanding tasks in NLP  Examples include dialog state tracking BIBREF16, BIBREF17, spoken language understanding BIBREF18, BIBREF19, text simplification BIBREF20, BIBREF21, BIBREF22, dictionary and thesaurus construction BIBREF23, BIBREF24 
Despite the proven usefulness of semantic similarity datasets, they are available only for a small and typologically narrow sample of resource-rich languages such as German, Italian, and Russian BIBREF25, whereas some language types and low-resource languages typically lack similar evaluation data  Even if some resources do exist, they are limited in their size (e g , 500 pairs in Turkish BIBREF26, 500 in Farsi BIBREF27, or 300 in Finnish BIBREF28) and coverage (e g , all datasets which originated from the original English SimLex-999 contain only high-frequent concepts, and are dominated by nouns)  This is why, as our departure point, we introduce a larger and more comprehensive English word similarity dataset spanning 1,888 concept pairs (see §SECREF4) 
Most importantly, semantic similarity datasets in different languages have been created using heterogeneous construction procedures with different guidelines for translation and annotation, as well as different rating scales  For instance, some datasets were obtained by directly translating the English SimLex-999 in its entirety BIBREF25, BIBREF16 or in part BIBREF28  Other datasets were created from scratch BIBREF26 and yet others sampled English concept pairs differently from SimLex-999 and then translated and reannotated them in target languages BIBREF27  This heterogeneity makes these datasets incomparable and precludes systematic cross-linguistic analyses  In this article, consolidating the lessons learned from previous dataset construction paradigms, we propose a carefully designed translation and annotation protocol for developing monolingual Multi-SimLex datasets with aligned concept pairs for typologically diverse languages  We apply this protocol to a set of 12 languages, including a mixture of major languages (e g , Mandarin, Russian, and French) as well as several low-resource ones (e g , Kiswahili, Welsh, and Yue Chinese)  We demonstrate that our proposed dataset creation procedure yields data with high inter-annotator agreement rates (e g , the average mean inter-annotator agreement for Welsh is 0 742) 
The unified construction protocol and alignment between concept pairs enables a series of quantitative analyses  Preliminary studies on the influence that polysemy and cross-lingual variation in lexical categories (see §SECREF6) have on similarity judgments are provided in §SECREF5  Data created according to Multi-SimLex protocol also allow for probing into whether similarity judgments are universal across languages, or rather depend on linguistic affinity (in terms of linguistic features, phylogeny, and geographical location)  We investigate this question in §SECREF25  Naturally, Multi-SimLex datasets can be used as an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms  We conduct a systematic evaluation of several state-of-the-art representation models in §SECREF7, showing that there are large gaps between human and system performance in all languages  The proposed construction paradigm also supports the automatic creation of 66 cross-lingual Multi-SimLex datasets by interleaving the monolingual ones  We outline the construction of the cross-lingual datasets in §SECREF6, and then present a quantitative evaluation of a series of cutting-edge cross-lingual representation models on this benchmark in §SECREF8 
Contributions  We now summarize the main contributions of this work:
1) Building on lessons learned from prior work, we create a more comprehensive lexical semantic similarity dataset for the English language spanning a total of 1,888 concept pairs balanced with respect to similarity, frequency, and concreteness, and covering four word classes: nouns, verbs, adjectives and, for the first time, adverbs  This dataset serves as the main source for the creation of equivalent datasets in several other languages 
2) We present a carefully designed and rigorous language-agnostic translation and annotation protocol  These well-defined guidelines will facilitate the development of future Multi-SimLex datasets for other languages  The proposed protocol eliminates some crucial issues with prior efforts focused on the creation of multi-lingual semantic resources, namely: i) limited coverage; ii) heterogeneous annotation guidelines; and iii) concept pairs which are semantically incomparable across different languages 
3) We offer to the community manually annotated evaluation sets of 1,888 concept pairs across 12 typologically diverse languages, and 66 large cross-lingual evaluation sets  To the best of our knowledge, Multi-SimLex is the most comprehensive evaluation resource to date focused on the relation of semantic similarity 
4) We benchmark a wide array of recent state-of-the-art monolingual and cross-lingual word representation models across our sample of languages  The results can serve as strong baselines that lay the foundation for future improvements 
5) We present a first large-scale evaluation study on the ability of encoders pretrained on language modeling (such as bert BIBREF29 and xlm BIBREF30) to reason over word-level semantic similarity in different languages  To our own surprise, the results show that monolingual pretrained encoders, even when presented with word types out of context, are sometimes competitive with static word embedding models such as fastText BIBREF31 or word2vec BIBREF32  The results also reveal a huge gap in performance between massively multilingual pretrained encoders and language-specific encoders in favor of the latter: our findings support other recent empirical evidence related to the “curse of multilinguality” BIBREF33, BIBREF34 in representation learning 
6) We make all of these resources available on a website which facilitates easy creation, submission and sharing of Multi-Simlex-style datasets for a larger number of languages  We hope that this will yield an even larger repository of semantic resources that inspire future advances in NLP within and across languages 
In light of the success of Universal Dependencies BIBREF3, we hope that our initiative will instigate a collaborative public effort with established and clear-cut guidelines that will result in additional Multi-SimLex datasets in a large number of languages in the near future  Moreover, we hope that it will provide means to advance our understanding of distributional and lexical semantics across a large number of languages  All monolingual and cross-lingual Multi-SimLex datasets–along with detailed translation and annotation guidelines–are available online at: https://multisimlex com/ 
Lexical Semantic Similarity ::: Similarity and Association
The focus of the Multi-SimLex initiative is on the lexical relation of pure semantic similarity  For any pair of words, this relation measures whether their referents share the same features  For instance, graffiti and frescos are similar to the extent that they are both forms of painting and appear on walls  This relation can be contrasted with the cognitive association between two words, which often depends on how much their referents interact in the real world, or are found in the same situations  For instance, a painter is easily associated with frescos, although they lack any physical commonalities  Association is also known in the literature under other names: relatedness BIBREF13, topical similarity BIBREF35, and domain similarity BIBREF36 
Semantic similarity and association overlap to some degree, but do not coincide BIBREF37, BIBREF38  In fact, there exist plenty of pairs that are intuitively associated but not similar  Pairs where the converse is true can also be encountered, although more rarely  An example are synonyms where a word is common and the other infrequent, such as to seize and to commandeer  BIBREF14 revealed that while similarity measures based on the WordNet graph BIBREF39 and human judgments of association in the University of South Florida Free Association Database BIBREF40 do correlate, a number of pairs follow opposite trends  Several studies on human cognition also point in the same direction  For instance, semantic priming can be triggered by similar words without association BIBREF41  On the other hand, a connection with cue words is established more quickly for topically related words rather than for similar words in free association tasks BIBREF42 
A key property of semantic similarity is its gradience: pairs of words can be similar to a different degree  On the other hand, the relation of synonymy is binary: pairs of words are synonyms if they can be substituted in all contexts (or most contexts, in a looser sense), otherwise they are not  While synonyms can be conceived as lying on one extreme of the semantic similarity continuum, it is crucial to note that their definition is stated in purely relational terms, rather than invoking their referential properties BIBREF43, BIBREF44, BIBREF45  This makes behavioral studies on semantic similarity fundamentally different from lexical resources like WordNet BIBREF46, which include paradigmatic relations (such as synonymy) 
Lexical Semantic Similarity ::: Similarity for NLP: Intrinsic Evaluation and Semantic Specialization
The ramifications of the distinction between similarity and association are profound for distributional semantics  This paradigm of lexical semantics is grounded in the distributional hypothesis, formulated by BIBREF47 and BIBREF48  According to this hypothesis, the meaning of a word can be recovered empirically from the contexts in which it occurs within a collection of texts  Since both pairs of topically related words and pairs of purely similar words tend to appear in the same contexts, their associated meaning confounds the two distinct relations BIBREF14, BIBREF49, BIBREF50  As a result, distributional methods obscure a crucial facet of lexical meaning  This limitation also reflects onto word embeddings (WEs), representations of words as low-dimensional vectors that have become indispensable for a wide range of NLP applications BIBREF51, BIBREF52, BIBREF53  In particular, it involves both static WEs learned from co-occurrence patterns BIBREF32, BIBREF54, BIBREF31 and contextualized WEs learned from modeling word sequences BIBREF55, BIBREF29  As a result, in the induced representations, geometrical closeness (measured e g  through cosine distance) conflates genuine similarity with broad relatedness  For instance, the vectors for antonyms such as sober and drunk, by definition dissimilar, might be neighbors in the semantic space under the distributional hypothesis  BIBREF36, BIBREF56, and BIBREF53 demonstrated that different choices of hyper-parameters in WE algorithms (such as context window) emphasize different relations in the resulting representations  Likewise, BIBREF57 and BIBREF54 discovered that WEs learned from texts annotated with syntactic information mirror similarity better than simple local bag-of-words neighborhoods 
The failure of WEs to capture semantic similarity, in turn, affects model performance in several NLP applications where such knowledge is crucial  In particular, Natural Language Understanding tasks such as statistical dialog modeling, text simplification, or semantic text similarity BIBREF58, BIBREF18, BIBREF59, among others, suffer the most  As a consequence, resources providing clean information on semantic similarity are key in mitigating the side effects of the distributional signal  In particular, such databases can be employed for the intrinsic evaluations of specific WE models as a proxy of their reliability for downstream applications BIBREF60, BIBREF61, BIBREF14; intuitively, the more WEs are misaligned with human judgments of similarity, the more their performance on actual tasks is expected to be degraded  Moreover, word representations can be specialized (a k a  retrofitted) by disentangling word relations of similarity and association  In particular, linguistic constraints sourced from external databases (such as synonyms from WordNet) can be injected into WEs BIBREF62, BIBREF63, BIBREF16, BIBREF22, BIBREF64 in order to enforce a particular relation in a distributional semantic space while preserving the original adjacency properties 
Lexical Semantic Similarity ::: Similarity and Language Variation: Semantic Typology
In this work, we tackle the concept of (true) semantic similarity from a multilingual perspective  While the same meaning representations may be shared by all human speakers at a deep cognitive level, there is no one-to-one mapping between the words in the lexicons of different languages  This makes the comparison of similarity judgments across languages difficult, since the meaning overlap of translationally equivalent words is sometimes far less than exact  This results from the fact that the way languages `partition' semantic fields is partially arbitrary BIBREF65, although constrained cross-lingually by common cognitive biases BIBREF66  For instance, consider the field of colors: English distinguishes between green and blue, whereas Murle (South Sudan) has a single word for both BIBREF67 
In general, semantic typology studies the variation in lexical semantics across the world's languages  According to BIBREF68, the ways languages categorize concepts into the lexicon follow three main axes: 1) granularity: what is the number of categories in a specific domain?; 2) boundary location: where do the lines marking different categories lie?; 3) grouping and dissection: what are the membership criteria of a category; which instances are considered to be more prototypical? Different choices with respect to these axes lead to different lexicalization patterns  For instance, distinct senses in a polysemous word in English, such as skin (referring to both the body and fruit), may be assigned separate words in other languages such as Italian pelle and buccia, respectively BIBREF70  We later analyze whether similarity scores obtained from native speakers also loosely follow the patterns described by semantic typology 
Previous Work and Evaluation Data
Word Pair Datasets  Rich expert-created resources such as WordNet BIBREF46, BIBREF71, VerbNet BIBREF72, BIBREF73, or FrameNet BIBREF74 encode a wealth of semantic and syntactic information, but are expensive and time-consuming to create  The scale of this problem gets multiplied by the number of languages in consideration  Therefore, crowd-sourcing with non-expert annotators has been adopted as a quicker alternative to produce smaller and more focused semantic resources and evaluation benchmarks  This alternative practice has had a profound impact on distributional semantics and representation learning BIBREF14  While some prominent English word pair datasets such as WordSim-353 BIBREF75, MEN BIBREF76, or Stanford Rare Words BIBREF77 did not discriminate between similarity and relatedness, the importance of this distinction was established by BIBREF14 through the creation of SimLex-999  This inspired other similar datasets which focused on different lexical properties  For instance, SimVerb-3500 BIBREF15 provided similarity ratings for 3,500 English verbs, whereas CARD-660 BIBREF78 aimed at measuring the semantic similarity of infrequent concepts 
Semantic Similarity Datasets in Other Languages  Motivated by the impact of datasets such as SimLex-999 and SimVerb-3500 on representation learning in English, a line of related work focused on creating similar resources in other languages  The dominant approach is translating and reannotating the entire original English SimLex-999 dataset, as done previously for German, Italian, and Russian BIBREF25, Hebrew and Croatian BIBREF16, and Polish BIBREF79  Venekoski:2017nodalida apply this process only to a subset of 300 concept pairs from the English SimLex-999  On the other hand, BIBREF27 sampled a new set of 500 English concept pairs to ensure wider topical coverage and balance across similarity spectra, and then translated those pairs to German, Italian, Spanish, and Farsi (SEMEVAL-500)  A similar approach was followed by BIBREF26 for Turkish, by BIBREF80 for Mandarin Chinese, and by BIBREF81 for Japanese  BIBREF82 translated the concatenation of SimLex-999, WordSim-353, and the English SEMEVAL-500 into Thai and then reannotated it  Finally, BIBREF83 translated English SimLex-999 and WordSim-353 to 11 resource-rich target languages (German, French, Russian, Italian, Dutch, Chinese, Portuguese, Swedish, Spanish, Arabic, Farsi), but they did not provide details concerning the translation process and the resolution of translation disagreements  More importantly, they also did not reannotate the translated pairs in the target languages  As we discussed in § SECREF6 and reiterate later in §SECREF5, semantic differences among languages can have a profound impact on the annotation scores; particulary, we show in §SECREF25 that these differences even roughly define language clusters based on language affinity 
A core issue with the current datasets concerns a lack of one unified procedure that ensures the comparability of resources in different languages  Further, concept pairs for different languages are sourced from different corpora (e g , direct translation of the English data versus sampling from scratch in the target language)  Moreover, the previous SimLex-based multilingual datasets inherit the main deficiencies of the English original version, such as the focus on nouns and highly frequent concepts  Finally, prior work mostly focused on languages that are widely spoken and do not account for the variety of the world's languages  Our long-term goal is devising a standardized methodology to extend the coverage also to languages that are resource-lean and/or typologically diverse (e g , Welsh, Kiswahili as in this work) 
Multilingual Datasets for Natural Language Understanding  The Multi-SimLex initiative and corresponding datasets are also aligned with the recent efforts on procuring multilingual benchmarks that can help advance computational modeling of natural language understanding across different languages  For instance, pretrained multilingual language models such as multilingual bert BIBREF29 or xlm BIBREF30 are typically probed on XNLI test data BIBREF84 for cross-lingual natural language inference  XNLI was created by translating examples from the English MultiNLI dataset, and projecting its sentence labels BIBREF85  Other recent multilingual datasets target the task of question answering based on reading comprehension: i) MLQA BIBREF86 includes 7 languages ii) XQuAD BIBREF87 10 languages; iii) TyDiQA BIBREF88 9 widely spoken typologically diverse languages  While MLQA and XQuAD result from the translation from an English dataset, TyDiQA was built independently in each language  Another multilingual dataset, PAWS-X BIBREF89, focused on the paraphrase identification task and was created translating the original English PAWS BIBREF90 into 6 languages  We believe that Multi-SimLex can substantially contribute to this endeavor by offering a comprehensive multilingual benchmark for the fundamental lexical level relation of semantic similarity  In future work, Multi-SimLex also offers an opportunity to investigate the correlations between word-level semantic similarity and performance in downstream tasks such as QA and NLI across different languages 
The Base for Multi-SimLex: Extending English SimLex-999
In this section, we discuss the design principles behind the English (eng) Multi-SimLex dataset, which is the basis for all the Multi-SimLex datasets in other languages, as detailed in §SECREF5  We first argue that a new, more balanced, and more comprehensive evaluation resource for lexical semantic similarity in English is necessary  We then describe how the 1,888 word pairs contained in the eng Multi-SimLex were selected in such a way as to represent various linguistic phenomena within a single integrated resource 
Construction Criteria  The following criteria have to be satisfied by any high-quality semantic evaluation resource, as argued by previous studies focused on the creation of such resources BIBREF14, BIBREF15, BIBREF91, BIBREF27:
(C1) Representative and diverse  The resource must cover the full range of diverse concepts occurring in natural language, including different word classes (e g , nouns, verbs, adjectives, adverbs), concrete and abstract concepts, a variety of lexical fields, and different frequency ranges 
(C2) Clearly defined  The resource must provide a clear understanding of which semantic relation exactly is annotated and measured, possibly contrasting it with other relations  For instance, the original SimLex-999 and SimVerb-3500 explicitly focus on true semantic similarity and distinguish it from broader relatedness captured by datasets such as MEN BIBREF76 or WordSim-353 BIBREF75 
(C3) Consistent and reliable  The resource must ensure consistent annotations obtained from non-expert native speakers following simple and precise annotation guidelines 
In choosing the word pairs and constructing eng Multi-SimLex, we adhere to these requirements  Moreover, we follow good practices established by the research on related resources  In particular, since the introduction of the original SimLex-999 dataset BIBREF14, follow-up works have improved its construction protocol across several aspects, including: 1) coverage of more lexical fields, e g , by relying on a diverse set of Wikipedia categories BIBREF27, 2) infrequent/rare words BIBREF78, 3) focus on particular word classes, e g , verbs BIBREF15, 4) annotation quality control BIBREF78  Our goal is to make use of these improvements towards a larger, more representative, and more reliable lexical similarity dataset in English and, consequently, in all other languages 
The Final Output: English Multi-SimLex  In order to ensure that the criterion C1 is satisfied, we consolidate and integrate the data already carefully sampled in prior work into a single, comprehensive, and representative dataset  This way, we can control for diversity, frequency, and other properties while avoiding to perform this time-consuming selection process from scratch  Note that, on the other hand, the word pairs chosen for English are scored from scratch as part of the entire Multi-SimLex annotation process, introduced later in §SECREF5  We now describe the external data sources for the final set of word pairs:
1) Source: SimLex-999  BIBREF14  The English Multi-SimLex has been initially conceived as an extension of the original SimLex-999 dataset  Therefore, we include all 999 word pairs from SimLex, which span 666 noun pairs, 222 verb pairs, and 111 adjective pairs  While SimLex-999 already provides examples representing different POS classes, it does not have a sufficient coverage of different linguistic phenomena: for instance, it contains only very frequent concepts, and it does not provide a representative set of verbs BIBREF15 
2) Source: SemEval-17: Task 2 BIBREF92  We start from the full dataset of 500 concept pairs to extract a total of 334 concept pairs for English Multi-SimLex a) which contain only single-word concepts, b) which are not named entities, c) where POS tags of the two concepts are the same, d) where both concepts occur in the top 250K most frequent word types in the English Wikipedia, and e) do not already occur in SimLex-999  The original concepts were sampled as to span all the 34 domains available as part of BabelDomains BIBREF93, which roughly correspond to the main high-level Wikipedia categories  This ensures topical diversity in our sub-sample 
3) Source: CARD-660 BIBREF78  67 word pairs are taken from this dataset focused on rare word similarity, applying the same selection criteria a) to e) employed for SEMEVAL-500  Words are controlled for frequency based on their occurrence counts from the Google News data and the ukWaC corpus BIBREF94  CARD-660 contains some words that are very rare (logboat), domain-specific (erythroleukemia) and slang (2mrw), which might be difficult to translate and annotate across a wide array of languages  Hence, we opt for retaining only the concept pairs above the threshold of top 250K most frequent Wikipedia concepts, as above 
4) Source: SimVerb-3500 BIBREF15 Since both CARD-660 and SEMEVAL-500 are heavily skewed towards noun pairs, and nouns also dominate the original SimLex-999, we also extract additional verb pairs from the verb-specific similarity dataset SimVerb-3500  We randomly sample 244 verb pairs from SimVerb-3500 that represent all similarity spectra  In particular, we add 61 verb pairs for each of the similarity intervals: $[0,1 5), [1 5,3), [3,4 5), [4 5, 6]$  Since verbs in SimVerb-3500 were originally chosen from VerbNet BIBREF95, BIBREF73, they cover a wide range of verb classes and their related linguistic phenomena 
5) Source: University of South Florida BIBREF96 norms, the largest database of free association for English  In order to improve the representation of different POS classes, we sample additional adjectives and adverbs from the USF norms following the procedure established by BIBREF14, BIBREF15  This yields additional 122 adjective pairs, but only a limited number of adverb pairs (e g , later – never, now – here, once – twice)  Therefore, we also create a set of adverb pairs semi-automatically by sampling adjectives that can be derivationally transformed into adverbs (e g  adding the suffix -ly) from the USF, and assessing the correctness of such derivation in WordNet  The resulting pairs include, for instance, primarily – mainly, softly – firmly, roughly – reliably, etc  We include a total of 123 adverb pairs into the final English Multi-SimLex  Note that this is the first time adverbs are included into any semantic similarity dataset 
Fulfillment of Construction Criteria  The final eng Multi-SimLex dataset spans 1,051 noun pairs, 469 verb pairs, 245 adjective pairs, and 123 adverb pairs  As mentioned above, the criterion C1 has been fulfilled by relying only on word pairs that already underwent meticulous sampling processes in prior work, integrating them into a single resource  As a consequence, Multi-SimLex allows for fine-grained analyses over different POS classes, concreteness levels, similarity spectra, frequency intervals, relation types, morphology, lexical fields, and it also includes some challenging orthographically similar examples (e g , infection – inflection)  We ensure that the criteria C2 and C3 are satisfied by using similar annotation guidelines as Simlex-999, SimVerb-3500, and SEMEVAL-500 that explicitly target semantic similarity  In what follows, we outline the carefully tailored process of translating and annotating Multi-SimLex datasets in all target languages 
Multi-SimLex: Translation and Annotation
We now detail the development of the final Multi-SimLex resource, describing our language selection process, as well as translation and annotation of the resource, including the steps taken to ensure and measure the quality of this resource  We also provide key data statistics and preliminary cross-lingual comparative analyses 
Language Selection  Multi-SimLex comprises eleven languages in addition to English  The main objective for our inclusion criteria has been to balance language prominence (by number of speakers of the language) for maximum impact of the resource, while simultaneously having a diverse suite of languages based on their typological features (such as morphological type and language family)  Table TABREF10 summarizes key information about the languages currently included in Multi-SimLex  We have included a mixture of fusional, agglutinative, isolating, and introflexive languages that come from eight different language families  This includes languages that are very widely used such as Chinese Mandarin and Spanish, and low-resource languages such as Welsh and Kiswahili  We hope to further include additional languages and inspire other researchers to contribute to the effort over the lifetime of this project 
The work on data collection can be divided into two crucial phases: 1) a translation phase where the extended English language dataset with 1,888 pairs (described in §SECREF4) is translated into eleven target languages, and 2) an annotation phase where human raters scored each pair in the translated set as well as the English set  Detailed guidelines for both phases are available online at: https://multisimlex com 
Multi-SimLex: Translation and Annotation ::: Word Pair Translation
Translators for each target language were instructed to find direct or approximate translations for the 1,888 word pairs that satisfy the following rules  (1) All pairs in the translated set must be unique (i e , no duplicate pairs); (2) Translating two words from the same English pair into the same word in the target language is not allowed (e g , it is not allowed to translate car and automobile to the same Spanish word coche)  (3) The translated pairs must preserve the semantic relations between the two words when possible  This means that, when multiple translations are possible, the translation that best conveys the semantic relation between the two words found in the original English pair is selected  (4) If it is not possible to use a single-word translation in the target language, then a multi-word expression (MWE) can be used to convey the nearest possible semantics given the above points (e g , the English word homework is translated into the Polish MWE praca domowa) 
Satisfying the above rules when finding appropriate translations for each pair–while keeping to the spirit of the intended semantic relation in the English version–is not always straightforward  For instance, kinship terminology in Sinitic languages (Mandarin and Yue) uses different terms depending on whether the family member is older or younger, and whether the family member comes from the mother’s side or the father’s side  UTF8gbsn In Mandarin, brother has no direct translation and can be translated as either: 哥哥 (older brother) or 弟弟 (younger brother)  Therefore, in such cases, the translators are asked to choose the best option given the semantic context (relation) expressed by the pair in English, otherwise select one of the translations arbitrarily  This is also used to remove duplicate pairs in the translated set, by differentiating the duplicates using a variant at each instance  Further, many translation instances were resolved using near-synonymous terms in the translation  For example, the words in the pair: wood – timber can only be directly translated in Estonian to puit, and are not distinguishable  Therefore, the translators approximated the translation for timber to the compound noun puitmaterjal (literally: wood material) in order to produce a valid pair in the target language  In some cases, a direct transliteration from English is used  For example, the pair: physician and doctor both translate to the same word in Estonian (arst); the less formal word doktor is used as a translation of doctor to generate a valid pair 
We measure the quality of the translated pairs by using a random sample set of 100 pairs (from the 1,888 pairs) to be translated by an independent translator for each target language  The sample is proportionally stratified according to the part-of-speech categories  The independent translator is given identical instructions to the main translator; we then measure the percentage of matched translated words between the two translations of the sample set  Table TABREF12 summarizes the inter-translator agreement results for all languages and by part-of-speech subsets  Overall across all languages, the agreement is 84 8%, which is similar to prior work BIBREF27, BIBREF97 
Multi-SimLex: Translation and Annotation ::: Guidelines and Word Pair Scoring
Across all languages, 145 human annotators were asked to score all 1,888 pairs (in their given language)  We finally collect at least ten valid annotations for each word pair in each language  All annotators were required to abide by the following instructions:
1  Each annotator must assign an integer score between 0 and 6 (inclusive) indicating how semantically similar the two words in a given pair are  A score of 6 indicates very high similarity (i e , perfect synonymy), while zero indicates no similarity 
2  Each annotator must score the entire set of 1,888 pairs in the dataset  The pairs must not be shared between different annotators 
3  Annotators are able to break the workload over a period of approximately 2-3 weeks, and are able to use external sources (e g  dictionaries, thesauri, WordNet) if required 
4  Annotators are kept anonymous, and are not able to communicate with each other during the annotation process 
The selection criteria for the annotators required that all annotators must be native speakers of the target language  Preference to annotators with university education was given, but not required  Annotators were asked to complete a spreadsheet containing the translated pairs of words, as well as the part-of-speech, and a column to enter the score  The annotators did not have access to the original pairs in English 
To ensure the quality of the collected ratings, we have employed an adjudication protocol similar to the one proposed and validated by Pilehvar:2018emnlp  It consists of the following three rounds:
Round 1: All annotators are asked to follow the instructions outlined above, and to rate all 1,888 pairs with integer scores between 0 and 6 
Round 2: We compare the scores of all annotators and identify the pairs for each annotator that have shown the most disagreement  We ask the annotators to reconsider the assigned scores for those pairs only  The annotators may chose to either change or keep the scores  As in the case with Round 1, the annotators have no access to the scores of the other annotators, and the process is anonymous  This process gives a chance for annotators to correct errors or reconsider their judgments, and has been shown to be very effective in reaching consensus, as reported by BIBREF78  We used a very similar procedure as BIBREF78 to identify the pairs with the most disagreement; for each annotator, we marked the $i$th pair if the rated score $s_i$ falls within: $s_i \ge \mu _i + 1 5$ or $s_i \le \mu _i - 1 5$, where $\mu _i$ is the mean of the other annotators' scores 
Round 3: We compute the average agreement for each annotator (with the other annotators), by measuring the average Spearman's correlation against all other annotators  We discard the scores of annotators that have shown the least average agreement with all other annotators, while we maintain at least ten annotators per language by the end of this round  The actual process is done in multiple iterations: (S1) we measure the average agreement for each annotator with every other annotator (this corresponds to the APIAA measure, see later); (S2) if we still have more than 10 valid annotators and the lowest average score is higher than in the previous iteration, we remove the lowest one, and rerun S1  Table TABREF14 shows the number of annotators at both the start (Round 1) and end (Round 3) of our process for each language 
We measure the agreement between annotators using two metrics, average pairwise inter-annotator agreement (APIAA), and average mean inter-annotator agreement (AMIAA)  Both of these use Spearman's correlation ($\rho $) between annotators scores, the only difference is how they are averaged  They are computed as follows:
where $\rho (s_i,s_j)$ is the Spearman's correlation between annotators $i$ and $j$'s scores ($s_i$,$s_j$) for all pairs in the dataset, and $N$ is the number of annotators  APIAA has been used widely as the standard measure for inter-annotator agreement, including in the original SimLex paper BIBREF14  It simply averages the pairwise Spearman's correlation between all annotators  On the other hand, AMIAA compares the average Spearman's correlation of one held-out annotator with the average of all the other $N-1$ annotators, and then averages across all $N$ `held-out' annotators  It smooths individual annotator effects and arguably serves as a better upper bound than APIAA BIBREF15, BIBREF91, BIBREF78 
We present the respective APIAA and AMIAA scores in Table TABREF16 and Table TABREF17 for all part-of-speech subsets, as well as the agreement for the full datasets  As reported in prior work BIBREF15, BIBREF91, AMIAA scores are typically higher than APIAA scores  Crucially, the results indicate `strong agreement' (across all languages) using both measurements  The languages with the highest annotator agreement were French (fra) and Yue Chinese (yue), while Russian (rus) had the lowest overall IAA scores  These scores, however, are still considered to be `moderately strong agreement' 
Multi-SimLex: Translation and Annotation ::: Data Analysis
Similarity Score Distributions  Across all languages, the average score (mean $=1 61$, median$=1 1$) is on the lower side of the similarity scale  However, looking closer at the scores of each language in Table TABREF19, we indicate notable differences in both the averages and the spread of scores  Notably, French has the highest average of similarity scores (mean$=2 61$, median$=2 5$), while Kiswahili has the lowest average (mean$=1 28$, median$=0 5$)  Russian has the lowest spread ($\sigma =1 37$), while Polish has the largest ($\sigma =1 62$)  All of the languages are strongly correlated with each other, as shown in Figure FIGREF20, where all of the Spearman's correlation coefficients are greater than 0 6 for all language pairs  Languages that share the same language family are highly correlated (e g, cmn-yue, rus-pol, est-fin)  In addition, we observe high correlations between English and most other languages, as expected  This is due to the effect of using English as the base/anchor language to create the dataset  In simple words, if one translates to two languages $L_1$ and $L_2$ starting from the same set of pairs in English, it is higly likely that $L_1$ and $L_2$ will diverge from English in different ways  Therefore, the similarity between $L_1$-eng and $L_2$-eng is expected to be higher than between $L_1$-$L_2$, especially if $L_1$ and $L_2$ are typologically dissimilar languages (e g , heb-cmn, see Figure FIGREF20)  This phenomenon is well documented in related prior work BIBREF25, BIBREF27, BIBREF16, BIBREF97  While we acknowledge this as a slight artifact of the dataset design, it would otherwise be impossible to construct a semantically aligned and comprehensive dataset across a large number of languages 
We also report differences in the distribution of the frequency of words among the languages in Multi-SimLex  Figure FIGREF22 shows six example languages, where each bar segment shows the proportion of words in each language that occur in the given frequency range  For example, the 10K-20K segment of the bars represents the proportion of words in the dataset that occur in the list of most frequent words between the frequency rank of 10,000 and 20,000 in that language; likewise with other intervals  Frequency lists for the presented languages are derived from Wikipedia and Common Crawl corpora  While many concept pairs are direct or approximate translations of English pairs, we can see that the frequency distribution does vary across different languages, and is also related to inherent language properties  For instance, in Finnish and Russian, while we use infinitive forms of all verbs, conjugated verb inflections are often more frequent in raw corpora than the corresponding infinitive forms  The variance can also be partially explained by the difference in monolingual corpora size used to derive the frequency rankings in the first place: absolute vocabulary sizes are expected to fluctuate across different languages  However, it is also important to note that the datasets also contain subsets of lower-frequency and rare words, which can be used for rare word evaluations in multiple languages, in the spirit of Pilehvar:2018emnlp's English rare word dataset 
Cross-Linguistic Differences  Table TABREF23 shows some examples of average similarity scores of English, Spanish, Kiswahili and Welsh concept pairs  Remember that the scores range from 0 to 6: the higher the score, the more similar the participants found the concepts in the pair  The examples from Table TABREF23 show evidence of both the stability of average similarity scores across languages (unlikely – friendly, book – literature, and vanish – disappear), as well as language-specific differences (care – caution)  Some differences in similarity scores seem to group languages into clusters  For example, the word pair regular – average has an average similarity score of 4 0 and 4 1 in English and Spanish, respectively, whereas in Kiswahili and Welsh the average similarity score of this pair is 0 5 and 0 8  We analyze this phenomenon in more detail in §SECREF25 
There are also examples for each of the four languages having a notably higher or lower similarity score for the same concept pair than the three other languages  For example, large – big in English has an average similarity score of 5 9, whereas Spanish, Kiswahili and Welsh speakers rate the closest concept pair in their native language to have a similarity score between 2 7 and 3 8  What is more, woman – wife receives an average similarity of 0 9 in English, 2 9 in Spanish, and greater than 4 0 in Kiswahili and Welsh  The examples from Spanish include banco – asiento (bank – seat) which receives an average similarity score 5 1, while in the other three languages the similarity score for this word pair does not exceed 0 1  At the same time, the average similarity score of espantosamente – fantásticamente (amazingly – fantastically) is much lower in Spanish (0 4) than in other languages (4 1 – 5 1)  In Kiswahili, an example of a word pair with a higher similarity score than the rest would be machweo – jioni (sunset – evening), having an average score of 5 5, while the other languages receive 2 8 or less, and a notably lower similarity score is given to wa ajabu - mkubwa sana (wonderful – terrific), getting 0 9, while the other languages receive 5 3 or more  Welsh examples include yn llwyr - yn gyfan gwbl (purely – completely), which scores 5 4 among Welsh speakers but 2 3 or less in other languages, while addo – tyngu (promise – swear) is rated as 0 by all Welsh annotators, but in the other three languages 4 3 or more on average 
There can be several explanations for the differences in similarity scores across languages, including but not limited to cultural context, polysemy, metonymy, translation, regional and generational differences, and most commonly, the fact that words and meanings do not exactly map onto each other across languages  For example, it is likely that the other three languages do not have two separate words for describing the concepts in the concept pair: big – large, and the translators had to opt for similar lexical items that were more distant in meaning, explaining why in English the concept pair received a much higher average similarity score than in other languages  A similar issue related to the mapping problem across languages arose in the Welsh concept pair yn llwye – yn gyfan gwbl, where Welsh speakers agreed that the two concepts are very similar  When asked, bilingual speakers considered the two Welsh concepts more similar than English equivalents purely – completely, potentially explaining why a higher average similarity score was reached in Welsh  The example of woman – wife can illustrate cultural differences or another translation-related issue where the word `wife' did not exist in some languages (for example, Estonian), and therefore had to be described using other words, affecting the comparability of the similarity scores  This was also the case with the football – soccer concept pair  The pair bank – seat demonstrates the effect of the polysemy mismatch across languages: while `bank' has two different meanings in English, neither of them is similar to the word `seat', but in Spanish, `banco' can mean `bank', but it can also mean `bench'  Quite naturally, Spanish speakers gave the pair banco – asiento a higher similarity score than the speakers of languages where this polysemy did not occur 
An example of metonymy affecting the average similarity score can be seen in the Kiswahili version of the word pair: sunset – evening (machweo – jioni)  The average similarity score for this pair is much higher in Kiswahili, likely because the word `sunset' can act as a metonym of `evening'  The low similarity score of wonderful – terrific in Kiswahili (wa ajabu - mkubwa sana) can be explained by the fact that while `mkubwa sana' can be used as `terrific' in Kiswahili, it technically means `very big', adding to the examples of translation- and mapping-related effects  The word pair amazingly – fantastically (espantosamente – fantásticamente) brings out another translation-related problem: the accuracy of the translation  While `espantosamente' could arguably be translated to `amazingly', more common meanings include: `frightfully', `terrifyingly', and `shockingly', explaining why the average similarity score differs from the rest of the languages  Another problem was brought out by addo – tyngu (promise – swear) in Welsh, where the `tyngu' may not have been a commonly used or even a known word choice for annotators, pointing out potential regional or generational differences in language use 
Table TABREF24 presents examples of concept pairs from English, Spanish, Kiswahili, and Welsh on which the participants agreed the most  For example, in English all participants rated the similarity of trial – test to be 4 or 5  In Spanish and Welsh, all participants rated start – begin to correspond to a score of 5 or 6  In Kiswahili, money – cash received a similarity rating of 6 from every participant  While there are numerous examples of concept pairs in these languages where the participants agreed on a similarity score of 4 or higher, it is worth noting that none of these languages had a single pair where all participants agreed on either 1-2, 2-3, or 3-4 similarity rating  Interestingly, in English all pairs where all the participants agreed on a 5-6 similarity score were adjectives 
Multi-SimLex: Translation and Annotation ::: Effect of Language Affinity on Similarity Scores
Based on the analysis in Figure FIGREF20 and inspecting the anecdotal examples in the previous section, it is evident that the correlation between similarity scores across languages is not random  To corroborate this intuition, we visualize the vectors of similarity scores for each single language by reducing their dimensionality to 2 via Principal Component Analysis BIBREF98  The resulting scatter plot in Figure FIGREF26 reveals that languages from the same family or branch have similar patterns in the scores  In particular, Russian and Polish (both Slavic), Finnish and Estonian (both Uralic), Cantonese and Mandarin Chinese (both Sinitic), and Spanish and French (both Romance) are all neighbors 
In order to quantify exactly the effect of language affinity on the similarity scores, we run correlation analyses between these and language features  In particular, we extract feature vectors from URIEL BIBREF99, a massively multilingual typological database that collects and normalizes information compiled by grammarians and field linguists about the world's languages  In particular, we focus on information about geography (the areas where the language speakers are concentrated), family (the phylogenetic tree each language belongs to), and typology (including syntax, phonological inventory, and phonology)  Moreover, we consider typological representations of languages that are not manually crafted by experts, but rather learned from texts  BIBREF100 proposed to construct such representations by training language-identifying vectors end-to-end as part of neural machine translation models 
The vector for similarity judgments and the vector of linguistic features for a given language have different dimensionality  Hence, we first construct a distance matrix for each vector space, such that both columns and rows are language indices, and each cell value is the cosine distance between the vectors of the corresponding language pair  Given a set of L languages, each resulting matrix $S$ has dimensionality of $\mathbb {R}^{|L| \times |L|}$ and is symmetrical  To estimate the correlation between the matrix for similarity judgments and each of the matrices for linguistic features, we run a Mantel test BIBREF101, a non-parametric statistical test based on matrix permutations that takes into account inter-dependencies among pairwise distances 
The results of the Mantel test reported in Table FIGREF26 show that there exist statistically significant correlations between similarity judgments and geography, family, and syntax, given that $p < 0 05$ and $z > 1 96$  The correlation coefficient is particularly strong for geography ($r = 0 647$) and syntax ($r = 0 649$)  The former result is intuitive, because languages in contact easily borrow and loan lexical units, and cultural interactions may result in similar cognitive categorizations  The result for syntax, instead, cannot be explained so easily, as formal properties of language do not affect lexical semantics  Instead, we conjecture that, while no causal relation is present, both syntactic features and similarity judgments might be linked to a common explanatory variable (such as geography)  In fact, several syntactic properties are not uniformly spread across the globe  For instance, verbs with Verb–Object–Subject word order are mostly concentrated in Oceania BIBREF102  In turn, geographical proximity leads to similar judgment patterns, as mentioned above  On the other hand, we find no correlation with phonology and inventory, as expected, nor with the bottom-up typological features from BIBREF100 
Cross-Lingual Multi-SimLex Datasets
A crucial advantage of having semantically aligned monolingual datasets across different languages is the potential to create cross-lingual semantic similarity datasets  Such datasets allow for probing the quality of cross-lingual representation learning algorithms BIBREF27, BIBREF103, BIBREF104, BIBREF105, BIBREF106, BIBREF30, BIBREF107 as an intrinsic evaluation task  However, the cross-lingual datasets previous work relied upon BIBREF27 were limited to a homogeneous set of high-resource languages (e g , English, German, Italian, Spanish) and a small number of concept pairs (all less than 1K pairs)  We address both problems by 1) using a typologically more diverse language sample, and 2) relying on a substantially larger English dataset as a source for the cross-lingual datasets: 1,888 pairs in this work versus 500 pairs in the work of Camacho:2017semeval  As a result, each of our cross-lingual datasets contains a substantially larger number of concept pairs, as shown in Table TABREF30  The cross-lingual Multi-Simlex datasets are constructed automatically, leveraging word pair translations and annotations collected in all 12 languages  This yields a total of 66 cross-lingual datasets, one for each possible combination of languages  Table TABREF30 provides the final number of concept pairs, which lie between 2,031 and 3,480 pairs for each cross-lingual dataset, whereas Table TABREF29 shows some sample pairs with their corresponding similarity scores 
The automatic creation and verification of cross-lingual datasets closely follows the procedure first outlined by Camacho:2015acl and later adopted by Camacho:2017semeval (for semantic similarity) and Vulic:2019acl (for graded lexical entailment)  First, given two languages, we intersect their aligned concept pairs obtained through translation  For instance, starting from the aligned pairs attroupement – foule in French and rahvasumm – rahvahulk in Estonian, we construct two cross-lingual pairs attroupement – rahvaluk and rahvasumm – foule  The scores of cross-lingual pairs are then computed as averages of the two corresponding monolingual scores  Finally, in order to filter out concept pairs whose semantic meaning was not preserved during this operation, we retain only cross-lingual pairs for which the corresponding monolingual scores $(s_s, s_t)$ differ at most by one fifth of the full scale (i e , $\mid s_s - s_t \mid \le 1 2$)  This heuristic mitigates the noise due to cross-lingual semantic shifts BIBREF27, BIBREF97  We refer the reader to the work of Camacho:2015acl for a detailed technical description of the procedure  UTF8gbsn
To assess the quality of the resulting cross-lingual datasets, we have conducted a verification experiment similar to Vulic:2019acl  We randomly sampled 300 concept pairs in the English-Spanish, English-French, and English-Mandarin cross-lingual datasets  Subsequently, we asked bilingual native speakers to provide similarity judgments of each pair  The Spearman's correlation score $\rho $ between automatically induced and manually collected ratings achieves $\rho \ge 0 90$ on all samples, which confirms the viability of the automatic construction procedure 
Score and Class Distributions  The summary of score and class distributions across all 66 cross-lingual datasets are provided in Figure FIGREF31 and Figure FIGREF31, respectively  First, it is obvious that the distribution over the four POS classes largely adheres to that of the original monolingual Multi-SimLex datasets, and that the variance is quite low: e g , the eng-fra dataset contains the lowest proportion of nouns (49 21%) and the highest proportion of verbs (27 1%), adjectives (15 28%), and adverbs (8 41%)  On the other hand, the distribution over similarity intervals in Figure FIGREF31 shows a much greater variance  This is again expected as this pattern resurfaces in monolingual datasets (see Table TABREF19)  It is also evident that the data are skewed towards lower-similarity concept pairs  However, due to the joint size of all cross-lingual datasets (see Table TABREF30), even the least represented intervals contain a substantial number of concept pairs  For instance, the rus-yue dataset contains the least highly similar concept pairs (in the interval $[4,6]$) of all 66 cross-lingual datasets  Nonetheless, the absolute number of pairs (138) in that interval for rus-yue is still substantial  If needed, this makes it possible to create smaller datasets which are balanced across the similarity spectra through sub-sampling 
Monolingual Evaluation of Representation Learning Models
After the numerical and qualitative analyses of the Multi-SimLex datasets provided in §§ SECREF18–SECREF25, we now benchmark a series of representation learning models on the new evaluation data  We evaluate standard static word embedding algorithms such as fastText BIBREF31, as well as a range of more recent text encoders pretrained on language modeling such as multilingual BERT BIBREF29  These experiments provide strong baseline scores on the new Multi-SimLex datasets and offer a first large-scale analysis of pretrained encoders on word-level semantic similarity across diverse languages  In addition, the experiments now enabled by Multi-SimLex aim to answer several important questions  (Q1) Is it viable to extract high-quality word-level representations from pretrained encoders receiving subword-level tokens as input? Are such representations competitive with standard static word-level embeddings? (Q2) What are the implications of monolingual pretraining versus (massively) multilingual pretraining for performance? (Q3) Do lightweight unsupervised post-processing techniques improve word representations consistently across different languages? (Q4) Can we effectively transfer available external lexical knowledge from resource-rich languages to resource-lean languages in order to learn word representations that distinguish between true similarity and conceptual relatedness (see the discussion in §SECREF6)?
Monolingual Evaluation of Representation Learning Models ::: Models in Comparison
Static Word Embeddings in Different Languages  First, we evaluate a standard method for inducing non-contextualized (i e , static) word embeddings across a plethora of different languages: fastText (ft) vectors BIBREF31 are currently the most popular and robust choice given 1) the availability of pretrained vectors in a large number of languages BIBREF108 trained on large Common Crawl (CC) plus Wikipedia (Wiki) data, and 2) their superior performance across a range of NLP tasks BIBREF109  In fact, fastText is an extension of the standard word-level CBOW and skip-gram word2vec models BIBREF32 that takes into account subword-level information, i e  the contituent character n-grams of each word BIBREF110  For this reason, fastText is also more suited for modeling rare words and morphologically rich languages 
We rely on 300-dimensional ft word vectors trained on CC+Wiki and available online for 157 languages  The word vectors for all languages are obtained by CBOW with position-weights, with character n-grams of length 5, a window of size 5, 10 negative examples, and 10 training epochs  We also probe another (older) collection of ft vectors, pretrained on full Wikipedia dumps of each language   The vectors are 300-dimensional, trained with the skip-gram objective for 5 epochs, with 5 negative examples, a window size set to 5, and relying on all character n-grams from length 3 to 6  Following prior work, we trim the vocabularies for all languages to the 200K most frequent words and compute representations for multi-word expressions by averaging the vectors of their constituent words 
Unsupervised Post-Processing  Further, we consider a variety of unsupervised post-processing steps that can be applied post-training on top of any pretrained input word embedding space without any external lexical semantic resource  So far, the usefulness of such methods has been verified only on the English language through benchmarks for lexical semantics and sentence-level tasks BIBREF113  In this paper, we assess if unsupervised post-processing is beneficial also in other languages  To this end, we apply the following post-hoc transformations on the initial word embeddings:
1) Mean centering (mc) is applied after unit length normalization to ensure that all vectors have a zero mean, and is commonly applied in data mining and analysis BIBREF114, BIBREF115 
2) All-but-the-top (abtt) BIBREF113, BIBREF116 eliminates the common mean vector and a few top dominating directions (according to PCA) from the input distributional word vectors, since they do not contribute towards distinguishing the actual semantic meaning of different words  The method contains a single (tunable) hyper-parameter $dd_{A}$ which denotes the number of the dominating directions to remove from the initial representations  Previous work has verified the usefulness of abtt in several English lexical semantic tasks such as semantic similarity, word analogies, and concept categorization, as well as in sentence-level text classification tasks BIBREF113 
3) uncovec BIBREF117 adjusts the similarity order of an arbitrary input word embedding space, and can emphasize either syntactic or semantic information in the transformed vectors  In short, it transforms the input space $\mathbf {X}$ into an adjusted space $\mathbf {X}\mathbf {W}_{\alpha }$ through a linear map $\mathbf {W}_{\alpha }$ controlled by a single hyper-parameter $\alpha $  The $n^{\text{th}}$-order similarity transformation of the input word vector space $\mathbf {X}$ (for which $n=1$) can be obtained as $\mathbf {M}_{n}(\mathbf {X}) = \mathbf {M}_1(\mathbf {X}\mathbf {W}_{(n-1)/2})$, with $\mathbf {W}_{\alpha }=\mathbf {Q}\mathbf {\Gamma }^{\alpha }$, where $\mathbf {Q}$ and $\mathbf {\Gamma }$ are the matrices obtained via eigendecomposition of $\mathbf {X}^T\mathbf {X}=\mathbf {Q}\mathbf {\Gamma }\mathbf {Q}^T$  $\mathbf {\Gamma }$ is a diagonal matrix containing eigenvalues of $\mathbf {X}^T\mathbf {X}$; $\mathbf {Q}$ is an orthogonal matrix with eigenvectors of $\mathbf {X}^T\mathbf {X}$ as columns  While the motivation for the uncovec methods does originate from adjusting discrete similarity orders, note that $\alpha $ is in fact a continuous real-valued hyper-parameter which can be carefully tuned  For more technical details we refer the reader to the original work of BIBREF117 
As mentioned, all post-processing methods can be seen as unsupervised retrofitting methods that, given an arbitrary input vector space $\mathbf {X}$, produce a perturbed/transformed output vector space $\mathbf {X}^{\prime }$, but unlike common retrofitting methods BIBREF62, BIBREF16, the perturbation is completely unsupervised (i e , self-contained) and does not inject any external (semantic similarity oriented) knowledge into the vector space  Note that different perturbations can also be stacked: e g , we can apply uncovec and then use abtt on top the output uncovec vectors  When using uncovec and abtt we always length-normalize and mean-center the data first (i e , we apply the simple mc normalization)  Finally, we tune the two hyper-parameters $d_A$ (for abtt) and $\alpha $ (uncovec) on the English Multi-SimLex and use the same values on the datasets of all other languages; we report results with $dd_A = 3$ or $dd_A = 10$, and $\alpha =-0 3$ 
Contextualized Word Embeddings  We also evaluate the capacity of unsupervised pretraining architectures based on language modeling objectives to reason over lexical semantic similarity  To the best of our knowledge, our article is the first study performing such analyses  State-of-the-art models such as bert BIBREF29, xlm BIBREF30, or roberta BIBREF118 are typically very deep neural networks based on the Transformer architecture BIBREF119  They receive subword-level tokens as inputs (such as WordPieces BIBREF120) to tackle data sparsity  In output, they return contextualized embeddings, dynamic representations for words in context 
To represent words or multi-word expressions through a pretrained model, we follow prior work BIBREF121 and compute an input item's representation by 1) feeding it to a pretrained model in isolation; then 2) averaging the $H$ last hidden representations for each of the item’s constituent subwords; and then finally 3) averaging the resulting subword representations to produce the final $d$-dimensional representation, where $d$ is the embedding and hidden-layer dimensionality (e g , $d=768$ with bert)  We opt for this approach due to its proven viability and simplicity BIBREF121, as it does not require any additional corpora to condition the induction of contextualized embeddings  Other ways to extract the representations from pretrained models BIBREF122, BIBREF123, BIBREF124 are beyond the scope of this work, and we will experiment with them in the future 
In other words, we treat each pretrained encoder enc as a black-box function to encode a single word or a multi-word expression $x$ in each language into a $d$-dimensional contextualized representation $\mathbf {x}_{\textsc {enc}} \in \mathbb {R}^d = \textsc {enc}(x)$ (e g , $d=768$ with bert)  As multilingual pretrained encoders, we experiment with the multilingual bert model (m-bert) BIBREF29 and xlm BIBREF30  m-bert is pretrained on monolingual Wikipedia corpora of 102 languages (comprising all Multi-SimLex languages) with a 12-layer Transformer network, and yields 768-dimensional representations  Since the concept pairs in Multi-SimLex are lowercased, we use the uncased version of m-bert  m-bert comprises all Multi-SimLex languages, and its evident ability to perform cross-lingual transfer BIBREF12, BIBREF125, BIBREF126 also makes it a convenient baseline model for cross-lingual experiments later in §SECREF8  The second multilingual model we consider, xlm-100, is pretrained on Wikipedia dumps of 100 languages, and encodes each concept into a $1,280$-dimensional representation  In contrast to m-bert, xlm-100 drops the next-sentence prediction objective and adds a cross-lingual masked language modeling objective  For both encoders, the representations of each concept are computed as averages over the last $H=4$ hidden layers in all experiments, as suggested by Wu:2019arxiv 
Besides m-bert and xlm, covering multiple languages, we also analyze the performance of “language-specific” bert and xlm models for the languages where they are available: Finnish, Spanish, English, Mandarin Chinese, and French  The main goal of this comparison is to study the differences in performance between multilingual “one-size-fits-all” encoders and language-specific encoders  For all experiments, we rely on the pretrained models released in the Transformers repository BIBREF127 
Unsupervised post-processing steps devised for static word embeddings (i e , mean-centering, abtt, uncovec) can also be applied on top of contextualized embeddings if we predefine a vocabulary of word types $V$ that will be represented in a word vector space $\mathbf {X}$  We construct such $V$ for each language as the intersection of word types covered by the corresponding CC+Wiki fastText vectors and the (single-word or multi-word) expressions appearing in the corresponding Multi-SimLex dataset 
Finally, note that it is not feasible to evaluate a full range of available pretrained encoders within the scope of this work  Our main intention is to provide the first set of baseline results on Multi-SimLex by benchmarking a sample of most popular encoders, at the same time also investigating other important questions such as performance of static versus contextualized word embeddings, or multilingual versus language-specific pretraining  Another purpose of the experiments is to outline the wide potential and applicability of the Multi-SimLex datasets for multilingual and cross-lingual representation learning evaluation 
Monolingual Evaluation of Representation Learning Models ::: Results and Discussion
The results we report are Spearman's $\rho $ coefficients of the correlation between the ranks derived from the scores of the evaluated models and the human scores provided in each Multi-SimLex dataset  The main results with static and contextualized word vectors for all test languages are summarized in Table TABREF43  The scores reveal several interesting patterns, and also pinpoint the main challenges for future work 
State-of-the-Art Representation Models  The absolute scores of CC+Wiki ft, Wiki ft, and m-bert are not directly comparable, because these models have different coverage  In particular, Multi-SimLex contains some out-of-vocabulary (OOV) words whose static ft embeddings are not available  On the other hand, m-bert has perfect coverage  A general comparison between CC+Wiki and Wiki ft vectors, however, supports the intuition that larger corpora (such as CC+Wiki) yield higher correlations  Another finding is that a single massively multilingual model such as m-bert cannot produce semantically rich word-level representations  Whether this actually happens because the training objective is different—or because the need to represent 100+ languages reduces its language-specific capacity—is investigated further below 
The overall results also clearly indicate that (i) there are differences in performance across different monolingual Multi-SimLex datasets, and (ii) unsupervised post-processing is universally useful, and can lead to huge improvements in correlation scores for many languages  In what follows, we also delve deeper into these analyses 
Impact of Unsupervised Post-Processing  First, the results in Table TABREF43 suggest that applying dimension-wise mean centering to the initial vector spaces has positive impact on word similarity scores in all test languages and for all models, both static and contextualized (see the +mc rows in Table TABREF43)  BIBREF128 show that distributional word vectors have a tendency towards narrow clusters in the vector space (i e , they occupy a narrow cone in the vector space and are therefore anisotropic BIBREF113, BIBREF129), and are prone to the undesired effect of hubness BIBREF130, BIBREF131  Applying dimension-wise mean centering has the effect of spreading the vectors across the hyper-plane and mitigating the hubness issue, which consequently improves word-level similarity, as it emerges from the reported results  Previous work has already validated the importance of mean centering for clustering-based tasks BIBREF132, bilingual lexicon induction with cross-lingual word embeddings BIBREF133, BIBREF134, BIBREF135, and for modeling lexical semantic change BIBREF136  However, to the best of our knowledge, the results summarized in Table TABREF43 are the first evidence that also confirms its importance for semantic similarity in a wide array of languages  In sum, as a general rule of thumb, we suggest to always mean-center representations for semantic tasks 
The results further indicate that additional post-processing methods such as abtt and uncovec on top of mean-centered vector spaces can lead to further gains in most languages  The gains are even visible for languages which start from high correlation scores: for instance , cmn with CC+Wiki ft increases from 0 534 to 0 583, from 0 315 to 0 526 with Wiki ft, and from 0 408 to 0 487 with m-bert  Similarly, for rus with CC+Wiki ft we can improve from 0 422 to 0 500, and for fra the scores improve from 0 578 to 0 613  There are additional similar cases reported in Table TABREF43 
Overall, the unsupervised post-processing techniques seem universally useful across languages, but their efficacy and relative performance does vary across different languages  Note that we have not carefully fine-tuned the hyper-parameters of the evaluated post-processing methods, so additional small improvements can be expected for some languages  The main finding, however, is that these post-processing techniques are robust to semantic similarity computations beyond English, and are truly language independent  For instance, removing dominant latent (PCA-based) components from word vectors emphasizes semantic differences between different concepts, as only shared non-informative latent semantic knowledge is removed from the representations 
In summary, pretrained word embeddings do contain more information pertaining to semantic similarity than revealed in the initial vectors  This way, we have corroborated the hypotheses from prior work BIBREF113, BIBREF117 which were not previously empirically verified on other languages due to a shortage of evaluation data; this gap has now been filled with the introduction of the Multi-SimLex datasets  In all follow-up experiments, we always explicitly denote which post-processing configuration is used in evaluation 
POS-Specific Subsets  We present the results for subsets of word pairs grouped by POS class in Table TABREF46  Prior work based on English data showed that representations for nouns are typically of higher quality than those for the other POS classes BIBREF49, BIBREF137, BIBREF50  We observe a similar trend in other languages as well  This pattern is consistent across different representation models and can be attributed to several reasons  First, verb representations need to express a rich range of syntactic and semantic behaviors rather than purely referential features BIBREF138, BIBREF139, BIBREF73  Second, low correlation scores on the adjective and adverb subsets in some languages (e g , pol, cym, swa) might be due to their low frequency in monolingual texts, which yields unreliable representations  In general, the variance in performance across different word classes warrants further research in class-specific representation learning BIBREF140, BIBREF50  The scores further attest the usefulness of unsupervised post-processing as almost all class-specific correlation scores are improved by applying mean-centering and abtt  Finally, the results for m-bert and xlm-100 in Table TABREF46 further confirm that massively multilingual pretraining cannot yield reasonable semantic representations for many languages: in fact, for some classes they display no correlation with human ratings at all 
Differences across Languages  Naturally, the results from Tables TABREF43 and TABREF46 also reveal that there is variation in performance of both static word embeddings and pretrained encoders across different languages  Among other causes, the lowest absolute scores with ft are reported for languages with least resources available to train monolingual word embeddings, such as Kiswahili, Welsh, and Estonian  The low performance on Welsh is especially indicative: Figure FIGREF20 shows that the ratings in the Welsh dataset match up very well with the English ratings, but we cannot achieve the same level of correlation in Welsh with Welsh ft word embeddings  Difference in performance between two closely related languages, est (low-resource) and fin (high-resource), provides additional evidence in this respect 
The highest reported scores with m-bert and xlm-100 are obtained for Mandarin Chinese and Yue Chinese: this effectively points to the weaknesses of massively multilingual training with a joint subword vocabulary spanning 102 and 100 languages  Due to the difference in scripts, “language-specific” subwords for yue and cmn do not need to be shared across a vast amount of languages and the quality of their representation remains unscathed  This effectively means that m-bert's subword vocabulary contains plenty of cmn-specific and yue-specific subwords which are exploited by the encoder when producing m-bert-based representations  Simultaneously, higher scores with m-bert (and xlm in Table TABREF46) are reported for resource-rich languages such as French, Spanish, and English, which are better represented in m-bert's training data  We also observe lower absolute scores (and a larger number of OOVs) for languages with very rich and productive morphological systems such as the two Slavic languages (Polish and Russian) and Finnish  Since Polish and Russian are known to have large Wikipedias and Common Crawl data BIBREF33 (e g , their Wikipedias are in the top 10 largest Wikipedias worldwide), the problem with coverage can be attributed exactly to the proliferation of morphological forms in those languages 
Finally, while Table TABREF43 does reveal that unsupervised post-processing is useful for all languages, it also demonstrates that peak scores are achieved with different post-processing configurations  This finding suggests that a more careful language-specific fine-tuning is indeed needed to refine word embeddings towards semantic similarity  We plan to inspect the relationship between post-processing techniques and linguistic properties in more depth in future work 
Multilingual vs  Language-Specific Contextualized Embeddings  Recent work has shown that—despite the usefulness of massively multilingual models such as m-bert and xlm-100 for zero-shot cross-lingual transfer BIBREF12, BIBREF125—stronger results in downstream tasks for a particular language can be achieved by pretraining language-specific models on language-specific data  In this experiment, motivated by the low results of m-bert and xlm-100 (see again Table TABREF46), we assess if monolingual pretrained encoders can produce higher-quality word-level representations than multilingual models  Therefore, we evaluate language-specific bert and xlm models for a subset of the Multi-SimLex languages for which such models are currently available: Finnish BIBREF141 (bert-base architecture, uncased), French BIBREF142 (the FlauBERT model based on xlm), English (bert-base, uncased), Mandarin Chinese (bert-base) BIBREF29 and Spanish (bert-base, uncased)  In addition, we also evaluate a series of pretrained encoders available for English: (i) bert-base, bert-large, and bert-large with whole word masking (wwm) from the original work on BERT BIBREF29, (ii) monolingual “English-specific” xlm BIBREF30, and (iii) two models which employ parameter reduction techniques to build more compact encoders: albert-b uses a configuration similar to bert-base, while albert-l is similar to bert-large, but with an $18\times $ reduction in the number of parameters BIBREF143 
From the results in Table FIGREF49, it is clear that monolingual pretrained encoders yield much more reliable word-level representations  The gains are visible even for languages such as cmn which showed reasonable performance with m-bert and are substantial on all test languages  This further confirms the validity of language-specific pretraining in lieu of multilingual training, if sufficient monolingual data are available  Moreover, a comparison of pretrained English encoders in Figure FIGREF49 largely follows the intuition: the larger bert-large model yields slight improvements over bert-base, and we can improve a bit more by relying on word-level (i e , lexical-level) masking Finally, light-weight albert model variants are quite competitive with the original bert models, with only modest drops reported, and albert-l again outperforms albert-b  Overall, it is interesting to note that the scores obtained with monolingual pretrained encoders are on a par with or even outperform static ft word embeddings: this is a very intriguing finding per se as it shows that such subword-level models trained on large corpora can implicitly capture rich lexical semantic knowledge 
Similarity-Specialized Word Embeddings  Conflating distinct lexico-semantic relations is a well-known property of distributional representations BIBREF144, BIBREF53  Semantic specialization fine-tunes distributional spaces to emphasize a particular lexico-semantic relation in the transformed space by injecting external lexical knowledge BIBREF145  Explicitly discerning between true semantic similarity (as captured in Multi-SimLex) and broad conceptual relatedness benefits a number of tasks, as discussed in §SECREF4  Since most languages lack dedicated lexical resources, however, one viable strategy to steer monolingual word vector spaces to emphasize semantic similarity is through cross-lingual transfer of lexical knowledge, usually through a shared cross-lingual word vector space BIBREF106  Therefore, we evaluate the effectiveness of specialization transfer methods using Multi-SimLex as our multilingual test bed 
We evaluate a current state-of-the-art cross-lingual specialization transfer method with minimal requirements, put forth recently by Ponti:2019emnlp  In a nutshell, their li-postspec method is a multi-step procedure that operates as follows  First, the knowledge about semantic similarity is extracted from WordNet in the form of triplets, that is, linguistic constraints $(w_1, w_2, r)$, where $w_1$ and $w_2$ are two concepts, and $r$ is a relation between them obtained from WordNet (e g , synonymy or antonymy)  The goal is to “attract” synonyms closer to each other in the transformed vector space as they reflect true semantic similarity, and “repel” antonyms further apart  In the second step, the linguistic constraints are translated from English to the target language via a shared cross-lingual word vector space  To this end, following Ponti:2019emnlp we rely on cross-lingual word embeddings (CLWEs) BIBREF146 available online, which are based on Wiki ft vectors  Following that, a constraint refinement step is applied in the target language which aims to eliminate the noise inserted during the translation process  This is done by training a relation classification tool: it is trained again on the English linguistic constraints and then used on the translated target language constraints, where the transfer is again enabled via a shared cross-lingual word vector space  Finally, a state-of-the-art monolingual specialization procedure from Ponti:2018emnlp injects the (now target language) linguistic constraints into the target language distributional space 
The scores are summarized in Table TABREF56  Semantic specialization with li-postspec leads to substantial improvements in correlation scores for the majority of the target languages, demonstrating the importance of external semantic similarity knowledge for semantic similarity reasoning  However, we also observe deteriorated performance for the three target languages which can be considered the lowest-resource ones in our set: cym, swa, yue  We hypothesize that this occurs due to the inferior quality of the underlying monolingual Wikipedia word embeddings, which generates a chain of error accumulations  In particular, poor distributional word estimates compromise the alignment of the embedding spaces, which in turn results in increased translation noise, and reduced refinement ability of the relation classifier  On a high level, this “poor get poorer” observation again points to the fact that one of the primary causes of low performance of resource-low languages in semantic tasks is the sheer lack of even unlabeled data for distributional training  On the other hand, as we see from Table TABREF46, typological dissimilarity between the source and the target does not deteriorate the effectiveness of semantic specialization  In fact, li-postspec does yield substantial gains also for the typologically distant targets such as heb, cmn, and est  The critical problem indeed seems to be insufficient raw data for monolingual distributional training 
Cross-Lingual Evaluation
Similar to monolingual evaluation in §SECREF7, we now evaluate several state-of-the-art cross-lingual representation models on the suite of 66 automatically constructed cross-lingual Multi-SimLex datasets  Again, note that evaluating a full range of cross-lingual models available in the rich prior work on cross-lingual representation learning is well beyond the scope of this article  We therefore focus our cross-lingual analyses on several well-established and indicative state-of-the-art cross-lingual models, again spanning both static and contextualized cross-lingual word embeddings 
Cross-Lingual Evaluation ::: Models in Comparison
Static Word Embeddings  We rely on a state-of-the-art mapping-based method for the induction of cross-lingual word embeddings (CLWEs): vecmap BIBREF148  The core idea behind such mapping-based or projection-based approaches is to learn a post-hoc alignment of independently trained monolingual word embeddings BIBREF106  Such methods have gained popularity due to their conceptual simplicity and competitive performance coupled with reduced bilingual supervision requirements: they support CLWE induction with only as much as a few thousand word translation pairs as the bilingual supervision BIBREF149, BIBREF150, BIBREF151, BIBREF107  More recent work has shown that CLWEs can be induced with even weaker supervision from small dictionaries spanning several hundred pairs BIBREF152, BIBREF135, identical strings BIBREF153, or even only shared numerals BIBREF154  In the extreme, fully unsupervised projection-based CLWEs extract such seed bilingual lexicons from scratch on the basis of monolingual data only BIBREF103, BIBREF148, BIBREF155, BIBREF156, BIBREF104, BIBREF157 
Recent empirical studies BIBREF158, BIBREF135, BIBREF159 have compared a variety of unsupervised and weakly supervised mapping-based CLWE methods, and vecmap emerged as the most robust and very competitive choice  Therefore, we focus on 1) its fully unsupervised variant (unsuper) in our comparisons  For several language pairs, we also report scores with two other vecmap model variants: 2) a supervised variant which learns a mapping based on an available seed lexicon (super), and 3) a supervised variant with self-learning (super+sl) which iteratively increases the seed lexicon and improves the mapping gradually  For a detailed description of these variants, we refer the reader to recent work BIBREF148, BIBREF135  We again use CC+Wiki ft vectors as initial monolingual word vectors, except for yue where Wiki ft is used  The seed dictionaries of two different sizes (1k and 5k translation pairs) are based on PanLex BIBREF160, and are taken directly from prior work BIBREF135, or extracted from PanLex following the same procedure as in the prior work 
Contextualized Cross-Lingual Word Embeddings  We again evaluate the capacity of (massively) multilingual pretrained language models, m-bert and xlm-100, to reason over cross-lingual lexical similarity  Implicitly, such an evaluation also evaluates “the intrinsic quality” of shared cross-lingual word-level vector spaces induced by these methods, and their ability to boost cross-lingual transfer between different language pairs  We rely on the same procedure of aggregating the models' subword-level parameters into word-level representations, already described in §SECREF34 
As in monolingual settings, we can apply unsupervised post-processing steps such as abtt to both static and contextualized cross-lingual word embeddings 
Cross-Lingual Evaluation ::: Results and Discussion
Main Results and Differences across Language Pairs  A summary of the results on the 66 cross-lingual Multi-SimLex datasets are provided in Table TABREF59 and Figure FIGREF60  The findings confirm several interesting findings from our previous monolingual experiments (§SECREF44), and also corroborate several hypotheses and findings from prior work, now on a large sample of language pairs and for the task of cross-lingual semantic similarity 
First, we observe that the fully unsupervised vecmap model, despite being the most robust fully unsupervised method at present, fails to produce a meaningful cross-lingual word vector space for a large number of language pairs (see the bottom triangle of Table TABREF59): many correlation scores are in fact no-correlation results, accentuating the problem of fully unsupervised cross-lingual learning for typologically diverse languages and with fewer amounts of monolingual data BIBREF135  The scores are particularly low across the board for lower-resource languages such as Welsh and Kiswahili  It also seems that the lack of monolingual data is a larger problem than typological dissimilarity between language pairs, as we do observe reasonably high correlation scores with vecmap for language pairs such as cmn-spa, heb-est, and rus-fin  However, typological differences (e g , morphological richness) still play an important role as we observe very low scores when pairing cmn with morphologically rich languages such fin, est, pol, and rus  Similar to prior work of Vulic:2019we and doval2019onthe, given the fact that unsupervised vecmap is the most robust unsupervised CLWE method at present BIBREF158, our results again question the usefulness of fully unsupervised approaches for a large number of languages, and call for further developments in the area of unsupervised and weakly supervised cross-lingual representation learning 
The scores of m-bert and xlm-100 lead to similar conclusions as in the monolingual settings  Reasonable correlation scores are achieved only for a small subset of resource-rich language pairs (e g , eng, fra, spa, cmn) which dominate the multilingual m-bert training  Interestingly, the scores indicate a much higher performance of language pairs where yue is one of the languages when we use m-bert instead of vecmap  This boils down again to the fact that yue, due to its specific language script, has a good representation of its words and subwords in the shared m-bert vocabulary  At the same time, a reliable vecmap mapping between yue and other languages cannot be found due to a small monolingual yue corpus  In cases when vecmap does not yield a degenerate cross-lingual vector space starting from two monolingual ones, the final correlation scores seem substantially higher than the ones obtained by the single massively multilingual m-bert model 
Finally, the results in Figure FIGREF60 again verify the usefulness of unsupervised post-processing also in cross-lingual settings  We observe improved performance with both m-bert and xlm-100 when mean centering (+mc) is applied, and further gains can be achieved by using abtt on the mean-centered vector spaces  A similar finding also holds for static cross-lingual word embeddings, where applying abbt (-10) yields higher scores on 61/66 language pairs 
Fully Unsupervised vs  Weakly Supervised Cross-Lingual Embeddings  The results in Table TABREF59 indicate that fully unsupervised cross-lingual learning fails for a large number of language pairs  However, recent work BIBREF135 has noted that these sub-optimal non-alignment solutions with the unsuper model can be avoided by relying on (weak) cross-lingual supervision spanning only several thousands or even hundreds of word translation pairs  Therefore, we examine 1) if we can further improve the results on cross-lingual Multi-SimLex resorting to (at least some) cross-lingual supervision for resource-rich language pairs; and 2) if such available word-level supervision can also be useful for a range of languages which displayed near-zero performance in Table TABREF59  In other words, we test if recent “tricks of the trade” used in the rich literature on CLWE learning reflect in gains on cross-lingual Multi-SimLex datasets 
First, we reassess the findings established on the bilingual lexicon induction task BIBREF161, BIBREF135: using at least some cross-lingual supervision is always beneficial compared to using no supervision at all  We report improvements over the unsuper model for all 10 language pairs in Table TABREF66, even though the unsuper method initially produced strong correlation scores  The importance of self-learning increases with decreasing available seed dictionary size, and the +sl model always outperforms unsuper with 1k seed pairs; we observe the same patterns also with even smaller dictionary sizes than reported in Table TABREF66 (250 and 500 seed pairs)  Along the same line, the results in Table TABREF67 indicate that at least some supervision is crucial for the success of static CLWEs on resource-leaner language pairs  We note substantial improvements on all language pairs; in fact, the vecmap model is able to learn a more reliable mapping starting from clean supervision  We again note large gains with self-learning 
Multilingual vs  Bilingual Contextualized Embeddings  Similar to the monolingual settings, we also inspect if massively multilingual training in fact dilutes the knowledge necessary for cross-lingual reasoning on a particular language pair  Therefore, we compare the 100-language xlm-100 model with i) a variant of the same model trained on a smaller set of 17 languages (xlm-17); ii) a variant of the same model trained specifically for the particular language pair (xlm-2); and iii) a variant of the bilingual xlm-2 model that also leverages bilingual knowledge from parallel data during joint training (xlm-2++)  We again use the pretrained models made available by Conneau:2019nips, and we refer to the original work for further technical details 
The results are summarized in Figure FIGREF60, and they confirm the intuition that massively multilingual pretraining can damage performance even on resource-rich languages and language pairs  We observe a steep rise in performance when the multilingual model is trained on a much smaller set of languages (17 versus 100), and further improvements can be achieved by training a dedicated bilingual model  Finally, leveraging bilingual parallel data seems to offer additional slight gains, but a tiny difference between xlm-2 and xlm-2++ also suggests that this rich bilingual information is not used in the optimal way within the xlm architecture for semantic similarity 
In summary, these results indicate that, in order to improve performance in cross-lingual transfer tasks, more work should be invested into 1) pretraining dedicated language pair-specific models, and 2) creative ways of leveraging available cross-lingual supervision (e g , word translation pairs, parallel or comparable corpora) BIBREF121, BIBREF123, BIBREF124 with pretraining paradigms such as bert and xlm  Using such cross-lingual supervision could lead to similar benefits as indicated by the results obtained with static cross-lingual word embeddings (see Table TABREF66 and Table TABREF67)  We believe that Multi-SimLex can serve as a valuable means to track and guide future progress in this research area 
Conclusion and Future Work
We have presented Multi-SimLex, a resource containing human judgments on the semantic similarity of word pairs for 12 monolingual and 66 cross-lingual datasets  The languages covered are typologically diverse and include also under-resourced ones, such as Welsh and Kiswahili  The resource covers an unprecedented amount of 1,888 word pairs, carefully balanced according to their similarity score, frequency, concreteness, part-of-speech class, and lexical field  In addition to Multi-Simlex, we release the detailed protocol we followed to create this resource  We hope that our consistent guidelines will encourage researchers to translate and annotate Multi-Simlex -style datasets for additional languages  This can help and create a hugely valuable, large-scale semantic resource for multilingual NLP research 
The core Multi-SimLex we release with this paper already enables researchers to carry out novel linguistic analysis as well as establishes a benchmark for evaluating representation learning models  Based on our preliminary analyses, we found that speakers of closely related languages tend to express equivalent similarity judgments  In particular, geographical proximity seems to play a greater role than family membership in determining the similarity of judgments across languages  Moreover, we tested several state-of-the-art word embedding models, both static and contextualized representations, as well as several (supervised and unsupervised) post-processing techniques, on the newly released Multi-SimLex  This enables future endeavors to improve multilingual representation learning with challenging baselines  In addition, our results provide several important insights for research on both monolingual and cross-lingual word representations:
1) Unsupervised post-processing techniques (mean centering, elimination of top principal components, adjusting similarity orders) are always beneficial independently of the language, although the combination leading to the best scores is language-specific and hence needs to be tuned 
2) Similarity rankings obtained from word embeddings for nouns are better aligned with human judgments than all the other part-of-speech classes considered here (verbs, adjectives, and, for the first time, adverbs)  This confirms previous generalizations based on experiments on English 
3) The factor having the greatest impact on the quality of word representations is the availability of raw texts to train them in the first place, rather than language properties (such as family, geographical area, typological features) 
4) Massively multilingual pretrained encoders such as m-bert BIBREF29 and xlm-100 BIBREF30 fare quite poorly on our benchmark, whereas pretrained encoders dedicated to a single language are more competitive with static word embeddings such as fastText BIBREF31  Moreover, for language-specific encoders, parameter reduction techniques reduce performance only marginally 
5) Techniques to inject clean lexical semantic knowledge from external resources into distributional word representations were proven to be effective in emphasizing the relation of semantic similarity  In particular, methods capable of transferring such knowledge from resource-rich to resource-lean languages BIBREF59 increased the correlation with human judgments for most languages, except for those with limited unlabelled data 
Future work can expand our preliminary, yet large-scale study on the ability of pretrained encoders to reason over word-level semantic similarity in different languages  For instance, we have highlighted how sharing the same encoder parameters across multiple languages may harm performance  However, it remains unclear if, and to what extent, the input language embeddings present in xlm-100 but absent in m-bert help mitigate this issue  In addition, pretrained language embeddings can be obtained both from typological databases BIBREF99 and from neural architectures BIBREF100  Plugging these embeddings into the encoders in lieu of embeddings trained end-to-end as suggested by prior work BIBREF162, BIBREF163, BIBREF164 might extend the coverage to more resource-lean languages 
Another important follow-up analysis might involve the comparison of the performance of representation learning models on multilingual datasets for both word-level semantic similarity and sentence-level Natural Language Understanding  In particular, Multi-SimLex fills a gap in available resources for multilingual NLP and might help understand how lexical and compositional semantics interact if put alongside existing resources such as XNLI BIBREF84 for natural language inference or PAWS-X BIBREF89 for cross-lingual paraphrase identification  Finally, the Multi-SimLex annotation could turn out to be a unique source of evidence to study the effects of polysemy in human judgments on semantic similarity: for equivalent word pairs in multiple languages, are the similarity scores affected by how many senses the two words (or multi-word expressions) incorporate?
In light of the success of initiatives like Universal Dependencies for multilingual treebanks, we hope that making Multi-SimLex and its guidelines available will encourage other researchers to expand our current sample of languages  We particularly encourage creation and submission of comparable Multi-SimLex datasets for under-resourced and typologically diverse languages in future work  In particular, we have made a Multi-Simlex community website available to facilitate easy creation, gathering, dissemination, and use of annotated datasets: https://multisimlex com/ 
This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909)  Thierry Poibeau is partly supported by a PRAIRIE 3IA Institute fellowship ("Investissements d'avenir" program, reference ANR-19-P3IA-0001) 
Context 3: Introduction
This work is licensed under a Creative Commons Attribution 4 0 International License  License details: http://creativecommons org/licenses/by/4 0/
Entity linking (EL), mapping entity mentions in texts to a given knowledge base (KB), serves as a fundamental role in many fields, such as question answering BIBREF0 , semantic search BIBREF1 , and information extraction BIBREF2 , BIBREF3   However, this task is non-trivial because entity mentions are usually ambiguous  As shown in Figure FIGREF1 , the mention England refers to three entities in KB, and an entity linking system should be capable of identifying the correct entity as England cricket team rather than England and England national football team 
Entity linking is typically broken down into two main phases: (i) candidate generation obtains a set of referent entities in KB for each mention, and (ii) named entity disambiguation selects the possible candidate entity by solving a ranking problem  The key challenge lies in the ranking model that computes the relevance between candidates and the corresponding mentions based on the information both in texts and KBs BIBREF4   In terms of the features used for ranking, we classify existing EL models into two groups: local models to resolve mentions independently relying on textual context information from the surrounding words BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , and global (collective) models, which are the main focus of this paper, that encourage the target entities of all mentions in a document to be topically coherent BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14  
Global models usually build an entity graph based on KBs to capture coherent entities for all identified mentions in a document, where the nodes are entities, and edges denote their relations  The graph provides highly discriminative semantic signals (e g , entity relatedness) that are unavailable to local model BIBREF15   For example (Figure FIGREF1 ), an EL model seemly cannot find sufficient disambiguation clues for the mention England from its surrounding words, unless it utilizes the coherence information of consistent topic “cricket" among adjacent mentions England, Hussain, and Essex  Although the global model has achieved significant improvements, its limitation is threefold:
To mitigate the first limitation, recent EL studies introduce neural network (NN) models due to its amazing feature abstraction and generalization ability  In such models, words/entities are represented by low dimensional vectors in a continuous space, and features for mention as well as candidate entities are automatically learned from data BIBREF4   However, existing NN-based methods for EL are either local models BIBREF16 , BIBREF17 or merely use word/entity embeddings for feature extraction and rely on another modules for collective disambiguation, which thus cannot fully utilize the power of NN models for collective EL BIBREF18 , BIBREF19 , BIBREF20  
The second drawback of the global approach has been alleviated through approximate optimization techniques, such as PageRank/random walks BIBREF21 , graph pruning BIBREF22 , ranking SVMs BIBREF23 , or loopy belief propagation (LBP) BIBREF18 , BIBREF24   However, these methods are not differentiable and thus difficult to be integrated into neural network models (the solution for the first limitation) 
To overcome the third issue of inadequate training data, BIBREF17 has explored a massive amount of hyperlinks in Wikipedia, but these potential annotations for EL contain much noise, which may distract a naive disambiguation model BIBREF6  
In this paper, we propose a novel Neural Collective Entity Linking model (NCEL), which performs global EL combining deep neural networks with Graph Convolutional Network (GCN) BIBREF25 , BIBREF26 that allows flexible encoding of entity graphs  It integrates both local contextual information and global interdependence of mentions in a document, and is efficiently trainable in an end-to-end fashion  Particularly, we introduce attention mechanism to robustly model local contextual information by selecting informative words and filtering out the noise  On the other hand, we apply GCNs to improve discriminative signals of candidate entities by exploiting the rich structure underlying the correct entities  To alleviate the global computations, we propose to convolute on the subgraph of adjacent mentions  Thus, the overall coherence shall be achieved in a chain-like way via a sliding window over the document  To the best of our knowledge, this is the first effort to develop a unified model for neural collective entity linking 
In experiments, we first verify the efficiency of NCEL via theoretically comparing its time complexity with other collective alternatives  Afterwards, we train our neural model using collected Wikipedia hyperlinks instead of dataset-specific annotations, and perform evaluations on five public available benchmarks  The results show that NCEL consistently outperforms various baselines with a favorable generalization ability  Finally, we further present the performance on a challenging dataset WW BIBREF19 as well as qualitative results, investigating the effectiveness of each key module 
Preliminaries and Framework
We denote INLINEFORM0 as a set of entity mentions in a document INLINEFORM1 , where INLINEFORM2 is either a word INLINEFORM3 or a mention INLINEFORM4   INLINEFORM5 is the entity graph for document INLINEFORM6 derived from the given knowledge base, where INLINEFORM7 is a set of entities, INLINEFORM8 denotes the relatedness between INLINEFORM9 and higher values indicate stronger relations  Based on INLINEFORM10 , we extract a subgraph INLINEFORM11 for INLINEFORM12 , where INLINEFORM13 denotes the set of candidate entities for INLINEFORM14   Note that we don't include the relations among candidates of the same mention in INLINEFORM15 because these candidates are mutually exclusive in disambiguation 
Formally, we define the entity linking problem as follows: Given a set of mentions INLINEFORM0 in a document INLINEFORM1 , and an entity graph INLINEFORM2 , the goal is to find an assignment INLINEFORM3  
To collectively find the best assignment, NCEL aims to improve the discriminability of candidates' local features by using entity relatedness within a document via GCN, which is capable of learning a function of features on the graph through shared parameters over all nodes  Figure FIGREF10 shows the framework of NCEL including three main components:
Example As shown in Figure FIGREF10 , for the current mention England, we utilize its surrounding words as local contexts (e g , surplus), and adjacent mentions (e g , Hussian) as global information  Collectively, we utilize the candidates of England INLINEFORM0 as well as those entities of its adjacencies INLINEFORM1 to construct feature vectors for INLINEFORM2 and the subgraph of relatedness as inputs of our neural model  Let darker blue indicate higher probability of being predicted, the correct candidate INLINEFORM3 becomes bluer due to its bluer neighbor nodes of other mentions INLINEFORM4   The dashed lines denote entity relations that have indirect impacts through the sliding adjacent window , and the overall structure shall be achieved via multiple sub-graphs by traversing all mentions 
Before introducing our model, we first describe the component of candidate generation 
Candidate Generation
Similar to previous work BIBREF24 , we use the prior probability INLINEFORM0 of entity INLINEFORM1 conditioned on mention INLINEFORM2 both as a local feature and to generate candidate entities: INLINEFORM3   We compute INLINEFORM4 based on statistics of mention-entity pairs from: (i) Wikipedia page titles, redirect titles and hyperlinks, (ii) the dictionary derived from a large Web Corpus BIBREF27 , and (iii) the YAGO dictionary with a uniform distribution BIBREF22   We pick up the maximal prior if a mention-entity pair occurs in different resources  In experiments, to optimize for memory and run time, we keep only top INLINEFORM5 entities based on INLINEFORM6   In the following two sections, we will present the key components of NECL, namely feature extraction and neural network for collective entity linking 
Feature Extraction
The main goal of NCEL is to find a solution for collective entity linking using an end-to-end neural model, rather than to improve the measurements of local textual similarity or global mention/entity relatedness  Therefore, we use joint embeddings of words and entities at sense level BIBREF28 to represent mentions and its contexts for feature extraction  In this section, we give a brief description of our embeddings followed by our features used in the neural model 
Learning Joint Embeddings of Word and Entity
Following BIBREF28 , we use Wikipedia articles, hyperlinks, and entity outlinks to jointly learn word/mention and entity embeddings in a unified vector space, so that similar words/mentions and entities have similar vectors  To address the ambiguity of words/mentions, BIBREF28 represents each word/mention with multiple vectors, and each vector denotes a sense referring to an entity in KB  The quality of the embeddings is verified on both textual similarity and entity relatedness tasks 
Formally, each word/mention has a global embedding INLINEFORM0 , and multiple sense embeddings INLINEFORM1   Each sense embedding INLINEFORM2 refers to an entity embedding INLINEFORM3 , while the difference between INLINEFORM4 and INLINEFORM5 is that INLINEFORM6 models the co-occurrence information of an entity in texts (via hyperlinks) and INLINEFORM7 encodes the structured entity relations in KBs  More details can be found in the original paper 
Local Features
Local features focus on how compatible the entity is mentioned in a piece of text (i e , the mention and the context words)  Except for the prior probability (Section SECREF9 ), we define two types of local features for each candidate entity INLINEFORM0 :
String Similarity Similar to BIBREF16 , we define string based features as follows: the edit distance between mention's surface form and entity title, and boolean features indicating whether they are equivalent, whether the mention is inside, starts with or ends with entity title and vice versa 
Compatibility We also measure the compatibility of INLINEFORM0 with the mention's context words INLINEFORM1 by computing their similarities based on joint embeddings: INLINEFORM2 and INLINEFORM3 , where INLINEFORM4 is the context embedding of INLINEFORM5 conditioned on candidate INLINEFORM6 and is defined as the average sum of word global vectors weighted by attentions: INLINEFORM7
where INLINEFORM0 is the INLINEFORM1 -th word's attention from INLINEFORM2   In this way, we automatically select informative words by assigning higher attention weights, and filter out irrelevant noise through small weights  The attention INLINEFORM3 is computed as follows: INLINEFORM4
where INLINEFORM0 is the similarity measurement, and we use cosine similarity in the presented work  We concatenate the prior probability, string based similarities, compatibility similarities and the embeddings of contexts as well as the entity as the local feature vectors 
Global Features
The key idea of collective EL is to utilize the topical coherence throughout the entire document  The consistency assumption behind it is that: all mentions in a document shall be on the same topic  However, this leads to exhaustive computations if the number of mentions is large  Based on the observation that the consistency attenuates along with the distance between two mentions, we argue that the adjacent mentions might be sufficient for supporting the assumption efficiently 
Formally, we define neighbor mentions as INLINEFORM0 adjacent mentions before and after current mention INLINEFORM1 : INLINEFORM2 , where INLINEFORM3 is the pre-defined window size  Thus, the topical coherence at document level shall be achieved in a chain-like way  As shown in Figure FIGREF10 ( INLINEFORM4 ), mentions Hussain and Essex, a cricket player and the cricket club, provide adequate disambiguation clues to induce the underlying topic “cricket" for the current mention England, which impacts positively on identifying the mention surrey as another cricket club via the common neighbor mention Essex 
A degraded case happens if INLINEFORM0 is large enough to cover the entire document, and the mentions used for global features become the same as the previous work, such as BIBREF21   In experiments, we heuristically found a suitable INLINEFORM1 which is much smaller than the total number of mentions  The benefits of efficiency are in two ways: (i) to decrease time complexity, and (ii) to trim the entity graph into a fixed size of subgraph that facilitates computation acceleration through GPUs and batch techniques, which will be discussed in Section SECREF24  
Given neighbor mentions INLINEFORM0 , we extract two types of vectorial global features and structured global features for each candidate INLINEFORM1 :
Neighbor Mention Compatibility Suppose neighbor mentions are topical coherent, a candidate entity shall also be compatible with neighbor mentions if it has a high compatibility score with the current mention, otherwise not  That is, we extract the vectorial global features by computing the similarities between INLINEFORM0 and all neighbor mentions: INLINEFORM1 , where INLINEFORM2 is the mention embedding by averaging the global vectors of words in its surface form: INLINEFORM3 , where INLINEFORM4 are tokenized words of mention INLINEFORM5  
Subgraph Structure The above features reflect the consistent semantics in texts (i e , mentions)  We now extract structured global features using the relations in KB, which facilitates the inference among candidates to find the most topical coherent subset

ground_truth: Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese

Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. LDA is an unsupervised method; is this paper introducing an unsupervised approach to spam detection?
Context 2: Introduction
Microblogging such as Twitter and Weibo is a popular social networking service, which allows users to post messages up to 140 characters  There are millions of active users on the platform who stay connected with friends  Unfortunately, spammers also use it as a tool to post malicious links, send unsolicited messages to legitimate users, etc  A certain amount of spammers could sway the public opinion and cause distrust of the social platform  Despite the use of rigid anti-spam rules, human-like spammers whose homepages having photos, detailed profiles etc  have emerged  Unlike previous "simple" spammers, whose tweets contain only malicious links, those "smart" spammers are more difficult to distinguish from legitimate users via content-based features alone BIBREF0  
There is a considerable amount of previous work on spammer detection on social platforms  Researcher from Twitter Inc  BIBREF1 collect bot accounts and perform analysis on the user behavior and user profile features  Lee et al  lee2011seven use the so-called social honeypot by alluring social spammers' retweet to build a benchmark dataset, which has been extensively explored in our paper  Some researchers focus on the clustering of urls in tweets and network graph of social spammers BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , showing the power of social relationship features As for content information modeling, BIBREF6 apply improved sparse learning methods  However, few studies have adopted topic-based features  Some researchers BIBREF7 discuss topic characteristics of spamming posts, indicating that spammers are highly likely to dwell on some certain topics such as promotion  But this may not be applicable to the current scenario of smart spammers 
In this paper, we propose an efficient feature extraction method  In this method, two new topic-based features are extracted and used to discriminate human-like spammers from legitimate users  We consider the historical tweets of each user as a document and use the Latent Dirichlet Allocation (LDA) model to compute the topic distribution for each user  Based on the calculated topic probability, two topic-based features, the Local Outlier Standard Score (LOSS) which captures the user's interests on different topics and the Global Outlier Standard Score (GOSS) which reveals the user's interests on specific topic in comparison with other users', are extracted  The two features contain both local and global information, and the combination of them can distinguish human-like spammers effectively 
To the best of our knowledge, it is the first time that features based on topic distributions are used in spammer classification  Experimental results on one public dataset and one self-collected dataset further validate that the two sets of extracted topic-based features get excellent performance on human-like spammer classification problem compared with other state-of-the-art methods  In addition, we build a Weibo dataset, which contains both legitimate users and spammers 
To summarize, our major contributions are two-fold:
In the following sections, we first propose the topic-based features extraction method in Section 2, and then introduce the two datasets in Section 3  Experimental results are discussed in Section 4, and we conclude the paper in Section 5  Future work is presented in Section 6 
Methodology
In this section, we first provide some observations we obtained after carefully exploring the social network, then the LDA model is introduced  Based on the LDA model, the ways to obtain the topic probability vector for each user and the two topic-based features are provided 
Observation
After exploring the homepages of a substantial number of spammers, we have two observations  1) social spammers can be divided into two categories  One is content polluters, and their tweets are all about certain kinds of advertisement and campaign  The other is fake accounts, and their tweets resemble legitimate users' but it seems they are simply random copies of others to avoid being detected by anti-spam rules  2) For legitimate users, content polluters and fake accounts, they show different patterns on topics which interest them 
Legitimate users mainly focus on limited topics which interest him  They seldom post contents unrelated to their concern 
Content polluters concentrate on certain topics 
Fake accounts focus on a wide range of topics due to random copying and retweeting of other users' tweets 
Spammers and legitimate users show different interests on some topics e g  commercial, weather, etc 
To better illustrate our observation, Figure  1 shows the topic distribution of spammers and legitimate users in two employed datasets(the Honeypot dataset and Weibo dataset)  We can see that on both topics (topic-3 and topic-11) there exists obvious difference between the red bars and green bars, representing spammers and legitimate users  On the Honeypot dataset, spammers have a narrower shape of distribution (the outliers on the red bar tail are not counted) than that of legitimate users  This is because there are more content polluters than fake accounts  In other word, spammers in this dataset tend to concentrate on limited topics  While on the Weibo dataset, fake accounts who are interested in different topics take large proportion of spammers  Their distribution is more flat (i e  red bars) than that of the legitimate users  Therefore we can detect spammers by means of the difference of their topic distribution patterns 
LDA model
Blei et al blei2003latent first presented Latent Dirichlet Allocation(LDA) as an example of topic model 
Each document $i$ is deemed as a bag of words $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $ and $M$ is the number of words  Each word is attributable to one of the document's topics $Z=\left\lbrace  z_{i1},z_{i2},   ,z_{iK}\right\rbrace $ and $K$ is the number of topics  $\psi _{k}$ is a multinomial distribution over words for topic $k$   $\theta _i$ is another multinomial distribution over topics for document $i$   The smoothed generative model is illustrated in Figure  2   $\alpha $ and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $0 are hyper parameter that affect scarcity of the document-topic and topic-word distributions  In this paper, $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $1 , $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $2 and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $3 are empirically set to 0 3, 0 01 and 15  The entire content of each Twitter user is regarded as one document  We adopt Gibbs Sampling BIBREF8 to speed up the inference of LDA  Based on LDA, we can get the topic probabilities for all users in the employed dataset as: $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $4 , where $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $5 is the number of users  Each element $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $6 is a topic probability vector for the $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $7 document  $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $8 is the raw topic probability vector and our features are developed on top of it 
Topic-based Features
Using the LDA model, each person in the dataset is with a topic probability vector $X_i$   Assume $x_{ik}\in X_{i}$ denotes the likelihood that the $\emph {i}^{th}$ tweet account favors $\emph {k}^{th}$ topic in the dataset  Our topic based features can be calculated as below 
Global Outlier Standard Score measures the degree that a user's tweet content is related to a certain topic compared to the other users  Specifically, the "GOSS" score of user $i$ on topic $k$ can be calculated as Eq ( 12 ):
$$\centering \begin{array}{ll} \mu \left(x_{k}\right)=\frac{\sum _{i=1}^{n} x_{ik}}{n},\\ GOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_k\right)}{\sqrt{\underset{i}{\sum }\left(x_{ik}-\mu \left(x_{k}\right)\right)^{2}}}  \end{array}$$   (Eq  12)
The value of $GOSS\left(x_{ik}\right)$ indicates the interesting degree of this person to the $\emph {k}^{th}$ topic  Specifically, if $GOSS\left(x_{ik}\right)$ > $GOSS\left(x_{jk}\right)$ , it means that the $\emph {i}^{th}$ person has more interest in topic $k$ than the $\emph {j}^{th}$ person  If the value $GOSS\left(x_{ik}\right)$ is extremely high or low, the $\emph {i}^{th}$ person showing extreme interest or no interest on topic $k$ which will probably be a distinctive pattern in the fowllowing classfication  Therefore, the topics interested or disliked by the $\emph {k}^{th}$0 person can be manifested by $\emph {k}^{th}$1 , from which the pattern of the interested topics with regarding to this person is found  Denote $\emph {k}^{th}$2 our first topic-based feature, and it hopefully can get good performance on spammer detection 
Local Outlier Standard Score measures the degree of interest someone shows to a certain topic by considering his own homepage content only  For instance, the "LOSS" score of account $i$ on topic $k$ can be calculated as Eq ( 13 ):
$$\centering \begin{array}{ll} \mu \left(x_{i}\right)=\frac{\sum _{k=1}^{K} x_{ik}}{K},\\ LOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_i\right)}{\sqrt{\underset{k}{\sum }\left(x_{ik}-\mu \left(x_{i}\right)\right)^{2}}}  \end{array}$$   (Eq  13)
$\mu (x_i)$ represents the averaged interesting degree for all topics with regarding to $\emph {i}^{th}$ user and his tweet content  Similarly to $GOSS$ , the topics interested or disliked by the $\emph {i}^{th}$ person via considering his single post information can be manifested by $f_{LOSS}^{i}=[LOSS(x_{i1})\cdots LOSS(x_{iK})]$ , and $LOSS$ becomes our second topic-based features for the $\emph {i}^{th}$ person 
Dataset
We use one public dataset Social Honeypot dataset and one self-collected dataset Weibo dataset to validate the effectiveness of our proposed features 
Social Honeypot Dataset: Lee et al  lee2010devils created and deployed 60 seed social accounts on Twitter to attract spammers by reporting back what accounts interact with them  They collected 19,276 legitimate users and 22,223 spammers in their datasets along with their tweet content in 7 months  This is our first test dataset 
Our Weibo Dataset: Sina Weibo is one of the most famous social platforms in China  It has implemented many features from Twitter  The 2197 legitimate user accounts in this dataset are provided by the Tianchi Competition held by Sina Weibo  The spammers are all purchased commercially from multiple vendors on the Internet  We checked them manually and collected 802 suitable "smart" spammers accounts 
Preprocessing: Before directly performing the experiments on the employed datasets, we first delete some accounts with few posts in the two employed since the number of tweets is highly indicative of spammers  For the English Honeypot dataset, we remove stopwords, punctuations, non-ASCII words and apply stemming  For the Chinese Weibo dataset, we perform segmentation with "Jieba", a Chinese text segmentation tool  After preprocessing steps, the Weibo dataset contains 2197 legitimate users and 802 spammers, and the honeypot dataset contains 2218 legitimate users and 2947 spammers  It is worth mentioning that the Honeypot dataset has been slashed because most of the Twitter accounts only have limited number of posts, which are not enough to show their interest inclination 
Evaluation Metrics
The evaluating indicators in our model are show in 2   We calculate precision, recall and F1-score (i e  F1 score) as in Eq  ( 19 )  Precision is the ratio of selected accounts that are spammers  Recall is the ratio of spammers that are detected so  F1-score is the harmonic mean of precision and recall 
$$precision =\frac{TP}{TP+FP}, recall =\frac{TP}{TP+FN}\nonumber \\ F1-score = \frac{2\times precision \times recall}{precision + recall}$$   (Eq  19)
Performance Comparisons with Baseline
Three baseline classification methods: Support Vector Machines (SVM), Adaboost, and Random Forests are adopted to evaluate our extracted features  We test each classification algorithm with scikit-learn BIBREF9 and run a 10-fold cross validation  On each dataset, the employed classifiers are trained with individual feature first, and then with the combination of the two features  From 1 , we can see that GOSS+LOSS achieves the best performance on F1-score among all others  Besides, the classification by combination of LOSS and GOSS can increase accuracy by more than 3% compared with raw topic distribution probability 
Comparison with Other Features
To compare our extracted features with previously used features for spammer detection, we use three most discriminative feature sets according to Lee et al  lee2011seven( 4 )  Two classifiers (Adaboost and SVM) are selected to conduct feature performance comparisons  Using Adaboost, our LOSS+GOSS features outperform all other features except for UFN which is 2% higher than ours with regard to precision on the Honeypot dataset  It is caused by the incorrectly classified spammers who are mostly news source after our manual check  They keep posting all kinds of news pieces covering diverse topics, which is similar to the behavior of fake accounts  However, UFN based on friendship networks is more useful for public accounts who possess large number of followers  The best recall value of our LOSS+GOSS features using SVM is up to 6% higher than the results by other feature groups  Regarding F1-score, our features outperform all other features  To further show the advantages of our proposed features, we compare our combined LOSS+GOSS with the combination of all the features from Lee et al  lee2011seven (UFN+UC+UH)  It's obvious that LOSS+GOSS have a great advantage over UFN+UC+UH in terms of recall and F1-score  Moreover, by combining our LOSS+GOSS features and UFN+UC+UH features together, we obtained another 7 1% and 2 3% performance gain with regard to precision and F1-score by Adaboost  Though there is a slight decline in terms of recall  By SVM, we get comparative results on recall and F1-score but about 3 5% improvement on precision 
Conclusion
In this paper, we propose a novel feature extraction method to effectively detect "smart" spammers who post seemingly legitimate tweets and are thus difficult to identify by existing spammer classification methods  Using the LDA model, we obtain the topic probability for each Twitter user  By utilizing the topic probability result, we extract our two topic-based features: GOSS and LOSS which represent the account with global and local information  Experimental results on a public dataset and a self-built Chinese microblog dataset validate the effectiveness of the proposed features 
Future Work
In future work, the combination method of local and global information can be further improved to maximize their individual strengths  We will also apply decision theory to enhancing the performance of our proposed features  Moreover, we are also building larger datasets on both Twitter and Weibo to validate our method  Moreover, larger datasets on both Twitter and Weibo will be built to further validate our method 
Context 3:  For each document, we obtain the entity graph INLINEFORM0 by taking candidate entities of all mentions INLINEFORM1 as nodes, and using entity embeddings to compute their similarities as edges INLINEFORM2   Then, we extract the subgraph structured features INLINEFORM3 for each entity INLINEFORM4 for efficiency 
Formally, we define the subgraph as: INLINEFORM0 , where INLINEFORM1   For example (Figure FIGREF1 ), for entity England cricket team, the subgraph contains the relation from it to all candidates of neighbor mentions: England cricket team, Nasser Hussain (rugby union), Nasser Hussain, Essex, Essex County Cricket Club and Essex, New York  To support batch-wise acceleration, we represent INLINEFORM2 in the form of adjacency table based vectors: INLINEFORM3 , where INLINEFORM4 is the number of candidates per mention 
Finally, for each candidate INLINEFORM0 , we concatenate local features and neighbor mention compatibility scores as the feature vector INLINEFORM1 , and construct the subgraph structure representation INLINEFORM2 as the inputs of NCEL 
Neural Collective Entity Linking
NCEL incorporates GCN into a deep neural network to utilize structured graph information for collectively feature abstraction, while differs from conventional GCN in the way of applying the graph  Instead of the entire graph, only a subset of nodes is “visible" to each node in our proposed method, and then the overall structured information shall be reached in a chain-like way  Fixing the size of the subset, NCEL is further speeded up by batch techniques and GPUs, and is efficient to large-scale data 
Graph Convolutional Network
GCNs are a type of neural network model that deals with structured data  It takes a graph as an input and output labels for each node  As a simplification of spectral graph convolutions, the main idea of BIBREF26 is similar to a propagation model: to enhance the features of a node according to its neighbor nodes  The formulation is as follows: INLINEFORM0
where INLINEFORM0 is a normalized adjacent matrix of the input graph with self-connection, INLINEFORM1 and INLINEFORM2 are the hidden states and weights in the INLINEFORM3 -th layer, and INLINEFORM4 is a non-linear activation, such as ReLu 
Model Architecture
As shown in Figure FIGREF10 , NCEL identifies the correct candidate INLINEFORM0 for the mention INLINEFORM1 by using vectorial features as well as structured relatedness with candidates of neighbor mentions INLINEFORM2   Given feature vector INLINEFORM3 and subgraph representation INLINEFORM4 of each candidate INLINEFORM5 , we stack them as inputs for mention INLINEFORM6 : INLINEFORM7 , and the adjacent matrix INLINEFORM8 , where INLINEFORM9 denotes the subgraph with self-connection  We normalize INLINEFORM10 such that all rows sum to one, denoted as INLINEFORM11 , avoiding the change in the scale of the feature vectors 
Given INLINEFORM0 and INLINEFORM1 , the goal of NCEL is to find the best assignment: INLINEFORM2
where INLINEFORM0 is the output variable of candidates, and INLINEFORM1 is a probability function as follows: INLINEFORM2
where INLINEFORM0 is the score function parameters by INLINEFORM1   NCEL learns the mapping INLINEFORM2 through a neural network including three main modules: encoder, sub-graph convolution network (sub-GCN) and decoder  Next, we introduce them in turn 
Encoder The function of this module is to integrate different features by a multi-layer perceptron (MLP): INLINEFORM0
where INLINEFORM0 is the hidden states of the current mention, INLINEFORM1 and INLINEFORM2 are trainable parameters and bias  We use ReLu as the non-linear activation INLINEFORM3  
Sub-Graph Convolution Network Similar to GCN, this module learns to abstract features from the hidden state of the mention itself as well as its neighbors  Suppose INLINEFORM0 is the hidden states of the neighbor INLINEFORM1 , we stack them to expand the current hidden states of INLINEFORM2 as INLINEFORM3 , such that each row corresponds to that in the subgraph adjacent matrix INLINEFORM4   We define sub-graph convolution as: INLINEFORM5
where INLINEFORM0 is a trainable parameter 
Decoder After INLINEFORM0 iterations of sub-graph convolution, the hidden states integrate both features of INLINEFORM1 and its neighbors  A fully connected decoder maps INLINEFORM2 to the number of candidates as follows: INLINEFORM3
where INLINEFORM0  
Training
The parameters of network are trained to minimize cross-entropy of the predicted and ground truth INLINEFORM0 : INLINEFORM1
Suppose there are INLINEFORM0 documents in training corpus, each document has a set of mentions INLINEFORM1 , leading to totally INLINEFORM2 mention sets  The overall objective function is as follows: INLINEFORM3
Experiments
To avoid overfitting with some dataset, we train NCEL using collected Wikipedia hyperlinks instead of specific annotated data  We then evaluate the trained model on five different benchmarks to verify the linking precision as well as the generalization ability  Furthermore, we investigate the effectiveness of key modules in NCEL and give qualitative results for comprehensive analysis 
Baselines and Datasets
We compare NCEL with the following state-of-the-art EL methods including three local models and three types of global models:
Local models: He BIBREF29 and Chisholm BIBREF6 beat many global models by using auto-encoders and web links, respectively, and NTEE BIBREF16 achieves the best performance based on joint embeddings of words and entities 
Iterative model: AIDA BIBREF22 links entities by iteratively finding a dense subgraph 
Loopy Belief Propagation: Globerson BIBREF18 and PBoH BIBREF30 introduce LBP BIBREF31 techniques for collective inference, and Ganea BIBREF24 solves the global training problem via truncated fitting LBP 
PageRank/Random Walk: Boosting BIBREF32 , AGDISTISG BIBREF33 , Babelfy BIBREF34 , WAT BIBREF35 , xLisa BIBREF36 and WNED BIBREF19 performs PageRank BIBREF37 or random walk BIBREF38 on the mention-entity graph and use the convergence score for disambiguation 
For fairly comparison, we report the original scores of the baselines in the papers  Following these methods, we evaluate NCEL on the following five datasets: (1) CoNLL-YAGO BIBREF22 : the CoNLL 2003 shared task including testa of 4791 mentions in 216 documents, and testb of 4485 mentions in 213 documents  (2) TAC2010 BIBREF39 : constructed for the Text Analysis Conference that comprises 676 mentions in 352 documents for testing  (3) ACE2004 BIBREF23 : a subset of ACE2004 co-reference documents including 248 mentions in 35 documents, which is annotated by Amazon Mechanical Turk  (4) AQUAINT BIBREF40 : 50 news articles including 699 mentions from three different news agencies  (5) WW BIBREF19 : a new benchmark with balanced prior distributions of mentions, leading to a hard case of disambiguation  It has 6374 mentions in 310 documents automatically extracted from Wikipedia 
Training Details and Running Time Analysis
Training We collect 50,000 Wikipedia articles according to the number of its hyperlinks as our training data  For efficiency, we trim the articles to the first three paragraphs leading to 1,035,665 mentions in total  Using CoNLL-Test A as the development set, we evaluate the trained NCEL on the above benchmarks  We set context window to 20, neighbor mention window to 6, and top INLINEFORM0 candidates for each mention  We use two layers with 2000 and 1 hidden units in MLP encoder, and 3 layers in sub-GCN  We use early stop and fine tune the embeddings  With a batch size of 16, nearly 3 epochs cost less than 15 minutes on the server with 20 core CPU and the GeForce GTX 1080Ti GPU with 12Gb memory  We use standard Precision, Recall and F1 at mention level (Micro) and at the document level (Macro) as measurements 
Complexity Analysis Compared with local methods, the main disadvantage of collective methods is high complexity and expensive costs  Suppose there are INLINEFORM0 mentions in documents on average, among these global models, NCEL not surprisingly has the lowest time complexity INLINEFORM1 since it only considers adjacent mentions, where INLINEFORM2 is the number of sub-GCN layers indicating the iterations until convergence  AIDA has the highest time complexity INLINEFORM3 in worst case due to exhaustive iteratively finding and sorting the graph  The LBP and PageRank/random walk based methods achieve similar high time complexity of INLINEFORM4 mainly because of the inference on the entire graph 
Results on GERBIL
GERBIL BIBREF41 is a benchmark entity annotation framework that aims to provide a unified comparison among different EL methods across datasets including ACE2004, AQUAINT and CoNLL  We compare NCEL with the global models that report the performance on GERBIL 
As shown in Table TABREF26 , NCEL achieves the best performance in most cases with an average gain of 2% on Micro F1 and 3% Macro F1  The baseline methods also achieve competitive results on some datasets but fail to adapt to the others  For example, AIDA and xLisa perform quite well on ACE2004 but poorly on other datasets, or WAT, PBoH, and WNED have a favorable performance on CoNLL but lower values on ACE2004 and AQUAINT  Our proposed method performs consistently well on all datasets that demonstrates the good generalization ability 
Results on TAC2010 and WW
In this section, we investigate the effectiveness of NCEL in the “easy" and “hard" datasets, respectively  Particularly, TAC2010, which has two mentions per document on average (Section SECREF19 ) and high prior probabilities of correct candidates (Figure FIGREF28 ), is regarded as the “easy" case for EL, and WW is the “hard" case since it has the most mentions with balanced prior probabilities BIBREF19   Besides, we further compare the impact of key modules by removing the following part from NCEL: global features (NCEL-local), attention (NCEL-noatt), embedding features (NCEL-noemb), and the impact of the prior probability (prior) 
The results are shown in Table FIGREF28 and Table FIGREF28   We can see the average linking precision (Micro) of WW is lower than that of TAC2010, and NCEL outperforms all baseline methods in both easy and hard cases  In the “easy" case, local models have similar performance with global models since only little global information is available (2 mentions per document)  Besides, NN-based models, NTEE and NCEL-local, perform significantly better than others including most global models, demonstrating that the effectiveness of neural models deals with the first limitation in the introduction 
Impact of NCEL Modules
As shown in Figure FIGREF28 , the prior probability performs quite well in TAC2010 but poorly in WW  Compared with NCEL-local, the global module in NCEL brings more improvements in the “hard" case than that for “easy" dataset, because local features are discriminative enough in most cases of TAC2010, and global information becomes quite helpful when local features cannot handle  That is, our propose collective model is robust and shows a good generalization ability to difficult EL  The improvements by each main module are relatively small in TAC2010, while the modules of attention and embedding features show non-negligible impacts in WW (even worse than local model), mainly because WW contains much noise, and these two modules are effective in improving the robustness to noise and the ability of generalization by selecting informative words and providing more accurate semantics, respectively 
Qualitative Analysis
The results of example in Figure FIGREF1 are shown in Table TABREF30 , which is from CoNLL testa dataset  For mention Essex, although both NCEL and NCEL-local correctly identify entity Essex County Cricket Club, NCEL outputs higher probability due to the enhancement of neighbor mentions  Moreover, for mention England, NCEL-local cannot find enough disambiguation clues from its context words, such as surplus and requirements, and thus assigns a higher probability of 0 42 to the country England according to the prior probability  Collectively, NCEL correctly identifies England cricket team with a probability of 0 72 as compared with 0 20 in NCEL-local with the help of its neighbor mention Essex 
Conclusion
In this paper, we propose a neural model for collective entity linking that is end-to-end trainable  It applies GCN on subgraphs instead of the entire entity graph to efficiently learn features from both local and global information  We design an attention mechanism that endows NCEL robust to noisy data  Trained on collected Wikipedia hyperlinks, NCEL outperforms the state-of-the-art collective methods across five different datasets  Besides, further analysis of the impacts of main modules as well as qualitative results demonstrates its effectiveness 
In the future, we will extend our method into cross-lingual settings to help link entities in low-resourced languages by exploiting rich knowledge from high-resourced languages, and deal with NIL entities to facilitate specific applications 
Acknowledgments
The work is supported by National Key Research and Development Program of China (2017YFB1002101), NSFC key project (U1736204, 61661146007), and THUNUS NExT Co-Lab 

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How effective is their NCEL approach overall?
Context 4:  We optimize marginal data likelihood directly using Adam BIBREF22   For both tasks in the fully unsupervised setting, we do not tune the hyper-parameters using supervised data 
Unsupervised POS tagging
For unsupervised POS tagging, we use a Markov-structured syntax model in our approach, which is a popular structure for unsupervised tagging tasks BIBREF9 , BIBREF10  
Following existing literature, we train and test on the entire WSJ corpus (49208 sentences, 1M tokens)  We use 45 tag clusters, the number of POS tags that appear in WSJ corpus  We train the discrete HMM and the Gaussian HMM BIBREF9 as baselines  For the Gaussian HMM, mean vectors of Gaussian emissions are initialized with the empirical mean of all word vectors with an additive noise  We assume diagonal covariance matrix for INLINEFORM0 and initialize it with the empirical variance of the word vectors  Following BIBREF9 , the covariance matrix is fixed during training  The multinomial probabilities are initialized as INLINEFORM1 , where INLINEFORM2   For our approach, we initialize the syntax model and Gaussian parameters with the pre-trained Gaussian HMM  The weights of layers in the rectified network are initialized from a uniform distribution with mean zero and a standard deviation of INLINEFORM3 , where INLINEFORM4 is the input dimension  We evaluate the performance of POS tagging with both Many-to-One (M-1) accuracy BIBREF23 and V-Measure (VM) BIBREF24   Given a model we found that the tagging performance is well-correlated with the training data likelihood, thus we use training data likelihood as a unsupervised criterion to select the trained model over 10 random restarts after training 50 epochs  We repeat this process 5 times and report the mean and standard deviation of performance 
We compare our approach with basic HMM, Gaussian HMM, and several state-of-the-art systems, including sophisticated HMM variants and clustering techniques with hand-engineered features  The results are presented in Table TABREF32   Through the introduced latent embeddings and additional neural projection, our approach improves over the Gaussian HMM by 5 4 points in M-1 and 5 6 points in VM  Neural HMM (NHMM) BIBREF10 is a baseline that also learns word representation jointly  Both their basic model and extended Conv version does not outperform the Gaussian HMM  Their best model incorporates another LSTM to model long distance dependency and breaks the Markov assumption, yet our approach still achieves substantial improvement over it without considering more context information  Moreover, our method outperforms the best published result that benefits from hand-engineered features BIBREF27 by 2 0 points on VM 
We found that most tagging errors happen in noun subcategories  Therefore, we do the one-to-one mapping between gold POS tags and induced clusters and plot the normalized confusion matrix of noun subcategories in Figure FIGREF35   The Gaussian HMM fails to identify “NN” and “NNS” correctly for most cases, and it often recognizes “NNPS” as “NNP”  In contrast, our approach corrects these errors well 
Unsupervised Dependency Parsing without gold POS tags
For the task of unsupervised dependency parse induction, we employ the Dependency Model with Valence (DMV) BIBREF2 as the syntax model in our approach  DMV is a generative model that defines a probability distribution over dependency parse trees and syntactic categories, generating tokens and dependencies in a head-outward fashion  While, traditionally, DMV is trained using gold POS tags as observed syntactic categories, in our approach, we treat each tag as a latent variable, as described in sec:general-neural 
Most existing approaches to this task are not fully unsupervised since they rely on gold POS tags following the original experimental setup for DMV  This is partially because automatically parsing from words is difficult even when using unsupervised syntactic categories BIBREF29   However, inducing dependencies from words alone represents a more realistic experimental condition since gold POS tags are often unavailable in practice  Previous work that has trained from words alone often requires additional linguistic constraints (like sentence internal boundaries) BIBREF29 , BIBREF30 , BIBREF31 , BIBREF32 , acoustic cues BIBREF33 , additional training data BIBREF4 , or annotated data from related languages BIBREF34   Our approach is naturally designed to train on word embeddings directly, thus we attempt to induce dependencies without using gold POS tags or other extra linguistic information 
Like previous work we use sections 02-21 of WSJ corpus as training data and evaluate on section 23, we remove punctuations and train the models on sentences of length INLINEFORM0 , “head-percolation” rules BIBREF39 are applied to obtain gold dependencies for evaluation  We train basic DMV, extended DMV (E-DMV) BIBREF35 and Gaussian DMV (which treats POS tag as unknown latent variables and generates observed word embeddings directly conditioned on them following Gaussian distribution) as baselines  Basic DMV and E-DMV are trained with Viterbi EM BIBREF40 on unsupervised POS tags induced from our Markov-structured model described in sec:pos  Multinomial parameters of the syntax model in both Gaussian DMV and our model are initialized with the pre-trained DMV baseline  Other parameters are initialized in the same way as in the POS tagging experiment  The directed dependency accuracy (DDA) is used for evaluation and we report accuracy on sentences of length INLINEFORM1 and all lengths  We train the parser until training data likelihood converges, and report the mean and standard deviation over 20 random restarts 
Our model directly observes word embeddings and does not require gold POS tags during training  Thus, results from related work trained on gold tags are not directly comparable  However, to measure how these systems might perform without gold tags, we run three recent state-of-the-art systems in our experimental setting: UR-A E-DMV BIBREF36 , Neural E-DMV BIBREF11 , and CRF Autoencoder (CRFAE) BIBREF37   We use unsupervised POS tags (induced from our Markov-structured model) in place of gold tags  We also train basic DMV on gold tags and include several state-of-the-art results on gold tags as reference points 
As shown in Table TABREF39 , our approach is able to improve over the Gaussian DMV by 4 8 points on length INLINEFORM0 and 4 8 points on all lengths, which suggests the additional latent embedding layer and neural projector are helpful  The proposed approach yields, to the best of our knowledge, state-of-the-art performance without gold POS annotation and without sentence-internal boundary information  DMV, UR-A E-DMV, Neural E-DMV, and CRFAE suffer a large decrease in performance when trained on unsupervised tags – an effect also seen in previous work BIBREF29 , BIBREF34   Since our approach induces latent POS tags jointly with dependency trees, it may be able to learn POS clusters that are more amenable to grammar induction than the unsupervised tags  We observe that CRFAE underperforms its gold-tag counterpart substantially  This may largely be a result of the model's reliance on prior linguistic rules that become unavailable when gold POS tag types are unknown  Many extensions to DMV can be considered orthogonal to our approach – they essentially focus on improving the syntax model  It is possible that incorporating these more sophisticated syntax models into our approach may lead to further improvements 
Sensitivity Analysis
In the above experiments we initialize the structured syntax components with the pre-trained Gaussian or discrete baseline, which is shown as a useful technique to help train our deep models  We further study the results with fully random initialization  In the POS tagging experiment, we report the results in Table TABREF48   While the performance with 4 layers is comparable to the pre-trained Gaussian initialization, deeper projections (8 or 16 layers) result in a dramatic drop in performance  This suggests that the structured syntax model with very deep projections is difficult to train from scratch, and a simpler projection might be a good compromise in the random initialization setting 
Different from the Markov prior in POS tagging experiments, our parsing model seems to be quite sensitive to the initialization  For example, directed accuracy of our approach on sentences of length INLINEFORM0 is below 40 0 with random initialization  This is consistent with previous work that has noted the importance of careful initialization for DMV-based models such as the commonly used harmonic initializer BIBREF2   However, it is not straightforward to apply the harmonic initializer for DMV directly in our model without using some kind of pre-training since we do not observe gold POS 
We investigate the effect of the choice of pre-trained embedding on performance while using our approach  To this end, we additionally include results using fastText embeddings BIBREF41 – which, in contrast with skip-gram embeddings, include character-level information  We set the context windows size to 1 and the dimension size to 100 as in the skip-gram training, while keeping other parameters set to their defaults  These results are summarized in Table TABREF50 and Table TABREF51   While fastText embeddings lead to reduced performance with our model, our approach still yields an improvement over the Gaussian baseline with the new observed embeddings space 
Qualitative Analysis of Embeddings
We perform qualitative analysis to understand how the latent embeddings help induce syntactic structures  First we filter out low-frequency words and punctuations in WSJ, and visualize the rest words (10k) with t-SNE BIBREF42 under different embeddings  We assign each word with its most likely gold POS tags in WSJ and color them according to the gold POS tags 
For our Markov-structured model, we have displayed the embedding space in Figure SECREF5 , where the gold POS clusters are well-formed  Further, we present five example target words and their five nearest neighbors in terms of cosine similarity  As shown in Table TABREF53 , the skip-gram embedding captures both semantic and syntactic aspects to some degree, yet our embeddings are able to focus especially on the syntactic aspects of words, in an unsupervised fashion without using any extra morphological information 
In Figure FIGREF54 we depict the learned latent embeddings with the DMV-structured syntax model  Unlike the Markov structure, the DMV structure maps a large subset of singular and plural nouns to the same overlapping region  However, two clusters of singular and plural nouns are actually separated  We inspect the two clusters and the overlapping region in Figure FIGREF54 , it turns out that the nouns in the separated clusters are words that can appear as subjects and, therefore, for which verb agreement is important to model  In contrast, the nouns in the overlapping region are typically objects  This demonstrates that the latent embeddings are focusing on aspects of language that are specifically important for modeling dependency without ever having seen examples of dependency parses  Some previous work has deliberately created embeddings to capture different notions of similarity BIBREF43 , BIBREF44 , while they use extra morphology or dependency annotations to guide the embedding learning, our approach provides a potential alternative to create new embeddings that are guided by structured syntax model, only using unlabeled text corpora 
Related Work
Our approach is related to flow-based generative models, which are first described in NICE BIBREF16 and have recently received more attention BIBREF17 , BIBREF19 , BIBREF18   This relevant work mostly adopts simple (e g  Gaussian) and fixed priors and does not attempt to learn interpretable latent structures  Another related generative model class is variational auto-encoders (VAEs) BIBREF45 that optimize a lower bound on the marginal data likelihood, and can be extended to learn latent structures BIBREF46 , BIBREF47   Against the flow-based models, VAEs remove the invertibility constraint but sacrifice the merits of exact inference and exact log likelihood computation, which potentially results in optimization challenges BIBREF48   Our approach can also be viewed in connection with generative adversarial networks (GANs) BIBREF49 that is a likelihood-free framework to learn implicit generative models  However, it is non-trivial for a gradient-based method like GANs to propagate gradients through discrete structures 
Conclusion
In this work, we define a novel generative approach to leverage continuous word representations for unsupervised learning of syntactic structure  Experiments on both POS induction and unsupervised dependency parsing tasks demonstrate the effectiveness of our proposed approach  Future work might explore more sophisticated invertible projections, or recurrent projections that jointly transform the entire input sequence 

ground_truth: NCEL consistently outperforms various baselines with a favorable generalization ability