prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the 12 languages covered?
Context 1: Introduction
The lack of annotated training and evaluation data for many tasks and domains hinders the development of computational models for the majority of the world's languages BIBREF0, BIBREF1, BIBREF2  The necessity to guide and advance multilingual and cross-lingual NLP through annotation efforts that follow cross-lingually consistent guidelines has been recently recognized by collaborative initiatives such as the Universal Dependency (UD) project BIBREF3  The latest version of UD (as of March 2020) covers more than 70 languages  Crucially, this resource continues to steadily grow and evolve through the contributions of annotators from across the world, extending the UD's reach to a wide array of typologically diverse languages  Besides steering research in multilingual parsing BIBREF4, BIBREF5, BIBREF6 and cross-lingual parser transfer BIBREF7, BIBREF8, BIBREF9, the consistent annotations and guidelines have also enabled a range of insightful comparative studies focused on the languages' syntactic (dis)similarities BIBREF10, BIBREF11, BIBREF12 
Inspired by the UD work and its substantial impact on research in (multilingual) syntax, in this article we introduce Multi-SimLex, a suite of manually and consistently annotated semantic datasets for 12 different languages, focused on the fundamental lexical relation of semantic similarity BIBREF13, BIBREF14  For any pair of words, this relation measures whether their referents share the same (functional) features, as opposed to general cognitive association captured by co-occurrence patterns in texts (i e , the distributional information)  Datasets that quantify the strength of true semantic similarity between concept pairs such as SimLex-999 BIBREF14 or SimVerb-3500 BIBREF15 have been instrumental in improving models for distributional semantics and representation learning  Discerning between semantic similarity and relatedness/association is not only crucial for theoretical studies on lexical semantics (see §SECREF2), but has also been shown to benefit a range of language understanding tasks in NLP  Examples include dialog state tracking BIBREF16, BIBREF17, spoken language understanding BIBREF18, BIBREF19, text simplification BIBREF20, BIBREF21, BIBREF22, dictionary and thesaurus construction BIBREF23, BIBREF24 
Despite the proven usefulness of semantic similarity datasets, they are available only for a small and typologically narrow sample of resource-rich languages such as German, Italian, and Russian BIBREF25, whereas some language types and low-resource languages typically lack similar evaluation data  Even if some resources do exist, they are limited in their size (e g , 500 pairs in Turkish BIBREF26, 500 in Farsi BIBREF27, or 300 in Finnish BIBREF28) and coverage (e g , all datasets which originated from the original English SimLex-999 contain only high-frequent concepts, and are dominated by nouns)  This is why, as our departure point, we introduce a larger and more comprehensive English word similarity dataset spanning 1,888 concept pairs (see §SECREF4) 
Most importantly, semantic similarity datasets in different languages have been created using heterogeneous construction procedures with different guidelines for translation and annotation, as well as different rating scales  For instance, some datasets were obtained by directly translating the English SimLex-999 in its entirety BIBREF25, BIBREF16 or in part BIBREF28  Other datasets were created from scratch BIBREF26 and yet others sampled English concept pairs differently from SimLex-999 and then translated and reannotated them in target languages BIBREF27  This heterogeneity makes these datasets incomparable and precludes systematic cross-linguistic analyses  In this article, consolidating the lessons learned from previous dataset construction paradigms, we propose a carefully designed translation and annotation protocol for developing monolingual Multi-SimLex datasets with aligned concept pairs for typologically diverse languages  We apply this protocol to a set of 12 languages, including a mixture of major languages (e g , Mandarin, Russian, and French) as well as several low-resource ones (e g , Kiswahili, Welsh, and Yue Chinese)  We demonstrate that our proposed dataset creation procedure yields data with high inter-annotator agreement rates (e g , the average mean inter-annotator agreement for Welsh is 0 742) 
The unified construction protocol and alignment between concept pairs enables a series of quantitative analyses  Preliminary studies on the influence that polysemy and cross-lingual variation in lexical categories (see §SECREF6) have on similarity judgments are provided in §SECREF5  Data created according to Multi-SimLex protocol also allow for probing into whether similarity judgments are universal across languages, or rather depend on linguistic affinity (in terms of linguistic features, phylogeny, and geographical location)  We investigate this question in §SECREF25  Naturally, Multi-SimLex datasets can be used as an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms  We conduct a systematic evaluation of several state-of-the-art representation models in §SECREF7, showing that there are large gaps between human and system performance in all languages  The proposed construction paradigm also supports the automatic creation of 66 cross-lingual Multi-SimLex datasets by interleaving the monolingual ones  We outline the construction of the cross-lingual datasets in §SECREF6, and then present a quantitative evaluation of a series of cutting-edge cross-lingual representation models on this benchmark in §SECREF8 
Contributions  We now summarize the main contributions of this work:
1) Building on lessons learned from prior work, we create a more comprehensive lexical semantic similarity dataset for the English language spanning a total of 1,888 concept pairs balanced with respect to similarity, frequency, and concreteness, and covering four word classes: nouns, verbs, adjectives and, for the first time, adverbs  This dataset serves as the main source for the creation of equivalent datasets in several other languages 
2) We present a carefully designed and rigorous language-agnostic translation and annotation protocol  These well-defined guidelines will facilitate the development of future Multi-SimLex datasets for other languages  The proposed protocol eliminates some crucial issues with prior efforts focused on the creation of multi-lingual semantic resources, namely: i) limited coverage; ii) heterogeneous annotation guidelines; and iii) concept pairs which are semantically incomparable across different languages 
3) We offer to the community manually annotated evaluation sets of 1,888 concept pairs across 12 typologically diverse languages, and 66 large cross-lingual evaluation sets  To the best of our knowledge, Multi-SimLex is the most comprehensive evaluation resource to date focused on the relation of semantic similarity 
4) We benchmark a wide array of recent state-of-the-art monolingual and cross-lingual word representation models across our sample of languages  The results can serve as strong baselines that lay the foundation for future improvements 
5) We present a first large-scale evaluation study on the ability of encoders pretrained on language modeling (such as bert BIBREF29 and xlm BIBREF30) to reason over word-level semantic similarity in different languages  To our own surprise, the results show that monolingual pretrained encoders, even when presented with word types out of context, are sometimes competitive with static word embedding models such as fastText BIBREF31 or word2vec BIBREF32  The results also reveal a huge gap in performance between massively multilingual pretrained encoders and language-specific encoders in favor of the latter: our findings support other recent empirical evidence related to the “curse of multilinguality” BIBREF33, BIBREF34 in representation learning 
6) We make all of these resources available on a website which facilitates easy creation, submission and sharing of Multi-Simlex-style datasets for a larger number of languages  We hope that this will yield an even larger repository of semantic resources that inspire future advances in NLP within and across languages 
In light of the success of Universal Dependencies BIBREF3, we hope that our initiative will instigate a collaborative public effort with established and clear-cut guidelines that will result in additional Multi-SimLex datasets in a large number of languages in the near future  Moreover, we hope that it will provide means to advance our understanding of distributional and lexical semantics across a large number of languages  All monolingual and cross-lingual Multi-SimLex datasets–along with detailed translation and annotation guidelines–are available online at: https://multisimlex com/ 
Lexical Semantic Similarity ::: Similarity and Association
The focus of the Multi-SimLex initiative is on the lexical relation of pure semantic similarity  For any pair of words, this relation measures whether their referents share the same features  For instance, graffiti and frescos are similar to the extent that they are both forms of painting and appear on walls  This relation can be contrasted with the cognitive association between two words, which often depends on how much their referents interact in the real world, or are found in the same situations  For instance, a painter is easily associated with frescos, although they lack any physical commonalities  Association is also known in the literature under other names: relatedness BIBREF13, topical similarity BIBREF35, and domain similarity BIBREF36 
Semantic similarity and association overlap to some degree, but do not coincide BIBREF37, BIBREF38  In fact, there exist plenty of pairs that are intuitively associated but not similar  Pairs where the converse is true can also be encountered, although more rarely  An example are synonyms where a word is common and the other infrequent, such as to seize and to commandeer  BIBREF14 revealed that while similarity measures based on the WordNet graph BIBREF39 and human judgments of association in the University of South Florida Free Association Database BIBREF40 do correlate, a number of pairs follow opposite trends  Several studies on human cognition also point in the same direction  For instance, semantic priming can be triggered by similar words without association BIBREF41  On the other hand, a connection with cue words is established more quickly for topically related words rather than for similar words in free association tasks BIBREF42 
A key property of semantic similarity is its gradience: pairs of words can be similar to a different degree  On the other hand, the relation of synonymy is binary: pairs of words are synonyms if they can be substituted in all contexts (or most contexts, in a looser sense), otherwise they are not  While synonyms can be conceived as lying on one extreme of the semantic similarity continuum, it is crucial to note that their definition is stated in purely relational terms, rather than invoking their referential properties BIBREF43, BIBREF44, BIBREF45  This makes behavioral studies on semantic similarity fundamentally different from lexical resources like WordNet BIBREF46, which include paradigmatic relations (such as synonymy) 
Lexical Semantic Similarity ::: Similarity for NLP: Intrinsic Evaluation and Semantic Specialization
The ramifications of the distinction between similarity and association are profound for distributional semantics  This paradigm of lexical semantics is grounded in the distributional hypothesis, formulated by BIBREF47 and BIBREF48  According to this hypothesis, the meaning of a word can be recovered empirically from the contexts in which it occurs within a collection of texts  Since both pairs of topically related words and pairs of purely similar words tend to appear in the same contexts, their associated meaning confounds the two distinct relations BIBREF14, BIBREF49, BIBREF50  As a result, distributional methods obscure a crucial facet of lexical meaning  This limitation also reflects onto word embeddings (WEs), representations of words as low-dimensional vectors that have become indispensable for a wide range of NLP applications BIBREF51, BIBREF52, BIBREF53  In particular, it involves both static WEs learned from co-occurrence patterns BIBREF32, BIBREF54, BIBREF31 and contextualized WEs learned from modeling word sequences BIBREF55, BIBREF29  As a result, in the induced representations, geometrical closeness (measured e g  through cosine distance) conflates genuine similarity with broad relatedness  For instance, the vectors for antonyms such as sober and drunk, by definition dissimilar, might be neighbors in the semantic space under the distributional hypothesis  BIBREF36, BIBREF56, and BIBREF53 demonstrated that different choices of hyper-parameters in WE algorithms (such as context window) emphasize different relations in the resulting representations  Likewise, BIBREF57 and BIBREF54 discovered that WEs learned from texts annotated with syntactic information mirror similarity better than simple local bag-of-words neighborhoods 
The failure of WEs to capture semantic similarity, in turn, affects model performance in several NLP applications where such knowledge is crucial  In particular, Natural Language Understanding tasks such as statistical dialog modeling, text simplification, or semantic text similarity BIBREF58, BIBREF18, BIBREF59, among others, suffer the most  As a consequence, resources providing clean information on semantic similarity are key in mitigating the side effects of the distributional signal  In particular, such databases can be employed for the intrinsic evaluations of specific WE models as a proxy of their reliability for downstream applications BIBREF60, BIBREF61, BIBREF14; intuitively, the more WEs are misaligned with human judgments of similarity, the more their performance on actual tasks is expected to be degraded  Moreover, word representations can be specialized (a k a  retrofitted) by disentangling word relations of similarity and association  In particular, linguistic constraints sourced from external databases (such as synonyms from WordNet) can be injected into WEs BIBREF62, BIBREF63, BIBREF16, BIBREF22, BIBREF64 in order to enforce a particular relation in a distributional semantic space while preserving the original adjacency properties 
Lexical Semantic Similarity ::: Similarity and Language Variation: Semantic Typology
In this work, we tackle the concept of (true) semantic similarity from a multilingual perspective  While the same meaning representations may be shared by all human speakers at a deep cognitive level, there is no one-to-one mapping between the words in the lexicons of different languages  This makes the comparison of similarity judgments across languages difficult, since the meaning overlap of translationally equivalent words is sometimes far less than exact  This results from the fact that the way languages `partition' semantic fields is partially arbitrary BIBREF65, although constrained cross-lingually by common cognitive biases BIBREF66  For instance, consider the field of colors: English distinguishes between green and blue, whereas Murle (South Sudan) has a single word for both BIBREF67 
In general, semantic typology studies the variation in lexical semantics across the world's languages  According to BIBREF68, the ways languages categorize concepts into the lexicon follow three main axes: 1) granularity: what is the number of categories in a specific domain?; 2) boundary location: where do the lines marking different categories lie?; 3) grouping and dissection: what are the membership criteria of a category; which instances are considered to be more prototypical? Different choices with respect to these axes lead to different lexicalization patterns  For instance, distinct senses in a polysemous word in English, such as skin (referring to both the body and fruit), may be assigned separate words in other languages such as Italian pelle and buccia, respectively BIBREF70  We later analyze whether similarity scores obtained from native speakers also loosely follow the patterns described by semantic typology 
Previous Work and Evaluation Data
Word Pair Datasets  Rich expert-created resources such as WordNet BIBREF46, BIBREF71, VerbNet BIBREF72, BIBREF73, or FrameNet BIBREF74 encode a wealth of semantic and syntactic information, but are expensive and time-consuming to create  The scale of this problem gets multiplied by the number of languages in consideration  Therefore, crowd-sourcing with non-expert annotators has been adopted as a quicker alternative to produce smaller and more focused semantic resources and evaluation benchmarks  This alternative practice has had a profound impact on distributional semantics and representation learning BIBREF14  While some prominent English word pair datasets such as WordSim-353 BIBREF75, MEN BIBREF76, or Stanford Rare Words BIBREF77 did not discriminate between similarity and relatedness, the importance of this distinction was established by BIBREF14 through the creation of SimLex-999  This inspired other similar datasets which focused on different lexical properties  For instance, SimVerb-3500 BIBREF15 provided similarity ratings for 3,500 English verbs, whereas CARD-660 BIBREF78 aimed at measuring the semantic similarity of infrequent concepts 
Semantic Similarity Datasets in Other Languages  Motivated by the impact of datasets such as SimLex-999 and SimVerb-3500 on representation learning in English, a line of related work focused on creating similar resources in other languages  The dominant approach is translating and reannotating the entire original English SimLex-999 dataset, as done previously for German, Italian, and Russian BIBREF25, Hebrew and Croatian BIBREF16, and Polish BIBREF79  Venekoski:2017nodalida apply this process only to a subset of 300 concept pairs from the English SimLex-999  On the other hand, BIBREF27 sampled a new set of 500 English concept pairs to ensure wider topical coverage and balance across similarity spectra, and then translated those pairs to German, Italian, Spanish, and Farsi (SEMEVAL-500)  A similar approach was followed by BIBREF26 for Turkish, by BIBREF80 for Mandarin Chinese, and by BIBREF81 for Japanese  BIBREF82 translated the concatenation of SimLex-999, WordSim-353, and the English SEMEVAL-500 into Thai and then reannotated it  Finally, BIBREF83 translated English SimLex-999 and WordSim-353 to 11 resource-rich target languages (German, French, Russian, Italian, Dutch, Chinese, Portuguese, Swedish, Spanish, Arabic, Farsi), but they did not provide details concerning the translation process and the resolution of translation disagreements  More importantly, they also did not reannotate the translated pairs in the target languages  As we discussed in § SECREF6 and reiterate later in §SECREF5, semantic differences among languages can have a profound impact on the annotation scores; particulary, we show in §SECREF25 that these differences even roughly define language clusters based on language affinity 
A core issue with the current datasets concerns a lack of one unified procedure that ensures the comparability of resources in different languages  Further, concept pairs for different languages are sourced from different corpora (e g , direct translation of the English data versus sampling from scratch in the target language)  Moreover, the previous SimLex-based multilingual datasets inherit the main deficiencies of the English original version, such as the focus on nouns and highly frequent concepts  Finally, prior work mostly focused on languages that are widely spoken and do not account for the variety of the world's languages  Our long-term goal is devising a standardized methodology to extend the coverage also to languages that are resource-lean and/or typologically diverse (e g , Welsh, Kiswahili as in this work) 
Multilingual Datasets for Natural Language Understanding  The Multi-SimLex initiative and corresponding datasets are also aligned with the recent efforts on procuring multilingual benchmarks that can help advance computational modeling of natural language understanding across different languages  For instance, pretrained multilingual language models such as multilingual bert BIBREF29 or xlm BIBREF30 are typically probed on XNLI test data BIBREF84 for cross-lingual natural language inference  XNLI was created by translating examples from the English MultiNLI dataset, and projecting its sentence labels BIBREF85  Other recent multilingual datasets target the task of question answering based on reading comprehension: i) MLQA BIBREF86 includes 7 languages ii) XQuAD BIBREF87 10 languages; iii) TyDiQA BIBREF88 9 widely spoken typologically diverse languages  While MLQA and XQuAD result from the translation from an English dataset, TyDiQA was built independently in each language  Another multilingual dataset, PAWS-X BIBREF89, focused on the paraphrase identification task and was created translating the original English PAWS BIBREF90 into 6 languages  We believe that Multi-SimLex can substantially contribute to this endeavor by offering a comprehensive multilingual benchmark for the fundamental lexical level relation of semantic similarity  In future work, Multi-SimLex also offers an opportunity to investigate the correlations between word-level semantic similarity and performance in downstream tasks such as QA and NLI across different languages 
The Base for Multi-SimLex: Extending English SimLex-999
In this section, we discuss the design principles behind the English (eng) Multi-SimLex dataset, which is the basis for all the Multi-SimLex datasets in other languages, as detailed in §SECREF5  We first argue that a new, more balanced, and more comprehensive evaluation resource for lexical semantic similarity in English is necessary  We then describe how the 1,888 word pairs contained in the eng Multi-SimLex were selected in such a way as to represent various linguistic phenomena within a single integrated resource 
Construction Criteria  The following criteria have to be satisfied by any high-quality semantic evaluation resource, as argued by previous studies focused on the creation of such resources BIBREF14, BIBREF15, BIBREF91, BIBREF27:
(C1) Representative and diverse  The resource must cover the full range of diverse concepts occurring in natural language, including different word classes (e g , nouns, verbs, adjectives, adverbs), concrete and abstract concepts, a variety of lexical fields, and different frequency ranges 
(C2) Clearly defined  The resource must provide a clear understanding of which semantic relation exactly is annotated and measured, possibly contrasting it with other relations  For instance, the original SimLex-999 and SimVerb-3500 explicitly focus on true semantic similarity and distinguish it from broader relatedness captured by datasets such as MEN BIBREF76 or WordSim-353 BIBREF75 
(C3) Consistent and reliable  The resource must ensure consistent annotations obtained from non-expert native speakers following simple and precise annotation guidelines 
In choosing the word pairs and constructing eng Multi-SimLex, we adhere to these requirements  Moreover, we follow good practices established by the research on related resources  In particular, since the introduction of the original SimLex-999 dataset BIBREF14, follow-up works have improved its construction protocol across several aspects, including: 1) coverage of more lexical fields, e g , by relying on a diverse set of Wikipedia categories BIBREF27, 2) infrequent/rare words BIBREF78, 3) focus on particular word classes, e g , verbs BIBREF15, 4) annotation quality control BIBREF78  Our goal is to make use of these improvements towards a larger, more representative, and more reliable lexical similarity dataset in English and, consequently, in all other languages 
The Final Output: English Multi-SimLex  In order to ensure that the criterion C1 is satisfied, we consolidate and integrate the data already carefully sampled in prior work into a single, comprehensive, and representative dataset  This way, we can control for diversity, frequency, and other properties while avoiding to perform this time-consuming selection process from scratch  Note that, on the other hand, the word pairs chosen for English are scored from scratch as part of the entire Multi-SimLex annotation process, introduced later in §SECREF5  We now describe the external data sources for the final set of word pairs:
1) Source: SimLex-999  BIBREF14  The English Multi-SimLex has been initially conceived as an extension of the original SimLex-999 dataset  Therefore, we include all 999 word pairs from SimLex, which span 666 noun pairs, 222 verb pairs, and 111 adjective pairs  While SimLex-999 already provides examples representing different POS classes, it does not have a sufficient coverage of different linguistic phenomena: for instance, it contains only very frequent concepts, and it does not provide a representative set of verbs BIBREF15 
2) Source: SemEval-17: Task 2 BIBREF92  We start from the full dataset of 500 concept pairs to extract a total of 334 concept pairs for English Multi-SimLex a) which contain only single-word concepts, b) which are not named entities, c) where POS tags of the two concepts are the same, d) where both concepts occur in the top 250K most frequent word types in the English Wikipedia, and e) do not already occur in SimLex-999  The original concepts were sampled as to span all the 34 domains available as part of BabelDomains BIBREF93, which roughly correspond to the main high-level Wikipedia categories  This ensures topical diversity in our sub-sample 
3) Source: CARD-660 BIBREF78  67 word pairs are taken from this dataset focused on rare word similarity, applying the same selection criteria a) to e) employed for SEMEVAL-500  Words are controlled for frequency based on their occurrence counts from the Google News data and the ukWaC corpus BIBREF94  CARD-660 contains some words that are very rare (logboat), domain-specific (erythroleukemia) and slang (2mrw), which might be difficult to translate and annotate across a wide array of languages  Hence, we opt for retaining only the concept pairs above the threshold of top 250K most frequent Wikipedia concepts, as above 
4) Source: SimVerb-3500 BIBREF15 Since both CARD-660 and SEMEVAL-500 are heavily skewed towards noun pairs, and nouns also dominate the original SimLex-999, we also extract additional verb pairs from the verb-specific similarity dataset SimVerb-3500  We randomly sample 244 verb pairs from SimVerb-3500 that represent all similarity spectra  In particular, we add 61 verb pairs for each of the similarity intervals: $[0,1 5), [1 5,3), [3,4 5), [4 5, 6]$  Since verbs in SimVerb-3500 were originally chosen from VerbNet BIBREF95, BIBREF73, they cover a wide range of verb classes and their related linguistic phenomena 
5) Source: University of South Florida BIBREF96 norms, the largest database of free association for English  In order to improve the representation of different POS classes, we sample additional adjectives and adverbs from the USF norms following the procedure established by BIBREF14, BIBREF15  This yields additional 122 adjective pairs, but only a limited number of adverb pairs (e g , later – never, now – here, once – twice)  Therefore, we also create a set of adverb pairs semi-automatically by sampling adjectives that can be derivationally transformed into adverbs (e g  adding the suffix -ly) from the USF, and assessing the correctness of such derivation in WordNet  The resulting pairs include, for instance, primarily – mainly, softly – firmly, roughly – reliably, etc  We include a total of 123 adverb pairs into the final English Multi-SimLex  Note that this is the first time adverbs are included into any semantic similarity dataset 
Fulfillment of Construction Criteria  The final eng Multi-SimLex dataset spans 1,051 noun pairs, 469 verb pairs, 245 adjective pairs, and 123 adverb pairs  As mentioned above, the criterion C1 has been fulfilled by relying only on word pairs that already underwent meticulous sampling processes in prior work, integrating them into a single resource  As a consequence, Multi-SimLex allows for fine-grained analyses over different POS classes, concreteness levels, similarity spectra, frequency intervals, relation types, morphology, lexical fields, and it also includes some challenging orthographically similar examples (e g , infection – inflection)  We ensure that the criteria C2 and C3 are satisfied by using similar annotation guidelines as Simlex-999, SimVerb-3500, and SEMEVAL-500 that explicitly target semantic similarity  In what follows, we outline the carefully tailored process of translating and annotating Multi-SimLex datasets in all target languages 
Multi-SimLex: Translation and Annotation
We now detail the development of the final Multi-SimLex resource, describing our language selection process, as well as translation and annotation of the resource, including the steps taken to ensure and measure the quality of this resource  We also provide key data statistics and preliminary cross-lingual comparative analyses 
Language Selection  Multi-SimLex comprises eleven languages in addition to English  The main objective for our inclusion criteria has been to balance language prominence (by number of speakers of the language) for maximum impact of the resource, while simultaneously having a diverse suite of languages based on their typological features (such as morphological type and language family)  Table TABREF10 summarizes key information about the languages currently included in Multi-SimLex  We have included a mixture of fusional, agglutinative, isolating, and introflexive languages that come from eight different language families  This includes languages that are very widely used such as Chinese Mandarin and Spanish, and low-resource languages such as Welsh and Kiswahili  We hope to further include additional languages and inspire other researchers to contribute to the effort over the lifetime of this project 
The work on data collection can be divided into two crucial phases: 1) a translation phase where the extended English language dataset with 1,888 pairs (described in §SECREF4) is translated into eleven target languages, and 2) an annotation phase where human raters scored each pair in the translated set as well as the English set  Detailed guidelines for both phases are available online at: https://multisimlex com 
Multi-SimLex: Translation and Annotation ::: Word Pair Translation
Translators for each target language were instructed to find direct or approximate translations for the 1,888 word pairs that satisfy the following rules  (1) All pairs in the translated set must be unique (i e , no duplicate pairs); (2) Translating two words from the same English pair into the same word in the target language is not allowed (e g , it is not allowed to translate car and automobile to the same Spanish word coche)  (3) The translated pairs must preserve the semantic relations between the two words when possible  This means that, when multiple translations are possible, the translation that best conveys the semantic relation between the two words found in the original English pair is selected  (4) If it is not possible to use a single-word translation in the target language, then a multi-word expression (MWE) can be used to convey the nearest possible semantics given the above points (e g , the English word homework is translated into the Polish MWE praca domowa) 
Satisfying the above rules when finding appropriate translations for each pair–while keeping to the spirit of the intended semantic relation in the English version–is not always straightforward  For instance, kinship terminology in Sinitic languages (Mandarin and Yue) uses different terms depending on whether the family member is older or younger, and whether the family member comes from the mother’s side or the father’s side  UTF8gbsn In Mandarin, brother has no direct translation and can be translated as either: 哥哥 (older brother) or 弟弟 (younger brother)  Therefore, in such cases, the translators are asked to choose the best option given the semantic context (relation) expressed by the pair in English, otherwise select one of the translations arbitrarily  This is also used to remove duplicate pairs in the translated set, by differentiating the duplicates using a variant at each instance  Further, many translation instances were resolved using near-synonymous terms in the translation  For example, the words in the pair: wood – timber can only be directly translated in Estonian to puit, and are not distinguishable  Therefore, the translators approximated the translation for timber to the compound noun puitmaterjal (literally: wood material) in order to produce a valid pair in the target language  In some cases, a direct transliteration from English is used  For example, the pair: physician and doctor both translate to the same word in Estonian (arst); the less formal word doktor is used as a translation of doctor to generate a valid pair 
We measure the quality of the translated pairs by using a random sample set of 100 pairs (from the 1,888 pairs) to be translated by an independent translator for each target language  The sample is proportionally stratified according to the part-of-speech categories  The independent translator is given identical instructions to the main translator; we then measure the percentage of matched translated words between the two translations of the sample set  Table TABREF12 summarizes the inter-translator agreement results for all languages and by part-of-speech subsets  Overall across all languages, the agreement is 84 8%, which is similar to prior work BIBREF27, BIBREF97 
Multi-SimLex: Translation and Annotation ::: Guidelines and Word Pair Scoring
Across all languages, 145 human annotators were asked to score all 1,888 pairs (in their given language)  We finally collect at least ten valid annotations for each word pair in each language  All annotators were required to abide by the following instructions:
1  Each annotator must assign an integer score between 0 and 6 (inclusive) indicating how semantically similar the two words in a given pair are  A score of 6 indicates very high similarity (i e , perfect synonymy), while zero indicates no similarity 
2  Each annotator must score the entire set of 1,888 pairs in the dataset  The pairs must not be shared between different annotators 
3  Annotators are able to break the workload over a period of approximately 2-3 weeks, and are able to use external sources (e g  dictionaries, thesauri, WordNet) if required 
4  Annotators are kept anonymous, and are not able to communicate with each other during the annotation process 
The selection criteria for the annotators required that all annotators must be native speakers of the target language  Preference to annotators with university education was given, but not required  Annotators were asked to complete a spreadsheet containing the translated pairs of words, as well as the part-of-speech, and a column to enter the score  The annotators did not have access to the original pairs in English 
To ensure the quality of the collected ratings, we have employed an adjudication protocol similar to the one proposed and validated by Pilehvar:2018emnlp  It consists of the following three rounds:
Round 1: All annotators are asked to follow the instructions outlined above, and to rate all 1,888 pairs with integer scores between 0 and 6 
Round 2: We compare the scores of all annotators and identify the pairs for each annotator that have shown the most disagreement  We ask the annotators to reconsider the assigned scores for those pairs only  The annotators may chose to either change or keep the scores  As in the case with Round 1, the annotators have no access to the scores of the other annotators, and the process is anonymous  This process gives a chance for annotators to correct errors or reconsider their judgments, and has been shown to be very effective in reaching consensus, as reported by BIBREF78  We used a very similar procedure as BIBREF78 to identify the pairs with the most disagreement; for each annotator, we marked the $i$th pair if the rated score $s_i$ falls within: $s_i \ge \mu _i + 1 5$ or $s_i \le \mu _i - 1 5$, where $\mu _i$ is the mean of the other annotators' scores 
Round 3: We compute the average agreement for each annotator (with the other annotators), by measuring the average Spearman's correlation against all other annotators  We discard the scores of annotators that have shown the least average agreement with all other annotators, while we maintain at least ten annotators per language by the end of this round  The actual process is done in multiple iterations: (S1) we measure the average agreement for each annotator with every other annotator (this corresponds to the APIAA measure, see later); (S2) if we still have more than 10 valid annotators and the lowest average score is higher than in the previous iteration, we remove the lowest one, and rerun S1  Table TABREF14 shows the number of annotators at both the start (Round 1) and end (Round 3) of our process for each language 
We measure the agreement between annotators using two metrics, average pairwise inter-annotator agreement (APIAA), and average mean inter-annotator agreement (AMIAA)  Both of these use Spearman's correlation ($\rho $) between annotators scores, the only difference is how they are averaged  They are computed as follows:
where $\rho (s_i,s_j)$ is the Spearman's correlation between annotators $i$ and $j$'s scores ($s_i$,$s_j$) for all pairs in the dataset, and $N$ is the number of annotators  APIAA has been used widely as the standard measure for inter-annotator agreement, including in the original SimLex paper BIBREF14  It simply averages the pairwise Spearman's correlation between all annotators  On the other hand, AMIAA compares the average Spearman's correlation of one held-out annotator with the average of all the other $N-1$ annotators, and then averages across all $N$ `held-out' annotators  It smooths individual annotator effects and arguably serves as a better upper bound than APIAA BIBREF15, BIBREF91, BIBREF78 
We present the respective APIAA and AMIAA scores in Table TABREF16 and Table TABREF17 for all part-of-speech subsets, as well as the agreement for the full datasets  As reported in prior work BIBREF15, BIBREF91, AMIAA scores are typically higher than APIAA scores  Crucially, the results indicate `strong agreement' (across all languages) using both measurements  The languages with the highest annotator agreement were French (fra) and Yue Chinese (yue), while Russian (rus) had the lowest overall IAA scores  These scores, however, are still considered to be `moderately strong agreement' 
Multi-SimLex: Translation and Annotation ::: Data Analysis
Similarity Score Distributions  Across all languages, the average score (mean $=1 61$, median$=1 1$) is on the lower side of the similarity scale  However, looking closer at the scores of each language in Table TABREF19, we indicate notable differences in both the averages and the spread of scores  Notably, French has the highest average of similarity scores (mean$=2 61$, median$=2 5$), while Kiswahili has the lowest average (mean$=1 28$, median$=0 5$)  Russian has the lowest spread ($\sigma =1 37$), while Polish has the largest ($\sigma =1 62$)  All of the languages are strongly correlated with each other, as shown in Figure FIGREF20, where all of the Spearman's correlation coefficients are greater than 0 6 for all language pairs  Languages that share the same language family are highly correlated (e g, cmn-yue, rus-pol, est-fin)  In addition, we observe high correlations between English and most other languages, as expected  This is due to the effect of using English as the base/anchor language to create the dataset  In simple words, if one translates to two languages $L_1$ and $L_2$ starting from the same set of pairs in English, it is higly likely that $L_1$ and $L_2$ will diverge from English in different ways  Therefore, the similarity between $L_1$-eng and $L_2$-eng is expected to be higher than between $L_1$-$L_2$, especially if $L_1$ and $L_2$ are typologically dissimilar languages (e g , heb-cmn, see Figure FIGREF20)  This phenomenon is well documented in related prior work BIBREF25, BIBREF27, BIBREF16, BIBREF97  While we acknowledge this as a slight artifact of the dataset design, it would otherwise be impossible to construct a semantically aligned and comprehensive dataset across a large number of languages 
We also report differences in the distribution of the frequency of words among the languages in Multi-SimLex  Figure FIGREF22 shows six example languages, where each bar segment shows the proportion of words in each language that occur in the given frequency range  For example, the 10K-20K segment of the bars represents the proportion of words in the dataset that occur in the list of most frequent words between the frequency rank of 10,000 and 20,000 in that language; likewise with other intervals  Frequency lists for the presented languages are derived from Wikipedia and Common Crawl corpora  While many concept pairs are direct or approximate translations of English pairs, we can see that the frequency distribution does vary across different languages, and is also related to inherent language properties  For instance, in Finnish and Russian, while we use infinitive forms of all verbs, conjugated verb inflections are often more frequent in raw corpora than the corresponding infinitive forms  The variance can also be partially explained by the difference in monolingual corpora size used to derive the frequency rankings in the first place: absolute vocabulary sizes are expected to fluctuate across different languages  However, it is also important to note that the datasets also contain subsets of lower-frequency and rare words, which can be used for rare word evaluations in multiple languages, in the spirit of Pilehvar:2018emnlp's English rare word dataset 
Cross-Linguistic Differences  Table TABREF23 shows some examples of average similarity scores of English, Spanish, Kiswahili and Welsh concept pairs  Remember that the scores range from 0 to 6: the higher the score, the more similar the participants found the concepts in the pair  The examples from Table TABREF23 show evidence of both the stability of average similarity scores across languages (unlikely – friendly, book – literature, and vanish – disappear), as well as language-specific differences (care – caution)  Some differences in similarity scores seem to group languages into clusters  For example, the word pair regular – average has an average similarity score of 4 0 and 4 1 in English and Spanish, respectively, whereas in Kiswahili and Welsh the average similarity score of this pair is 0 5 and 0 8  We analyze this phenomenon in more detail in §SECREF25 
There are also examples for each of the four languages having a notably higher or lower similarity score for the same concept pair than the three other languages  For example, large – big in English has an average similarity score of 5 9, whereas Spanish, Kiswahili and Welsh speakers rate the closest concept pair in their native language to have a similarity score between 2 7 and 3 8  What is more, woman – wife receives an average similarity of 0 9 in English, 2 9 in Spanish, and greater than 4 0 in Kiswahili and Welsh  The examples from Spanish include banco – asiento (bank – seat) which receives an average similarity score 5 1, while in the other three languages the similarity score for this word pair does not exceed 0 1  At the same time, the average similarity score of espantosamente – fantásticamente (amazingly – fantastically) is much lower in Spanish (0 4) than in other languages (4 1 – 5 1)  In Kiswahili, an example of a word pair with a higher similarity score than the rest would be machweo – jioni (sunset – evening), having an average score of 5 5, while the other languages receive 2 8 or less, and a notably lower similarity score is given to wa ajabu - mkubwa sana (wonderful – terrific), getting 0 9, while the other languages receive 5 3 or more  Welsh examples include yn llwyr - yn gyfan gwbl (purely – completely), which scores 5 4 among Welsh speakers but 2 3 or less in other languages, while addo – tyngu (promise – swear) is rated as 0 by all Welsh annotators, but in the other three languages 4 3 or more on average 
There can be several explanations for the differences in similarity scores across languages, including but not limited to cultural context, polysemy, metonymy, translation, regional and generational differences, and most commonly, the fact that words and meanings do not exactly map onto each other across languages  For example, it is likely that the other three languages do not have two separate words for describing the concepts in the concept pair: big – large, and the translators had to opt for similar lexical items that were more distant in meaning, explaining why in English the concept pair received a much higher average similarity score than in other languages  A similar issue related to the mapping problem across languages arose in the Welsh concept pair yn llwye – yn gyfan gwbl, where Welsh speakers agreed that the two concepts are very similar  When asked, bilingual speakers considered the two Welsh concepts more similar than English equivalents purely – completely, potentially explaining why a higher average similarity score was reached in Welsh  The example of woman – wife can illustrate cultural differences or another translation-related issue where the word `wife' did not exist in some languages (for example, Estonian), and therefore had to be described using other words, affecting the comparability of the similarity scores  This was also the case with the football – soccer concept pair  The pair bank – seat demonstrates the effect of the polysemy mismatch across languages: while `bank' has two different meanings in English, neither of them is similar to the word `seat', but in Spanish, `banco' can mean `bank', but it can also mean `bench'  Quite naturally, Spanish speakers gave the pair banco – asiento a higher similarity score than the speakers of languages where this polysemy did not occur 
An example of metonymy affecting the average similarity score can be seen in the Kiswahili version of the word pair: sunset – evening (machweo – jioni)  The average similarity score for this pair is much higher in Kiswahili, likely because the word `sunset' can act as a metonym of `evening'  The low similarity score of wonderful – terrific in Kiswahili (wa ajabu - mkubwa sana) can be explained by the fact that while `mkubwa sana' can be used as `terrific' in Kiswahili, it technically means `very big', adding to the examples of translation- and mapping-related effects  The word pair amazingly – fantastically (espantosamente – fantásticamente) brings out another translation-related problem: the accuracy of the translation  While `espantosamente' could arguably be translated to `amazingly', more common meanings include: `frightfully', `terrifyingly', and `shockingly', explaining why the average similarity score differs from the rest of the languages  Another problem was brought out by addo – tyngu (promise – swear) in Welsh, where the `tyngu' may not have been a commonly used or even a known word choice for annotators, pointing out potential regional or generational differences in language use 
Table TABREF24 presents examples of concept pairs from English, Spanish, Kiswahili, and Welsh on which the participants agreed the most  For example, in English all participants rated the similarity of trial – test to be 4 or 5  In Spanish and Welsh, all participants rated start – begin to correspond to a score of 5 or 6  In Kiswahili, money – cash received a similarity rating of 6 from every participant  While there are numerous examples of concept pairs in these languages where the participants agreed on a similarity score of 4 or higher, it is worth noting that none of these languages had a single pair where all participants agreed on either 1-2, 2-3, or 3-4 similarity rating  Interestingly, in English all pairs where all the participants agreed on a 5-6 similarity score were adjectives 
Multi-SimLex: Translation and Annotation ::: Effect of Language Affinity on Similarity Scores
Based on the analysis in Figure FIGREF20 and inspecting the anecdotal examples in the previous section, it is evident that the correlation between similarity scores across languages is not random  To corroborate this intuition, we visualize the vectors of similarity scores for each single language by reducing their dimensionality to 2 via Principal Component Analysis BIBREF98  The resulting scatter plot in Figure FIGREF26 reveals that languages from the same family or branch have similar patterns in the scores  In particular, Russian and Polish (both Slavic), Finnish and Estonian (both Uralic), Cantonese and Mandarin Chinese (both Sinitic), and Spanish and French (both Romance) are all neighbors 
In order to quantify exactly the effect of language affinity on the similarity scores, we run correlation analyses between these and language features  In particular, we extract feature vectors from URIEL BIBREF99, a massively multilingual typological database that collects and normalizes information compiled by grammarians and field linguists about the world's languages  In particular, we focus on information about geography (the areas where the language speakers are concentrated), family (the phylogenetic tree each language belongs to), and typology (including syntax, phonological inventory, and phonology)  Moreover, we consider typological representations of languages that are not manually crafted by experts, but rather learned from texts  BIBREF100 proposed to construct such representations by training language-identifying vectors end-to-end as part of neural machine translation models 
The vector for similarity judgments and the vector of linguistic features for a given language have different dimensionality  Hence, we first construct a distance matrix for each vector space, such that both columns and rows are language indices, and each cell value is the cosine distance between the vectors of the corresponding language pair  Given a set of L languages, each resulting matrix $S$ has dimensionality of $\mathbb {R}^{|L| \times |L|}$ and is symmetrical  To estimate the correlation between the matrix for similarity judgments and each of the matrices for linguistic features, we run a Mantel test BIBREF101, a non-parametric statistical test based on matrix permutations that takes into account inter-dependencies among pairwise distances 
The results of the Mantel test reported in Table FIGREF26 show that there exist statistically significant correlations between similarity judgments and geography, family, and syntax, given that $p < 0 05$ and $z > 1 96$  The correlation coefficient is particularly strong for geography ($r = 0 647$) and syntax ($r = 0 649$)  The former result is intuitive, because languages in contact easily borrow and loan lexical units, and cultural interactions may result in similar cognitive categorizations  The result for syntax, instead, cannot be explained so easily, as formal properties of language do not affect lexical semantics  Instead, we conjecture that, while no causal relation is present, both syntactic features and similarity judgments might be linked to a common explanatory variable (such as geography)  In fact, several syntactic properties are not uniformly spread across the globe  For instance, verbs with Verb–Object–Subject word order are mostly concentrated in Oceania BIBREF102  In turn, geographical proximity leads to similar judgment patterns, as mentioned above  On the other hand, we find no correlation with phonology and inventory, as expected, nor with the bottom-up typological features from BIBREF100 
Cross-Lingual Multi-SimLex Datasets
A crucial advantage of having semantically aligned monolingual datasets across different languages is the potential to create cross-lingual semantic similarity datasets  Such datasets allow for probing the quality of cross-lingual representation learning algorithms BIBREF27, BIBREF103, BIBREF104, BIBREF105, BIBREF106, BIBREF30, BIBREF107 as an intrinsic evaluation task  However, the cross-lingual datasets previous work relied upon BIBREF27 were limited to a homogeneous set of high-resource languages (e g , English, German, Italian, Spanish) and a small number of concept pairs (all less than 1K pairs)  We address both problems by 1) using a typologically more diverse language sample, and 2) relying on a substantially larger English dataset as a source for the cross-lingual datasets: 1,888 pairs in this work versus 500 pairs in the work of Camacho:2017semeval  As a result, each of our cross-lingual datasets contains a substantially larger number of concept pairs, as shown in Table TABREF30  The cross-lingual Multi-Simlex datasets are constructed automatically, leveraging word pair translations and annotations collected in all 12 languages  This yields a total of 66 cross-lingual datasets, one for each possible combination of languages  Table TABREF30 provides the final number of concept pairs, which lie between 2,031 and 3,480 pairs for each cross-lingual dataset, whereas Table TABREF29 shows some sample pairs with their corresponding similarity scores 
The automatic creation and verification of cross-lingual datasets closely follows the procedure first outlined by Camacho:2015acl and later adopted by Camacho:2017semeval (for semantic similarity) and Vulic:2019acl (for graded lexical entailment)  First, given two languages, we intersect their aligned concept pairs obtained through translation  For instance, starting from the aligned pairs attroupement – foule in French and rahvasumm – rahvahulk in Estonian, we construct two cross-lingual pairs attroupement – rahvaluk and rahvasumm – foule  The scores of cross-lingual pairs are then computed as averages of the two corresponding monolingual scores  Finally, in order to filter out concept pairs whose semantic meaning was not preserved during this operation, we retain only cross-lingual pairs for which the corresponding monolingual scores $(s_s, s_t)$ differ at most by one fifth of the full scale (i e , $\mid s_s - s_t \mid \le 1 2$)  This heuristic mitigates the noise due to cross-lingual semantic shifts BIBREF27, BIBREF97  We refer the reader to the work of Camacho:2015acl for a detailed technical description of the procedure  UTF8gbsn
To assess the quality of the resulting cross-lingual datasets, we have conducted a verification experiment similar to Vulic:2019acl  We randomly sampled 300 concept pairs in the English-Spanish, English-French, and English-Mandarin cross-lingual datasets  Subsequently, we asked bilingual native speakers to provide similarity judgments of each pair  The Spearman's correlation score $\rho $ between automatically induced and manually collected ratings achieves $\rho \ge 0 90$ on all samples, which confirms the viability of the automatic construction procedure 
Score and Class Distributions  The summary of score and class distributions across all 66 cross-lingual datasets are provided in Figure FIGREF31 and Figure FIGREF31, respectively  First, it is obvious that the distribution over the four POS classes largely adheres to that of the original monolingual Multi-SimLex datasets, and that the variance is quite low: e g , the eng-fra dataset contains the lowest proportion of nouns (49 21%) and the highest proportion of verbs (27 1%), adjectives (15 28%), and adverbs (8 41%)  On the other hand, the distribution over similarity intervals in Figure FIGREF31 shows a much greater variance  This is again expected as this pattern resurfaces in monolingual datasets (see Table TABREF19)  It is also evident that the data are skewed towards lower-similarity concept pairs  However, due to the joint size of all cross-lingual datasets (see Table TABREF30), even the least represented intervals contain a substantial number of concept pairs  For instance, the rus-yue dataset contains the least highly similar concept pairs (in the interval $[4,6]$) of all 66 cross-lingual datasets  Nonetheless, the absolute number of pairs (138) in that interval for rus-yue is still substantial  If needed, this makes it possible to create smaller datasets which are balanced across the similarity spectra through sub-sampling 
Monolingual Evaluation of Representation Learning Models
After the numerical and qualitative analyses of the Multi-SimLex datasets provided in §§ SECREF18–SECREF25, we now benchmark a series of representation learning models on the new evaluation data  We evaluate standard static word embedding algorithms such as fastText BIBREF31, as well as a range of more recent text encoders pretrained on language modeling such as multilingual BERT BIBREF29  These experiments provide strong baseline scores on the new Multi-SimLex datasets and offer a first large-scale analysis of pretrained encoders on word-level semantic similarity across diverse languages  In addition, the experiments now enabled by Multi-SimLex aim to answer several important questions  (Q1) Is it viable to extract high-quality word-level representations from pretrained encoders receiving subword-level tokens as input? Are such representations competitive with standard static word-level embeddings? (Q2) What are the implications of monolingual pretraining versus (massively) multilingual pretraining for performance? (Q3) Do lightweight unsupervised post-processing techniques improve word representations consistently across different languages? (Q4) Can we effectively transfer available external lexical knowledge from resource-rich languages to resource-lean languages in order to learn word representations that distinguish between true similarity and conceptual relatedness (see the discussion in §SECREF6)?
Monolingual Evaluation of Representation Learning Models ::: Models in Comparison
Static Word Embeddings in Different Languages  First, we evaluate a standard method for inducing non-contextualized (i e , static) word embeddings across a plethora of different languages: fastText (ft) vectors BIBREF31 are currently the most popular and robust choice given 1) the availability of pretrained vectors in a large number of languages BIBREF108 trained on large Common Crawl (CC) plus Wikipedia (Wiki) data, and 2) their superior performance across a range of NLP tasks BIBREF109  In fact, fastText is an extension of the standard word-level CBOW and skip-gram word2vec models BIBREF32 that takes into account subword-level information, i e  the contituent character n-grams of each word BIBREF110  For this reason, fastText is also more suited for modeling rare words and morphologically rich languages 
We rely on 300-dimensional ft word vectors trained on CC+Wiki and available online for 157 languages  The word vectors for all languages are obtained by CBOW with position-weights, with character n-grams of length 5, a window of size 5, 10 negative examples, and 10 training epochs  We also probe another (older) collection of ft vectors, pretrained on full Wikipedia dumps of each language   The vectors are 300-dimensional, trained with the skip-gram objective for 5 epochs, with 5 negative examples, a window size set to 5, and relying on all character n-grams from length 3 to 6  Following prior work, we trim the vocabularies for all languages to the 200K most frequent words and compute representations for multi-word expressions by averaging the vectors of their constituent words 
Unsupervised Post-Processing  Further, we consider a variety of unsupervised post-processing steps that can be applied post-training on top of any pretrained input word embedding space without any external lexical semantic resource  So far, the usefulness of such methods has been verified only on the English language through benchmarks for lexical semantics and sentence-level tasks BIBREF113  In this paper, we assess if unsupervised post-processing is beneficial also in other languages  To this end, we apply the following post-hoc transformations on the initial word embeddings:
1) Mean centering (mc) is applied after unit length normalization to ensure that all vectors have a zero mean, and is commonly applied in data mining and analysis BIBREF114, BIBREF115 
2) All-but-the-top (abtt) BIBREF113, BIBREF116 eliminates the common mean vector and a few top dominating directions (according to PCA) from the input distributional word vectors, since they do not contribute towards distinguishing the actual semantic meaning of different words  The method contains a single (tunable) hyper-parameter $dd_{A}$ which denotes the number of the dominating directions to remove from the initial representations  Previous work has verified the usefulness of abtt in several English lexical semantic tasks such as semantic similarity, word analogies, and concept categorization, as well as in sentence-level text classification tasks BIBREF113 
3) uncovec BIBREF117 adjusts the similarity order of an arbitrary input word embedding space, and can emphasize either syntactic or semantic information in the transformed vectors  In short, it transforms the input space $\mathbf {X}$ into an adjusted space $\mathbf {X}\mathbf {W}_{\alpha }$ through a linear map $\mathbf {W}_{\alpha }$ controlled by a single hyper-parameter $\alpha $  The $n^{\text{th}}$-order similarity transformation of the input word vector space $\mathbf {X}$ (for which $n=1$) can be obtained as $\mathbf {M}_{n}(\mathbf {X}) = \mathbf {M}_1(\mathbf {X}\mathbf {W}_{(n-1)/2})$, with $\mathbf {W}_{\alpha }=\mathbf {Q}\mathbf {\Gamma }^{\alpha }$, where $\mathbf {Q}$ and $\mathbf {\Gamma }$ are the matrices obtained via eigendecomposition of $\mathbf {X}^T\mathbf {X}=\mathbf {Q}\mathbf {\Gamma }\mathbf {Q}^T$  $\mathbf {\Gamma }$ is a diagonal matrix containing eigenvalues of $\mathbf {X}^T\mathbf {X}$; $\mathbf {Q}$ is an orthogonal matrix with eigenvectors of $\mathbf {X}^T\mathbf {X}$ as columns  While the motivation for the uncovec methods does originate from adjusting discrete similarity orders, note that $\alpha $ is in fact a continuous real-valued hyper-parameter which can be carefully tuned  For more technical details we refer the reader to the original work of BIBREF117 
As mentioned, all post-processing methods can be seen as unsupervised retrofitting methods that, given an arbitrary input vector space $\mathbf {X}$, produce a perturbed/transformed output vector space $\mathbf {X}^{\prime }$, but unlike common retrofitting methods BIBREF62, BIBREF16, the perturbation is completely unsupervised (i e , self-contained) and does not inject any external (semantic similarity oriented) knowledge into the vector space  Note that different perturbations can also be stacked: e g , we can apply uncovec and then use abtt on top the output uncovec vectors  When using uncovec and abtt we always length-normalize and mean-center the data first (i e , we apply the simple mc normalization)  Finally, we tune the two hyper-parameters $d_A$ (for abtt) and $\alpha $ (uncovec) on the English Multi-SimLex and use the same values on the datasets of all other languages; we report results with $dd_A = 3$ or $dd_A = 10$, and $\alpha =-0 3$ 
Contextualized Word Embeddings  We also evaluate the capacity of unsupervised pretraining architectures based on language modeling objectives to reason over lexical semantic similarity  To the best of our knowledge, our article is the first study performing such analyses  State-of-the-art models such as bert BIBREF29, xlm BIBREF30, or roberta BIBREF118 are typically very deep neural networks based on the Transformer architecture BIBREF119  They receive subword-level tokens as inputs (such as WordPieces BIBREF120) to tackle data sparsity  In output, they return contextualized embeddings, dynamic representations for words in context 
To represent words or multi-word expressions through a pretrained model, we follow prior work BIBREF121 and compute an input item's representation by 1) feeding it to a pretrained model in isolation; then 2) averaging the $H$ last hidden representations for each of the item’s constituent subwords; and then finally 3) averaging the resulting subword representations to produce the final $d$-dimensional representation, where $d$ is the embedding and hidden-layer dimensionality (e g , $d=768$ with bert)  We opt for this approach due to its proven viability and simplicity BIBREF121, as it does not require any additional corpora to condition the induction of contextualized embeddings  Other ways to extract the representations from pretrained models BIBREF122, BIBREF123, BIBREF124 are beyond the scope of this work, and we will experiment with them in the future 
In other words, we treat each pretrained encoder enc as a black-box function to encode a single word or a multi-word expression $x$ in each language into a $d$-dimensional contextualized representation $\mathbf {x}_{\textsc {enc}} \in \mathbb {R}^d = \textsc {enc}(x)$ (e g , $d=768$ with bert)  As multilingual pretrained encoders, we experiment with the multilingual bert model (m-bert) BIBREF29 and xlm BIBREF30  m-bert is pretrained on monolingual Wikipedia corpora of 102 languages (comprising all Multi-SimLex languages) with a 12-layer Transformer network, and yields 768-dimensional representations  Since the concept pairs in Multi-SimLex are lowercased, we use the uncased version of m-bert  m-bert comprises all Multi-SimLex languages, and its evident ability to perform cross-lingual transfer BIBREF12, BIBREF125, BIBREF126 also makes it a convenient baseline model for cross-lingual experiments later in §SECREF8  The second multilingual model we consider, xlm-100, is pretrained on Wikipedia dumps of 100 languages, and encodes each concept into a $1,280$-dimensional representation  In contrast to m-bert, xlm-100 drops the next-sentence prediction objective and adds a cross-lingual masked language modeling objective  For both encoders, the representations of each concept are computed as averages over the last $H=4$ hidden layers in all experiments, as suggested by Wu:2019arxiv 
Besides m-bert and xlm, covering multiple languages, we also analyze the performance of “language-specific” bert and xlm models for the languages where they are available: Finnish, Spanish, English, Mandarin Chinese, and French  The main goal of this comparison is to study the differences in performance between multilingual “one-size-fits-all” encoders and language-specific encoders  For all experiments, we rely on the pretrained models released in the Transformers repository BIBREF127 
Unsupervised post-processing steps devised for static word embeddings (i e , mean-centering, abtt, uncovec) can also be applied on top of contextualized embeddings if we predefine a vocabulary of word types $V$ that will be represented in a word vector space $\mathbf {X}$  We construct such $V$ for each language as the intersection of word types covered by the corresponding CC+Wiki fastText vectors and the (single-word or multi-word) expressions appearing in the corresponding Multi-SimLex dataset 
Finally, note that it is not feasible to evaluate a full range of available pretrained encoders within the scope of this work  Our main intention is to provide the first set of baseline results on Multi-SimLex by benchmarking a sample of most popular encoders, at the same time also investigating other important questions such as performance of static versus contextualized word embeddings, or multilingual versus language-specific pretraining  Another purpose of the experiments is to outline the wide potential and applicability of the Multi-SimLex datasets for multilingual and cross-lingual representation learning evaluation 
Monolingual Evaluation of Representation Learning Models ::: Results and Discussion
The results we report are Spearman's $\rho $ coefficients of the correlation between the ranks derived from the scores of the evaluated models and the human scores provided in each Multi-SimLex dataset  The main results with static and contextualized word vectors for all test languages are summarized in Table TABREF43  The scores reveal several interesting patterns, and also pinpoint the main challenges for future work 
State-of-the-Art Representation Models  The absolute scores of CC+Wiki ft, Wiki ft, and m-bert are not directly comparable, because these models have different coverage  In particular, Multi-SimLex contains some out-of-vocabulary (OOV) words whose static ft embeddings are not available  On the other hand, m-bert has perfect coverage  A general comparison between CC+Wiki and Wiki ft vectors, however, supports the intuition that larger corpora (such as CC+Wiki) yield higher correlations  Another finding is that a single massively multilingual model such as m-bert cannot produce semantically rich word-level representations  Whether this actually happens because the training objective is different—or because the need to represent 100+ languages reduces its language-specific capacity—is investigated further below 
The overall results also clearly indicate that (i) there are differences in performance across different monolingual Multi-SimLex datasets, and (ii) unsupervised post-processing is universally useful, and can lead to huge improvements in correlation scores for many languages  In what follows, we also delve deeper into these analyses 
Impact of Unsupervised Post-Processing  First, the results in Table TABREF43 suggest that applying dimension-wise mean centering to the initial vector spaces has positive impact on word similarity scores in all test languages and for all models, both static and contextualized (see the +mc rows in Table TABREF43)  BIBREF128 show that distributional word vectors have a tendency towards narrow clusters in the vector space (i e , they occupy a narrow cone in the vector space and are therefore anisotropic BIBREF113, BIBREF129), and are prone to the undesired effect of hubness BIBREF130, BIBREF131  Applying dimension-wise mean centering has the effect of spreading the vectors across the hyper-plane and mitigating the hubness issue, which consequently improves word-level similarity, as it emerges from the reported results  Previous work has already validated the importance of mean centering for clustering-based tasks BIBREF132, bilingual lexicon induction with cross-lingual word embeddings BIBREF133, BIBREF134, BIBREF135, and for modeling lexical semantic change BIBREF136  However, to the best of our knowledge, the results summarized in Table TABREF43 are the first evidence that also confirms its importance for semantic similarity in a wide array of languages  In sum, as a general rule of thumb, we suggest to always mean-center representations for semantic tasks 
The results further indicate that additional post-processing methods such as abtt and uncovec on top of mean-centered vector spaces can lead to further gains in most languages  The gains are even visible for languages which start from high correlation scores: for instance , cmn with CC+Wiki ft increases from 0 534 to 0 583, from 0 315 to 0 526 with Wiki ft, and from 0 408 to 0 487 with m-bert  Similarly, for rus with CC+Wiki ft we can improve from 0 422 to 0 500, and for fra the scores improve from 0 578 to 0 613  There are additional similar cases reported in Table TABREF43 
Overall, the unsupervised post-processing techniques seem universally useful across languages, but their efficacy and relative performance does vary across different languages  Note that we have not carefully fine-tuned the hyper-parameters of the evaluated post-processing methods, so additional small improvements can be expected for some languages  The main finding, however, is that these post-processing techniques are robust to semantic similarity computations beyond English, and are truly language independent  For instance, removing dominant latent (PCA-based) components from word vectors emphasizes semantic differences between different concepts, as only shared non-informative latent semantic knowledge is removed from the representations 
In summary, pretrained word embeddings do contain more information pertaining to semantic similarity than revealed in the initial vectors  This way, we have corroborated the hypotheses from prior work BIBREF113, BIBREF117 which were not previously empirically verified on other languages due to a shortage of evaluation data; this gap has now been filled with the introduction of the Multi-SimLex datasets  In all follow-up experiments, we always explicitly denote which post-processing configuration is used in evaluation 
POS-Specific Subsets  We present the results for subsets of word pairs grouped by POS class in Table TABREF46  Prior work based on English data showed that representations for nouns are typically of higher quality than those for the other POS classes BIBREF49, BIBREF137, BIBREF50  We observe a similar trend in other languages as well  This pattern is consistent across different representation models and can be attributed to several reasons  First, verb representations need to express a rich range of syntactic and semantic behaviors rather than purely referential features BIBREF138, BIBREF139, BIBREF73  Second, low correlation scores on the adjective and adverb subsets in some languages (e g , pol, cym, swa) might be due to their low frequency in monolingual texts, which yields unreliable representations  In general, the variance in performance across different word classes warrants further research in class-specific representation learning BIBREF140, BIBREF50  The scores further attest the usefulness of unsupervised post-processing as almost all class-specific correlation scores are improved by applying mean-centering and abtt  Finally, the results for m-bert and xlm-100 in Table TABREF46 further confirm that massively multilingual pretraining cannot yield reasonable semantic representations for many languages: in fact, for some classes they display no correlation with human ratings at all 
Differences across Languages  Naturally, the results from Tables TABREF43 and TABREF46 also reveal that there is variation in performance of both static word embeddings and pretrained encoders across different languages  Among other causes, the lowest absolute scores with ft are reported for languages with least resources available to train monolingual word embeddings, such as Kiswahili, Welsh, and Estonian  The low performance on Welsh is especially indicative: Figure FIGREF20 shows that the ratings in the Welsh dataset match up very well with the English ratings, but we cannot achieve the same level of correlation in Welsh with Welsh ft word embeddings  Difference in performance between two closely related languages, est (low-resource) and fin (high-resource), provides additional evidence in this respect 
The highest reported scores with m-bert and xlm-100 are obtained for Mandarin Chinese and Yue Chinese: this effectively points to the weaknesses of massively multilingual training with a joint subword vocabulary spanning 102 and 100 languages  Due to the difference in scripts, “language-specific” subwords for yue and cmn do not need to be shared across a vast amount of languages and the quality of their representation remains unscathed  This effectively means that m-bert's subword vocabulary contains plenty of cmn-specific and yue-specific subwords which are exploited by the encoder when producing m-bert-based representations  Simultaneously, higher scores with m-bert (and xlm in Table TABREF46) are reported for resource-rich languages such as French, Spanish, and English, which are better represented in m-bert's training data  We also observe lower absolute scores (and a larger number of OOVs) for languages with very rich and productive morphological systems such as the two Slavic languages (Polish and Russian) and Finnish  Since Polish and Russian are known to have large Wikipedias and Common Crawl data BIBREF33 (e g , their Wikipedias are in the top 10 largest Wikipedias worldwide), the problem with coverage can be attributed exactly to the proliferation of morphological forms in those languages 
Finally, while Table TABREF43 does reveal that unsupervised post-processing is useful for all languages, it also demonstrates that peak scores are achieved with different post-processing configurations  This finding suggests that a more careful language-specific fine-tuning is indeed needed to refine word embeddings towards semantic similarity  We plan to inspect the relationship between post-processing techniques and linguistic properties in more depth in future work 
Multilingual vs  Language-Specific Contextualized Embeddings  Recent work has shown that—despite the usefulness of massively multilingual models such as m-bert and xlm-100 for zero-shot cross-lingual transfer BIBREF12, BIBREF125—stronger results in downstream tasks for a particular language can be achieved by pretraining language-specific models on language-specific data  In this experiment, motivated by the low results of m-bert and xlm-100 (see again Table TABREF46), we assess if monolingual pretrained encoders can produce higher-quality word-level representations than multilingual models  Therefore, we evaluate language-specific bert and xlm models for a subset of the Multi-SimLex languages for which such models are currently available: Finnish BIBREF141 (bert-base architecture, uncased), French BIBREF142 (the FlauBERT model based on xlm), English (bert-base, uncased), Mandarin Chinese (bert-base) BIBREF29 and Spanish (bert-base, uncased)  In addition, we also evaluate a series of pretrained encoders available for English: (i) bert-base, bert-large, and bert-large with whole word masking (wwm) from the original work on BERT BIBREF29, (ii) monolingual “English-specific” xlm BIBREF30, and (iii) two models which employ parameter reduction techniques to build more compact encoders: albert-b uses a configuration similar to bert-base, while albert-l is similar to bert-large, but with an $18\times $ reduction in the number of parameters BIBREF143 
From the results in Table FIGREF49, it is clear that monolingual pretrained encoders yield much more reliable word-level representations  The gains are visible even for languages such as cmn which showed reasonable performance with m-bert and are substantial on all test languages  This further confirms the validity of language-specific pretraining in lieu of multilingual training, if sufficient monolingual data are available  Moreover, a comparison of pretrained English encoders in Figure FIGREF49 largely follows the intuition: the larger bert-large model yields slight improvements over bert-base, and we can improve a bit more by relying on word-level (i e , lexical-level) masking Finally, light-weight albert model variants are quite competitive with the original bert models, with only modest drops reported, and albert-l again outperforms albert-b  Overall, it is interesting to note that the scores obtained with monolingual pretrained encoders are on a par with or even outperform static ft word embeddings: this is a very intriguing finding per se as it shows that such subword-level models trained on large corpora can implicitly capture rich lexical semantic knowledge 
Similarity-Specialized Word Embeddings  Conflating distinct lexico-semantic relations is a well-known property of distributional representations BIBREF144, BIBREF53  Semantic specialization fine-tunes distributional spaces to emphasize a particular lexico-semantic relation in the transformed space by injecting external lexical knowledge BIBREF145  Explicitly discerning between true semantic similarity (as captured in Multi-SimLex) and broad conceptual relatedness benefits a number of tasks, as discussed in §SECREF4  Since most languages lack dedicated lexical resources, however, one viable strategy to steer monolingual word vector spaces to emphasize semantic similarity is through cross-lingual transfer of lexical knowledge, usually through a shared cross-lingual word vector space BIBREF106  Therefore, we evaluate the effectiveness of specialization transfer methods using Multi-SimLex as our multilingual test bed 
We evaluate a current state-of-the-art cross-lingual specialization transfer method with minimal requirements, put forth recently by Ponti:2019emnlp  In a nutshell, their li-postspec method is a multi-step procedure that operates as follows  First, the knowledge about semantic similarity is extracted from WordNet in the form of triplets, that is, linguistic constraints $(w_1, w_2, r)$, where $w_1$ and $w_2$ are two concepts, and $r$ is a relation between them obtained from WordNet (e g , synonymy or antonymy)  The goal is to “attract” synonyms closer to each other in the transformed vector space as they reflect true semantic similarity, and “repel” antonyms further apart  In the second step, the linguistic constraints are translated from English to the target language via a shared cross-lingual word vector space  To this end, following Ponti:2019emnlp we rely on cross-lingual word embeddings (CLWEs) BIBREF146 available online, which are based on Wiki ft vectors  Following that, a constraint refinement step is applied in the target language which aims to eliminate the noise inserted during the translation process  This is done by training a relation classification tool: it is trained again on the English linguistic constraints and then used on the translated target language constraints, where the transfer is again enabled via a shared cross-lingual word vector space  Finally, a state-of-the-art monolingual specialization procedure from Ponti:2018emnlp injects the (now target language) linguistic constraints into the target language distributional space 
The scores are summarized in Table TABREF56  Semantic specialization with li-postspec leads to substantial improvements in correlation scores for the majority of the target languages, demonstrating the importance of external semantic similarity knowledge for semantic similarity reasoning  However, we also observe deteriorated performance for the three target languages which can be considered the lowest-resource ones in our set: cym, swa, yue  We hypothesize that this occurs due to the inferior quality of the underlying monolingual Wikipedia word embeddings, which generates a chain of error accumulations  In particular, poor distributional word estimates compromise the alignment of the embedding spaces, which in turn results in increased translation noise, and reduced refinement ability of the relation classifier  On a high level, this “poor get poorer” observation again points to the fact that one of the primary causes of low performance of resource-low languages in semantic tasks is the sheer lack of even unlabeled data for distributional training  On the other hand, as we see from Table TABREF46, typological dissimilarity between the source and the target does not deteriorate the effectiveness of semantic specialization  In fact, li-postspec does yield substantial gains also for the typologically distant targets such as heb, cmn, and est  The critical problem indeed seems to be insufficient raw data for monolingual distributional training 
Cross-Lingual Evaluation
Similar to monolingual evaluation in §SECREF7, we now evaluate several state-of-the-art cross-lingual representation models on the suite of 66 automatically constructed cross-lingual Multi-SimLex datasets  Again, note that evaluating a full range of cross-lingual models available in the rich prior work on cross-lingual representation learning is well beyond the scope of this article  We therefore focus our cross-lingual analyses on several well-established and indicative state-of-the-art cross-lingual models, again spanning both static and contextualized cross-lingual word embeddings 
Cross-Lingual Evaluation ::: Models in Comparison
Static Word Embeddings  We rely on a state-of-the-art mapping-based method for the induction of cross-lingual word embeddings (CLWEs): vecmap BIBREF148  The core idea behind such mapping-based or projection-based approaches is to learn a post-hoc alignment of independently trained monolingual word embeddings BIBREF106  Such methods have gained popularity due to their conceptual simplicity and competitive performance coupled with reduced bilingual supervision requirements: they support CLWE induction with only as much as a few thousand word translation pairs as the bilingual supervision BIBREF149, BIBREF150, BIBREF151, BIBREF107  More recent work has shown that CLWEs can be induced with even weaker supervision from small dictionaries spanning several hundred pairs BIBREF152, BIBREF135, identical strings BIBREF153, or even only shared numerals BIBREF154  In the extreme, fully unsupervised projection-based CLWEs extract such seed bilingual lexicons from scratch on the basis of monolingual data only BIBREF103, BIBREF148, BIBREF155, BIBREF156, BIBREF104, BIBREF157 
Recent empirical studies BIBREF158, BIBREF135, BIBREF159 have compared a variety of unsupervised and weakly supervised mapping-based CLWE methods, and vecmap emerged as the most robust and very competitive choice  Therefore, we focus on 1) its fully unsupervised variant (unsuper) in our comparisons  For several language pairs, we also report scores with two other vecmap model variants: 2) a supervised variant which learns a mapping based on an available seed lexicon (super), and 3) a supervised variant with self-learning (super+sl) which iteratively increases the seed lexicon and improves the mapping gradually  For a detailed description of these variants, we refer the reader to recent work BIBREF148, BIBREF135  We again use CC+Wiki ft vectors as initial monolingual word vectors, except for yue where Wiki ft is used  The seed dictionaries of two different sizes (1k and 5k translation pairs) are based on PanLex BIBREF160, and are taken directly from prior work BIBREF135, or extracted from PanLex following the same procedure as in the prior work 
Contextualized Cross-Lingual Word Embeddings  We again evaluate the capacity of (massively) multilingual pretrained language models, m-bert and xlm-100, to reason over cross-lingual lexical similarity  Implicitly, such an evaluation also evaluates “the intrinsic quality” of shared cross-lingual word-level vector spaces induced by these methods, and their ability to boost cross-lingual transfer between different language pairs  We rely on the same procedure of aggregating the models' subword-level parameters into word-level representations, already described in §SECREF34 
As in monolingual settings, we can apply unsupervised post-processing steps such as abtt to both static and contextualized cross-lingual word embeddings 
Cross-Lingual Evaluation ::: Results and Discussion
Main Results and Differences across Language Pairs  A summary of the results on the 66 cross-lingual Multi-SimLex datasets are provided in Table TABREF59 and Figure FIGREF60  The findings confirm several interesting findings from our previous monolingual experiments (§SECREF44), and also corroborate several hypotheses and findings from prior work, now on a large sample of language pairs and for the task of cross-lingual semantic similarity 
First, we observe that the fully unsupervised vecmap model, despite being the most robust fully unsupervised method at present, fails to produce a meaningful cross-lingual word vector space for a large number of language pairs (see the bottom triangle of Table TABREF59): many correlation scores are in fact no-correlation results, accentuating the problem of fully unsupervised cross-lingual learning for typologically diverse languages and with fewer amounts of monolingual data BIBREF135  The scores are particularly low across the board for lower-resource languages such as Welsh and Kiswahili  It also seems that the lack of monolingual data is a larger problem than typological dissimilarity between language pairs, as we do observe reasonably high correlation scores with vecmap for language pairs such as cmn-spa, heb-est, and rus-fin  However, typological differences (e g , morphological richness) still play an important role as we observe very low scores when pairing cmn with morphologically rich languages such fin, est, pol, and rus  Similar to prior work of Vulic:2019we and doval2019onthe, given the fact that unsupervised vecmap is the most robust unsupervised CLWE method at present BIBREF158, our results again question the usefulness of fully unsupervised approaches for a large number of languages, and call for further developments in the area of unsupervised and weakly supervised cross-lingual representation learning 
The scores of m-bert and xlm-100 lead to similar conclusions as in the monolingual settings  Reasonable correlation scores are achieved only for a small subset of resource-rich language pairs (e g , eng, fra, spa, cmn) which dominate the multilingual m-bert training  Interestingly, the scores indicate a much higher performance of language pairs where yue is one of the languages when we use m-bert instead of vecmap  This boils down again to the fact that yue, due to its specific language script, has a good representation of its words and subwords in the shared m-bert vocabulary  At the same time, a reliable vecmap mapping between yue and other languages cannot be found due to a small monolingual yue corpus  In cases when vecmap does not yield a degenerate cross-lingual vector space starting from two monolingual ones, the final correlation scores seem substantially higher than the ones obtained by the single massively multilingual m-bert model 
Finally, the results in Figure FIGREF60 again verify the usefulness of unsupervised post-processing also in cross-lingual settings  We observe improved performance with both m-bert and xlm-100 when mean centering (+mc) is applied, and further gains can be achieved by using abtt on the mean-centered vector spaces  A similar finding also holds for static cross-lingual word embeddings, where applying abbt (-10) yields higher scores on 61/66 language pairs 
Fully Unsupervised vs  Weakly Supervised Cross-Lingual Embeddings  The results in Table TABREF59 indicate that fully unsupervised cross-lingual learning fails for a large number of language pairs  However, recent work BIBREF135 has noted that these sub-optimal non-alignment solutions with the unsuper model can be avoided by relying on (weak) cross-lingual supervision spanning only several thousands or even hundreds of word translation pairs  Therefore, we examine 1) if we can further improve the results on cross-lingual Multi-SimLex resorting to (at least some) cross-lingual supervision for resource-rich language pairs; and 2) if such available word-level supervision can also be useful for a range of languages which displayed near-zero performance in Table TABREF59  In other words, we test if recent “tricks of the trade” used in the rich literature on CLWE learning reflect in gains on cross-lingual Multi-SimLex datasets 
First, we reassess the findings established on the bilingual lexicon induction task BIBREF161, BIBREF135: using at least some cross-lingual supervision is always beneficial compared to using no supervision at all  We report improvements over the unsuper model for all 10 language pairs in Table TABREF66, even though the unsuper method initially produced strong correlation scores  The importance of self-learning increases with decreasing available seed dictionary size, and the +sl model always outperforms unsuper with 1k seed pairs; we observe the same patterns also with even smaller dictionary sizes than reported in Table TABREF66 (250 and 500 seed pairs)  Along the same line, the results in Table TABREF67 indicate that at least some supervision is crucial for the success of static CLWEs on resource-leaner language pairs  We note substantial improvements on all language pairs; in fact, the vecmap model is able to learn a more reliable mapping starting from clean supervision  We again note large gains with self-learning 
Multilingual vs  Bilingual Contextualized Embeddings  Similar to the monolingual settings, we also inspect if massively multilingual training in fact dilutes the knowledge necessary for cross-lingual reasoning on a particular language pair  Therefore, we compare the 100-language xlm-100 model with i) a variant of the same model trained on a smaller set of 17 languages (xlm-17); ii) a variant of the same model trained specifically for the particular language pair (xlm-2); and iii) a variant of the bilingual xlm-2 model that also leverages bilingual knowledge from parallel data during joint training (xlm-2++)  We again use the pretrained models made available by Conneau:2019nips, and we refer to the original work for further technical details 
The results are summarized in Figure FIGREF60, and they confirm the intuition that massively multilingual pretraining can damage performance even on resource-rich languages and language pairs  We observe a steep rise in performance when the multilingual model is trained on a much smaller set of languages (17 versus 100), and further improvements can be achieved by training a dedicated bilingual model  Finally, leveraging bilingual parallel data seems to offer additional slight gains, but a tiny difference between xlm-2 and xlm-2++ also suggests that this rich bilingual information is not used in the optimal way within the xlm architecture for semantic similarity 
In summary, these results indicate that, in order to improve performance in cross-lingual transfer tasks, more work should be invested into 1) pretraining dedicated language pair-specific models, and 2) creative ways of leveraging available cross-lingual supervision (e g , word translation pairs, parallel or comparable corpora) BIBREF121, BIBREF123, BIBREF124 with pretraining paradigms such as bert and xlm  Using such cross-lingual supervision could lead to similar benefits as indicated by the results obtained with static cross-lingual word embeddings (see Table TABREF66 and Table TABREF67)  We believe that Multi-SimLex can serve as a valuable means to track and guide future progress in this research area 
Conclusion and Future Work
We have presented Multi-SimLex, a resource containing human judgments on the semantic similarity of word pairs for 12 monolingual and 66 cross-lingual datasets  The languages covered are typologically diverse and include also under-resourced ones, such as Welsh and Kiswahili  The resource covers an unprecedented amount of 1,888 word pairs, carefully balanced according to their similarity score, frequency, concreteness, part-of-speech class, and lexical field  In addition to Multi-Simlex, we release the detailed protocol we followed to create this resource  We hope that our consistent guidelines will encourage researchers to translate and annotate Multi-Simlex -style datasets for additional languages  This can help and create a hugely valuable, large-scale semantic resource for multilingual NLP research 
The core Multi-SimLex we release with this paper already enables researchers to carry out novel linguistic analysis as well as establishes a benchmark for evaluating representation learning models  Based on our preliminary analyses, we found that speakers of closely related languages tend to express equivalent similarity judgments  In particular, geographical proximity seems to play a greater role than family membership in determining the similarity of judgments across languages  Moreover, we tested several state-of-the-art word embedding models, both static and contextualized representations, as well as several (supervised and unsupervised) post-processing techniques, on the newly released Multi-SimLex  This enables future endeavors to improve multilingual representation learning with challenging baselines  In addition, our results provide several important insights for research on both monolingual and cross-lingual word representations:
1) Unsupervised post-processing techniques (mean centering, elimination of top principal components, adjusting similarity orders) are always beneficial independently of the language, although the combination leading to the best scores is language-specific and hence needs to be tuned 
2) Similarity rankings obtained from word embeddings for nouns are better aligned with human judgments than all the other part-of-speech classes considered here (verbs, adjectives, and, for the first time, adverbs)  This confirms previous generalizations based on experiments on English 
3) The factor having the greatest impact on the quality of word representations is the availability of raw texts to train them in the first place, rather than language properties (such as family, geographical area, typological features) 
4) Massively multilingual pretrained encoders such as m-bert BIBREF29 and xlm-100 BIBREF30 fare quite poorly on our benchmark, whereas pretrained encoders dedicated to a single language are more competitive with static word embeddings such as fastText BIBREF31  Moreover, for language-specific encoders, parameter reduction techniques reduce performance only marginally 
5) Techniques to inject clean lexical semantic knowledge from external resources into distributional word representations were proven to be effective in emphasizing the relation of semantic similarity  In particular, methods capable of transferring such knowledge from resource-rich to resource-lean languages BIBREF59 increased the correlation with human judgments for most languages, except for those with limited unlabelled data 
Future work can expand our preliminary, yet large-scale study on the ability of pretrained encoders to reason over word-level semantic similarity in different languages  For instance, we have highlighted how sharing the same encoder parameters across multiple languages may harm performance  However, it remains unclear if, and to what extent, the input language embeddings present in xlm-100 but absent in m-bert help mitigate this issue  In addition, pretrained language embeddings can be obtained both from typological databases BIBREF99 and from neural architectures BIBREF100  Plugging these embeddings into the encoders in lieu of embeddings trained end-to-end as suggested by prior work BIBREF162, BIBREF163, BIBREF164 might extend the coverage to more resource-lean languages 
Another important follow-up analysis might involve the comparison of the performance of representation learning models on multilingual datasets for both word-level semantic similarity and sentence-level Natural Language Understanding  In particular, Multi-SimLex fills a gap in available resources for multilingual NLP and might help understand how lexical and compositional semantics interact if put alongside existing resources such as XNLI BIBREF84 for natural language inference or PAWS-X BIBREF89 for cross-lingual paraphrase identification  Finally, the Multi-SimLex annotation could turn out to be a unique source of evidence to study the effects of polysemy in human judgments on semantic similarity: for equivalent word pairs in multiple languages, are the similarity scores affected by how many senses the two words (or multi-word expressions) incorporate?
In light of the success of initiatives like Universal Dependencies for multilingual treebanks, we hope that making Multi-SimLex and its guidelines available will encourage other researchers to expand our current sample of languages  We particularly encourage creation and submission of comparable Multi-SimLex datasets for under-resourced and typologically diverse languages in future work  In particular, we have made a Multi-Simlex community website available to facilitate easy creation, gathering, dissemination, and use of annotated datasets: https://multisimlex com/ 
This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909)  Thierry Poibeau is partly supported by a PRAIRIE 3IA Institute fellowship ("Investissements d'avenir" program, reference ANR-19-P3IA-0001) 
Context 2: Introduction
Multi-document summarization (MDS), the transformation of a set of documents into a short text containing their most important aspects, is a long-studied problem in NLP  Generated summaries have been shown to support humans dealing with large document collections in information seeking tasks BIBREF0 , BIBREF1 , BIBREF2   However, when exploring a set of documents manually, humans rarely write a fully-formulated summary for themselves  Instead, user studies BIBREF3 , BIBREF4 show that they note down important keywords and phrases, try to identify relationships between them and organize them accordingly  Therefore, we believe that the study of summarization with similarly structured outputs is an important extension of the traditional task 
A representation that is more in line with observed user behavior is a concept map BIBREF5 , a labeled graph showing concepts as nodes and relationships between them as edges (Figure FIGREF2 )  Introduced in 1972 as a teaching tool BIBREF6 , concept maps have found many applications in education BIBREF7 , BIBREF8 , for writing assistance BIBREF9 or to structure information repositories BIBREF10 , BIBREF11   For summarization, concept maps make it possible to represent a summary concisely and clearly reveal relationships  Moreover, we see a second interesting use case that goes beyond the capabilities of textual summaries: When concepts and relations are linked to corresponding locations in the documents they have been extracted from, the graph can be used to navigate in a document collection, similar to a table of contents  An implementation of this idea has been recently described by BIBREF12  
The corresponding task that we propose is concept-map-based MDS, the summarization of a document cluster in the form of a concept map  In order to develop and evaluate methods for the task, gold-standard corpora are necessary, but no suitable corpus is available  The manual creation of such a dataset is very time-consuming, as the annotation includes many subtasks  In particular, an annotator would need to manually identify all concepts in the documents, while only a few of them will eventually end up in the summary 
To overcome these issues, we present a corpus creation method that effectively combines automatic preprocessing, scalable crowdsourcing and high-quality expert annotations  Using it, we can avoid the high effort for single annotators, allowing us to scale to document clusters that are 15 times larger than in traditional summarization corpora  We created a new corpus of 30 topics, each with around 40 source documents on educational topics and a summarizing concept map that is the consensus of many crowdworkers (see Figure FIGREF3 ) 
As a crucial step of the corpus creation, we developed a new crowdsourcing scheme called low-context importance annotation  In contrast to traditional approaches, it allows us to determine important elements in a document cluster without requiring annotators to read all documents, making it feasible to crowdsource the task and overcome quality issues observed in previous work BIBREF13   We show that the approach creates reliable data for our focused summarization scenario and, when tested on traditional summarization corpora, creates annotations that are similar to those obtained by earlier efforts 
To summarize, we make the following contributions: (1) We propose a novel task, concept-map-based MDS (§ SECREF2 ), (2) present a new crowdsourcing scheme to create reference summaries (§ SECREF4 ), (3) publish a new dataset for the proposed task (§ SECREF5 ) and (4) provide an evaluation protocol and baseline (§ SECREF7 )  We make these resources publicly available under a permissive license 
Task
Concept-map-based MDS is defined as follows: Given a set of related documents, create a concept map that represents its most important content, satisfies a specified size limit and is connected 
We define a concept map as a labeled graph showing concepts as nodes and relationships between them as edges  Labels are arbitrary sequences of tokens taken from the documents, making the summarization task extractive  A concept can be an entity, abstract idea, event or activity, designated by its unique label  Good maps should be propositionally coherent, meaning that every relation together with the two connected concepts form a meaningful proposition 
The task is complex, consisting of several interdependent subtasks  One has to extract appropriate labels for concepts and relations and recognize different expressions that refer to the same concept across multiple documents  Further, one has to select the most important concepts and relations for the summary and finally organize them in a graph satisfying the connectedness and size constraints 
Related Work
Some attempts have been made to automatically construct concept maps from text, working with either single documents BIBREF14 , BIBREF9 , BIBREF15 , BIBREF16 or document clusters BIBREF17 , BIBREF18 , BIBREF19   These approaches extract concept and relation labels from syntactic structures and connect them to build a concept map  However, common task definitions and comparable evaluations are missing  In addition, only a few of them, namely Villalon 2012 and Valerio 2006, define summarization as their goal and try to compress the input to a substantially smaller size  Our newly proposed task and the created large-cluster dataset fill these gaps as they emphasize the summarization aspect of the task 
For the subtask of selecting summary-worthy concepts and relations, techniques developed for traditional summarization BIBREF20 and keyphrase extraction BIBREF21 are related and applicable  Approaches that build graphs of propositions to create a summary BIBREF22 , BIBREF23 , BIBREF24 , BIBREF25 seem to be particularly related, however, there is one important difference: While they use graphs as an intermediate representation from which a textual summary is then generated, the goal of the proposed task is to create a graph that is directly interpretable and useful for a user  In contrast, these intermediate graphs, e g  AMR, are hardly useful for a typical, non-linguist user 
For traditional summarization, the most well-known datasets emerged out of the DUC and TAC competitions  They provide clusters of news articles with gold-standard summaries  Extending these efforts, several more specialized corpora have been created: With regard to size, Nakano 2010 present a corpus of summaries for large-scale collections of web pages  Recently, corpora with more heterogeneous documents have been suggested, e g  BIBREF26 and BIBREF27   The corpus we present combines these aspects, as it has large clusters of heterogeneous documents, and provides a necessary benchmark to evaluate the proposed task 
For concept map generation, one corpus with human-created summary concept maps for student essays has been created BIBREF28   In contrast to our corpus, it only deals with single documents, requires a two orders of magnitude smaller amount of compression of the input and is not publicly available  
Other types of information representation that also model concepts and their relationships are knowledge bases, such as Freebase BIBREF29 , and ontologies  However, they both differ in important aspects: Whereas concept maps follow an open label paradigm and are meant to be interpretable by humans, knowledge bases and ontologies are usually more strictly typed and made to be machine-readable  Moreover, approaches to automatically construct them from text typically try to extract as much information as possible, while we want to summarize a document 
Low-Context Importance Annotation
Lloret 2013 describe several experiments to crowdsource reference summaries  Workers are asked to read 10 documents and then select 10 summary sentences from them for a reward of $0 05  They discovered several challenges, including poor work quality and the subjectiveness of the annotation task, indicating that crowdsourcing is not useful for this purpose 
To overcome these issues, we introduce a new task design, low-context importance annotation, to determine summary-worthy parts of documents  Compared to Lloret et al 's approach, it is more in line with crowdsourcing best practices, as the tasks are simple, intuitive and small BIBREF30 and workers receive reasonable payment BIBREF31   Most importantly, it is also much more efficient and scalable, as it does not require workers to read all documents in a cluster 
Task Design
We break down the task of importance annotation to the level of single propositions  The goal of our crowdsourcing scheme is to obtain a score for each proposition indicating its importance in a document cluster, such that a ranking according to the score would reveal what is most important and should be included in a summary  In contrast to other work, we do not show the documents to the workers at all, but provide only a description of the document cluster's topic along with the propositions  This ensures that tasks are small, simple and can be done quickly (see Figure FIGREF4 ) 
In preliminary tests, we found that this design, despite the minimal context, works reasonably on our focused clusters on common educational topics  For instance, consider Figure FIGREF4 : One can easily say that P1 is more important than P2 without reading the documents 
We distinguish two task variants:
Instead of enforcing binary importance decisions, we use a 5-point Likert-scale to allow more fine-grained annotations  The obtained labels are translated into scores (5  1) and the average of all scores for a proposition is used as an estimate for its importance  This follows the idea that while single workers might find the task subjective, the consensus of multiple workers, represented in the average score, tends to be less subjective due to the “wisdom of the crowd”  We randomly group five propositions into a task 
As an alternative, we use a second task design based on pairwise comparisons  Comparisons are known to be easier to make and more consistent BIBREF32 , but also more expensive, as the number of pairs grows quadratically with the number of objects  To reduce the cost, we group five propositions into a task and ask workers to order them by importance per drag-and-drop  From the results, we derive pairwise comparisons and use TrueSkill BIBREF35 , a powerful Bayesian rank induction model BIBREF34 , to obtain importance estimates for each proposition 
Pilot Study
To verify the proposed approach, we conducted a pilot study on Amazon Mechanical Turk using data from TAC2008 BIBREF36   We collected importance estimates for 474 propositions extracted from the first three clusters using both task designs  Each Likert-scale task was assigned to 5 different workers and awarded $0 06  For comparison tasks, we also collected 5 labels each, paid $0 05 and sampled around 7% of all possible pairs  We submitted them in batches of 100 pairs and selected pairs for subsequent batches based on the confidence of the TrueSkill model 
Following the observations of Lloret 2013, we established several measures for quality control  First, we restricted our tasks to workers from the US with an approval rate of at least 95%  Second, we identified low quality workers by measuring the correlation of each worker's Likert-scores with the average of the other four scores  The worst workers (at most 5% of all labels) were removed 
In addition, we included trap sentences, similar as in BIBREF13 , in around 80 of the tasks  In contrast to Lloret et al 's findings, both an obvious trap sentence (This sentence is not important) and a less obvious but unimportant one (Barack Obama graduated from Harvard Law) were consistently labeled as unimportant (1 08 and 1 14), indicating that the workers did the task properly 
For Likert-scale tasks, we follow Snow 2008 and calculate agreement as the average Pearson correlation of a worker's Likert-score with the average score of the remaining workers  This measure is less strict than exact label agreement and can account for close labels and high- or low-scoring workers  We observe a correlation of 0 81, indicating substantial agreement  For comparisons, the majority agreement is 0 73  To further examine the reliability of the collected data, we followed the approach of Kiritchenko 2016 and simply repeated the crowdsourcing for one of the three topics  Between the importance estimates calculated from the first and second run, we found a Pearson correlation of 0 82 (Spearman 0 78) for Likert-scale tasks and 0 69 (Spearman 0 66) for comparison tasks  This shows that the approach, despite the subjectiveness of the task, allows us to collect reliable annotations 
In addition to the reliability studies, we extrinsically evaluated the annotations in the task of summary evaluation  For each of the 58 peer summaries in TAC2008, we calculated a score as the sum of the importance estimates of the propositions it contains  Table TABREF13 shows how these peer scores, averaged over the three topics, correlate with the manual responsiveness scores assigned during TAC in comparison to ROUGE-2 and Pyramid scores  The results demonstrate that with both task designs, we obtain importance annotations that are similarly useful for summary evaluation as pyramid annotations or gold-standard summaries (used for ROUGE) 
Based on the pilot study, we conclude that the proposed crowdsourcing scheme allows us to obtain proper importance annotations for propositions  As workers are not required to read all documents, the annotation is much more efficient and scalable as with traditional methods 
Corpus Creation
This section presents the corpus construction process, as outlined in Figure FIGREF16 , combining automatic preprocessing, scalable crowdsourcing and high-quality expert annotations to be able to scale to the size of our document clusters  For every topic, we spent about $150 on crowdsourcing and 1 5h of expert annotations, while just a single annotator would already need over 8 hours (at 200 words per minute) to read all documents of a topic
Context 3: 10pt
1 10pt
[ Characterizing Political Fake News in Twitter by its Meta-DataJulio Amador Díaz LópezAxel Oehmichen Miguel Molina-Solana( j amador, axelfrancois oehmichen11, mmolinas@imperial ac uk ) Imperial College London This article presents a preliminary approach towards characterizing political fake news on Twitter through the analysis of their meta-data  In particular, we focus on more than 1 5M tweets collected on the day of the election of Donald Trump as 45th president of the United States of America  We use the meta-data embedded within those tweets in order to look for differences between tweets containing fake news and tweets not containing them  Specifically, we perform our analysis only on tweets that went viral, by studying proxies for users' exposure to the tweets, by characterizing accounts spreading fake news, and by looking at their polarization  We found significant differences on the distribution of followers, the number of URLs on tweets, and the verification of the users 
]
Introduction
While fake news, understood as deliberately misleading pieces of information, have existed since long ago (e g  it is not unusual to receive news falsely claiming the death of a celebrity), the term reached the mainstream, particularly so in politics, during the 2016 presidential election in the United States BIBREF0   Since then, governments and corporations alike (e g  Google BIBREF1 and Facebook BIBREF2 ) have begun efforts to tackle fake news as they can affect political decisions BIBREF3   Yet, the ability to define, identify and stop fake news from spreading is limited 
Since the Obama campaign in 2008, social media has been pervasive in the political arena in the United States  Studies report that up to 62% of American adults receive their news from social media BIBREF4   The wide use of platforms such as Twitter and Facebook has facilitated the diffusion of fake news by simplifying the process of receiving content with no significant third party filtering, fact-checking or editorial judgement  Such characteristics make these platforms suitable means for sharing news that, disguised as legit ones, try to confuse readers 
Such use and their prominent rise has been confirmed by Craig Silverman, a Canadian journalist who is a prominent figure on fake news BIBREF5 : “In the final three months of the US presidential campaign, the top-performing fake election news stories on Facebook generated more engagement than the top stories from major news outlet” 
Our current research hence departs from the assumption that social media is a conduit for fake news and asks the question of whether fake news (as spam was some years ago) can be identified, modelled and eventually blocked  In order to do so, we use a sample of more that 1 5M tweets collected on November 8th 2016 —election day in the United States— with the goal of identifying features that tweets containing fake news are likely to have  As such, our paper aims to provide a preliminary characterization of fake news in Twitter by looking into meta-data embedded in tweets  Considering meta-data as a relevant factor of analysis is in line with findings reported by Morris et al  BIBREF6   We argue that understanding differences between tweets containing fake news and regular tweets will allow researchers to design mechanisms to block fake news in Twitter 
Specifically, our goals are: 1) compare the characteristics of tweets labelled as containing fake news to tweets labelled as not containing them, 2) characterize, through their meta-data, viral tweets containing fake news and the accounts from which they originated, and 3) determine the extent to which tweets containing fake news expressed polarized political views 
For our study, we used the number of retweets to single-out those that went viral within our sample  Tweets within that subset (viral tweets hereafter) are varied and relate to different topics  We consider that a tweet contains fake news if its text falls within any of the following categories described by Rubin et al  BIBREF7 (see next section for the details of such categories): serious fabrication, large-scale hoaxes, jokes taken at face value, slanted reporting of real facts and stories where the truth is contentious  The dataset BIBREF8 , manually labelled by an expert, has been publicly released and is available to researchers and interested parties 
From our results, the following main observations can be made:
Our findings resonate with similar work done on fake news such as the one from Allcot and Gentzkow BIBREF9   Therefore, even if our study is a preliminary attempt at characterizing fake news on Twitter using only their meta-data, our results provide external validity to previous research  Moreover, our work not only stresses the importance of using meta-data, but also underscores which parameters may be useful to identify fake news on Twitter 
The rest of the paper is organized as follows  The next section briefly discusses where this work is located within the literature on fake news and contextualizes the type of fake news we are studying  Then, we present our hypotheses, the data, and the methodology we follow  Finally, we present our findings, conclusions of this study, and future lines of work 
Defining Fake news
Our research is connected to different strands of academic knowledge related to the phenomenon of fake news  In relation to Computer Science, a recent survey by Conroy and colleagues BIBREF10 identifies two popular approaches to single-out fake news  On the one hand, the authors pointed to linguistic approaches consisting in using text, its linguistic characteristics and machine learning techniques to automatically flag fake news  On the other, these researchers underscored the use of network approaches, which make use of network characteristics and meta-data, to identify fake news 
With respect to social sciences, efforts from psychology, political science and sociology, have been dedicated to understand why people consume and/or believe misinformation BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14   Most of these studies consistently reported that psychological biases such as priming effects and confirmation bias play an important role in people ability to discern misinformation 
In relation to the production and distribution of fake news, a recent paper in the field of Economics BIBREF9 found that most fake news sites use names that resemble those of legitimate organizations, and that sites supplying fake news tend to be short-lived

ground_truth: Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese

Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What type of evaluation is proposed for this task?
Context 2: 
Source Data
As a starting point, we used the DIP corpus BIBREF37 , a collection of 49 clusters of 100 web pages on educational topics (e g  bullying, homeschooling, drugs) with a short description of each topic  It was created from a large web crawl using state-of-the-art information retrieval  We selected 30 of the topics for which we created the necessary concept map annotations 
Proposition Extraction
As concept maps consist of propositions expressing the relation between concepts (see Figure FIGREF2 ), we need to impose such a structure upon the plain text in the document clusters  This could be done by manually annotating spans representing concepts and relations, however, the size of our clusters makes this a huge effort: 2288 sentences per topic (69k in total) need to be processed  Therefore, we resort to an automatic approach 
The Open Information Extraction paradigm BIBREF38 offers a representation very similar to the desired one  For instance, from
Students with bad credit history should not lose hope and apply for federal loans with the FAFSA 
Open IE systems extract tuples of two arguments and a relation phrase representing propositions:
(s  with bad credit history, should not lose, hope)
(s  with bad credit history, apply for, federal loans with the FAFSA)
While the relation phrase is similar to a relation in a concept map, many arguments in these tuples represent useful concepts  We used Open IE 4, a state-of-the-art system BIBREF39 to process all sentences  After removing duplicates, we obtained 4137 tuples per topic 
Since we want to create a gold-standard corpus, we have to ensure that we produce high-quality data  We therefore made use of the confidence assigned to every extracted tuple to filter out low quality ones  To ensure that we do not filter too aggressively (and miss important aspects in the final summary), we manually annotated 500 tuples sampled from all topics for correctness  On the first 250 of them, we tuned the filter threshold to 0 5, which keeps 98 7% of the correct extractions in the unseen second half  After filtering, a topic had on average 2850 propositions (85k in total) 
Proposition Filtering
Despite the similarity of the Open IE paradigm, not every extracted tuple is a suitable proposition for a concept map  To reduce the effort in the subsequent steps, we therefore want to filter out unsuitable ones  A tuple is suitable if it (1) is a correct extraction, (2) is meaningful without any context and (3) has arguments that represent proper concepts  We created a guideline explaining when to label a tuple as suitable for a concept map and performed a small annotation study  Three annotators independently labeled 500 randomly sampled tuples  The agreement was 82% ( INLINEFORM0 )  We found tuples to be unsuitable mostly because they had unresolvable pronouns, conflicting with (2), or arguments that were full clauses or propositions, conflicting with (3), while (1) was mostly taken care of by the confidence filtering in § SECREF21  
Due to the high number of tuples we decided to automate the filtering step  We trained a linear SVM on the majority voted annotations  As features, we used the extraction confidence, length of arguments and relations as well as part-of-speech tags, among others  To ensure that the automatic classification does not remove suitable propositions, we tuned the classifier to avoid false negatives  In particular, we introduced class weights, improving precision on the negative class at the cost of a higher fraction of positive classifications  Additionally, we manually verified a certain number of the most uncertain negative classifications to further improve performance  When 20% of the classifications are manually verified and corrected, we found that our model trained on 350 labeled instances achieves 93% precision on negative classifications on the unseen 150 instances  We found this to be a reasonable trade-off of automation and data quality and applied the model to the full dataset 
The classifier filtered out 43% of the propositions, leaving 1622 per topic  We manually examined the 17k least confident negative classifications and corrected 955 of them  We also corrected positive classifications for certain types of tuples for which we knew the classifier to be imprecise  Finally, each topic was left with an average of 1554 propositions (47k in total) 
Importance Annotation
Given the propositions identified in the previous step, we now applied our crowdsourcing scheme as described in § SECREF4 to determine their importance  To cope with the large number of propositions, we combine the two task designs: First, we collect Likert-scores from 5 workers for each proposition, clean the data and calculate average scores  Then, using only the top 100 propositions according to these scores, we crowdsource 10% of all possible pairwise comparisons among them  Using TrueSkill, we obtain a fine-grained ranking of the 100 most important propositions 
For Likert-scores, the average agreement over all topics is 0 80, while the majority agreement for comparisons is 0 78  We repeated the data collection for three randomly selected topics and found the Pearson correlation between both runs to be 0 73 (Spearman 0 73) for Likert-scores and 0 72 (Spearman 0 71) for comparisons  These figures show that the crowdsourcing approach works on this dataset as reliably as on the TAC documents 
In total, we uploaded 53k scoring and 12k comparison tasks to Mechanical Turk, spending $4425 45 including fees  From the fine-grained ranking of the 100 most important propositions, we select the top 50 per topic to construct a summary concept map in the subsequent steps 
Proposition Revision
Having a manageable number of propositions, an annotator then applied a few straightforward transformations that correct common errors of the Open IE system  First, we break down propositions with conjunctions in either of the arguments into separate propositions per conjunct, which the Open IE system sometimes fails to do  And second, we correct span errors that might occur in the argument or relation phrases, especially when sentences were not properly segmented  As a result, we have a set of high quality propositions for our concept map, consisting of, due to the first transformation, 56 1 propositions per topic on average 
Concept Map Construction
In this final step, we connect the set of important propositions to form a graph  For instance, given the following two propositions
(student, may borrow, Stafford Loan)
(the student, does not have, a credit history)
one can easily see, although the first arguments differ slightly, that both labels describe the concept student, allowing us to build a concept map with the concepts student, Stafford Loan and credit history  The annotation task thus involves deciding which of the available propositions to include in the map, which of their concepts to merge and, when merging, which of the available labels to use  As these decisions highly depend upon each other and require context, we decided to use expert annotators rather than crowdsource the subtasks 
Annotators were given the topic description and the most important, ranked propositions  Using a simple annotation tool providing a visualization of the graph, they could connect the propositions step by step  They were instructed to reach a size of 25 concepts, the recommended maximum size for a concept map BIBREF6   Further, they should prefer more important propositions and ensure connectedness  When connecting two propositions, they were asked to keep the concept label that was appropriate for both propositions  To support the annotators, the tool used ADW BIBREF40 , a state-of-the-art approach for semantic similarity, to suggest possible connections  The annotation was carried out by graduate students with a background in NLP after receiving an introduction into the guidelines and tool and annotating a first example 
If an annotator was not able to connect 25 concepts, she was allowed to create up to three synthetic relations with freely defined labels, making the maps slightly abstractive  On average, the constructed maps have 0 77 synthetic relations, mostly connecting concepts whose relation is too obvious to be explicitly stated in text (e g  between Montessori teacher and Montessori education) 
To assess the reliability of this annotation step, we had the first three maps created by two annotators  We casted the task of selecting propositions to be included in the map as a binary decision task and observed an agreement of 84% ( INLINEFORM0 )  Second, we modeled the decision which concepts to join as a binary decision on all pairs of common concepts, observing an agreement of 95% ( INLINEFORM1 )  And finally, we compared which concept labels the annotators decided to include in the final map, observing 85% ( INLINEFORM2 ) agreement  Hence, the annotation shows substantial agreement BIBREF41  
Corpus Analysis
In this section, we describe our newly created corpus, which, in addition to having summaries in the form of concept maps, differs from traditional summarization corpora in several aspects 
Document Clusters
The corpus consists of document clusters for 30 different topics  Each of them contains around 40 documents with on average 2413 tokens, which leads to an average cluster size of 97,880 token  With these characteristics, the document clusters are 15 times larger than typical DUC clusters of ten documents and five times larger than the 25-document-clusters (Table TABREF26 )  In addition, the documents are also more variable in terms of length, as the (length-adjusted) standard deviation is twice as high as in the other corpora  With these properties, the corpus represents an interesting challenge towards real-world application scenarios, in which users typically have to deal with much more than ten documents 
Because we used a large web crawl as the source for our corpus, it contains documents from a variety of genres  To further analyze this property, we categorized a sample of 50 documents from the corpus  Among them, we found professionally written articles and blog posts (28%), educational material for parents and kids (26%), personal blog posts (16%), forum discussions and comments (12%), commented link collections (12%) and scientific articles (6%) 
In addition to the variety of genres, the documents also differ in terms of language use  To capture this property, we follow Zopf 2016 and compute, for every topic, the average Jensen-Shannon divergence between the word distribution of one document and the word distribution in the remaining documents  The higher this value is, the more the language differs between documents  We found the average divergence over all topics to be 0 3490, whereas it is 0 3019 in DUC 2004 and 0 3188 in TAC 2008A 
Concept Maps
As Table TABREF33 shows, each of the 30 reference concept maps has exactly 25 concepts and between 24 and 28 relations  Labels for both concepts and relations consist on average of 3 2 tokens, whereas the latter are a bit shorter in characters 
To obtain a better picture of what kind of text spans have been used as labels, we automatically tagged them with their part-of-speech and determined their head with a dependency parser  Concept labels tend to be headed by nouns (82%) or verbs (15%), while they also contain adjectives, prepositions and determiners  Relation labels, on the other hand, are almost always headed by a verb (94%) and contain prepositions, nouns and particles in addition  These distributions are very similar to those reported by Villalon 2010 for their (single-document) concept map corpus 
Analyzing the graph structure of the maps, we found that all of them are connected  They have on average 7 2 central concepts with more than one relation, while the remaining ones occur in only one proposition  We found that achieving a higher number of connections would mean compromising importance, i e  including less important propositions, and decided against it 
Baseline Experiments
In this section, we briefly describe a baseline and evaluation scripts that we release, with a detailed documentation, along with the corpus 
Conclusion
In this work, we presented low-context importance annotation, a novel crowdsourcing scheme that we used to create a new benchmark corpus for concept-map-based MDS  The corpus has large-scale document clusters of heterogeneous web documents, posing a challenging summarization task  Together with the corpus, we provide implementations of a baseline method and evaluation scripts and hope that our efforts facilitate future research on this variant of summarization 
Acknowledgments
We would like to thank Teresa Botschen, Andreas Hanselowski and Markus Zopf for their help with the annotation work and Christian Meyer for his valuable feedback  This work has been supported by the German Research Foundation as part of the Research Training Group “Adaptive Preparation of Information from Heterogeneous Sources” (AIPHES) under grant No  GRK 1994/1 
Context 3:  These authors also noticed that fake news items are more likely shared than legitimate articles coming from trusted sources, and they tend to exhibit a larger level of polarization 
The conceptual issue of how to define fake news is a serious and unresolved issue  As the focus of our work is not attempting to offer light on this, we will rely on work by other authors to describe what we consider as fake news  In particular, we use the categorization provided by Rubin et al  BIBREF7   The five categories they described, together with illustrative examples from our dataset, are as follows:
Research Hypotheses
Previous works on the area (presented in the section above) suggest that there may be important determinants for the adoption and diffusion of fake news  Our hypotheses builds on them and identifies three important dimensions that may help distinguishing fake news from legit information:
Taking those three dimensions into account, we propose the following hypotheses about the features that we believe can help to identify tweets containing fake news from those not containing them  They will be later tested over our collected dataset 
Exposure 
Characterization 
Polarization 
Data and Methodology
For this study, we collected publicly available tweets using Twitter's public API  Given the nature of the data, it is important to emphasize that such tweets are subject to Twitter's terms and conditions which indicate that users consent to the collection, transfer, manipulation, storage, and disclosure of data  Therefore, we do not expect ethical, legal, or social implications from the usage of the tweets  Our data was collected using search terms related to the presidential election held in the United States on November 8th 2016  Particularly, we queried Twitter's streaming API, more precisely the filter endpoint of the streaming API, using the following hashtags and user handles: #MyVote2016, #ElectionDay, #electionnight, @realDonaldTrump and @HillaryClinton  The data collection ran for just one day (Nov 8th 2016) 
One straightforward way of sharing information on Twitter is by using the retweet functionality, which enables a user to share a exact copy of a tweet with his followers  Among the reasons for retweeting, Body et al  BIBREF15 reported the will to: 1) spread tweets to a new audience, 2) to show one’s role as a listener, and 3) to agree with someone or validate the thoughts of others  As indicated, our initial interest is to characterize tweets containing fake news that went viral (as they are the most harmful ones, as they reach a wider audience), and understand how it differs from other viral tweets (that do not contain fake news)  For our study, we consider that a tweet went viral if it was retweeted more than 1000 times 
Once we have the dataset of viral tweets, we eliminated duplicates (some of the tweets were collected several times because they had several handles) and an expert manually inspected the text field within the tweets to label them as containing fake news, or not containing them (according to the characterization presented before)  This annotated dataset BIBREF8 is publicly available and can be freely reused 
Finally, we use the following fields within tweets (from the ones returned by Twitter's API) to compare their distributions and look for differences between viral tweets containing fake news and viral tweets not containing fake news:
In the following section, we provide graphical descriptions of the distribution of each of the identified attributes for the two sets of tweets (those labelled as containing fake news and those labelled as not containing them)  Where appropriate, we normalized and/or took logarithms of the data for better representation  To gain a better understanding of the significance of those differences, we use the Kolmogorov-Smirnov test with the null hypothesis that both distributions are equal 
Results
The sample collected consisted on 1 785 855 tweets published by 848 196 different users  Within our sample, we identified 1327 tweets that went viral (retweeted more than 1000 times by the 8th of November 2016) produced by 643 users  Such small subset of viral tweets were retweeted on 290 841 occasions in the observed time-window 
The 1327 `viral' tweets were manually annotated as containing fake news or not  The annotation was carried out by a single person in order to obtain a consistent annotation throughout the dataset  Out of those 1327 tweets, we identified 136 as potentially containing fake news (according to the categories previously described), and the rest were classified as `non containing fake news'  Note that the categorization is far from being perfect given the ambiguity of fake news themselves and human judgement involved in the process of categorization  Because of this, we do not claim that this dataset can be considered a ground truth 
The following results detail characteristics of these tweets along the previously mentioned dimensions  Table TABREF23 reports the actual differences (together with their associated p-values) of the distributions of viral tweets containing fake news and viral tweets not containing them for every variable considered 
Exposure
Figure FIGREF24 shows that, in contrast to other kinds of viral tweets, those containing fake news were created more recently  As such, Twitter users were exposed to fake news related to the election for a shorter period of time 
However, in terms of retweets, Figure FIGREF25 shows no apparent difference between containing fake news or not containing them  That is confirmed by the Kolmogorov-Smirnoff test, which does not discard the hypothesis that the associated distributions are equal 
In relation to the number of favourites, users that generated at least a viral tweet containing fake news appear to have, on average, less favourites than users that do not generate them  Figure FIGREF26 shows the distribution of favourites  Despite the apparent visual differences, the difference are not statistically significant 
Finally, the number of hashtags used in viral fake news appears to be larger than those in other viral tweets  Figure FIGREF27 shows the density distribution of the number of hashtags used  However, once again, we were not able to find any statistical difference between the average number of hashtags in a viral tweet and the average number of hashtags in viral fake news 
Characterization
We found that 82 users within our sample were spreading fake news (i e  they produced at least one tweet which was labelled as fake news)

ground_truth: Answer with content missing: (Evaluation Metrics section) Precision, Recall, F1-scores, Strict match, METEOR, ROUGE-2

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How is the ground truth for fake news established?
Context 3:  Out of those, 34 had verified accounts, and the rest were unverified  From the 48 unverified accounts, 6 have been suspended by Twitter at the date of writing, 3 tried to imitate legitimate accounts of others, and 4 accounts have been already deleted  Figure FIGREF28 shows the proportion of verified accounts to unverified accounts for viral tweets (containing fake news vs  not containing fake news)  From the chart, it is clear that there is a higher chance of fake news coming from unverified accounts 
Turning to friends, accounts distributing fake news appear to have, on average, the same number of friends than those distributing tweets with no fake news  However, the density distribution of friends from the accounts (Figure FIGREF29 ) shows that there is indeed a statistically significant difference in their distributions 
If we take into consideration the number of followers, accounts generating viral tweets with fake news do have a very different distribution on this dimension, compared to those accounts generating viral tweets with no fake news (see Figure FIGREF30 )  In fact, such differences are statistically significant 
A useful representation for friends and followers is the ratio between friends/followers  Figures FIGREF31 and FIGREF32 show this representation  Notice that accounts spreading viral tweets with fake news have, on average, a larger ratio of friends/followers  The distribution of those accounts not generating fake news is more evenly distributed 
With respect to the number of mentions, Figure FIGREF33 shows that viral tweets labelled as containing fake news appear to use mentions to other users less frequently than viral tweets not containing fake news  In other words, tweets containing fake news mostly contain 1 mention, whereas other tweets tend to have two)  Such differences are statistically significant 
The analysis (Figure FIGREF34 ) of the presence of media in the tweets in our dataset shows that tweets labelled as not containing fake news appear to present more media elements than those labelled as fake news  However, the difference is not statistically significant 
On the other hand, Figure FIGREF35 shows that viral tweets containing fake news appear to include more URLs to other sites than viral tweets that do not contain fake news  In fact, the difference between the two distributions is statistically significant (assuming INLINEFORM0 ) 
Polarization
Finally, manual inspection of the text field of those viral tweets labelled as containing fake news shows that 117 of such tweets expressed support for Donald Trump, while only 8 supported Hillary Clinton  The remaining tweets contained fake news related to other topics, not expressing support for any of the candidates 
Discussion
As a summary, and constrained by our existing dataset, we made the following observations regarding differences between viral tweets labelled as containing fake news and viral tweets labelled as not containing them:
These findings (related to our initial hypothesis in Table TABREF44 ) clearly suggest that there are specific pieces of meta-data about tweets that may allow the identification of fake news  One such parameter is the time of exposure  Viral tweets containing fake news are shorter-lived than those containing other type of content  This notion seems to resonate with our findings showing that a number of accounts spreading fake news have already been deleted or suspended by Twitter by the time of writing  If one considers that researchers using different data have found similar results BIBREF9 , it appears that the lifetime of accounts, together with the age of the questioned viral content could be useful to identify fake news  In the light of this finding, accounts newly created should probably put under higher scrutiny than older ones  This in fact, would be a nice a-priori bias for a Bayesian classifier 
Accounts spreading fake news appear to have a larger proportion of friends/followers (i e  they have, on average, the same number of friends but a smaller number of followers) than those spreading viral content only  Together with the fact that, on average, tweets containing fake news have more URLs than those spreading viral content, it is possible to hypothesize that, both, the ratio of friends/followers of the account producing a viral tweet and number of URLs contained in such a tweet could be useful to single-out fake news in Twitter  Not only that, but our finding related to the number of URLs is in line with intuitions behind the incentives to create fake news commonly found in the literature BIBREF9 (in particular that of obtaining revenue through click-through advertising) 
Finally, it is interesting to notice that the content of viral fake news was highly polarized  This finding is also in line with those of Alcott et al  BIBREF9   This feature suggests that textual sentiment analysis of the content of tweets (as most researchers do), together with the above mentioned parameters from meta-data, may prove useful for identifying fake news 
Conclusions
With the election of Donald Trump as President of the United States, the concept of fake news has become a broadly-known phenomenon that is getting tremendous attention from governments and media companies  We have presented a preliminary study on the meta-data of a publicly available dataset of tweets that became viral during the day of the 2016 US presidential election  Our aim is to advance the understanding of which features might be characteristic of viral tweets containing fake news in comparison with viral tweets without fake news 
We believe that the only way to automatically identify those deceitful tweets (i e  containing fake news) is by actually understanding and modelling them  Only then, the automation of the processes of tagging and blocking these tweets can be successfully performed  In the same way that spam was fought, we anticipate fake news will suffer a similar evolution, with social platforms implementing tools to deal with them  With most works so far focusing on the actual content of the tweets, ours is a novel attempt from a different, but also complementary, angle 
Within the used dataset, we found there are differences around exposure, characteristics of accounts spreading fake news and the tone of the content  Those findings suggest that it is indeed possible to model and automatically detect fake news  We plan to replicate and validate our experiments in an extended sample of tweets (until 4 months after the US election), and tests the predictive power of the features we found relevant within our sample 
Author Disclosure Statement
No competing financial interest exist 

ground_truth: Ground truth is not established in the paper