prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Does their NER model learn NER from both text and images?
Context 1: Introduction
Social media with abundant user-generated posts provide a rich platform for understanding events, opinions and preferences of groups and individuals  These insights are primarily hidden in unstructured forms of social media posts, such as in free-form text or images without tags  Named entity recognition (NER), the task of recognizing named entities from free-form text, is thus a critical step for building structural information, allowing for its use in personalized assistance, recommendations, advertisement, etc 
While many previous approaches BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 on NER have shown success for well-formed text in recognizing named entities via word context resolution (e g  LSTM with word embeddings) combined with character-level features (e g  CharLSTM/CNN), several additional challenges remain for recognizing named entities from extremely short and coarse text found in social media posts  For instance, short social media posts often do not provide enough textual contexts to resolve polysemous entities (e g  “monopoly is da best ", where `monopoly' may refer to a board game (named entity) or a term in economics)  In addition, noisy text includes a huge number of unknown tokens due to inconsistent lexical notations and frequent mentions of various newly trending entities (e g  “xoxo Marshmelloooo ", where `Marshmelloooo' is a mis-spelling of a known entity `Marshmello', a music producer), making word embeddings based neural networks NER models vulnerable 
To address the challenges above for social media posts, we build upon the state-of-the-art neural architecture for NER with the following two novel approaches (Figure FIGREF1 )  First, we propose to leverage auxiliary modalities for additional context resolution of entities  For example, many popular social media platforms now provide ways to compose a post in multiple modalities - specifically image and text (e g  Snapchat captions, Twitter posts with image URLs), from which we can obtain additional context for understanding posts  While “monopoly" in the previous example is ambiguous in its textual form, an accompanying snap image of a board game can help disambiguate among polysemous entities, thereby correctly recognizing it as a named entity 
Second, we also propose a general modality attention module which chooses per decoding step the most informative modality among available ones (in our case, word embeddings, character embeddings, or visual features) to extract context from  For example, the modality attention module lets the decoder attenuate the word-level signals for unknown word tokens (“Marshmellooooo" with trailing `o's) and amplifies character-level features intsead (capitalized first letter, lexical similarity to other known named entity token `Marshmello', etc ), thereby suppressing noise information (“UNK" token embedding) in decoding steps  Note that most of the previous literature in NER or other NLP tasks combine word and character-level information with naive concatenation, which is vulnerable to noisy social media posts  When an auxiliary image is available, the modality attention module determines to amplify this visual context in disambiguating polysemous entities, or to attenuate visual contexts when they are irrelevant to target named entities, selfies, etc  Note that the proposed modality attention module is distinct from how attention is used in other sequence-to-sequence literature (e g  attending to a specific token within an input sequence)  Section SECREF2 provides the detailed literature review 
Our contributions are three-fold: we propose (1) an LSTM-CNN hybrid multimodal NER network that takes as input both image and text for recognition of a named entity in text input  To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks  (2) We propose a general modality attention module that selectively chooses modalities to extract primary context from, maximizing information gain and suppressing irrelevant contexts from each modality (we treat words, characters, and images as separate modalities)  (3) We show that the proposed approaches outperform the state-of-the-art NER models (both with and without using additional visual contexts) on our new MNER dataset SnapCaptions, a large collection of informal and extremely short social media posts paired with unique images 
Related Work
Neural models for NER have been recently proposed, producing state-of-the-art performance on standard NER tasks  For example, some of the end-to-end NER systems BIBREF4 , BIBREF2 , BIBREF3 , BIBREF0 , BIBREF1 use a recurrent neural network usually with a CRF BIBREF5 , BIBREF6 for sequence labeling, accompanied with feature extractors for words and characters (CNN, LSTMs, etc ), and achieve the state-of-the-art performance mostly without any use of gazetteers information  Note that most of these work aggregate textual contexts via concatenation of word embeddings and character embeddings  Recently, several work have addressed the NER task specifically on noisy short text segments such as Tweets, etc  BIBREF7 , BIBREF8   They report performance gains from leveraging external sources of information such as lexical information (POS tags, etc ) and/or from several preprocessing steps (token substitution, etc )  Our model builds upon these state-of-the-art neural models for NER tasks, and improves the model in two critical ways: (1) incorporation of visual contexts to provide auxiliary information for short media posts, and (2) addition of the modality attention module, which better incorporates word embeddings and character embeddings, especially when there are many missing tokens in the given word embedding matrix  Note that we do not explore the use of gazetteers information or other auxiliary information (POS tags, etc ) BIBREF9 as it is not the focus of our study 
Attention modules are widely applied in several deep learning tasks BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13   For example, they use an attention module to attend to a subset within a single input (a part/region of an image, a specific token in an input sequence of tokens, etc ) at each decoding step in an encoder-decoder framework for image captioning tasks, etc  BIBREF14 explore various attention mechanisms in NLP tasks, but do not incorporate visual components or investigate the impact of such models on noisy social media data  BIBREF15 propose to use attention for a subset of discrete source samples in transfer learning settings  Our modality attention differs from the previous approaches in that we attenuate or amplifies each modality input as a whole among multiple available modalities, and that we use the attention mechanism essentially to map heterogeneous modalities in a single joint embedding space  Our approach also allows for re-use of the same model for predicting labels even when some of the modalities are missing in input, as other modalities would still preserve the same semantics in the embeddings space 
Multimodal learning is studied in various domains and applications, aimed at building a joint model that extracts contextual information from multiple modalities (views) of parallel datasets 
The most relevant task to our multimodal NER system is the task of multimodal machine translation BIBREF16 , BIBREF17 , which aims at building a better machine translation system by taking as input a sentence in a source language as well as a corresponding image  Several standard sequence-to-sequence architectures are explored (a target-language LSTM decoder that takes as input an image first)  Other previous literature include study of Canonical Correlation Analysis (CCA) BIBREF18 to learn feature correlations among multiple modalities, which is widely used in many applications  Other applications include image captioning BIBREF10 , audio-visual recognition BIBREF19 , visual question answering systems BIBREF20 , etc 
To the best of our knowledge, our approach is the first work to incorporate visual contexts for named entity recognition tasks 
Proposed Methods
Figure FIGREF2 illustrates the proposed multimodal NER (MNER) model  First, we obtain word embeddings, character embeddings, and visual features (Section SECREF3 )  A Bi-LSTM-CRF model then takes as input a sequence of tokens, each of which comprises a word token, a character sequence, and an image, in their respective representation (Section SECREF4 )  At each decoding step, representations from each modality are combined via the modality attention module to produce an entity label for each token ( SECREF5 )  We formulate each component of the model in the following subsections 
Notations: Let INLINEFORM0 a sequence of input tokens with length INLINEFORM1 , with a corresponding label sequence INLINEFORM2 indicating named entities (e g  in standard BIO formats)  Each input token is composed of three modalities: INLINEFORM3 for word embeddings, character embeddings, and visual embeddings representations, respectively 
Features
Similar to the state-of-the-art NER approaches BIBREF0 , BIBREF1 , BIBREF8 , BIBREF4 , BIBREF2 , BIBREF3 , we use both word embeddings and character embeddings 
Word embeddings are obtained from an unsupervised learning model that learns co-occurrence statistics of words from a large external corpus, yielding word embeddings as distributional semantics BIBREF21   Specifically, we use pre-trained embeddings from GloVE BIBREF22  
Character embeddings are obtained from a Bi-LSTM which takes as input a sequence of characters of each token, similarly to BIBREF0   An alternative approach for obtaining character embeddings is using a convolutional neural network as in BIBREF1 , but we find that Bi-LSTM representation of characters yields empirically better results in our experiments 
Visual embeddings: To extract features from an image, we take the final hidden layer representation of a modified version of the convolutional network model called Inception (GoogLeNet) BIBREF23 , BIBREF24 trained on the ImageNet dataset BIBREF25 to classify multiple objects in the scene  Our implementation of the Inception model has deep 22 layers, training of which is made possible via “network in network" principles and several dimension reduction techniques to improve computing resource utilization  The final layer representation encodes discriminative information describing what objects are shown in an image, which provide auxiliary contexts for understanding textual tokens and entities in accompanying captions 
Incorporating this visual information onto the traditional NER system is an open challenge, and multiple approaches can be considered  For instance, one may provide visual contexts only as an initial input to decoder as in some encoder-decoder image captioning systems BIBREF26   However, we empirically observe that an NER decoder which takes as input the visual embeddings at every decoding step (Section SECREF4 ), combined with the modality attention module (Section SECREF5 ), yields better results 
Lastly, we add a transform layer for each feature INLINEFORM0 before it is fed to the NER entity LSTM 
Bi-LSTM + CRF for Multimodal NER
Our MNER model is built on a Bi-LSTM and CRF hybrid model  We use the following implementation for the entity Bi-LSTM 
it = (Wxiht-1 + Wcict-1)
ct = (1-it) ct-1
+ it tanh(Wxcxt + Whcht-1)
ot = (Wxoxt + Whoht-1 + Wcoct)
ht = LSTM(xt)
= ot tanh(ct)
where INLINEFORM0 is a weighted average of three modalities INLINEFORM1 via the modality attention module, which will be defined in Section SECREF5   Bias terms for gates are omitted here for simplicity of notation 
We then obtain bi-directional entity token representations INLINEFORM0 by concatenating its left and right context representations  To enforce structural correlations between labels in sequence decoding, INLINEFORM1 is then passed to a conditional random field (CRF) to produce a label for each token maximizing the following objective  y* = y p(y|h; WCRF)
p(y|h; WCRF) = t t (yt-1,yt;h) y' t t (y't-1,y't;h)
where INLINEFORM0 is a potential function, INLINEFORM1 is a set of parameters that defines the potential functions and weight vectors for label pairs ( INLINEFORM2 )  Bias terms are omitted for brevity of formulation 
The model can be trained via log-likelihood maximization for the training set INLINEFORM0 :
L(WCRF) = i p(y|h; W)
Modality Attention
The modality attention module learns a unified representation space for multiple available modalities (words, characters, images, etc ), and produces a single vector representation with aggregated knowledge among multiple modalities, based on their weighted importance  We motivate this module from the following observations 
A majority of the previous literature combine the word and character-level contexts by simply concatenating the word and character embeddings at each decoding step, e g  INLINEFORM0 in Eq  SECREF4   However, this naive concatenation of two modalities (word and characters) results in inaccurate decoding, specifically for unknown word token embeddings (an all-zero vector INLINEFORM1 or a random vector INLINEFORM2 is assigned for any unknown token INLINEFORM3 , thus INLINEFORM4 or INLINEFORM5 )  While this concatenation approach does not cause significant errors for well-formatted text, we observe that it induces performance degradation for our social media post datasets which contain a significant number of missing tokens 
Similarly, naive merging of textual and visual information ( INLINEFORM0 ) yields suboptimal results as each modality is treated equally informative, whereas in our datasets some of the images may contain irrelevant contexts to textual modalities  Hence, ideally there needs a mechanism in which the model can effectively turn the switch on and off the modalities adaptive to each sample 
To this end, we propose a general modality attention module, which adaptively attenuates or emphasizes each modality as a whole at each decoding step INLINEFORM0 , and produces a soft-attended context vector INLINEFORM1 as an input token for the entity LSTM  [at(w),at(c),at(v)] = (Wm[xt(w); xt(c); xt(v)] + bm )
t(m) = (at(m))m'{w,c,v}(at(m')) m {w,c,v}
xt = m{w,c,v} t(m)xt(m)
where INLINEFORM0 is an attention vector at each decoding step INLINEFORM1 , and INLINEFORM2 is a final context vector at INLINEFORM3 that maximizes information gain for INLINEFORM4   Note that the optimization of the objective function (Eq  SECREF4 ) with modality attention (Eq  SECREF5 ) requires each modality to have the same dimension ( INLINEFORM5 ), and that the transformation via INLINEFORM6 essentially enforces each modality to be mapped into the same unified subspace, where the weighted average of which encodes discrimitive features for recognition of named entities 
When visual context is not provided with each token (as in the traditional NER task), we can define the modality attention for word and character embeddings only in a similar way: [at(w),at(c)] = (Wm[xt(w); xt(c)] + bm )
t(m) = (at(m))m'{w,c}(at(m')) m {w,c}
xt = m{w,c} t(m)xt(m)
Note that while we apply this modality attention module to the Bi-LSTM+CRF architecture (Section SECREF4 ) for its empirical superiority, the module itself is flexible and thus can work with other NER architectures or for other multimodal applications 
SnapCaptions Dataset
The SnapCaptions dataset is composed of 10K user-generated image (snap) and textual caption pairs where named entities in captions are manually labeled by expert human annotators (entity types: PER, LOC, ORG, MISC)  These captions are collected exclusively from snaps submitted to public and crowd-sourced stories (aka Snapchat Live Stories or Our Stories)  Examples of such public crowd-sourced stories are “New York Story” or “Thanksgiving Story”, which comprise snaps that are aggregated for various public events, venues, etc  All snaps were posted between year 2016 and 2017, and do not contain raw images or other associated information (only textual captions and obfuscated visual descriptor features extracted from the pre-trained InceptionNet are available)  We split the dataset into train (70%), validation (15%), and test sets (15%)  The captions data have average length of 30 7 characters (5 81 words) with vocabulary size 15,733, where 6,612 are considered unknown tokens from Stanford GloVE embeddings BIBREF22   Named entities annotated in the SnapCaptions dataset include many of new and emerging entities, and they are found in various surface forms (various nicknames, typos, etc ) To the best of our knowledge, SnapCaptions is the only dataset that contains natural image-caption pairs with expert-annotated named entities 
Baselines
Task: given a caption and a paired image (if used), the goal is to label every token in a caption in BIO scheme (B: beginning, I: inside, O: outside) BIBREF27   We report the performance of the following state-of-the-art NER models as baselines, as well as several configurations of our proposed approach to examine contributions of each component (W: word, C: char, V: visual) 
Bi-LSTM/CRF (W only): only takes word token embeddings (Stanford GloVE) as input  The rest of the architecture is kept the same 
Bi-LSTM/CRF + Bi-CharLSTM (C only): only takes a character sequence of each word token as input  (No word embeddings)
Bi-LSTM/CRF + Bi-CharLSTM (W+C) BIBREF0 : takes as input both word embeddings and character embeddings extracted from a Bi-CharLSTM  Entity LSTM takes concatenated vectors of word and character embeddings as input tokens 
Bi-LSTM/CRF + CharCNN (W+C) BIBREF1 : uses character embeddings extracted from a CNN instead 
Bi-LSTM/CRF + CharCNN (W+C) + Multi-task BIBREF8 : trains the model to perform both recognition (into multiple entity types) as well as segmentation (binary) tasks 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM with modality attention (W+C): uses the modality attention to merge word and character embeddings 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception (W+C+V): takes as input visual contexts extracted from InceptionNet as well, concatenated with word and char vectors 
(proposed) Bi-LSTM/CRF + Bi-CharLSTM + Inception with modality attention (W+C+V): uses the modality attention to merge word, character, and visual embeddings as input to entity LSTM 
Results: SnapCaptions Dataset
Table TABREF6 shows the NER performance on the Snap Captions dataset  We report both entity types recognition (PER, LOC, ORG, MISC) and named entity segmentation (named entity or not) results 
Parameters: We tune the parameters of each model with the following search space (bold indicate the choice for our final model): character embeddings dimension: {25, 50, 100, 150, 200, 300}, word embeddings size: {25, 50, 100, 150, 200, 300}, LSTM hidden states: {25, 50, 100, 150, 200, 300}, and INLINEFORM0 dimension: {25, 50, 100, 150, 200, 300}  We optimize the parameters with Adagrad BIBREF28 with batch size 10, learning rate 0 02, epsilon INLINEFORM1 , and decay 0 0 
Main Results: When visual context is available (W+C+V), we see that the model performance greatly improves over the textual models (W+C), showing that visual contexts are complimentary to textual information in named entity recognition tasks  In addition, it can be seen that the modality attention module further improves the entity type recognition performance for (W+C+V)  This result indicates that the modality attention is able to focus on the most effective modality (visual, words, or characters) adaptive to each sample to maximize information gain  Note that our text-only model (W+C) with the modality attention module also significantly outperform the state-of-the-art baselines BIBREF8 , BIBREF1 , BIBREF0 that use the same textual modalities (W+C), showing the effectiveness of the modality attention module for textual models as well 
Error Analysis: Table TABREF17 shows example cases where incorporation of visual contexts affects prediction of named entities  For example, the token `curry' in the caption “The curry's " is polysemous and may refer to either a type of food or a famous basketball player `Stephen Curry', and the surrounding textual contexts do not provide enough information to disambiguate it  On the other hand, visual contexts (visual tags: `parade', `urban area',    ) provide similarities to the token's distributional semantics from other training examples (snaps from “NBA Championship Parade Story"), and thus the model successfully predicts the token as a named entity  Similarly, while the text-only model erroneously predicts `Apple' in the caption “Grandma w dat lit Apple Crisp" as an organization (Apple Inc ), the visual contexts (describing objects related to food) help disambiguate the token, making the model predict it correctly as a non-named entity (a fruit)  Trending entities (musicians or DJs such as `CID', `Duke Dumont', `Marshmello', etc ) are also recognized correctly with strengthened contexts from visual information (describing concert scenes) despite lack of surrounding textual contexts  A few cases where visual contexts harmed the performance mostly include visual tags that are unrelated to a token or its surrounding textual contexts 
Visualization of Modality Attention: Figure FIGREF19 visualizes the modality attention module at each decoding step (each column), where amplified modality is represented with darker color, and attenuated modality is represented with lighter color 
For the image-aided model (W+C+V; upper row in Figure FIGREF19 ), we confirm that the modality attention successfully attenuates irrelevant signals (selfies, etc ) and amplifies relevant modality-based contexts in prediction of a given token  In the example of “disney word essential = coffee" with visual tags selfie, phone, person, the modality attention successfully attenuates distracting visual signals and focuses on textual modalities, consequently making correct predictions  The named entities in the examples of “Beautiful night atop The Space Needle" and “Splash Mountain" are challenging to predict because they are composed of common nouns (space, needle, splash, mountain), and thus they often need additional contexts to correctly predict  In the training data, visual contexts make stronger indicators for these named entities (space needle, splash mountain), and the modality attention module successfully attends more to stronger signals 
For text-only model (W+C), we observe that performance gains mostly come from the modality attention module better handling tokens unseen during training or unknown tokens from the pre-trained word embeddings matrix  For example, while WaRriOoOrs and Kooler Matic are missing tokens in the word embeddings matrix, it successfully amplifies character-based contexts (capitalized first letters, similarity to known entities `Golden State Warriors') and suppresses word-based contexts (word embeddings for unknown tokens `WaRriOoOrs'), leading to correct predictions  This result is significant because it shows performance of the model, with an almost identical architecture, can still improve without having to scale the word embeddings matrix indefinitely 
Figure FIGREF19 (b) shows the cases where the modality attention led to incorrect predictions  For example, the model predicts missing tokens HUUUGE and Shampooer incorrectly as named entities by amplifying misleading character-based contexts (capitalized first letters) or visual contexts (concert scenes, associated contexts of which often include named entities in the training dataset) 
Sensitivity to Word Embeddings Vocabulary Size: In order to isolate the effectiveness of the modality attention module on textual models in handling missing tokens, we report the performance with varying word embeddings vocabulary sizes in Table TABREF20   By increasing the number of missing tokens artificially by randomly removing words from the word embeddings matrix (original vocab size: 400K), we observe that while the overall performance degrades, the modality attention module is able to suppress the peformance degradation  Note also that the performance gap generally gets bigger as we decrease the vocabulary size of the word embeddings matrix  This result is significant in that the modality attention is able to improve the model more robust to missing tokens without having to train an indefinitely large word embeddings matrix for arbitrarily noisy social media text datasets 
Conclusions
We proposed a new multimodal NER (MNER: image + text) task on short social media posts  We demonstrated for the first time an effective MNER system, where visual information is combined with textual information to outperform traditional text-based NER baselines  Our work can be applied to myriads of social media posts or other articles across multiple platforms which often include both text and accompanying images  In addition, we proposed the modality attention module, a new neural mechanism which learns optimal integration of different modes of correlated information  In essence, the modality attention learns to attenuate irrelevant or uninformative modal information while amplifying the primary modality to extract better overall representations  We showed that the modality attention based model outperforms other state-of-the-art baselines when text was the only modality available, by better combining word and character level information 

ground_truth: Yes

Yes

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the 12 languages covered?
Context 2: Introduction
The lack of annotated training and evaluation data for many tasks and domains hinders the development of computational models for the majority of the world's languages BIBREF0, BIBREF1, BIBREF2  The necessity to guide and advance multilingual and cross-lingual NLP through annotation efforts that follow cross-lingually consistent guidelines has been recently recognized by collaborative initiatives such as the Universal Dependency (UD) project BIBREF3  The latest version of UD (as of March 2020) covers more than 70 languages  Crucially, this resource continues to steadily grow and evolve through the contributions of annotators from across the world, extending the UD's reach to a wide array of typologically diverse languages  Besides steering research in multilingual parsing BIBREF4, BIBREF5, BIBREF6 and cross-lingual parser transfer BIBREF7, BIBREF8, BIBREF9, the consistent annotations and guidelines have also enabled a range of insightful comparative studies focused on the languages' syntactic (dis)similarities BIBREF10, BIBREF11, BIBREF12 
Inspired by the UD work and its substantial impact on research in (multilingual) syntax, in this article we introduce Multi-SimLex, a suite of manually and consistently annotated semantic datasets for 12 different languages, focused on the fundamental lexical relation of semantic similarity BIBREF13, BIBREF14  For any pair of words, this relation measures whether their referents share the same (functional) features, as opposed to general cognitive association captured by co-occurrence patterns in texts (i e , the distributional information)  Datasets that quantify the strength of true semantic similarity between concept pairs such as SimLex-999 BIBREF14 or SimVerb-3500 BIBREF15 have been instrumental in improving models for distributional semantics and representation learning  Discerning between semantic similarity and relatedness/association is not only crucial for theoretical studies on lexical semantics (see §SECREF2), but has also been shown to benefit a range of language understanding tasks in NLP  Examples include dialog state tracking BIBREF16, BIBREF17, spoken language understanding BIBREF18, BIBREF19, text simplification BIBREF20, BIBREF21, BIBREF22, dictionary and thesaurus construction BIBREF23, BIBREF24 
Despite the proven usefulness of semantic similarity datasets, they are available only for a small and typologically narrow sample of resource-rich languages such as German, Italian, and Russian BIBREF25, whereas some language types and low-resource languages typically lack similar evaluation data  Even if some resources do exist, they are limited in their size (e g , 500 pairs in Turkish BIBREF26, 500 in Farsi BIBREF27, or 300 in Finnish BIBREF28) and coverage (e g , all datasets which originated from the original English SimLex-999 contain only high-frequent concepts, and are dominated by nouns)  This is why, as our departure point, we introduce a larger and more comprehensive English word similarity dataset spanning 1,888 concept pairs (see §SECREF4) 
Most importantly, semantic similarity datasets in different languages have been created using heterogeneous construction procedures with different guidelines for translation and annotation, as well as different rating scales  For instance, some datasets were obtained by directly translating the English SimLex-999 in its entirety BIBREF25, BIBREF16 or in part BIBREF28  Other datasets were created from scratch BIBREF26 and yet others sampled English concept pairs differently from SimLex-999 and then translated and reannotated them in target languages BIBREF27  This heterogeneity makes these datasets incomparable and precludes systematic cross-linguistic analyses  In this article, consolidating the lessons learned from previous dataset construction paradigms, we propose a carefully designed translation and annotation protocol for developing monolingual Multi-SimLex datasets with aligned concept pairs for typologically diverse languages  We apply this protocol to a set of 12 languages, including a mixture of major languages (e g , Mandarin, Russian, and French) as well as several low-resource ones (e g , Kiswahili, Welsh, and Yue Chinese)  We demonstrate that our proposed dataset creation procedure yields data with high inter-annotator agreement rates (e g , the average mean inter-annotator agreement for Welsh is 0 742) 
The unified construction protocol and alignment between concept pairs enables a series of quantitative analyses  Preliminary studies on the influence that polysemy and cross-lingual variation in lexical categories (see §SECREF6) have on similarity judgments are provided in §SECREF5  Data created according to Multi-SimLex protocol also allow for probing into whether similarity judgments are universal across languages, or rather depend on linguistic affinity (in terms of linguistic features, phylogeny, and geographical location)  We investigate this question in §SECREF25  Naturally, Multi-SimLex datasets can be used as an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms  We conduct a systematic evaluation of several state-of-the-art representation models in §SECREF7, showing that there are large gaps between human and system performance in all languages  The proposed construction paradigm also supports the automatic creation of 66 cross-lingual Multi-SimLex datasets by interleaving the monolingual ones  We outline the construction of the cross-lingual datasets in §SECREF6, and then present a quantitative evaluation of a series of cutting-edge cross-lingual representation models on this benchmark in §SECREF8 
Contributions  We now summarize the main contributions of this work:
1) Building on lessons learned from prior work, we create a more comprehensive lexical semantic similarity dataset for the English language spanning a total of 1,888 concept pairs balanced with respect to similarity, frequency, and concreteness, and covering four word classes: nouns, verbs, adjectives and, for the first time, adverbs  This dataset serves as the main source for the creation of equivalent datasets in several other languages 
2) We present a carefully designed and rigorous language-agnostic translation and annotation protocol  These well-defined guidelines will facilitate the development of future Multi-SimLex datasets for other languages  The proposed protocol eliminates some crucial issues with prior efforts focused on the creation of multi-lingual semantic resources, namely: i) limited coverage; ii) heterogeneous annotation guidelines; and iii) concept pairs which are semantically incomparable across different languages 
3) We offer to the community manually annotated evaluation sets of 1,888 concept pairs across 12 typologically diverse languages, and 66 large cross-lingual evaluation sets  To the best of our knowledge, Multi-SimLex is the most comprehensive evaluation resource to date focused on the relation of semantic similarity 
4) We benchmark a wide array of recent state-of-the-art monolingual and cross-lingual word representation models across our sample of languages  The results can serve as strong baselines that lay the foundation for future improvements 
5) We present a first large-scale evaluation study on the ability of encoders pretrained on language modeling (such as bert BIBREF29 and xlm BIBREF30) to reason over word-level semantic similarity in different languages  To our own surprise, the results show that monolingual pretrained encoders, even when presented with word types out of context, are sometimes competitive with static word embedding models such as fastText BIBREF31 or word2vec BIBREF32  The results also reveal a huge gap in performance between massively multilingual pretrained encoders and language-specific encoders in favor of the latter: our findings support other recent empirical evidence related to the “curse of multilinguality” BIBREF33, BIBREF34 in representation learning 
6) We make all of these resources available on a website which facilitates easy creation, submission and sharing of Multi-Simlex-style datasets for a larger number of languages  We hope that this will yield an even larger repository of semantic resources that inspire future advances in NLP within and across languages 
In light of the success of Universal Dependencies BIBREF3, we hope that our initiative will instigate a collaborative public effort with established and clear-cut guidelines that will result in additional Multi-SimLex datasets in a large number of languages in the near future  Moreover, we hope that it will provide means to advance our understanding of distributional and lexical semantics across a large number of languages  All monolingual and cross-lingual Multi-SimLex datasets–along with detailed translation and annotation guidelines–are available online at: https://multisimlex com/ 
Lexical Semantic Similarity ::: Similarity and Association
The focus of the Multi-SimLex initiative is on the lexical relation of pure semantic similarity  For any pair of words, this relation measures whether their referents share the same features  For instance, graffiti and frescos are similar to the extent that they are both forms of painting and appear on walls  This relation can be contrasted with the cognitive association between two words, which often depends on how much their referents interact in the real world, or are found in the same situations  For instance, a painter is easily associated with frescos, although they lack any physical commonalities  Association is also known in the literature under other names: relatedness BIBREF13, topical similarity BIBREF35, and domain similarity BIBREF36 
Semantic similarity and association overlap to some degree, but do not coincide BIBREF37, BIBREF38  In fact, there exist plenty of pairs that are intuitively associated but not similar  Pairs where the converse is true can also be encountered, although more rarely  An example are synonyms where a word is common and the other infrequent, such as to seize and to commandeer  BIBREF14 revealed that while similarity measures based on the WordNet graph BIBREF39 and human judgments of association in the University of South Florida Free Association Database BIBREF40 do correlate, a number of pairs follow opposite trends  Several studies on human cognition also point in the same direction  For instance, semantic priming can be triggered by similar words without association BIBREF41  On the other hand, a connection with cue words is established more quickly for topically related words rather than for similar words in free association tasks BIBREF42 
A key property of semantic similarity is its gradience: pairs of words can be similar to a different degree  On the other hand, the relation of synonymy is binary: pairs of words are synonyms if they can be substituted in all contexts (or most contexts, in a looser sense), otherwise they are not  While synonyms can be conceived as lying on one extreme of the semantic similarity continuum, it is crucial to note that their definition is stated in purely relational terms, rather than invoking their referential properties BIBREF43, BIBREF44, BIBREF45  This makes behavioral studies on semantic similarity fundamentally different from lexical resources like WordNet BIBREF46, which include paradigmatic relations (such as synonymy) 
Lexical Semantic Similarity ::: Similarity for NLP: Intrinsic Evaluation and Semantic Specialization
The ramifications of the distinction between similarity and association are profound for distributional semantics  This paradigm of lexical semantics is grounded in the distributional hypothesis, formulated by BIBREF47 and BIBREF48  According to this hypothesis, the meaning of a word can be recovered empirically from the contexts in which it occurs within a collection of texts  Since both pairs of topically related words and pairs of purely similar words tend to appear in the same contexts, their associated meaning confounds the two distinct relations BIBREF14, BIBREF49, BIBREF50  As a result, distributional methods obscure a crucial facet of lexical meaning  This limitation also reflects onto word embeddings (WEs), representations of words as low-dimensional vectors that have become indispensable for a wide range of NLP applications BIBREF51, BIBREF52, BIBREF53  In particular, it involves both static WEs learned from co-occurrence patterns BIBREF32, BIBREF54, BIBREF31 and contextualized WEs learned from modeling word sequences BIBREF55, BIBREF29  As a result, in the induced representations, geometrical closeness (measured e g  through cosine distance) conflates genuine similarity with broad relatedness  For instance, the vectors for antonyms such as sober and drunk, by definition dissimilar, might be neighbors in the semantic space under the distributional hypothesis  BIBREF36, BIBREF56, and BIBREF53 demonstrated that different choices of hyper-parameters in WE algorithms (such as context window) emphasize different relations in the resulting representations  Likewise, BIBREF57 and BIBREF54 discovered that WEs learned from texts annotated with syntactic information mirror similarity better than simple local bag-of-words neighborhoods 
The failure of WEs to capture semantic similarity, in turn, affects model performance in several NLP applications where such knowledge is crucial  In particular, Natural Language Understanding tasks such as statistical dialog modeling, text simplification, or semantic text similarity BIBREF58, BIBREF18, BIBREF59, among others, suffer the most  As a consequence, resources providing clean information on semantic similarity are key in mitigating the side effects of the distributional signal  In particular, such databases can be employed for the intrinsic evaluations of specific WE models as a proxy of their reliability for downstream applications BIBREF60, BIBREF61, BIBREF14; intuitively, the more WEs are misaligned with human judgments of similarity, the more their performance on actual tasks is expected to be degraded  Moreover, word representations can be specialized (a k a  retrofitted) by disentangling word relations of similarity and association  In particular, linguistic constraints sourced from external databases (such as synonyms from WordNet) can be injected into WEs BIBREF62, BIBREF63, BIBREF16, BIBREF22, BIBREF64 in order to enforce a particular relation in a distributional semantic space while preserving the original adjacency properties 
Lexical Semantic Similarity ::: Similarity and Language Variation: Semantic Typology
In this work, we tackle the concept of (true) semantic similarity from a multilingual perspective  While the same meaning representations may be shared by all human speakers at a deep cognitive level, there is no one-to-one mapping between the words in the lexicons of different languages  This makes the comparison of similarity judgments across languages difficult, since the meaning overlap of translationally equivalent words is sometimes far less than exact  This results from the fact that the way languages `partition' semantic fields is partially arbitrary BIBREF65, although constrained cross-lingually by common cognitive biases BIBREF66  For instance, consider the field of colors: English distinguishes between green and blue, whereas Murle (South Sudan) has a single word for both BIBREF67 
In general, semantic typology studies the variation in lexical semantics across the world's languages  According to BIBREF68, the ways languages categorize concepts into the lexicon follow three main axes: 1) granularity: what is the number of categories in a specific domain?; 2) boundary location: where do the lines marking different categories lie?; 3) grouping and dissection: what are the membership criteria of a category; which instances are considered to be more prototypical? Different choices with respect to these axes lead to different lexicalization patterns  For instance, distinct senses in a polysemous word in English, such as skin (referring to both the body and fruit), may be assigned separate words in other languages such as Italian pelle and buccia, respectively BIBREF70  We later analyze whether similarity scores obtained from native speakers also loosely follow the patterns described by semantic typology 
Previous Work and Evaluation Data
Word Pair Datasets  Rich expert-created resources such as WordNet BIBREF46, BIBREF71, VerbNet BIBREF72, BIBREF73, or FrameNet BIBREF74 encode a wealth of semantic and syntactic information, but are expensive and time-consuming to create  The scale of this problem gets multiplied by the number of languages in consideration  Therefore, crowd-sourcing with non-expert annotators has been adopted as a quicker alternative to produce smaller and more focused semantic resources and evaluation benchmarks  This alternative practice has had a profound impact on distributional semantics and representation learning BIBREF14  While some prominent English word pair datasets such as WordSim-353 BIBREF75, MEN BIBREF76, or Stanford Rare Words BIBREF77 did not discriminate between similarity and relatedness, the importance of this distinction was established by BIBREF14 through the creation of SimLex-999  This inspired other similar datasets which focused on different lexical properties  For instance, SimVerb-3500 BIBREF15 provided similarity ratings for 3,500 English verbs, whereas CARD-660 BIBREF78 aimed at measuring the semantic similarity of infrequent concepts 
Semantic Similarity Datasets in Other Languages  Motivated by the impact of datasets such as SimLex-999 and SimVerb-3500 on representation learning in English, a line of related work focused on creating similar resources in other languages  The dominant approach is translating and reannotating the entire original English SimLex-999 dataset, as done previously for German, Italian, and Russian BIBREF25, Hebrew and Croatian BIBREF16, and Polish BIBREF79  Venekoski:2017nodalida apply this process only to a subset of 300 concept pairs from the English SimLex-999  On the other hand, BIBREF27 sampled a new set of 500 English concept pairs to ensure wider topical coverage and balance across similarity spectra, and then translated those pairs to German, Italian, Spanish, and Farsi (SEMEVAL-500)  A similar approach was followed by BIBREF26 for Turkish, by BIBREF80 for Mandarin Chinese, and by BIBREF81 for Japanese  BIBREF82 translated the concatenation of SimLex-999, WordSim-353, and the English SEMEVAL-500 into Thai and then reannotated it  Finally, BIBREF83 translated English SimLex-999 and WordSim-353 to 11 resource-rich target languages (German, French, Russian, Italian, Dutch, Chinese, Portuguese, Swedish, Spanish, Arabic, Farsi), but they did not provide details concerning the translation process and the resolution of translation disagreements  More importantly, they also did not reannotate the translated pairs in the target languages  As we discussed in § SECREF6 and reiterate later in §SECREF5, semantic differences among languages can have a profound impact on the annotation scores; particulary, we show in §SECREF25 that these differences even roughly define language clusters based on language affinity 
A core issue with the current datasets concerns a lack of one unified procedure that ensures the comparability of resources in different languages  Further, concept pairs for different languages are sourced from different corpora (e g , direct translation of the English data versus sampling from scratch in the target language)  Moreover, the previous SimLex-based multilingual datasets inherit the main deficiencies of the English original version, such as the focus on nouns and highly frequent concepts  Finally, prior work mostly focused on languages that are widely spoken and do not account for the variety of the world's languages  Our long-term goal is devising a standardized methodology to extend the coverage also to languages that are resource-lean and/or typologically diverse (e g , Welsh, Kiswahili as in this work) 
Multilingual Datasets for Natural Language Understanding  The Multi-SimLex initiative and corresponding datasets are also aligned with the recent efforts on procuring multilingual benchmarks that can help advance computational modeling of natural language understanding across different languages  For instance, pretrained multilingual language models such as multilingual bert BIBREF29 or xlm BIBREF30 are typically probed on XNLI test data BIBREF84 for cross-lingual natural language inference  XNLI was created by translating examples from the English MultiNLI dataset, and projecting its sentence labels BIBREF85  Other recent multilingual datasets target the task of question answering based on reading comprehension: i) MLQA BIBREF86 includes 7 languages ii) XQuAD BIBREF87 10 languages; iii) TyDiQA BIBREF88 9 widely spoken typologically diverse languages  While MLQA and XQuAD result from the translation from an English dataset, TyDiQA was built independently in each language  Another multilingual dataset, PAWS-X BIBREF89, focused on the paraphrase identification task and was created translating the original English PAWS BIBREF90 into 6 languages  We believe that Multi-SimLex can substantially contribute to this endeavor by offering a comprehensive multilingual benchmark for the fundamental lexical level relation of semantic similarity  In future work, Multi-SimLex also offers an opportunity to investigate the correlations between word-level semantic similarity and performance in downstream tasks such as QA and NLI across different languages 
The Base for Multi-SimLex: Extending English SimLex-999
In this section, we discuss the design principles behind the English (eng) Multi-SimLex dataset, which is the basis for all the Multi-SimLex datasets in other languages, as detailed in §SECREF5  We first argue that a new, more balanced, and more comprehensive evaluation resource for lexical semantic similarity in English is necessary  We then describe how the 1,888 word pairs contained in the eng Multi-SimLex were selected in such a way as to represent various linguistic phenomena within a single integrated resource 
Construction Criteria  The following criteria have to be satisfied by any high-quality semantic evaluation resource, as argued by previous studies focused on the creation of such resources BIBREF14, BIBREF15, BIBREF91, BIBREF27:
(C1) Representative and diverse  The resource must cover the full range of diverse concepts occurring in natural language, including different word classes (e g , nouns, verbs, adjectives, adverbs), concrete and abstract concepts, a variety of lexical fields, and different frequency ranges 
(C2) Clearly defined  The resource must provide a clear understanding of which semantic relation exactly is annotated and measured, possibly contrasting it with other relations  For instance, the original SimLex-999 and SimVerb-3500 explicitly focus on true semantic similarity and distinguish it from broader relatedness captured by datasets such as MEN BIBREF76 or WordSim-353 BIBREF75 
(C3) Consistent and reliable  The resource must ensure consistent annotations obtained from non-expert native speakers following simple and precise annotation guidelines 
In choosing the word pairs and constructing eng Multi-SimLex, we adhere to these requirements  Moreover, we follow good practices established by the research on related resources  In particular, since the introduction of the original SimLex-999 dataset BIBREF14, follow-up works have improved its construction protocol across several aspects, including: 1) coverage of more lexical fields, e g , by relying on a diverse set of Wikipedia categories BIBREF27, 2) infrequent/rare words BIBREF78, 3) focus on particular word classes, e g , verbs BIBREF15, 4) annotation quality control BIBREF78  Our goal is to make use of these improvements towards a larger, more representative, and more reliable lexical similarity dataset in English and, consequently, in all other languages 
The Final Output: English Multi-SimLex  In order to ensure that the criterion C1 is satisfied, we consolidate and integrate the data already carefully sampled in prior work into a single, comprehensive, and representative dataset  This way, we can control for diversity, frequency, and other properties while avoiding to perform this time-consuming selection process from scratch  Note that, on the other hand, the word pairs chosen for English are scored from scratch as part of the entire Multi-SimLex annotation process, introduced later in §SECREF5  We now describe the external data sources for the final set of word pairs:
1) Source: SimLex-999  BIBREF14  The English Multi-SimLex has been initially conceived as an extension of the original SimLex-999 dataset  Therefore, we include all 999 word pairs from SimLex, which span 666 noun pairs, 222 verb pairs, and 111 adjective pairs  While SimLex-999 already provides examples representing different POS classes, it does not have a sufficient coverage of different linguistic phenomena: for instance, it contains only very frequent concepts, and it does not provide a representative set of verbs BIBREF15 
2) Source: SemEval-17: Task 2 BIBREF92  We start from the full dataset of 500 concept pairs to extract a total of 334 concept pairs for English Multi-SimLex a) which contain only single-word concepts, b) which are not named entities, c) where POS tags of the two concepts are the same, d) where both concepts occur in the top 250K most frequent word types in the English Wikipedia, and e) do not already occur in SimLex-999  The original concepts were sampled as to span all the 34 domains available as part of BabelDomains BIBREF93, which roughly correspond to the main high-level Wikipedia categories  This ensures topical diversity in our sub-sample 
3) Source: CARD-660 BIBREF78  67 word pairs are taken from this dataset focused on rare word similarity, applying the same selection criteria a) to e) employed for SEMEVAL-500  Words are controlled for frequency based on their occurrence counts from the Google News data and the ukWaC corpus BIBREF94  CARD-660 contains some words that are very rare (logboat), domain-specific (erythroleukemia) and slang (2mrw), which might be difficult to translate and annotate across a wide array of languages  Hence, we opt for retaining only the concept pairs above the threshold of top 250K most frequent Wikipedia concepts, as above 
4) Source: SimVerb-3500 BIBREF15 Since both CARD-660 and SEMEVAL-500 are heavily skewed towards noun pairs, and nouns also dominate the original SimLex-999, we also extract additional verb pairs from the verb-specific similarity dataset SimVerb-3500  We randomly sample 244 verb pairs from SimVerb-3500 that represent all similarity spectra  In particular, we add 61 verb pairs for each of the similarity intervals: $[0,1 5), [1 5,3), [3,4 5), [4 5, 6]$  Since verbs in SimVerb-3500 were originally chosen from VerbNet BIBREF95, BIBREF73, they cover a wide range of verb classes and their related linguistic phenomena 
5) Source: University of South Florida BIBREF96 norms, the largest database of free association for English  In order to improve the representation of different POS classes, we sample additional adjectives and adverbs from the USF norms following the procedure established by BIBREF14, BIBREF15  This yields additional 122 adjective pairs, but only a limited number of adverb pairs (e g , later – never, now – here, once – twice)  Therefore, we also create a set of adverb pairs semi-automatically by sampling adjectives that can be derivationally transformed into adverbs (e g  adding the suffix -ly) from the USF, and assessing the correctness of such derivation in WordNet  The resulting pairs include, for instance, primarily – mainly, softly – firmly, roughly – reliably, etc  We include a total of 123 adverb pairs into the final English Multi-SimLex  Note that this is the first time adverbs are included into any semantic similarity dataset 
Fulfillment of Construction Criteria  The final eng Multi-SimLex dataset spans 1,051 noun pairs, 469 verb pairs, 245 adjective pairs, and 123 adverb pairs  As mentioned above, the criterion C1 has been fulfilled by relying only on word pairs that already underwent meticulous sampling processes in prior work, integrating them into a single resource  As a consequence, Multi-SimLex allows for fine-grained analyses over different POS classes, concreteness levels, similarity spectra, frequency intervals, relation types, morphology, lexical fields, and it also includes some challenging orthographically similar examples (e g , infection – inflection)  We ensure that the criteria C2 and C3 are satisfied by using similar annotation guidelines as Simlex-999, SimVerb-3500, and SEMEVAL-500 that explicitly target semantic similarity  In what follows, we outline the carefully tailored process of translating and annotating Multi-SimLex datasets in all target languages 
Multi-SimLex: Translation and Annotation
We now detail the development of the final Multi-SimLex resource, describing our language selection process, as well as translation and annotation of the resource, including the steps taken to ensure and measure the quality of this resource  We also provide key data statistics and preliminary cross-lingual comparative analyses 
Language Selection  Multi-SimLex comprises eleven languages in addition to English  The main objective for our inclusion criteria has been to balance language prominence (by number of speakers of the language) for maximum impact of the resource, while simultaneously having a diverse suite of languages based on their typological features (such as morphological type and language family)  Table TABREF10 summarizes key information about the languages currently included in Multi-SimLex  We have included a mixture of fusional, agglutinative, isolating, and introflexive languages that come from eight different language families  This includes languages that are very widely used such as Chinese Mandarin and Spanish, and low-resource languages such as Welsh and Kiswahili  We hope to further include additional languages and inspire other researchers to contribute to the effort over the lifetime of this project 
The work on data collection can be divided into two crucial phases: 1) a translation phase where the extended English language dataset with 1,888 pairs (described in §SECREF4) is translated into eleven target languages, and 2) an annotation phase where human raters scored each pair in the translated set as well as the English set  Detailed guidelines for both phases are available online at: https://multisimlex com 
Multi-SimLex: Translation and Annotation ::: Word Pair Translation
Translators for each target language were instructed to find direct or approximate translations for the 1,888 word pairs that satisfy the following rules  (1) All pairs in the translated set must be unique (i e , no duplicate pairs); (2) Translating two words from the same English pair into the same word in the target language is not allowed (e g , it is not allowed to translate car and automobile to the same Spanish word coche)  (3) The translated pairs must preserve the semantic relations between the two words when possible  This means that, when multiple translations are possible, the translation that best conveys the semantic relation between the two words found in the original English pair is selected  (4) If it is not possible to use a single-word translation in the target language, then a multi-word expression (MWE) can be used to convey the nearest possible semantics given the above points (e g , the English word homework is translated into the Polish MWE praca domowa) 
Satisfying the above rules when finding appropriate translations for each pair–while keeping to the spirit of the intended semantic relation in the English version–is not always straightforward  For instance, kinship terminology in Sinitic languages (Mandarin and Yue) uses different terms depending on whether the family member is older or younger, and whether the family member comes from the mother’s side or the father’s side  UTF8gbsn In Mandarin, brother has no direct translation and can be translated as either: 哥哥 (older brother) or 弟弟 (younger brother)  Therefore, in such cases, the translators are asked to choose the best option given the semantic context (relation) expressed by the pair in English, otherwise select one of the translations arbitrarily  This is also used to remove duplicate pairs in the translated set, by differentiating the duplicates using a variant at each instance  Further, many translation instances were resolved using near-synonymous terms in the translation  For example, the words in the pair: wood – timber can only be directly translated in Estonian to puit, and are not distinguishable  Therefore, the translators approximated the translation for timber to the compound noun puitmaterjal (literally: wood material) in order to produce a valid pair in the target language  In some cases, a direct transliteration from English is used  For example, the pair: physician and doctor both translate to the same word in Estonian (arst); the less formal word doktor is used as a translation of doctor to generate a valid pair 
We measure the quality of the translated pairs by using a random sample set of 100 pairs (from the 1,888 pairs) to be translated by an independent translator for each target language  The sample is proportionally stratified according to the part-of-speech categories  The independent translator is given identical instructions to the main translator; we then measure the percentage of matched translated words between the two translations of the sample set  Table TABREF12 summarizes the inter-translator agreement results for all languages and by part-of-speech subsets  Overall across all languages, the agreement is 84 8%, which is similar to prior work BIBREF27, BIBREF97 
Multi-SimLex: Translation and Annotation ::: Guidelines and Word Pair Scoring
Across all languages, 145 human annotators were asked to score all 1,888 pairs (in their given language)  We finally collect at least ten valid annotations for each word pair in each language  All annotators were required to abide by the following instructions:
1  Each annotator must assign an integer score between 0 and 6 (inclusive) indicating how semantically similar the two words in a given pair are  A score of 6 indicates very high similarity (i e , perfect synonymy), while zero indicates no similarity 
2  Each annotator must score the entire set of 1,888 pairs in the dataset  The pairs must not be shared between different annotators 
3  Annotators are able to break the workload over a period of approximately 2-3 weeks, and are able to use external sources (e g  dictionaries, thesauri, WordNet) if required 
4  Annotators are kept anonymous, and are not able to communicate with each other during the annotation process 
The selection criteria for the annotators required that all annotators must be native speakers of the target language  Preference to annotators with university education was given, but not required  Annotators were asked to complete a spreadsheet containing the translated pairs of words, as well as the part-of-speech, and a column to enter the score  The annotators did not have access to the original pairs in English 
To ensure the quality of the collected ratings, we have employed an adjudication protocol similar to the one proposed and validated by Pilehvar:2018emnlp  It consists of the following three rounds:
Round 1: All annotators are asked to follow the instructions outlined above, and to rate all 1,888 pairs with integer scores between 0 and 6 
Round 2: We compare the scores of all annotators and identify the pairs for each annotator that have shown the most disagreement  We ask the annotators to reconsider the assigned scores for those pairs only  The annotators may chose to either change or keep the scores  As in the case with Round 1, the annotators have no access to the scores of the other annotators, and the process is anonymous  This process gives a chance for annotators to correct errors or reconsider their judgments, and has been shown to be very effective in reaching consensus, as reported by BIBREF78  We used a very similar procedure as BIBREF78 to identify the pairs with the most disagreement; for each annotator, we marked the $i$th pair if the rated score $s_i$ falls within: $s_i \ge \mu _i + 1 5$ or $s_i \le \mu _i - 1 5$, where $\mu _i$ is the mean of the other annotators' scores 
Round 3: We compute the average agreement for each annotator (with the other annotators), by measuring the average Spearman's correlation against all other annotators  We discard the scores of annotators that have shown the least average agreement with all other annotators, while we maintain at least ten annotators per language by the end of this round  The actual process is done in multiple iterations: (S1) we measure the average agreement for each annotator with every other annotator (this corresponds to the APIAA measure, see later); (S2) if we still have more than 10 valid annotators and the lowest average score is higher than in the previous iteration, we remove the lowest one, and rerun S1  Table TABREF14 shows the number of annotators at both the start (Round 1) and end (Round 3) of our process for each language 
We measure the agreement between annotators using two metrics, average pairwise inter-annotator agreement (APIAA), and average mean inter-annotator agreement (AMIAA)  Both of these use Spearman's correlation ($\rho $) between annotators scores, the only difference is how they are averaged  They are computed as follows:
where $\rho (s_i,s_j)$ is the Spearman's correlation between annotators $i$ and $j$'s scores ($s_i$,$s_j$) for all pairs in the dataset, and $N$ is the number of annotators  APIAA has been used widely as the standard measure for inter-annotator agreement, including in the original SimLex paper BIBREF14  It simply averages the pairwise Spearman's correlation between all annotators  On the other hand, AMIAA compares the average Spearman's correlation of one held-out annotator with the average of all the other $N-1$ annotators, and then averages across all $N$ `held-out' annotators  It smooths individual annotator effects and arguably serves as a better upper bound than APIAA BIBREF15, BIBREF91, BIBREF78 
We present the respective APIAA and AMIAA scores in Table TABREF16 and Table TABREF17 for all part-of-speech subsets, as well as the agreement for the full datasets  As reported in prior work BIBREF15, BIBREF91, AMIAA scores are typically higher than APIAA scores  Crucially, the results indicate `strong agreement' (across all languages) using both measurements  The languages with the highest annotator agreement were French (fra) and Yue Chinese (yue), while Russian (rus) had the lowest overall IAA scores  These scores, however, are still considered to be `moderately strong agreement' 
Multi-SimLex: Translation and Annotation ::: Data Analysis
Similarity Score Distributions  Across all languages, the average score (mean $=1 61$, median$=1 1$) is on the lower side of the similarity scale  However, looking closer at the scores of each language in Table TABREF19, we indicate notable differences in both the averages and the spread of scores  Notably, French has the highest average of similarity scores (mean$=2 61$, median$=2 5$), while Kiswahili has the lowest average (mean$=1 28$, median$=0 5$)  Russian has the lowest spread ($\sigma =1 37$), while Polish has the largest ($\sigma =1 62$)  All of the languages are strongly correlated with each other, as shown in Figure FIGREF20, where all of the Spearman's correlation coefficients are greater than 0 6 for all language pairs  Languages that share the same language family are highly correlated (e g, cmn-yue, rus-pol, est-fin)  In addition, we observe high correlations between English and most other languages, as expected  This is due to the effect of using English as the base/anchor language to create the dataset  In simple words, if one translates to two languages $L_1$ and $L_2$ starting from the same set of pairs in English, it is higly likely that $L_1$ and $L_2$ will diverge from English in different ways  Therefore, the similarity between $L_1$-eng and $L_2$-eng is expected to be higher than between $L_1$-$L_2$, especially if $L_1$ and $L_2$ are typologically dissimilar languages (e g , heb-cmn, see Figure FIGREF20)  This phenomenon is well documented in related prior work BIBREF25, BIBREF27, BIBREF16, BIBREF97  While we acknowledge this as a slight artifact of the dataset design, it would otherwise be impossible to construct a semantically aligned and comprehensive dataset across a large number of languages 
We also report differences in the distribution of the frequency of words among the languages in Multi-SimLex  Figure FIGREF22 shows six example languages, where each bar segment shows the proportion of words in each language that occur in the given frequency range  For example, the 10K-20K segment of the bars represents the proportion of words in the dataset that occur in the list of most frequent words between the frequency rank of 10,000 and 20,000 in that language; likewise with other intervals  Frequency lists for the presented languages are derived from Wikipedia and Common Crawl corpora  While many concept pairs are direct or approximate translations of English pairs, we can see that the frequency distribution does vary across different languages, and is also related to inherent language properties  For instance, in Finnish and Russian, while we use infinitive forms of all verbs, conjugated verb inflections are often more frequent in raw corpora than the corresponding infinitive forms  The variance can also be partially explained by the difference in monolingual corpora size used to derive the frequency rankings in the first place: absolute vocabulary sizes are expected to fluctuate across different languages  However, it is also important to note that the datasets also contain subsets of lower-frequency and rare words, which can be used for rare word evaluations in multiple languages, in the spirit of Pilehvar:2018emnlp's English rare word dataset 
Cross-Linguistic Differences  Table TABREF23 shows some examples of average similarity scores of English, Spanish, Kiswahili and Welsh concept pairs  Remember that the scores range from 0 to 6: the higher the score, the more similar the participants found the concepts in the pair  The examples from Table TABREF23 show evidence of both the stability of average similarity scores across languages (unlikely – friendly, book – literature, and vanish – disappear), as well as language-specific differences (care – caution)  Some differences in similarity scores seem to group languages into clusters  For example, the word pair regular – average has an average similarity score of 4 0 and 4 1 in English and Spanish, respectively, whereas in Kiswahili and Welsh the average similarity score of this pair is 0 5 and 0 8  We analyze this phenomenon in more detail in §SECREF25 
There are also examples for each of the four languages having a notably higher or lower similarity score for the same concept pair than the three other languages  For example, large – big in English has an average similarity score of 5 9, whereas Spanish, Kiswahili and Welsh speakers rate the closest concept pair in their native language to have a similarity score between 2 7 and 3 8  What is more, woman – wife receives an average similarity of 0 9 in English, 2 9 in Spanish, and greater than 4 0 in Kiswahili and Welsh  The examples from Spanish include banco – asiento (bank – seat) which receives an average similarity score 5 1, while in the other three languages the similarity score for this word pair does not exceed 0 1  At the same time, the average similarity score of espantosamente – fantásticamente (amazingly – fantastically) is much lower in Spanish (0 4) than in other languages (4 1 – 5 1)  In Kiswahili, an example of a word pair with a higher similarity score than the rest would be machweo – jioni (sunset – evening), having an average score of 5 5, while the other languages receive 2 8 or less, and a notably lower similarity score is given to wa ajabu - mkubwa sana (wonderful – terrific), getting 0 9, while the other languages receive 5 3 or more  Welsh examples include yn llwyr - yn gyfan gwbl (purely – completely), which scores 5 4 among Welsh speakers but 2 3 or less in other languages, while addo – tyngu (promise – swear) is rated as 0 by all Welsh annotators, but in the other three languages 4 3 or more on average 
There can be several explanations for the differences in similarity scores across languages, including but not limited to cultural context, polysemy, metonymy, translation, regional and generational differences, and most commonly, the fact that words and meanings do not exactly map onto each other across languages  For example, it is likely that the other three languages do not have two separate words for describing the concepts in the concept pair: big – large, and the translators had to opt for similar lexical items that were more distant in meaning, explaining why in English the concept pair received a much higher average similarity score than in other languages  A similar issue related to the mapping problem across languages arose in the Welsh concept pair yn llwye – yn gyfan gwbl, where Welsh speakers agreed that the two concepts are very similar  When asked, bilingual speakers considered the two Welsh concepts more similar than English equivalents purely – completely, potentially explaining why a higher average similarity score was reached in Welsh  The example of woman – wife can illustrate cultural differences or another translation-related issue where the word `wife' did not exist in some languages (for example, Estonian), and therefore had to be described using other words, affecting the comparability of the similarity scores  This was also the case with the football – soccer concept pair  The pair bank – seat demonstrates the effect of the polysemy mismatch across languages: while `bank' has two different meanings in English, neither of them is similar to the word `seat', but in Spanish, `banco' can mean `bank', but it can also mean `bench'  Quite naturally, Spanish speakers gave the pair banco – asiento a higher similarity score than the speakers of languages where this polysemy did not occur 
An example of metonymy affecting the average similarity score can be seen in the Kiswahili version of the word pair: sunset – evening (machweo – jioni)  The average similarity score for this pair is much higher in Kiswahili, likely because the word `sunset' can act as a metonym of `evening'  The low similarity score of wonderful – terrific in Kiswahili (wa ajabu - mkubwa sana) can be explained by the fact that while `mkubwa sana' can be used as `terrific' in Kiswahili, it technically means `very big', adding to the examples of translation- and mapping-related effects  The word pair amazingly – fantastically (espantosamente – fantásticamente) brings out another translation-related problem: the accuracy of the translation  While `espantosamente' could arguably be translated to `amazingly', more common meanings include: `frightfully', `terrifyingly', and `shockingly', explaining why the average similarity score differs from the rest of the languages  Another problem was brought out by addo – tyngu (promise – swear) in Welsh, where the `tyngu' may not have been a commonly used or even a known word choice for annotators, pointing out potential regional or generational differences in language use 
Table TABREF24 presents examples of concept pairs from English, Spanish, Kiswahili, and Welsh on which the participants agreed the most  For example, in English all participants rated the similarity of trial – test to be 4 or 5  In Spanish and Welsh, all participants rated start – begin to correspond to a score of 5 or 6  In Kiswahili, money – cash received a similarity rating of 6 from every participant  While there are numerous examples of concept pairs in these languages where the participants agreed on a similarity score of 4 or higher, it is worth noting that none of these languages had a single pair where all participants agreed on either 1-2, 2-3, or 3-4 similarity rating  Interestingly, in English all pairs where all the participants agreed on a 5-6 similarity score were adjectives 
Multi-SimLex: Translation and Annotation ::: Effect of Language Affinity on Similarity Scores
Based on the analysis in Figure FIGREF20 and inspecting the anecdotal examples in the previous section, it is evident that the correlation between similarity scores across languages is not random  To corroborate this intuition, we visualize the vectors of similarity scores for each single language by reducing their dimensionality to 2 via Principal Component Analysis BIBREF98  The resulting scatter plot in Figure FIGREF26 reveals that languages from the same family or branch have similar patterns in the scores  In particular, Russian and Polish (both Slavic), Finnish and Estonian (both Uralic), Cantonese and Mandarin Chinese (both Sinitic), and Spanish and French (both Romance) are all neighbors 
In order to quantify exactly the effect of language affinity on the similarity scores, we run correlation analyses between these and language features  In particular, we extract feature vectors from URIEL BIBREF99, a massively multilingual typological database that collects and normalizes information compiled by grammarians and field linguists about the world's languages  In particular, we focus on information about geography (the areas where the language speakers are concentrated), family (the phylogenetic tree each language belongs to), and typology (including syntax, phonological inventory, and phonology)  Moreover, we consider typological representations of languages that are not manually crafted by experts, but rather learned from texts  BIBREF100 proposed to construct such representations by training language-identifying vectors end-to-end as part of neural machine translation models 
The vector for similarity judgments and the vector of linguistic features for a given language have different dimensionality  Hence, we first construct a distance matrix for each vector space, such that both columns and rows are language indices, and each cell value is the cosine distance between the vectors of the corresponding language pair  Given a set of L languages, each resulting matrix $S$ has dimensionality of $\mathbb {R}^{|L| \times |L|}$ and is symmetrical  To estimate the correlation between the matrix for similarity judgments and each of the matrices for linguistic features, we run a Mantel test BIBREF101, a non-parametric statistical test based on matrix permutations that takes into account inter-dependencies among pairwise distances 
The results of the Mantel test reported in Table FIGREF26 show that there exist statistically significant correlations between similarity judgments and geography, family, and syntax, given that $p < 0 05$ and $z > 1 96$  The correlation coefficient is particularly strong for geography ($r = 0 647$) and syntax ($r = 0 649$)  The former result is intuitive, because languages in contact easily borrow and loan lexical units, and cultural interactions may result in similar cognitive categorizations  The result for syntax, instead, cannot be explained so easily, as formal properties of language do not affect lexical semantics  Instead, we conjecture that, while no causal relation is present, both syntactic features and similarity judgments might be linked to a common explanatory variable (such as geography)  In fact, several syntactic properties are not uniformly spread across the globe  For instance, verbs with Verb–Object–Subject word order are mostly concentrated in Oceania BIBREF102  In turn, geographical proximity leads to similar judgment patterns, as mentioned above  On the other hand, we find no correlation with phonology and inventory, as expected, nor with the bottom-up typological features from BIBREF100 
Cross-Lingual Multi-SimLex Datasets
A crucial advantage of having semantically aligned monolingual datasets across different languages is the potential to create cross-lingual semantic similarity datasets  Such datasets allow for probing the quality of cross-lingual representation learning algorithms BIBREF27, BIBREF103, BIBREF104, BIBREF105, BIBREF106, BIBREF30, BIBREF107 as an intrinsic evaluation task  However, the cross-lingual datasets previous work relied upon BIBREF27 were limited to a homogeneous set of high-resource languages (e g , English, German, Italian, Spanish) and a small number of concept pairs (all less than 1K pairs)  We address both problems by 1) using a typologically more diverse language sample, and 2) relying on a substantially larger English dataset as a source for the cross-lingual datasets: 1,888 pairs in this work versus 500 pairs in the work of Camacho:2017semeval  As a result, each of our cross-lingual datasets contains a substantially larger number of concept pairs, as shown in Table TABREF30  The cross-lingual Multi-Simlex datasets are constructed automatically, leveraging word pair translations and annotations collected in all 12 languages  This yields a total of 66 cross-lingual datasets, one for each possible combination of languages  Table TABREF30 provides the final number of concept pairs, which lie between 2,031 and 3,480 pairs for each cross-lingual dataset, whereas Table TABREF29 shows some sample pairs with their corresponding similarity scores 
The automatic creation and verification of cross-lingual datasets closely follows the procedure first outlined by Camacho:2015acl and later adopted by Camacho:2017semeval (for semantic similarity) and Vulic:2019acl (for graded lexical entailment)  First, given two languages, we intersect their aligned concept pairs obtained through translation  For instance, starting from the aligned pairs attroupement – foule in French and rahvasumm – rahvahulk in Estonian, we construct two cross-lingual pairs attroupement – rahvaluk and rahvasumm – foule  The scores of cross-lingual pairs are then computed as averages of the two corresponding monolingual scores  Finally, in order to filter out concept pairs whose semantic meaning was not preserved during this operation, we retain only cross-lingual pairs for which the corresponding monolingual scores $(s_s, s_t)$ differ at most by one fifth of the full scale (i e , $\mid s_s - s_t \mid \le 1 2$)  This heuristic mitigates the noise due to cross-lingual semantic shifts BIBREF27, BIBREF97  We refer the reader to the work of Camacho:2015acl for a detailed technical description of the procedure  UTF8gbsn
To assess the quality of the resulting cross-lingual datasets, we have conducted a verification experiment similar to Vulic:2019acl  We randomly sampled 300 concept pairs in the English-Spanish, English-French, and English-Mandarin cross-lingual datasets  Subsequently, we asked bilingual native speakers to provide similarity judgments of each pair  The Spearman's correlation score $\rho $ between automatically induced and manually collected ratings achieves $\rho \ge 0 90$ on all samples, which confirms the viability of the automatic construction procedure 
Score and Class Distributions  The summary of score and class distributions across all 66 cross-lingual datasets are provided in Figure FIGREF31 and Figure FIGREF31, respectively  First, it is obvious that the distribution over the four POS classes largely adheres to that of the original monolingual Multi-SimLex datasets, and that the variance is quite low: e g , the eng-fra dataset contains the lowest proportion of nouns (49 21%) and the highest proportion of verbs (27 1%), adjectives (15 28%), and adverbs (8 41%)  On the other hand, the distribution over similarity intervals in Figure FIGREF31 shows a much greater variance  This is again expected as this pattern resurfaces in monolingual datasets (see Table TABREF19)  It is also evident that the data are skewed towards lower-similarity concept pairs  However, due to the joint size of all cross-lingual datasets (see Table TABREF30), even the least represented intervals contain a substantial number of concept pairs  For instance, the rus-yue dataset contains the least highly similar concept pairs (in the interval $[4,6]$) of all 66 cross-lingual datasets  Nonetheless, the absolute number of pairs (138) in that interval for rus-yue is still substantial  If needed, this makes it possible to create smaller datasets which are balanced across the similarity spectra through sub-sampling 
Monolingual Evaluation of Representation Learning Models
After the numerical and qualitative analyses of the Multi-SimLex datasets provided in §§ SECREF18–SECREF25, we now benchmark a series of representation learning models on the new evaluation data  We evaluate standard static word embedding algorithms such as fastText BIBREF31, as well as a range of more recent text encoders pretrained on language modeling such as multilingual BERT BIBREF29  These experiments provide strong baseline scores on the new Multi-SimLex datasets and offer a first large-scale analysis of pretrained encoders on word-level semantic similarity across diverse languages  In addition, the experiments now enabled by Multi-SimLex aim to answer several important questions  (Q1) Is it viable to extract high-quality word-level representations from pretrained encoders receiving subword-level tokens as input? Are such representations competitive with standard static word-level embeddings? (Q2) What are the implications of monolingual pretraining versus (massively) multilingual pretraining for performance? (Q3) Do lightweight unsupervised post-processing techniques improve word representations consistently across different languages? (Q4) Can we effectively transfer available external lexical knowledge from resource-rich languages to resource-lean languages in order to learn word representations that distinguish between true similarity and conceptual relatedness (see the discussion in §SECREF6)?
Monolingual Evaluation of Representation Learning Models ::: Models in Comparison
Static Word Embeddings in Different Languages  First, we evaluate a standard method for inducing non-contextualized (i e , static) word embeddings across a plethora of different languages: fastText (ft) vectors BIBREF31 are currently the most popular and robust choice given 1) the availability of pretrained vectors in a large number of languages BIBREF108 trained on large Common Crawl (CC) plus Wikipedia (Wiki) data, and 2) their superior performance across a range of NLP tasks BIBREF109  In fact, fastText is an extension of the standard word-level CBOW and skip-gram word2vec models BIBREF32 that takes into account subword-level information, i e  the contituent character n-grams of each word BIBREF110  For this reason, fastText is also more suited for modeling rare words and morphologically rich languages 
We rely on 300-dimensional ft word vectors trained on CC+Wiki and available online for 157 languages  The word vectors for all languages are obtained by CBOW with position-weights, with character n-grams of length 5, a window of size 5, 10 negative examples, and 10 training epochs  We also probe another (older) collection of ft vectors, pretrained on full Wikipedia dumps of each language   The vectors are 300-dimensional, trained with the skip-gram objective for 5 epochs, with 5 negative examples, a window size set to 5, and relying on all character n-grams from length 3 to 6  Following prior work, we trim the vocabularies for all languages to the 200K most frequent words and compute representations for multi-word expressions by averaging the vectors of their constituent words 
Unsupervised Post-Processing  Further, we consider a variety of unsupervised post-processing steps that can be applied post-training on top of any pretrained input word embedding space without any external lexical semantic resource  So far, the usefulness of such methods has been verified only on the English language through benchmarks for lexical semantics and sentence-level tasks BIBREF113  In this paper, we assess if unsupervised post-processing is beneficial also in other languages  To this end, we apply the following post-hoc transformations on the initial word embeddings:
1) Mean centering (mc) is applied after unit length normalization to ensure that all vectors have a zero mean, and is commonly applied in data mining and analysis BIBREF114, BIBREF115 
2) All-but-the-top (abtt) BIBREF113, BIBREF116 eliminates the common mean vector and a few top dominating directions (according to PCA) from the input distributional word vectors, since they do not contribute towards distinguishing the actual semantic meaning of different words  The method contains a single (tunable) hyper-parameter $dd_{A}$ which denotes the number of the dominating directions to remove from the initial representations  Previous work has verified the usefulness of abtt in several English lexical semantic tasks such as semantic similarity, word analogies, and concept categorization, as well as in sentence-level text classification tasks BIBREF113 
3) uncovec BIBREF117 adjusts the similarity order of an arbitrary input word embedding space, and can emphasize either syntactic or semantic information in the transformed vectors  In short, it transforms the input space $\mathbf {X}$ into an adjusted space $\mathbf {X}\mathbf {W}_{\alpha }$ through a linear map $\mathbf {W}_{\alpha }$ controlled by a single hyper-parameter $\alpha $  The $n^{\text{th}}$-order similarity transformation of the input word vector space $\mathbf {X}$ (for which $n=1$) can be obtained as $\mathbf {M}_{n}(\mathbf {X}) = \mathbf {M}_1(\mathbf {X}\mathbf {W}_{(n-1)/2})$, with $\mathbf {W}_{\alpha }=\mathbf {Q}\mathbf {\Gamma }^{\alpha }$, where $\mathbf {Q}$ and $\mathbf {\Gamma }$ are the matrices obtained via eigendecomposition of $\mathbf {X}^T\mathbf {X}=\mathbf {Q}\mathbf {\Gamma }\mathbf {Q}^T$  $\mathbf {\Gamma }$ is a diagonal matrix containing eigenvalues of $\mathbf {X}^T\mathbf {X}$; $\mathbf {Q}$ is an orthogonal matrix with eigenvectors of $\mathbf {X}^T\mathbf {X}$ as columns  While the motivation for the uncovec methods does originate from adjusting discrete similarity orders, note that $\alpha $ is in fact a continuous real-valued hyper-parameter which can be carefully tuned  For more technical details we refer the reader to the original work of BIBREF117 
As mentioned, all post-processing methods can be seen as unsupervised retrofitting methods that, given an arbitrary input vector space $\mathbf {X}$, produce a perturbed/transformed output vector space $\mathbf {X}^{\prime }$, but unlike common retrofitting methods BIBREF62, BIBREF16, the perturbation is completely unsupervised (i e , self-contained) and does not inject any external (semantic similarity oriented) knowledge into the vector space  Note that different perturbations can also be stacked: e g , we can apply uncovec and then use abtt on top the output uncovec vectors  When using uncovec and abtt we always length-normalize and mean-center the data first (i e , we apply the simple mc normalization)  Finally, we tune the two hyper-parameters $d_A$ (for abtt) and $\alpha $ (uncovec) on the English Multi-SimLex and use the same values on the datasets of all other languages; we report results with $dd_A = 3$ or $dd_A = 10$, and $\alpha =-0 3$ 
Contextualized Word Embeddings  We also evaluate the capacity of unsupervised pretraining architectures based on language modeling objectives to reason over lexical semantic similarity  To the best of our knowledge, our article is the first study performing such analyses  State-of-the-art models such as bert BIBREF29, xlm BIBREF30, or roberta BIBREF118 are typically very deep neural networks based on the Transformer architecture BIBREF119  They receive subword-level tokens as inputs (such as WordPieces BIBREF120) to tackle data sparsity  In output, they return contextualized embeddings, dynamic representations for words in context 
To represent words or multi-word expressions through a pretrained model, we follow prior work BIBREF121 and compute an input item's representation by 1) feeding it to a pretrained model in isolation; then 2) averaging the $H$ last hidden representations for each of the item’s constituent subwords; and then finally 3) averaging the resulting subword representations to produce the final $d$-dimensional representation, where $d$ is the embedding and hidden-layer dimensionality (e g , $d=768$ with bert)  We opt for this approach due to its proven viability and simplicity BIBREF121, as it does not require any additional corpora to condition the induction of contextualized embeddings  Other ways to extract the representations from pretrained models BIBREF122, BIBREF123, BIBREF124 are beyond the scope of this work, and we will experiment with them in the future 
In other words, we treat each pretrained encoder enc as a black-box function to encode a single word or a multi-word expression $x$ in each language into a $d$-dimensional contextualized representation $\mathbf {x}_{\textsc {enc}} \in \mathbb {R}^d = \textsc {enc}(x)$ (e g , $d=768$ with bert)  As multilingual pretrained encoders, we experiment with the multilingual bert model (m-bert) BIBREF29 and xlm BIBREF30  m-bert is pretrained on monolingual Wikipedia corpora of 102 languages (comprising all Multi-SimLex languages) with a 12-layer Transformer network, and yields 768-dimensional representations  Since the concept pairs in Multi-SimLex are lowercased, we use the uncased version of m-bert  m-bert comprises all Multi-SimLex languages, and its evident ability to perform cross-lingual transfer BIBREF12, BIBREF125, BIBREF126 also makes it a convenient baseline model for cross-lingual experiments later in §SECREF8  The second multilingual model we consider, xlm-100, is pretrained on Wikipedia dumps of 100 languages, and encodes each concept into a $1,280$-dimensional representation  In contrast to m-bert, xlm-100 drops the next-sentence prediction objective and adds a cross-lingual masked language modeling objective  For both encoders, the representations of each concept are computed as averages over the last $H=4$ hidden layers in all experiments, as suggested by Wu:2019arxiv 
Besides m-bert and xlm, covering multiple languages, we also analyze the performance of “language-specific” bert and xlm models for the languages where they are available: Finnish, Spanish, English, Mandarin Chinese, and French  The main goal of this comparison is to study the differences in performance between multilingual “one-size-fits-all” encoders and language-specific encoders  For all experiments, we rely on the pretrained models released in the Transformers repository BIBREF127 
Unsupervised post-processing steps devised for static word embeddings (i e , mean-centering, abtt, uncovec) can also be applied on top of contextualized embeddings if we predefine a vocabulary of word types $V$ that will be represented in a word vector space $\mathbf {X}$  We construct such $V$ for each language as the intersection of word types covered by the corresponding CC+Wiki fastText vectors and the (single-word or multi-word) expressions appearing in the corresponding Multi-SimLex dataset 
Finally, note that it is not feasible to evaluate a full range of available pretrained encoders within the scope of this work  Our main intention is to provide the first set of baseline results on Multi-SimLex by benchmarking a sample of most popular encoders, at the same time also investigating other important questions such as performance of static versus contextualized word embeddings, or multilingual versus language-specific pretraining  Another purpose of the experiments is to outline the wide potential and applicability of the Multi-SimLex datasets for multilingual and cross-lingual representation learning evaluation 
Monolingual Evaluation of Representation Learning Models ::: Results and Discussion
The results we report are Spearman's $\rho $ coefficients of the correlation between the ranks derived from the scores of the evaluated models and the human scores provided in each Multi-SimLex dataset  The main results with static and contextualized word vectors for all test languages are summarized in Table TABREF43  The scores reveal several interesting patterns, and also pinpoint the main challenges for future work 
State-of-the-Art Representation Models  The absolute scores of CC+Wiki ft, Wiki ft, and m-bert are not directly comparable, because these models have different coverage  In particular, Multi-SimLex contains some out-of-vocabulary (OOV) words whose static ft embeddings are not available  On the other hand, m-bert has perfect coverage  A general comparison between CC+Wiki and Wiki ft vectors, however, supports the intuition that larger corpora (such as CC+Wiki) yield higher correlations  Another finding is that a single massively multilingual model such as m-bert cannot produce semantically rich word-level representations  Whether this actually happens because the training objective is different—or because the need to represent 100+ languages reduces its language-specific capacity—is investigated further below 
The overall results also clearly indicate that (i) there are differences in performance across different monolingual Multi-SimLex datasets, and (ii) unsupervised post-processing is universally useful, and can lead to huge improvements in correlation scores for many languages  In what follows, we also delve deeper into these analyses 
Impact of Unsupervised Post-Processing  First, the results in Table TABREF43 suggest that applying dimension-wise mean centering to the initial vector spaces has positive impact on word similarity scores in all test languages and for all models, both static and contextualized (see the +mc rows in Table TABREF43)  BIBREF128 show that distributional word vectors have a tendency towards narrow clusters in the vector space (i e , they occupy a narrow cone in the vector space and are therefore anisotropic BIBREF113, BIBREF129), and are prone to the undesired effect of hubness BIBREF130, BIBREF131  Applying dimension-wise mean centering has the effect of spreading the vectors across the hyper-plane and mitigating the hubness issue, which consequently improves word-level similarity, as it emerges from the reported results  Previous work has already validated the importance of mean centering for clustering-based tasks BIBREF132, bilingual lexicon induction with cross-lingual word embeddings BIBREF133, BIBREF134, BIBREF135, and for modeling lexical semantic change BIBREF136  However, to the best of our knowledge, the results summarized in Table TABREF43 are the first evidence that also confirms its importance for semantic similarity in a wide array of languages  In sum, as a general rule of thumb, we suggest to always mean-center representations for semantic tasks 
The results further indicate that additional post-processing methods such as abtt and uncovec on top of mean-centered vector spaces can lead to further gains in most languages  The gains are even visible for languages which start from high correlation scores: for instance , cmn with CC+Wiki ft increases from 0 534 to 0 583, from 0 315 to 0 526 with Wiki ft, and from 0 408 to 0 487 with m-bert  Similarly, for rus with CC+Wiki ft we can improve from 0 422 to 0 500, and for fra the scores improve from 0 578 to 0 613  There are additional similar cases reported in Table TABREF43 
Overall, the unsupervised post-processing techniques seem universally useful across languages, but their efficacy and relative performance does vary across different languages  Note that we have not carefully fine-tuned the hyper-parameters of the evaluated post-processing methods, so additional small improvements can be expected for some languages  The main finding, however, is that these post-processing techniques are robust to semantic similarity computations beyond English, and are truly language independent  For instance, removing dominant latent (PCA-based) components from word vectors emphasizes semantic differences between different concepts, as only shared non-informative latent semantic knowledge is removed from the representations 
In summary, pretrained word embeddings do contain more information pertaining to semantic similarity than revealed in the initial vectors  This way, we have corroborated the hypotheses from prior work BIBREF113, BIBREF117 which were not previously empirically verified on other languages due to a shortage of evaluation data; this gap has now been filled with the introduction of the Multi-SimLex datasets  In all follow-up experiments, we always explicitly denote which post-processing configuration is used in evaluation 
POS-Specific Subsets  We present the results for subsets of word pairs grouped by POS class in Table TABREF46  Prior work based on English data showed that representations for nouns are typically of higher quality than those for the other POS classes BIBREF49, BIBREF137, BIBREF50  We observe a similar trend in other languages as well  This pattern is consistent across different representation models and can be attributed to several reasons  First, verb representations need to express a rich range of syntactic and semantic behaviors rather than purely referential features BIBREF138, BIBREF139, BIBREF73  Second, low correlation scores on the adjective and adverb subsets in some languages (e g , pol, cym, swa) might be due to their low frequency in monolingual texts, which yields unreliable representations  In general, the variance in performance across different word classes warrants further research in class-specific representation learning BIBREF140, BIBREF50  The scores further attest the usefulness of unsupervised post-processing as almost all class-specific correlation scores are improved by applying mean-centering and abtt  Finally, the results for m-bert and xlm-100 in Table TABREF46 further confirm that massively multilingual pretraining cannot yield reasonable semantic representations for many languages: in fact, for some classes they display no correlation with human ratings at all 
Differences across Languages  Naturally, the results from Tables TABREF43 and TABREF46 also reveal that there is variation in performance of both static word embeddings and pretrained encoders across different languages  Among other causes, the lowest absolute scores with ft are reported for languages with least resources available to train monolingual word embeddings, such as Kiswahili, Welsh, and Estonian  The low performance on Welsh is especially indicative: Figure FIGREF20 shows that the ratings in the Welsh dataset match up very well with the English ratings, but we cannot achieve the same level of correlation in Welsh with Welsh ft word embeddings  Difference in performance between two closely related languages, est (low-resource) and fin (high-resource), provides additional evidence in this respect 
The highest reported scores with m-bert and xlm-100 are obtained for Mandarin Chinese and Yue Chinese: this effectively points to the weaknesses of massively multilingual training with a joint subword vocabulary spanning 102 and 100 languages  Due to the difference in scripts, “language-specific” subwords for yue and cmn do not need to be shared across a vast amount of languages and the quality of their representation remains unscathed  This effectively means that m-bert's subword vocabulary contains plenty of cmn-specific and yue-specific subwords which are exploited by the encoder when producing m-bert-based representations  Simultaneously, higher scores with m-bert (and xlm in Table TABREF46) are reported for resource-rich languages such as French, Spanish, and English, which are better represented in m-bert's training data  We also observe lower absolute scores (and a larger number of OOVs) for languages with very rich and productive morphological systems such as the two Slavic languages (Polish and Russian) and Finnish  Since Polish and Russian are known to have large Wikipedias and Common Crawl data BIBREF33 (e g , their Wikipedias are in the top 10 largest Wikipedias worldwide), the problem with coverage can be attributed exactly to the proliferation of morphological forms in those languages 
Finally, while Table TABREF43 does reveal that unsupervised post-processing is useful for all languages, it also demonstrates that peak scores are achieved with different post-processing configurations  This finding suggests that a more careful language-specific fine-tuning is indeed needed to refine word embeddings towards semantic similarity  We plan to inspect the relationship between post-processing techniques and linguistic properties in more depth in future work 
Multilingual vs  Language-Specific Contextualized Embeddings  Recent work has shown that—despite the usefulness of massively multilingual models such as m-bert and xlm-100 for zero-shot cross-lingual transfer BIBREF12, BIBREF125—stronger results in downstream tasks for a particular language can be achieved by pretraining language-specific models on language-specific data  In this experiment, motivated by the low results of m-bert and xlm-100 (see again Table TABREF46), we assess if monolingual pretrained encoders can produce higher-quality word-level representations than multilingual models  Therefore, we evaluate language-specific bert and xlm models for a subset of the Multi-SimLex languages for which such models are currently available: Finnish BIBREF141 (bert-base architecture, uncased), French BIBREF142 (the FlauBERT model based on xlm), English (bert-base, uncased), Mandarin Chinese (bert-base) BIBREF29 and Spanish (bert-base, uncased)  In addition, we also evaluate a series of pretrained encoders available for English: (i) bert-base, bert-large, and bert-large with whole word masking (wwm) from the original work on BERT BIBREF29, (ii) monolingual “English-specific” xlm BIBREF30, and (iii) two models which employ parameter reduction techniques to build more compact encoders: albert-b uses a configuration similar to bert-base, while albert-l is similar to bert-large, but with an $18\times $ reduction in the number of parameters BIBREF143 
From the results in Table FIGREF49, it is clear that monolingual pretrained encoders yield much more reliable word-level representations  The gains are visible even for languages such as cmn which showed reasonable performance with m-bert and are substantial on all test languages  This further confirms the validity of language-specific pretraining in lieu of multilingual training, if sufficient monolingual data are available  Moreover, a comparison of pretrained English encoders in Figure FIGREF49 largely follows the intuition: the larger bert-large model yields slight improvements over bert-base, and we can improve a bit more by relying on word-level (i e , lexical-level) masking Finally, light-weight albert model variants are quite competitive with the original bert models, with only modest drops reported, and albert-l again outperforms albert-b  Overall, it is interesting to note that the scores obtained with monolingual pretrained encoders are on a par with or even outperform static ft word embeddings: this is a very intriguing finding per se as it shows that such subword-level models trained on large corpora can implicitly capture rich lexical semantic knowledge 
Similarity-Specialized Word Embeddings  Conflating distinct lexico-semantic relations is a well-known property of distributional representations BIBREF144, BIBREF53  Semantic specialization fine-tunes distributional spaces to emphasize a particular lexico-semantic relation in the transformed space by injecting external lexical knowledge BIBREF145  Explicitly discerning between true semantic similarity (as captured in Multi-SimLex) and broad conceptual relatedness benefits a number of tasks, as discussed in §SECREF4  Since most languages lack dedicated lexical resources, however, one viable strategy to steer monolingual word vector spaces to emphasize semantic similarity is through cross-lingual transfer of lexical knowledge, usually through a shared cross-lingual word vector space BIBREF106  Therefore, we evaluate the effectiveness of specialization transfer methods using Multi-SimLex as our multilingual test bed 
We evaluate a current state-of-the-art cross-lingual specialization transfer method with minimal requirements, put forth recently by Ponti:2019emnlp  In a nutshell, their li-postspec method is a multi-step procedure that operates as follows  First, the knowledge about semantic similarity is extracted from WordNet in the form of triplets, that is, linguistic constraints $(w_1, w_2, r)$, where $w_1$ and $w_2$ are two concepts, and $r$ is a relation between them obtained from WordNet (e g , synonymy or antonymy)  The goal is to “attract” synonyms closer to each other in the transformed vector space as they reflect true semantic similarity, and “repel” antonyms further apart  In the second step, the linguistic constraints are translated from English to the target language via a shared cross-lingual word vector space  To this end, following Ponti:2019emnlp we rely on cross-lingual word embeddings (CLWEs) BIBREF146 available online, which are based on Wiki ft vectors  Following that, a constraint refinement step is applied in the target language which aims to eliminate the noise inserted during the translation process  This is done by training a relation classification tool: it is trained again on the English linguistic constraints and then used on the translated target language constraints, where the transfer is again enabled via a shared cross-lingual word vector space  Finally, a state-of-the-art monolingual specialization procedure from Ponti:2018emnlp injects the (now target language) linguistic constraints into the target language distributional space 
The scores are summarized in Table TABREF56  Semantic specialization with li-postspec leads to substantial improvements in correlation scores for the majority of the target languages, demonstrating the importance of external semantic similarity knowledge for semantic similarity reasoning  However, we also observe deteriorated performance for the three target languages which can be considered the lowest-resource ones in our set: cym, swa, yue  We hypothesize that this occurs due to the inferior quality of the underlying monolingual Wikipedia word embeddings, which generates a chain of error accumulations  In particular, poor distributional word estimates compromise the alignment of the embedding spaces, which in turn results in increased translation noise, and reduced refinement ability of the relation classifier  On a high level, this “poor get poorer” observation again points to the fact that one of the primary causes of low performance of resource-low languages in semantic tasks is the sheer lack of even unlabeled data for distributional training  On the other hand, as we see from Table TABREF46, typological dissimilarity between the source and the target does not deteriorate the effectiveness of semantic specialization  In fact, li-postspec does yield substantial gains also for the typologically distant targets such as heb, cmn, and est  The critical problem indeed seems to be insufficient raw data for monolingual distributional training 
Cross-Lingual Evaluation
Similar to monolingual evaluation in §SECREF7, we now evaluate several state-of-the-art cross-lingual representation models on the suite of 66 automatically constructed cross-lingual Multi-SimLex datasets  Again, note that evaluating a full range of cross-lingual models available in the rich prior work on cross-lingual representation learning is well beyond the scope of this article  We therefore focus our cross-lingual analyses on several well-established and indicative state-of-the-art cross-lingual models, again spanning both static and contextualized cross-lingual word embeddings 
Cross-Lingual Evaluation ::: Models in Comparison
Static Word Embeddings  We rely on a state-of-the-art mapping-based method for the induction of cross-lingual word embeddings (CLWEs): vecmap BIBREF148  The core idea behind such mapping-based or projection-based approaches is to learn a post-hoc alignment of independently trained monolingual word embeddings BIBREF106  Such methods have gained popularity due to their conceptual simplicity and competitive performance coupled with reduced bilingual supervision requirements: they support CLWE induction with only as much as a few thousand word translation pairs as the bilingual supervision BIBREF149, BIBREF150, BIBREF151, BIBREF107  More recent work has shown that CLWEs can be induced with even weaker supervision from small dictionaries spanning several hundred pairs BIBREF152, BIBREF135, identical strings BIBREF153, or even only shared numerals BIBREF154  In the extreme, fully unsupervised projection-based CLWEs extract such seed bilingual lexicons from scratch on the basis of monolingual data only BIBREF103, BIBREF148, BIBREF155, BIBREF156, BIBREF104, BIBREF157 
Recent empirical studies BIBREF158, BIBREF135, BIBREF159 have compared a variety of unsupervised and weakly supervised mapping-based CLWE methods, and vecmap emerged as the most robust and very competitive choice  Therefore, we focus on 1) its fully unsupervised variant (unsuper) in our comparisons  For several language pairs, we also report scores with two other vecmap model variants: 2) a supervised variant which learns a mapping based on an available seed lexicon (super), and 3) a supervised variant with self-learning (super+sl) which iteratively increases the seed lexicon and improves the mapping gradually  For a detailed description of these variants, we refer the reader to recent work BIBREF148, BIBREF135  We again use CC+Wiki ft vectors as initial monolingual word vectors, except for yue where Wiki ft is used  The seed dictionaries of two different sizes (1k and 5k translation pairs) are based on PanLex BIBREF160, and are taken directly from prior work BIBREF135, or extracted from PanLex following the same procedure as in the prior work 
Contextualized Cross-Lingual Word Embeddings  We again evaluate the capacity of (massively) multilingual pretrained language models, m-bert and xlm-100, to reason over cross-lingual lexical similarity  Implicitly, such an evaluation also evaluates “the intrinsic quality” of shared cross-lingual word-level vector spaces induced by these methods, and their ability to boost cross-lingual transfer between different language pairs  We rely on the same procedure of aggregating the models' subword-level parameters into word-level representations, already described in §SECREF34 
As in monolingual settings, we can apply unsupervised post-processing steps such as abtt to both static and contextualized cross-lingual word embeddings 
Cross-Lingual Evaluation ::: Results and Discussion
Main Results and Differences across Language Pairs  A summary of the results on the 66 cross-lingual Multi-SimLex datasets are provided in Table TABREF59 and Figure FIGREF60  The findings confirm several interesting findings from our previous monolingual experiments (§SECREF44), and also corroborate several hypotheses and findings from prior work, now on a large sample of language pairs and for the task of cross-lingual semantic similarity 
First, we observe that the fully unsupervised vecmap model, despite being the most robust fully unsupervised method at present, fails to produce a meaningful cross-lingual word vector space for a large number of language pairs (see the bottom triangle of Table TABREF59): many correlation scores are in fact no-correlation results, accentuating the problem of fully unsupervised cross-lingual learning for typologically diverse languages and with fewer amounts of monolingual data BIBREF135  The scores are particularly low across the board for lower-resource languages such as Welsh and Kiswahili  It also seems that the lack of monolingual data is a larger problem than typological dissimilarity between language pairs, as we do observe reasonably high correlation scores with vecmap for language pairs such as cmn-spa, heb-est, and rus-fin  However, typological differences (e g , morphological richness) still play an important role as we observe very low scores when pairing cmn with morphologically rich languages such fin, est, pol, and rus  Similar to prior work of Vulic:2019we and doval2019onthe, given the fact that unsupervised vecmap is the most robust unsupervised CLWE method at present BIBREF158, our results again question the usefulness of fully unsupervised approaches for a large number of languages, and call for further developments in the area of unsupervised and weakly supervised cross-lingual representation learning 
The scores of m-bert and xlm-100 lead to similar conclusions as in the monolingual settings  Reasonable correlation scores are achieved only for a small subset of resource-rich language pairs (e g , eng, fra, spa, cmn) which dominate the multilingual m-bert training  Interestingly, the scores indicate a much higher performance of language pairs where yue is one of the languages when we use m-bert instead of vecmap  This boils down again to the fact that yue, due to its specific language script, has a good representation of its words and subwords in the shared m-bert vocabulary  At the same time, a reliable vecmap mapping between yue and other languages cannot be found due to a small monolingual yue corpus  In cases when vecmap does not yield a degenerate cross-lingual vector space starting from two monolingual ones, the final correlation scores seem substantially higher than the ones obtained by the single massively multilingual m-bert model 
Finally, the results in Figure FIGREF60 again verify the usefulness of unsupervised post-processing also in cross-lingual settings  We observe improved performance with both m-bert and xlm-100 when mean centering (+mc) is applied, and further gains can be achieved by using abtt on the mean-centered vector spaces  A similar finding also holds for static cross-lingual word embeddings, where applying abbt (-10) yields higher scores on 61/66 language pairs 
Fully Unsupervised vs  Weakly Supervised Cross-Lingual Embeddings  The results in Table TABREF59 indicate that fully unsupervised cross-lingual learning fails for a large number of language pairs  However, recent work BIBREF135 has noted that these sub-optimal non-alignment solutions with the unsuper model can be avoided by relying on (weak) cross-lingual supervision spanning only several thousands or even hundreds of word translation pairs  Therefore, we examine 1) if we can further improve the results on cross-lingual Multi-SimLex resorting to (at least some) cross-lingual supervision for resource-rich language pairs; and 2) if such available word-level supervision can also be useful for a range of languages which displayed near-zero performance in Table TABREF59  In other words, we test if recent “tricks of the trade” used in the rich literature on CLWE learning reflect in gains on cross-lingual Multi-SimLex datasets 
First, we reassess the findings established on the bilingual lexicon induction task BIBREF161, BIBREF135: using at least some cross-lingual supervision is always beneficial compared to using no supervision at all  We report improvements over the unsuper model for all 10 language pairs in Table TABREF66, even though the unsuper method initially produced strong correlation scores  The importance of self-learning increases with decreasing available seed dictionary size, and the +sl model always outperforms unsuper with 1k seed pairs; we observe the same patterns also with even smaller dictionary sizes than reported in Table TABREF66 (250 and 500 seed pairs)  Along the same line, the results in Table TABREF67 indicate that at least some supervision is crucial for the success of static CLWEs on resource-leaner language pairs  We note substantial improvements on all language pairs; in fact, the vecmap model is able to learn a more reliable mapping starting from clean supervision  We again note large gains with self-learning 
Multilingual vs  Bilingual Contextualized Embeddings  Similar to the monolingual settings, we also inspect if massively multilingual training in fact dilutes the knowledge necessary for cross-lingual reasoning on a particular language pair  Therefore, we compare the 100-language xlm-100 model with i) a variant of the same model trained on a smaller set of 17 languages (xlm-17); ii) a variant of the same model trained specifically for the particular language pair (xlm-2); and iii) a variant of the bilingual xlm-2 model that also leverages bilingual knowledge from parallel data during joint training (xlm-2++)  We again use the pretrained models made available by Conneau:2019nips, and we refer to the original work for further technical details 
The results are summarized in Figure FIGREF60, and they confirm the intuition that massively multilingual pretraining can damage performance even on resource-rich languages and language pairs  We observe a steep rise in performance when the multilingual model is trained on a much smaller set of languages (17 versus 100), and further improvements can be achieved by training a dedicated bilingual model  Finally, leveraging bilingual parallel data seems to offer additional slight gains, but a tiny difference between xlm-2 and xlm-2++ also suggests that this rich bilingual information is not used in the optimal way within the xlm architecture for semantic similarity 
In summary, these results indicate that, in order to improve performance in cross-lingual transfer tasks, more work should be invested into 1) pretraining dedicated language pair-specific models, and 2) creative ways of leveraging available cross-lingual supervision (e g , word translation pairs, parallel or comparable corpora) BIBREF121, BIBREF123, BIBREF124 with pretraining paradigms such as bert and xlm  Using such cross-lingual supervision could lead to similar benefits as indicated by the results obtained with static cross-lingual word embeddings (see Table TABREF66 and Table TABREF67)  We believe that Multi-SimLex can serve as a valuable means to track and guide future progress in this research area 
Conclusion and Future Work
We have presented Multi-SimLex, a resource containing human judgments on the semantic similarity of word pairs for 12 monolingual and 66 cross-lingual datasets  The languages covered are typologically diverse and include also under-resourced ones, such as Welsh and Kiswahili  The resource covers an unprecedented amount of 1,888 word pairs, carefully balanced according to their similarity score, frequency, concreteness, part-of-speech class, and lexical field  In addition to Multi-Simlex, we release the detailed protocol we followed to create this resource  We hope that our consistent guidelines will encourage researchers to translate and annotate Multi-Simlex -style datasets for additional languages  This can help and create a hugely valuable, large-scale semantic resource for multilingual NLP research 
The core Multi-SimLex we release with this paper already enables researchers to carry out novel linguistic analysis as well as establishes a benchmark for evaluating representation learning models  Based on our preliminary analyses, we found that speakers of closely related languages tend to express equivalent similarity judgments  In particular, geographical proximity seems to play a greater role than family membership in determining the similarity of judgments across languages  Moreover, we tested several state-of-the-art word embedding models, both static and contextualized representations, as well as several (supervised and unsupervised) post-processing techniques, on the newly released Multi-SimLex  This enables future endeavors to improve multilingual representation learning with challenging baselines  In addition, our results provide several important insights for research on both monolingual and cross-lingual word representations:
1) Unsupervised post-processing techniques (mean centering, elimination of top principal components, adjusting similarity orders) are always beneficial independently of the language, although the combination leading to the best scores is language-specific and hence needs to be tuned 
2) Similarity rankings obtained from word embeddings for nouns are better aligned with human judgments than all the other part-of-speech classes considered here (verbs, adjectives, and, for the first time, adverbs)  This confirms previous generalizations based on experiments on English 
3) The factor having the greatest impact on the quality of word representations is the availability of raw texts to train them in the first place, rather than language properties (such as family, geographical area, typological features) 
4) Massively multilingual pretrained encoders such as m-bert BIBREF29 and xlm-100 BIBREF30 fare quite poorly on our benchmark, whereas pretrained encoders dedicated to a single language are more competitive with static word embeddings such as fastText BIBREF31  Moreover, for language-specific encoders, parameter reduction techniques reduce performance only marginally 
5) Techniques to inject clean lexical semantic knowledge from external resources into distributional word representations were proven to be effective in emphasizing the relation of semantic similarity  In particular, methods capable of transferring such knowledge from resource-rich to resource-lean languages BIBREF59 increased the correlation with human judgments for most languages, except for those with limited unlabelled data 
Future work can expand our preliminary, yet large-scale study on the ability of pretrained encoders to reason over word-level semantic similarity in different languages  For instance, we have highlighted how sharing the same encoder parameters across multiple languages may harm performance  However, it remains unclear if, and to what extent, the input language embeddings present in xlm-100 but absent in m-bert help mitigate this issue  In addition, pretrained language embeddings can be obtained both from typological databases BIBREF99 and from neural architectures BIBREF100  Plugging these embeddings into the encoders in lieu of embeddings trained end-to-end as suggested by prior work BIBREF162, BIBREF163, BIBREF164 might extend the coverage to more resource-lean languages 
Another important follow-up analysis might involve the comparison of the performance of representation learning models on multilingual datasets for both word-level semantic similarity and sentence-level Natural Language Understanding  In particular, Multi-SimLex fills a gap in available resources for multilingual NLP and might help understand how lexical and compositional semantics interact if put alongside existing resources such as XNLI BIBREF84 for natural language inference or PAWS-X BIBREF89 for cross-lingual paraphrase identification  Finally, the Multi-SimLex annotation could turn out to be a unique source of evidence to study the effects of polysemy in human judgments on semantic similarity: for equivalent word pairs in multiple languages, are the similarity scores affected by how many senses the two words (or multi-word expressions) incorporate?
In light of the success of initiatives like Universal Dependencies for multilingual treebanks, we hope that making Multi-SimLex and its guidelines available will encourage other researchers to expand our current sample of languages  We particularly encourage creation and submission of comparable Multi-SimLex datasets for under-resourced and typologically diverse languages in future work  In particular, we have made a Multi-Simlex community website available to facilitate easy creation, gathering, dissemination, and use of annotated datasets: https://multisimlex com/ 
This work is supported by the ERC Consolidator Grant LEXICAL: Lexical Acquisition Across Languages (no 648909)  Thierry Poibeau is partly supported by a PRAIRIE 3IA Institute fellowship ("Investissements d'avenir" program, reference ANR-19-P3IA-0001) 
Context 3: Introduction
Relation classification is the task of assigning sentences with two marked entities to a predefined set of relations  The sentence “We poured the <e1>milk</e1> into the <e2>pumpkin mixture</e2> ”, for example, expresses the relation Entity-Destination(e1,e2)  While early research mostly focused on support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1 , recent research showed performance improvements by applying neural networks (NNs) BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 on the benchmark data from SemEval 2010 shared task 8 BIBREF8  
This study investigates two different types of NNs: recurrent neural networks (RNNs) and convolutional neural networks (CNNs) as well as their combination  We make the following contributions:
(1) We propose extended middle context, a new context representation for CNNs for relation classification  The extended middle context uses all parts of the sentence (the relation arguments, left of the relation arguments, between the arguments, right of the arguments) and pays special attention to the middle part 
(2) We present connectionist bi-directional RNN models which are especially suited for sentence classification tasks since they combine all intermediate hidden layers for their final decision  Furthermore, the ranking loss function is introduced for the RNN model optimization which has not been investigated in the literature for relation classification before 
(3) Finally, we combine CNNs and RNNs using a simple voting scheme and achieve new state-of-the-art results on the SemEval 2010 benchmark dataset 
Related Work
In 2010, manually annotated data for relation classification was released in the context of a SemEval shared task BIBREF8   Shared task participants used, i a , support vector machines or maximum entropy classifiers BIBREF0 , BIBREF1   Recently, their results on this data set were outperformed by applying NNs BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6  
zeng2014 built a CNN based only on the context between the relation arguments and extended it with several lexical features  kim2014 and others used convolutional filters of different sizes for CNNs  nguyen applied this to relation classification and obtained improvements over single filter sizes  deSantos2015 replaced the softmax layer of the CNN with a ranking layer  They showed improvements and published the best result so far on the SemEval dataset, to our knowledge 
socher used another NN architecture for relation classification: recursive neural networks that built recursive sentence representations based on syntactic parsing  In contrast, zhang investigated a temporal structured RNN with only words as input  They used a bi-directional model with a pooling layer on top 
Convolutional Neural Networks (CNN)
CNNs perform a discrete convolution on an input matrix with a set of different filters  For NLP tasks, the input matrix represents a sentence: Each column of the matrix stores the word embedding of the corresponding word  By applying a filter with a width of, e g , three columns, three neighboring words (trigram) are convolved  Afterwards, the results of the convolution are pooled  Following collobertWeston, we perform max-pooling which extracts the maximum value for each filter and, thus, the most informative n-gram for the following steps  Finally, the resulting values are concatenated and used for classifying the relation expressed in the sentence 
Input: Extended Middle Context
One of our contributions is a new input representation especially designed for relation classification  The contexts are split into three disjoint regions based on the two relation arguments: the left context, the middle context and the right context  Since in most cases the middle context contains the most relevant information for the relation, we want to focus on it but not ignore the other regions completely  Hence, we propose to use two contexts: (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context  Due to the repetition of the middle context, we force the network to pay special attention to it  The two contexts are processed by two independent convolutional and max-pooling layers  After pooling, the results are concatenated to form the sentence representation  Figure FIGREF3 depicts this procedure  It shows an examplary sentence: “He had chest pain and <e1>headaches</e1> from <e2>mold</e2> in the bedroom ” If we only considered the middle context “from”, the network might be tempted to predict a relation like Entity-Origin(e1,e2)  However, by also taking the left and right context into account, the model can detect the relation Cause-Effect(e2,e1)  While this could also be achieved by integrating the whole context into the model, using the whole context can have disadvantages for longer sentences: The max pooling step can easily choose a value from a part of the sentence which is far away from the mention of the relation  With splitting the context into two parts, we reduce this danger  Repeating the middle context increases the chance for the max pooling step to pick a value from the middle context 
Convolutional Layer
Following previous work (e g , BIBREF5 , BIBREF6 ), we use 2D filters spanning all embedding dimensions  After convolution, a max pooling operation is applied that stores only the highest activation of each filter  We apply filters with different window sizes 2-5 (multi-windows) as in BIBREF5 , i e  spanning a different number of input words 
Recurrent Neural Networks (RNN)
Traditional RNNs consist of an input vector, a history vector and an output vector  Based on the representation of the current input word and the previous history vector, a new history is computed  Then, an output is predicted (e g , using a softmax layer)  In contrast to most traditional RNN architectures, we use the RNN for sentence modeling, i e , we predict an output vector only after processing the whole sentence and not after each word  Training is performed using backpropagation through time BIBREF9 which unfolds the recurrent computations of the history vector for a certain number of time steps  To avoid exploding gradients, we use gradient clipping with a threshold of 10 BIBREF10  
Input of the RNNs
Initial experiments showed that using trigrams as input instead of single words led to superior results  Hence, at timestep INLINEFORM0 we do not only give word INLINEFORM1 to the model but the trigram INLINEFORM2 by concatenating the corresponding word embeddings 
Connectionist Bi-directional RNNs
Especially for relation classification, the processing of the relation arguments might be easier with knowledge of the succeeding words  Therefore in bi-directional RNNs, not only a history vector of word INLINEFORM0 is regarded but also a future vector  This leads to the following conditioned probability for the history INLINEFORM1 at time step INLINEFORM2 : DISPLAYFORM0
Thus, the network can be split into three parts: a forward pass which processes the original sentence word by word (Equation EQREF6 ); a backward pass which processes the reversed sentence word by word (Equation ); and a combination of both (Equation )  All three parts are trained jointly  This is also depicted in Figure FIGREF7  
Combining forward and backward pass by adding their hidden layer is similar to BIBREF7   We, however, also add a connection to the previous combined hidden layer with weight INLINEFORM0 to be able to include all intermediate hidden layers into the final decision of the network (see Equation )  We call this “connectionist bi-directional RNN” 
In our experiments, we compare this RNN with uni-directional RNNs and bi-directional RNNs without additional hidden layer connections 
Word Representations
Words are represented by concatenated vectors: a word embedding and a position feature vector 
Pretrained word embeddings  In this study, we used the word2vec toolkit BIBREF11 to train embeddings on an English Wikipedia from May 2014

ground_truth: Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese

Chinese Mandarin, Welsh, English, Estonian, Finnish, French, Hebrew, Polish, Russian, Spanish, Kiswahili, Yue Chinese

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. How do they obtain the new context represetation?
Context 3:  We only considered words appearing more than 100 times and added a special PADDING token for convolution  This results in an embedding training text of about 485,000 terms and INLINEFORM0 tokens  During model training, the embeddings are updated 
Position features  We incorporate randomly initialized position embeddings similar to zeng2014, nguyen and deSantos2015  In our RNN experiments, we investigate different possibilities of integrating position information: position embeddings, position embeddings with entity presence flags (flags indicating whether the current word is one of the relation arguments), and position indicators BIBREF7  
Objective Function: Ranking Loss
Ranking  We applied the ranking loss function proposed in deSantos2015 to train our models  It maximizes the distance between the true label INLINEFORM0 and the best competitive label INLINEFORM1 given a data point INLINEFORM2   The objective function is DISPLAYFORM0
with INLINEFORM0 and INLINEFORM1 being the scores for the classes INLINEFORM2 and INLINEFORM3 respectively  The parameter INLINEFORM4 controls the penalization of the prediction errors and INLINEFORM5 and INLINEFORM6 are margins for the correct and incorrect classes  Following deSantos2015, we set INLINEFORM7   We do not learn a pattern for the class Other but increase its difference to the best competitive label by using only the second summand in Equation EQREF10 during training 
Experiments and Results
We used the relation classification dataset of the SemEval 2010 task 8 BIBREF8   It consists of sentences which have been manually labeled with 19 relations (9 directed relations and one artificial class Other)  8,000 sentences have been distributed as training set and 2,717 sentences served as test set  For evaluation, we applied the official scoring script and report the macro F1 score which also served as the official result of the shared task 
RNN and CNN models were implemented with theano BIBREF12 , BIBREF13   For all our models, we use L2 regularization with a weight of 0 0001  For CNN training, we use mini batches of 25 training examples while we perform stochastic gradient descent for the RNN  The initial learning rates are 0 2 for the CNN and 0 01 for the RNN  We train the models for 10 (CNN) and 50 (RNN) epochs without early stopping  As activation function, we apply tanh for the CNN and capped ReLU for the RNN  For tuning the hyperparameters, we split the training data into two parts: 6 5k (training) and 1 5k (development) sentences  We also tuned the learning rate schedule on dev 
Beside of training single models, we also report ensemble results for which we combined the presented single models with a voting process 
Performance of CNNs
As a baseline system, we implemented a CNN similar to the one described by zeng2014  It consists of a standard convolutional layer with filters with only one window size, followed by a softmax layer  As input it uses the middle context  In contrast to zeng2014, our CNN does not have an additional fully connected hidden layer  Therefore, we increased the number of convolutional filters to 1200 to keep the number of parameters comparable  With this, we obtain a baseline result of 73 0  After including 5 dimensional position features, the performance was improved to 78 6 (comparable to 78 9 as reported by zeng2014 without linguistic features) 
In the next step, we investigate how this result changes if we successively add further features to our CNN: multi-windows for convolution (window sizes: 2,3,4,5 and 300 feature maps each), ranking layer instead of softmax and our proposed extended middle context  Table TABREF12 shows the results  Note that all numbers are produced by CNNs with a comparable number of parameters  We also report F1 for increasing the word embedding dimensionality from 50 to 400  The position embedding dimensionality is 5 in combination with 50 dimensional word embeddings and 35 with 400 dimensional word embeddings  Our results show that especially the ranking layer and the embedding size have an important impact on the performance 
Performance of RNNs
As a baseline for the RNN models, we apply a uni-directional RNN which predicts the relation after processing the whole sentence  With this model, we achieve an F1 score of 61 2 on the SemEval test set 
Afterwards, we investigate the impact of different position features on the performance of uni-directional RNNs (position embeddings, position embeddings concatenated with a flag indicating whether the current word is an entity or not, and position indicators BIBREF7 )  The results indicate that position indicators (i e  artificial words that indicate the entity presence) perform the best on the SemEval data  We achieve an F1 score of 73 4 with them  However, the difference to using position embeddings with entity flags is not statistically significant 
Similar to our CNN experiments, we successively vary the RNN models by using bi-directionality, by adding connections between the hidden layers (“connectionist”), by applying ranking instead of softmax to predict the relation and by increasing the word embedding dimension to 400 
The results in Table TABREF14 show that all of these variations lead to statistically significant improvements  Especially the additional hidden layer connections and the integration of the ranking layer have a large impact on the performance 
Combination of CNNs and RNNs
Finally, we combine our CNN and RNN models using a voting process  For each sentence in the test set, we apply several CNN and RNN models presented in Tables TABREF12 and TABREF14 and predict the class with the most votes  In case of a tie, we pick one of the most frequent classes randomly  The combination achieves an F1 score of 84 9 which is better than the performance of the two NN types alone  It, thus, confirms our assumption that the networks provide complementary information: while the RNN computes a weighted combination of all words in the sentence, the CNN extracts the most informative n-grams for the relation and only considers their resulting activations 
Comparison with State of the Art
Table TABREF16 shows the results of our models ER-CNN (extended ranking CNN) and R-RNN (ranking RNN) in the context of other state-of-the-art models  Our proposed models obtain state-of-the-art results on the SemEval 2010 task 8 data set without making use of any linguistic features 
Conclusion
In this paper, we investigated different features and architectural choices for convolutional and recurrent neural networks for relation classification without using any linguistic features  For convolutional neural networks, we presented a new context representation for relation classification  Furthermore, we introduced connectionist recurrent neural networks for sentence classification tasks and performed the first experiments with ranking recurrent neural networks  Finally, we showed that even a simple combination of convolutional and recurrent neural networks improved results  With our neural models, we achieved new state-of-the-art results on the SemEval 2010 task 8 benchmark data 
Acknowledgments
Heike Adel is a recipient of the Google European Doctoral Fellowship in Natural Language Processing and this research is supported by this fellowship 
This research was also supported by Deutsche Forschungsgemeinschaft: grant SCHU 2246/4-2 

ground_truth: They use two independent convolutional and max-pooling layers on (1) a combination of the left context, the left entity and the middle context; and (2) a combination of the middle context, the right entity and the right context. They concatenated the two results after pooling to get the new context representation.