dataesr/scientific-paragraphs-categorization huggingface dataset:
2018). Our goal is to make use of these improvements towards a larger, more representative, and more reliable lexical similarity dataset in English and, consequently, in all other languages. The Final Output: English Multi-SimLex. In order to ensure that the criterion C1 is satisfied, we consolidate and integrate the data already carefully sampled in prior work into a single, comprehensive, and representative dataset. This way, we can control for diversity, frequency, and other properties while avoiding to perform this time-consuming selection process from scratch. Note that, on the other hand, the word pairs chosen for English are scored from scratch as part of the entire Multi-SimLex annotation process, introduced later in §5. We now describe the external data sources for the final set of word pairs: 1) Source: SimLex-999.
(Hill, Reichart, and Korhonen 2015). The English Multi-SimLex has been initially conceived as an extension of the original SimLex-999 dataset. Therefore, we include all 999 word pairs from SimLex, which span 666 noun pairs, 222 verb pairs, and 111 adjective pairs. While SimLex-999 already provides examples representing different POS classes, it does not have a sufficient coverage of different linguistic phenomena: for instance, it contains only very frequent concepts, and it does not provide a representative set of verbs  (Gerz et al. 2016). 2) Source: SemEval-17: Task 2 (henceforth SEMEVAL-500;  Camacho-Collados et al. 2017). We start from the full dataset of 500 concept pairs to extract a total of 334 concept pairs for English Multi-SimLex a) which contain only single-word concepts, b) which are not named entities, c) where POS tags of the two concepts are the same, d) where both concepts occur in the top 250K most frequent word types in the English Wikipedia, and e) do not already occur in SimLex-999.
The original concepts were sampled as to span all the 34 domains available as part of BabelDomains (Camacho-Collados and Navigli 2017), which roughly correspond to the main high-level Wikipedia categories. This ensures topical diversity in our sub-sample. 3) Source: CARD-660  (Pilehvar et al. 2018). 67 word pairs are taken from this dataset focused on rare word similarity, applying the same selection criteria a) to e) employed for SEMEVAL-500. Words are controlled for frequency based on their occurrence counts from the Google News data and the ukWaC corpus  (Baroni et al. 2009). CARD-660 contains some words that are very rare (logboat), domain-specific (erythroleukemia) and slang (2mrw), which might be difficult to translate and annotate across a wide array of languages. Hence, we opt for retaining only the concept pairs above the threshold of top 250K most frequent Wikipedia concepts, as above.
4) Source: SimVerb-3500 (Gerz et al. 2016) Since both CARD-660 and SEMEVAL-500 are heavily skewed towards noun pairs, and nouns also dominate the original SimLex-999, we also extract additional verb pairs from the verb-specific similarity dataset SimVerb-3500. We randomly sample 244 verb pairs from SimVerb-3500 that represent all similarity spectra. In particular, we add 61 verb pairs for each of the similarity intervals: [0, 1. 5),  [1. 5, 3), [3, 4. 5 ),  [4. 5, 6]. Since verbs in SimVerb-3500 were originally chosen from VerbNet  (Kipper, Snyder, and Palmer 2004; Kipper et al. 2008) , they cover a wide range of verb classes and their related linguistic phenomena. 5) Source: University of South Florida (USF; Nelson, McEvoy, and Schreiber 2004) norms, the largest database of free association for English.
In order to improve the representation of different POS classes, we sample additional adjectives and adverbs from the USF norms following the procedure established by  Hill, Reichart, and Korhonen (2015); Gerz et al. (2016). This yields additional 122 adjective pairs, but only a limited number of adverb pairs (e. g. , later -never, now -here, once -twice). Therefore, we also create a set of adverb pairs semi-automatically by sampling adjectives that can be derivationally transformed into adverbs (e. g. adding the suffix -ly) from the USF, and assessing the correctness of such derivation in WordNet. The resulting pairs include, for instance, primarily -mainly, softly -firmly, roughly -reliably, etc. We include a total of 123 adverb pairs into the final English Multi-SimLex. Note that this is the first time adverbs are included into any semantic similarity dataset.
Fulfillment of Construction Criteria. The final ENG Multi-SimLex dataset spans 1,051 noun pairs, 469 verb pairs, 245 adjective pairs, and 123 adverb pairs. 2 As mentioned above, the criterion C1 has been fulfilled by relying only on word pairs that already underwent meticulous sampling processes in prior work, integrating them into a single resource. As a consequence, Multi-SimLex allows for fine-grained analyses over different POS classes, concreteness levels, similarity spectra, frequency intervals, relation types, morphology, lexical fields, and it also includes some challenging orthographically similar examples (e. g. , infection -inflection). 3 We ensure that the criteria C2 and C3 are satisfied by using similar annotation guidelines as Simlex-999, SimVerb-3500, and SEMEVAL-500 that explicitly target semantic similarity.
Multi-SimLex: Translation and Annotation We now detail the development of the final Multi-SimLex resource, describing our language selection process, as well as translation and annotation of the resource, including the steps taken to ensure and measure the quality of this resource. We also provide key data statistics and preliminary cross-lingual comparative analyses. Language Selection. Multi-SimLex comprises eleven languages in addition to English. The main objective for our inclusion criteria has been to balance language prominence (by number of speakers of the language) for maximum impact of the resource, while simultaneously having a diverse suite of languages based on their typological features (such as morphological type and language family). Table  1  summarizes key information about the languages currently included in Multi-SimLex.
We have included a mixture of fusional, agglutinative, isolating, and introflexive languages that come from eight different language families. This includes languages that are very widely used such as Chinese Mandarin and Spanish, and low-resource languages such as Welsh and Kiswahili. We hope to further include additional languages and inspire other researchers to contribute to the effort over the lifetime of this project. The work on data collection can be divided into two crucial phases: 1) a translation phase where the extended English language dataset with 1,888 pairs (described in §4) is translated into eleven target languages, and 2) an annotation phase where human raters scored each pair in the translated set as well as the English set. Detailed guidelines for both phases are available online at: https://multisimlex.
This is also used to remove duplicate pairs in the translated set, by differentiating the duplicates using a variant at each instance. Further, many translation instances were resolved using near-synonymous terms in the translation. For example, the words in the pair: wood -timber can only be directly translated in Estonian to puit, and are not distinguishable. Therefore, the translators approximated the translation for timber to the compound noun puitmaterjal (literally: wood material) in order to produce a valid pair in the target language. In some cases, a direct transliteration from English is used. For example, the pair: physician and Table  2 : Inter-translator agreement (% of matched translated words) by independent translators using a randomly selected 100-pair English sample from the Multi-SimLex dataset, and the corresponding 100-pair samples from the other datasets.
Guidelines and Word Pair Scoring Across all languages, 145 human annotators were asked to score all 1,888 pairs (in their given language). We finally collect at least ten valid annotations for each word pair in each language. All annotators were required to abide by the following instructions: 1. Each annotator must assign an integer score between 0 and 6 (inclusive) indicating how semantically similar the two words in a given pair are. A score of 6 indicates very high similarity (i. e. , perfect synonymy), while zero indicates no similarity. 2. Each annotator must score the entire set of 1,888 pairs in the dataset. The pairs must not be shared between different annotators. 3. Annotators are able to break the workload over a period of approximately 2-3 weeks, and are able to use external sources (e.
5 , where µ i is the mean of the other annotators' scores. Round 3: We compute the average agreement for each annotator (with the other annotators), by measuring the average Spearman's correlation against all other annotators. We discard the scores of annotators that have shown the least average agreement with all other annotators, while we maintain at least ten annotators per language by the end of this round. The actual process is done in multiple iterations: (S1) we measure the average agreement for each annotator with every other annotator (this corresponds to the APIAA measure, see later); (S2) if we still have more than 10 valid annotators and the lowest average score is higher than in the previous iteration, we remove the lowest one, and rerun S1. Table  3  shows the number of annotators at both the start (Round 1) and end (Round 3) of our process for each language.
We measure the agreement between annotators using two metrics, average pairwise inter-annotator agreement (APIAA), and average mean inter-annotator agreement (AMIAA). Both of these use Spearman's correlation (ρ) between annotators scores, the only difference is how they are averaged. They are computed as follows: 1)APIAA = 2 i,j ρ(s i , s j ) N (N -1) 2)AMIAA = i ρ(s i , µ i ) N , where: µ i = j,j =i s j N -1 (1) where ρ(s i , s j ) is the Spearman's correlation between annotators i and j's scores (s i ,s j ) for all pairs in the dataset, and N is the number of annotators. APIAA has been used widely as the standard measure for inter-annotator agreement, including in the original SimLex paper  (Hill, Reichart, and Korhonen 2015). It simply averages the pairwise Spearman's correlation between all annotators.
On the other hand, AMIAA compares the average Spearman's correlation of one held-out annotator with the average of all the other N -1 annotators, and then averages across all N 'held-out' annotators. It smooths individual annotator effects and arguably serves as a better upper bound than APIAA  (Gerz et al. 2016; Vulić et al. 2017a; Pilehvar et al. 2018, inter alia). We present the respective APIAA and AMIAA scores in Table  4  and Table 5  for all part-of-speech subsets, as well as the agreement for the full datasets. As reported in prior work  (Gerz et al. 2016; Vulić et al. 2017a ), AMIAA scores are typically higher than APIAA scores. Crucially, the results indicate 'strong agreement' (across all languages) using both measurements. The languages with the highest annotator agreement were French (FRA) and Yue Chinese (YUE), while Russian (RUS) had the lowest overall IAA scores.
g, CMN-YUE, RUS-POL, EST-FIN). In addition, we observe high correlations between English and most other languages, as expected. This is due to the effect of using English as the base/anchor language to create the dataset. In simple words, if one translates to two languages L 1 and L 2 starting from the same set of pairs in English, it is higly likely that L 1 and L 2 will diverge from English in different ways. Therefore, the similarity between L 1 -ENG and L 2 -ENG is expected to be higher than between L 1 -L 2 , especially if L 1 and L 2 are typologically dissimilar languages (e. g. , HEB-CMN, see Figure  1 ). This phenomenon is well documented in related prior work  (Leviant and Reichart 2015; Camacho-Collados et al. 2017; Mrkšić et al. 2017; Vulić, Ponzetto, and Glavaš 2019). While we acknowledge this as a slight artifact of the dataset design, it would otherwise be impossible to construct a semantically aligned and comprehensive dataset across a large number of languages.
We also report differences in the distribution of the frequency of words among the languages in Multi-SimLex. Figure  2  shows six example languages, where each bar segment shows the proportion of words in each language that occur in the given frequency range. For example, the 10K-20K segment of the bars represents the proportion  of words in the dataset that occur in the list of most frequent words between the frequency rank of 10,000 and 20,000 in that language; likewise with other intervals. Frequency lists for the presented languages are derived from Wikipedia and Common Crawl corpora. 4 hile many concept pairs are direct or approximate translations of English pairs, we can see that the frequency distribution does vary across different languages, and is also related to inherent language properties.
For instance, in Finnish and Russian, while we use infinitive forms of all verbs, conjugated verb inflections are often more frequent in raw corpora than the corresponding infinitive forms. The variance can also be partially explained by the difference in monolingual corpora size used to derive the frequency rankings in the first place: absolute vocabulary sizes are expected to fluctuate across different languages. However, it is also important to note that the datasets also contain subsets of lower-frequency and rare words, which can be used for rare word evaluations in multiple languages, in the spirit of  Pilehvar et al. (2018) 's English rare word dataset. Cross-Linguistic Differences. Table  7  shows some examples of average similarity scores of English, Spanish, Kiswahili and Welsh concept pairs.
In particular, we extract feature vectors from URIEL  (Littell et al. 2017 ), a massively multilingual typological database that collects and normalizes information compiled by grammarians and field linguists about the world's languages. In particular, we focus on information about geography (the areas where the language speakers are concentrated), family (the phylogenetic tree each language belongs to), and typology (including syntax, phonological inventory, and phonology). 5 Moreover, we consider typological representations of languages that are not manually crafted by experts, but rather learned from texts. Malaviya, Neubig, and   Littell (2017)  proposed to construct such representations by training language-identifying vectors end-to-end as part of neural machine translation models. The vector for similarity judgments and the vector of linguistic features for a given language have different dimensionality.
Cross-Lingual Multi-SimLex Datasets A crucial advantage of having semantically aligned monolingual datasets across different languages is the potential to create cross-lingual semantic similarity datasets. ) as an intrinsic evaluation task. However, the cross-lingual datasets previous work relied upon  (Camacho-Collados et al. 2017)  were limited to a homogeneous set of high-resource languages (e. g. , English, German, Italian, Spanish) and a small number of concept pairs (all less than 1K pairs). We address both problems by 1) using a typologically more diverse language sample, and 2) relying on a substantially larger English dataset as a source for the cross-lingual datasets: 1,888 pairs in this work versus 500 pairs in the work of  Camacho-Collados et al. (2017). As a result, each of our cross-lingual datasets contains a substantially larger number of concept pairs, as shown in Table  11.
The cross-lingual Multi-Simlex datasets are constructed automatically, leveraging word pair translations and annotations collected in all 12 languages. This yields a total of 66 cross-lingual datasets, one for each possible combination of languages. Table  11  provides the final number of concept pairs, which lie between 2,031 and 3,480 pairs for each cross-lingual dataset, whereas Table  10  shows some sample pairs with their corresponding similarity scores. The automatic creation and verification of cross-lingual datasets closely follows the procedure first outlined by Camacho-Collados,  Pilehvar, and Navigli (2015)  and later adopted by  Camacho-Collados et al. (2017)  (for semantic similarity) and  Vulić, Ponzetto, and Glavaš (2019)  (for graded lexical entailment). First, given two languages, we intersect their aligned concept pairs obtained through translation.
For instance, starting from the aligned pairs attroupement -foule in French and rahvasumm -rahvahulk in Estonian, we construct two cross-lingual pairs attroupement -rahvaluk and rahvasumm -foule. The scores of cross-lingual pairs are then computed as averages of the two corresponding monolingual scores. Finally, in order to filter out concept pairs whose semantic meaning was not preserved during this operation, we retain only cross-lingual pairs for which the  To assess the quality of the resulting cross-lingual datasets, we have conducted a verification experiment similar to  Vulić, Ponzetto, and Glavaš (2019). We randomly sampled 300 concept pairs in the English-Spanish, English-French, and English-Mandarin cross-lingual datasets. Subsequently, we asked bilingual native speakers to provide similarity judgments of each pair.
The Spearman's correlation score ρ between automatically induced and manually collected ratings achieves ρ ≥ 0. 90 on all samples, which confirms the viability of the automatic construction procedure. Score and Class Distributions. The summary of score and class distributions across all 66 cross-lingual datasets are provided in Figure  4a  and Figure  4b , respectively. First, it is obvious that the distribution over the four POS classes largely adheres to that of the original monolingual Multi-SimLex datasets, and that the variance is quite low: e. g. , the ENG-FRA dataset contains the lowest proportion of nouns (49. 21%) and the highest proportion of verbs (27. 1%), adjectives (15. 28%), and adverbs (8. 41%). On the other hand, the distribution over similarity intervals in Figure  4a  shows a much greater variance.
This is again expected as this pattern resurfaces in monolingual datasets (see Table  6 ). It is also evident that the data are skewed towards lower-similarity concept pairs. However, due to the joint size of all cross-lingual datasets (see Table  11 ), even the least represented intervals contain a substantial number of concept pairs. For instance, the RUS-YUE dataset contains the least highly similar concept pairs (in the interval  [4, 6] ) of all 66 cross-lingual datasets. Nonetheless, the absolute number of pairs (138) in that interval for RUS-YUE is still substantial. If needed, this makes it possible to create smaller datasets which are balanced across the similarity spectra through sub-sampling. [0, 1) [1, 2) [2, 3) [3, 4) [4,.
Monolingual Evaluation of Representation Learning Models After the numerical and qualitative analyses of the Multi-SimLex datasets provided in § § 5. 3-5. 4, we now benchmark a series of representation learning models on the new evaluation data. We evaluate standard static word embedding algorithms such as fastText  (Bojanowski et al. 2017) , as well as a range of more recent text encoders pretrained on language modeling such as multilingual BERT  (Devlin et al. 2019). These experiments provide strong baseline scores on the new Multi-SimLex datasets and offer a first largescale analysis of pretrained encoders on word-level semantic similarity across diverse languages. In addition, the experiments now enabled by Multi-SimLex aim to answer several important questions. (Q1) Is it viable to extract high-quality word-level representations from pretrained encoders receiving subword-level tokens as input? Are such representations competitive with standard static word-level embeddings? (Q2) What are the implications of monolingual pretraining versus (massively) multilingual pretraining for performance? (Q3) Do lightweight unsupervised post-processing techniques improve word representations consistently across different languages? (Q4) Can we effectively transfer available external lexical knowledge from resource-rich languages to resourcelean languages in order to learn word representations that distinguish between true similarity and conceptual relatedness (see the discussion in §2.
Models in Comparison Static Word Embeddings in Different Languages. First, we evaluate a standard method for inducing non-contextualized (i. e. , static) word embeddings across a plethora of different languages: FASTTEXT (FT) vectors  (Bojanowski et al. 2017 ) are currently the most popular and robust choice given 1) the availability of pretrained vectors in a large number of languages  (Grave et al. 2018 ) trained on large Common Crawl (CC) plus Wikipedia (Wiki) data, and 2) their superior performance across a range of NLP tasks  (Mikolov et al. 2018 ). In fact, FASTTEXT is an extension of the standard word-level CBOW and skip-gram word2vec models  (Mikolov et al. 2013 ) that takes into account subwordlevel information, i. e. the contituent character n-grams of each word  (Zhu, Vulić, and Korhonen 2019).
In short, it transforms the input space X into an adjusted space XW α through a linear map W α controlled by a single hyper-parameter α. The n th -6 We have also trained standard word-level CBOW and skip-gram with negative sampling (SGNS) on full Wikipedia dumps for several languages, but our preliminary experiments have verified that they under-perform compared to FASTTEXT. This finding is consistent with other recent studies demonstrating the usefulness of subword-level information  (Vania and Lopez 2017; Mikolov et al. 2018; Zhu, Vulić, and Korhonen 2019; Zhu et al. 2019). Therefore, we do not report the results with CBOW and SGNS for brevity. 7 https://fasttext. cc/docs/en/crawl-vectors. html 8 https://fasttext. cc/docs/en/pretrained-vectors. html order similarity transformation of the input word vector space X (for which n = 1) can be obtained as M n (X) = M 1 (XW (n-1)/2 ), with W α = QΓ α , where Q and Γ are the matrices obtained via eigendecomposition of X T X = QΓQ T.
Note that different perturbations can also be stacked: e. g. , we can apply UNCOVEC and then use ABTT on top the output UNCOVEC vectors. When using UNCOVEC and ABTT we always length-normalize and mean-center the data first (i. e. , we apply the simple MC normalization). Finally, we tune the two hyper-parameters d A (for ABTT) and α (UNCOVEC) on the English Multi-SimLex and use the same values on the datasets of all other languages; we report results with dd A = 3 or dd A = 10, and α = -0. 3. Contextualized Word Embeddings. We also evaluate the capacity of unsupervised pretraining architectures based on language modeling objectives to reason over lexical semantic similarity. To the best of our knowledge, our article is the first study performing such analyses. State-of-the-art models such as BERT  (Devlin et al.
M-BERT is pretrained on monolingual Wikipedia corpora of 102 languages (comprising all Multi-SimLex languages) with a 12-layer Transformer network, and yields 768-dimensional representations. Since the concept pairs in Multi-SimLex are lowercased, we use the uncased version of M-BERT. 10 M-BERT comprises all Multi-SimLex languages, and its evident ability to perform cross-lingual transfer  (Pires, Schlinger, and Garrette 2019; Wu and Dredze 2019; Wang et al. 2020 ) also makes it a convenient baseline model for cross-lingual experiments later in §8. The second multilingual model we consider, XLM-100, 11 is pretrained on Wikipedia dumps of 100 languages, and encodes each concept into a 1, 280dimensional representation. In contrast to M-BERT, XLM-100 drops the next-sentence prediction objective and adds a cross-lingual masked language modeling objective.