dataesr/scientific-paragraphs-categorization huggingface dataset:
We construct such V for each language as the intersection of word types covered by the corresponding CC+Wiki fastText vectors and the (single-word or multi-word) expressions appearing in the corresponding Multi-SimLex dataset. Finally, note that it is not feasible to evaluate a full range of available pretrained encoders within the scope of this work. Our main intention is to provide the first set of baseline results on Multi-SimLex by benchmarking a sample of most popular encoders, at the same time also investigating other important questions such as performance of static versus contextualized word embeddings, or multilingual versus language-specific pretraining. Another purpose of the experiments is to outline the wide potential and applicability of the Multi-SimLex datasets for multilingual and cross-lingual representation learning evaluation.
Results and Discussion The results we report are Spearman's ρ coefficients of the correlation between the ranks derived from the scores of the evaluated models and the human scores provided in each Multi-SimLex dataset. The main results with static and contextualized word vectors for all test languages are summarized in Table  12. The scores reveal several interesting patterns, and also pinpoint the main challenges for future work. 315. 318. 436. 400. 575. 444. 428. 370. 359. 432. 332. 376 (2)   FT:+MC. 373. 337. 445. 404. 583. 463. 447. 383. 378. 447. 373. 427 (3)  FT:+ABTT  (-3). 459. 343. 453. 404. 584. 487. 447. 387. 394. 456. 423. 429 (4) FT:+ABTT  (-10). 496. 323. 460. 385. 581. 494. 460. 401. 400. 477. 406. 399 (5)   FT:+UNCOVEC. 518. 328. 469. 375. 568. 483. 449. 389. 387. 469. 386.
)+(3). 526. 323. 470. 369. 564. 495. 448. 392. 392. 473. 388. 388 (1)+(  2 )+(  5 )+(  4 ). 526. 307. 471. 355. 548. 495. 450. 394. 394. 476. 382. 396. 458. 044. 256. 122. 173. 183. 128. 097. 123. 203. 128. 469 (3) M-BERT:+ABTT  (-3). 487. 056. 321. 137. 200. 287. 144. 126. 197. 299. 135. 492 (4) M-BERT:+ABTT  (-10). 456. 056. 329. 122. 164. 306. 121. 126. 183. 315. 136. 467 (5)   2 )+(  5 )+(3) M-BERT (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (0) (. 464. 083. 326. 130. 201. 304. 149. 122. 199. 295. 148. 456 (1)+(  2 )+(  5 )+  (4). 444. 086. 326. 112. 179. 305. 135. 127. 187. 285. 119. 447 Table  12 : A summary of results (Spearman's ρ correlation scores) on the full monolingual Multi-SimLex datasets for 12 languages. We benchmark fastText word embeddings trained on two different corpora (CC+Wiki and only Wiki) as well the multilingual M-BERT model (see §7.
Another finding is that a single massively multilingual model such as M-BERT cannot produce semantically rich word-level representations. Whether this actually happens because the training objective is different-or because the need to represent 100+ languages reduces its language-specific capacity-is investigated further below. The overall results also clearly indicate that (i) there are differences in performance across different monolingual Multi-SimLex datasets, and (ii) unsupervised post-   (469). 511. 265. 408. 379. 527. 551. 458. 384. 464. 499. 391. 252 ADJ  (245). 448. 338. 564. 401. 546. 616. 467. 284. 349. 401. 344. 288 ADV  (123). 622. 187. 482. 378. 547. 648. 491. 266. 514. 423. 172 Table  13 : Spearman's ρ correlation scores over the four POS classes represented in Multi-SimLex datasets.
In addition to the word vectors considered earlier in Table  12 , we also report scores for another contextualized model, XLM-100. The numbers in parentheses refer to the total number of POS-class pairs in the original ENG dataset and, consequently, in all other monolingual datasets. processing is universally useful, and can lead to huge improvements in correlation scores for many languages. In what follows, we also delve deeper into these analyses. Impact of Unsupervised Post-Processing. First, the results in Table  12  suggest that applying dimension-wise mean centering to the initial vector spaces has positive impact on word similarity scores in all test languages and for all models, both static and contextualized (see the +MC rows in Table  12 ). Mimno and Thompson (2017)  show that distributional word vectors have a tendency towards narrow clusters in the vector space (i.
In summary, pretrained word embeddings do contain more information pertaining to semantic similarity than revealed in the initial vectors. This way, we have corroborated the hypotheses from prior work  (Mu, Bhat, and Viswanath 2018; Artetxe et al. 2018)  which were not previously empirically verified on other languages due to a shortage of evaluation data; this gap has now been filled with the introduction of the Multi-SimLex datasets. In all follow-up experiments, we always explicitly denote which post-processing configuration is used in evaluation. POS-Specific Subsets. We present the results for subsets of word pairs grouped by POS class in Table  13. Prior work based on English data showed that representations for nouns are typically of higher quality than those for the other POS classes  (Schwartz, Reichart, and Rappoport 2015, 2016; Vulić et al.
Finally, the results for M-BERT and XLM-100 in Table  13  further confirm that massively multilingual pretraining cannot yield reasonable semantic representations for many languages: in fact, for some classes they display no correlation with human ratings at all. Differences across Languages. Naturally, the results from Tables 12 and 13 also reveal that there is variation in performance of both static word embeddings and pretrained encoders across different languages. Among other causes, the lowest absolute scores with FT are reported for languages with least resources available to train monolingual word embeddings, such as Kiswahili, Welsh, and Estonian. The low performance on Welsh is especially indicative: Figure  1  shows that the ratings in the Welsh dataset match up very well with the English ratings, but we cannot achieve the same level of correlation in Welsh with Welsh FT word embeddings.
Similarity-Specialized Word Embeddings. Conflating distinct lexico-semantic relations is a well-known property of distributional representations  (Turney and Pantel 2010; Melamud et al. 2016). Semantic specialization fine-tunes distributional spaces to emphasize a particular lexico-semantic relation in the transformed space by injecting external lexical knowledge  (Glavaš, Ponti, and Vulić 2019). Explicitly discerning between true semantic similarity (as captured in Multi-SimLex) and broad conceptual relatedness benefits a number of tasks, as discussed in §2. 1. 17 Since most languages lack dedicated lexical resources, however, one viable strategy to steer monolingual word vector spaces to emphasize semantic similarity is through cross-lingual transfer of lexical knowledge, usually through a shared cross-lingual word vector space  (Ruder, Vulić, and Søgaard 2019).
Cross-Lingual Evaluation Similar to monolingual evaluation in §7, we now evaluate several state-of-the-art crosslingual representation models on the suite of 66 automatically constructed cross-lingual Multi-SimLex datasets. Again, note that evaluating a full range of cross-lingual models Table  14 : The impact of vector space specialization for semantic similarity. The scores are reported using the current state-of-the-art specialization transfer LI-POSTSPEC method of  Ponti et al. (2019c) , relying on English as a resource-rich source language and the external lexical semantic knowledge from the English WordNet. available in the rich prior work on cross-lingual representation learning is well beyond the scope of this article. We therefore focus our cross-lingual analyses on several wellestablished and indicative state-of-the-art cross-lingual models, again spanning both static and contextualized cross-lingual word embeddings.
The seed dictionaries of two different sizes (1k and 5k translation pairs) are based on PanLex  (Kamholz, Pool, and Colowick 2014)   2) The scores above the main diagonal are computed by obtaining 768-dimensional word-level vectors from pretrained multilingual BERT (M-BERT) following the procedure described in §7. 1. For both fully unsupervised VECMAP and M-BERT, we report the results with unsupervised postprocessing enabled: all 2 × 66 reported scores are obtained using the +ABBT (-10) variant. taken directly from prior work  (Vulić et al. 2019 ), 21  or extracted from PanLex following the same procedure as in the prior work. Contextualized Cross-Lingual Word Embeddings. We again evaluate the capacity of (massively) multilingual pretrained language models, M-BERT and XLM-100, to reason over cross-lingual lexical similarity.
Results and Discussion Main Results and Differences across Language Pairs. A summary of the results on the 66 cross-lingual Multi-SimLex datasets are provided in Table  15  and Figure  6a. The findings confirm several interesting findings from our previous monolingual experiments ( §7. 2), and also corroborate several hypotheses and findings from prior work, now on a large sample of language pairs and for the task of cross-lingual semantic similarity. First, we observe that the fully unsupervised VECMAP model, despite being the most robust fully unsupervised method at present, fails to produce a meaningful crosslingual word vector space for a large number of language pairs (see the bottom triangle A comparison of various pretrained encoders available for the English-French language pair, see the main text for a short description of each benchmarked pretrained encoder.
A similar finding also holds for static cross-lingual word embeddings  23  , where applying ABBT (-10) yields higher scores on 61/66 language pairs. Fully Unsupervised vs. Weakly Supervised Cross-Lingual Embeddings. The results in Table  15  indicate that fully unsupervised cross-lingual learning fails for a large number of language pairs. However, recent work  (Vulić et al. 2019 ) has noted that these sub-optimal non-alignment solutions with the UNSUPER model can be avoided by relying on (weak) cross-lingual supervision spanning only several thousands or even hundreds of word translation pairs. Therefore, we examine 1) if we can further improve the results on cross-lingual Multi-SimLex resorting to (at least some) cross-lingual supervision for resource-rich language pairs; and 2) if such available word-level supervision can also be useful for a range of languages which displayed near-zero performance in Table  15.
In other words, we test if recent "tricks of the trade" used in the rich literature on CLWE learning reflect in gains on cross-lingual Multi-SimLex datasets. First, we reassess the findings established on the bilingual lexicon induction task  (Søgaard, Ruder, and Vulić 2018; Vulić et al. 2019 ): using at least some cross-lingual supervision is always beneficial compared to using no supervision at all. We report improvements over the UNSUPER model for all 10 language pairs in Table  16 , even though the UNSUPER method initially produced strong correlation scores. The importance of self-learning increases with decreasing available seed dictionary size, and the +SL model always outperforms UNSUPER with 1k seed pairs; we observe the same patterns also with even smaller dictionary sizes than reported in Table  16  (250 and 500 seed pairs).
Conclusion and Future Work We have presented Multi-SimLex, a resource containing human judgments on the semantic similarity of word pairs for 12 monolingual and 66 cross-lingual datasets. The languages covered are typologically diverse and include also under-resourced ones, such as Welsh and Kiswahili. The resource covers an unprecedented amount of 1,888 word pairs, carefully balanced according to their similarity score, frequency, concreteness, part-of-speech class, and lexical field. In addition to Multi-Simlex, we release the detailed protocol we followed to create this resource. We hope that our consistent guidelines will encourage researchers to translate and annotate Multi-Simlex -style datasets for additional languages. This can help and create a hugely valuable, large-scale semantic resource for multilingual NLP research.
The core Multi-SimLex we release with this paper already enables researchers to carry out novel linguistic analysis as well as establishes a benchmark for evaluating representation learning models. Based on our preliminary analyses, we found that speakers of closely related languages tend to express equivalent similarity judgments. In particular, geographical proximity seems to play a greater role than family membership in determining the similarity of judgments across languages. Moreover, we tested several state-of-the-art word embedding models, both static and contextualized representations, as well as several (supervised and unsupervised) post-processing techniques, on the newly released Multi-SimLex. This enables future endeavors to improve multilingual representation learning with challenging baselines.
Future work can expand our preliminary, yet large-scale study on the ability of pretrained encoders to reason over word-level semantic similarity in different languages. For instance, we have highlighted how sharing the same encoder parameters across multiple languages may harm performance. However, it remains unclear if, and to what extent, the input language embeddings present in XLM-100 but absent in M-BERT help mitigate this issue. In addition, pretrained language embeddings can be obtained both from typological databases  (Littell et al. 2017 ) and from neural architectures  (Malaviya, Neubig, and Littell 2017). Plugging these embeddings into the encoders in lieu of embeddings trained end-to-end as suggested by prior work  (Tsvetkov et al. 2016; Ammar et al. 2016; Ponti et al. 2019b ) might extend the coverage to more resource-lean languages.
Another important follow-up analysis might involve the comparison of the performance of representation learning models on multilingual datasets for both word-level semantic similarity and sentence-level Natural Language Understanding. In particular, Multi-SimLex fills a gap in available resources for multilingual NLP and might help understand how lexical and compositional semantics interact if put alongside existing resources such as XNLI  (Conneau et al. 2018b ) for natural language inference or PAWS-X  (Yang et al. 2019)  for cross-lingual paraphrase identification. Finally, the Multi-SimLex annotation could turn out to be a unique source of evidence to study the effects of polysemy in human judgments on semantic similarity: for equivalent word pairs in multiple languages, are the similarity scores affected by how many senses the two words (or multi-word expressions) incorporate? In light of the success of initiatives like Universal Dependencies for multilingual treebanks, we hope that making Multi-SimLex and its guidelines available will encourage other researchers to expand our current sample of languages.
Spearman's ρ correlation scores on all 66 cross-lingual datasets. 1)The scores below the main diagonal are computed based on cross-lingual word embeddings (CLWEs) induced by aligning CC+Wiki FT in all languages (except for YUE where we use Wiki FT) in a fully unsupervised way (i. e. , without any bilingual supervision). We rely on a standard CLWE mapping-based (i. e. , alignment) approach: VECMAP(Artetxe,  Labaka, and Agirre 2018b).
SEMEVAL-500 pairs; and similarly for adjectives. 3 Unlike SEMEVAL-500 and CARD-660, we do not explicitly control for the equal representation of concept pairs across each similarity interval for several reasons: a) Multi-SimLex contains a substantially larger number of concept pairs, so it is possible to extract balanced samples from the full data; b) such balance, even if imposed on the English dataset, would be distorted in all other monolingual and cross-lingual datasets; c) balancing over similarity intervals arguably does not reflect a true distribution "in the wild" where most concepts are only loosely related or completely unrelated.
Basal stem rot caused by Ganoderma boninense is the major threat to oil palm cultivation in South-East Asia, which accounts for 80% of palm oil production worldwide, and this disease is increasing in Africa. The use of resistant planting material as part of an integrated pest management of this disease is one sustainable solution. However, breeding for Ganoderma resistance requires long-term and costly research, which could greatly benefit from marker assisted selection (MAS). In this study, we evaluated the effectiveness of an in silico genetic mapping approach that took advantage of extensive data recorded in an ongoing breeding program. A pedigree-based QTL mapping approach applied to more than 10 years' worth of data collected during pre-nursery tests revealed the quantitative nature of Ganoderma resistance and identified underlying loci segregating in genetic diversity that is directly relevant for the breeding program supporting the study.
2014; Pootakham et al. 2015). This is even more problematic for field studies that are indispensable to assess genetic diversity in an agronomic context, whose implementation is very costly and would result in lower production income due to the disease context. A powerful and cost-effective approach is to directly use the databases compiled in ongoing breeding programs, which are typically large and obtained from diverse relevant genetic backgrounds, to map in silico the QTLs for the traits of interest  (Parisseaux and Bernardo 2004). Despite the potential of this approach, data from breeding programs are unique, mainly because of a complex genetic design that may be biased due to selection, or unbalanced phenotyping coverage. Thus, they require appropriate statistical models for their development and evaluation in contrasted contexts, which are currently an active research topic  (Würschum 2012; Garin et al.
2017; Korontzis et al. 2020). In oil palm, an in silico QTL mapping approach based on the two step variance component approach considering identity by descent (IBD) information  (George et al. 2000; van Eeuwijk et al. 2010 ) yielded promising results on production traits recorded in large scale evaluation genetic trials  (Tisné et al. 2015). This approach was successfully extended to survival data and applied to a multi-parent population to detect Ganoderma resistance QTLs in the field  (Tisné et al. 2017). A Bayesian approach to pedigree based QTL mapping using IBD information was also developed in the 2000s and implemented in the FlexQTL software (van de  Weg et al. 2004; Bink et al. 2008). This made it possible to carry out increasing numbers of studies in several crops that share the constraints and potential described above for oil palm, in particular for disease resistance in strawberry  (Mangandi et al.
2017; Anciro et al. 2018)  or in apple (van de  Weg et al. 2018). In this study, we evaluated the potential of an in silico approach based on the large existing databases of a longterm oil palm breeding program for the study of Ganoderma resistance. We genotyped an existing DNA bank primarily established for identity checking purpose and performed a pedigree-based QTL mapping using data recorded in Ganoderma pre-nursery trials over a period of more than ten years. We then assessed the consistency of pre-nursery QTL effects in natural field conditions using a database recording the Ganoderma infection status over years for the palms planted in genealogical gardens. Thus, using a cost-effective approach that is directly relevant to the breeding program, we were able to study two major issues, i.
Inoculated seedlings were observed every four weeks for the appearance of the first external disease symptom, on average between 8 and 12 weeks after inoculation of the germinated seeds, after which the disease symptoms were recorded at two weekly intervals as (1) infected and (0) if not infected. The trial was stopped when the average percentage of infected seedlings within the group of control crosses reached 30%, usually around 34 weeks after inoculation of the germinated seeds. This study included 102 Ganoderma pre-nursery screening test trial, covering 10 years of data recording. The trials performed between 2007 and 2017 represented the evaluation of 4,017 unique crosses, from either GA×GA, GA×GB or GB×GB genetic background. Considering that the purpose of this study was to assess the genetic bases of Ganoderma resistance in the commercial genetic material, only the GA×GB crosses were taken into consideration (n=3,792), derived from 2,037 and 340 individuals from the GA and GB respectively.
Note that this second model is a linear mixed model (LMM). Both models enabled prediction of the best linear unbiased predictor (BLUP) for each GB individual used in the QTL mapping,  % being replaced by an identity matrix in order to avoid using the pedigree information that was subsequently used in the QTL analysis. Both statistical models were performed using ASReml-R software  (Butler et al. 2007, V4)  and resulted in two vectors of BLUP for group B individuals that were used in subsequent QTL mapping analysis.
Molecular data and genetic map construction The 334 freeze-dried oil palm leaf samples available at the Cirad DNA-bank for the GB individuals included in the analysis were genotyped with 199 SSR markers developed in different studies. Among the 199 markers, 177 markers were developed by Cirad  (Billotte et al. 2005) , four markers were developed by the Malaysian Palm Oil Board (MPOB)  (Zaki et al. 2012 ) and 18 expressed sequence tags markers were developed by IRD (Institut de Recherche pour le Développement) and Cirad  (Tranbarger et al. 2012 ). These markers were selected based on a previous integrated pedigree-based genetic map constructed from a population of related individuals  (Cochard et al. 2015). Selection was for a uniform distribution in the genome and the highest level of polymorphism in both LM and YBI genetic backgrounds.
DNA extraction, evaluation of the DNA concentrations and microsatellite fragment amplification were performed using the protocol described in  Cochard et al. (2015). Genemapper© V4. 1 (Applied Biosystems, USA) software was used to determine the size of the alleles. Three genetic maps were constructed, one for each of LM and YBI population and one integrated map using the pedigree-based linkage mapping software CRI-MAP v2. 4  (Green et al. 1990 ), as described in  Cochard et al. (2015). Consistency of marker calling across pedigrees and absence of spurious rates of double recombination events were checked using both CRI-MAP and FlexQTL TM , and data were improved where necessary. Genetic maps were drawn using MapChart v2. 0 software (Voorrips 2002) and are presented in Supporting Information Figure  S1.
(2017). The appearance of the first Ganoderma symptom (T1S, first observation of score 2-6) was recorded and the associated time was considered as survival time, i. e. , time from planting to the time the event occurred. The survival data were analyzed using the Cox model integrating a fixed effect for the date of planting: 𝜆(𝑡, 𝑋) = 𝜆 / (𝑡)𝑒 0 2 (1) where  is the time to the event or censoring,  / denotes the baseline hazard function,  is the  ×  design matrix relating the survival outcome for individuals to date of planting effects ( = 8) and  = ( " , … ,  3 ) is a  × 1 unknown vector. The effects of pre-nursery QTL were evaluated using the likelihood ratio test, for which the limiting distribution follows a chi-squared distribution, between the model (1) and the following model (2): 𝜆(𝑡, 𝑋) = 𝜆 / (𝑡)𝑒 0240 $ -(2) with  -being the {0,1,2} vector of pre-nursery-based QTL genotypes for the individuals and  the QTL effect.