dataesr/scientific-paragraphs-categorization huggingface dataset:
The overall quality of the MERIT model has been tested at most gravity station heights. The differences can be seen in Fig. C4  and basic statistical data in Table  C5. Largest differences were observed in Croatia, Czech Republic, France, and Hungary most likely due to the low quality of the position of gravity stations. Table  C1. List of DEMs used for test and mass correction calculations in the "most inner zone" of the TOPOSK program (Appendix D) of the individual countries; the grid spacing, sources and internet references are given. The letters stand for the techniques used in the DEM compilation: "L" for lidar, "P" for Photogrammetry, "TM" for heights from digitized topographic maps, and "MERIT" and "SRTM" for the radar data.
Source Grid     The Toposk software  (Zahorec et al. , 2017a)  is designed for the calculation of the gravitational effect of the near-terrain masses for both "near-terrain effect" (NTE) and "mass correction" (MC), i. e. , the total masses between the topography and the zero level -geoid or ellipsoid (we point out the difference from the terrain correction (TC), which represents only masses exceeding the classical "Bouguer shell"). The program is suitable for highly accurate calculations in rugged terrain using high-resolution DTMs. Different DTMs, with increasing resolution towards the calculation point, are used within particular zones. By default the program uses the following zoning: T1: inner zone (0-250 m from the calculation point), T2: intermediate zone (250-5240 m) and outer zones: T31 (5.
The TriTop software  (Holzrichter et al. , 2019)  is an adaptive algorithm for MC based on a triangulated polyhedral representation of the topography. The runtime of the algorithm is improved by an automatic resampling of topography. The topography is resampled in a quadtree structure. The high resolution of the topography is only considered if it has a significant influence on the gravitational effect at the station and not only by the distance to the station. Therefore, there are no default zone radius definitions, but the resolution depends only on the gravitational effect and differs for each station. In comparison to Toposk, TriTop does not consider a high-resolution zone (T1; see above) and does not interpolate topography in this zone in dependence to station height. The DTM heights are not modified.
The programs were compared to each other on different sets of points from Slovakia and Austria. Mainly the second comparison was important because of the typical Alpine terrain character of the majority of the territory in Austria. The obtained results by the Toposk and TriTop software were compared with previously computed mass corrections (NTE) from the Austrian gravity database. This comparison was realized on a set of 28 420 points with the ellipsoidal heights ranging from 158. 35 to 2898. 78 m. The character of the differences between mass corrections from the Austrian gravity database and NTE calculations by means of the Toposk and TriTop programs is visible from histograms in Fig. D1. Finally, the Toposk software was selected for the recalculation of MC effects due to better statistical parameters (median and standard deviations) and the absence of outliers in the calculations.
The differences in MC of both algorithms are observed in areas where stations are located close to steep slopes in topography. The differences of the results in Austria are caused by the main difference of both algorithms and in particular the handling of the inner zone T1. TriTop does not change or interpolate the topography around the station. This might lead to larger correction values in areas of highly rugged terrain due to steep slopes close to the station or even in cases in which the station height is slightly below the DTM. The comparison shows that in the area of highly rugged terrain the inner zone just around a station should be handled separately from the rest. Therefore, we decided to perform mass corrections with the Toposk software.
D2 Comparison of mass corrections For most countries, we used the available local detailed DEMs (Appendix C) with the resolution of 10-20 m (derived mainly from lidar data) for calculation in the innermost Toposk zone (T1). For all other zones we chose the best available global DEMs. We got good results with SRTM models for outer zones. For the intermediate zone T2, we decided to use the MERIT model based on our tests (Appendix C). MERIT was also used for the inner zone if local models were unavailable. This model (resampled to a 1 arcsec resolution) showed better height accuracy compared to other global models (based on the height residues at the points of the databases tested) and consequently minor differences in MC compared to local models (Fig. D2 ). The mentioned height residues of individual points of the databases in relation to local (or MERIT) models were subsequently used as a control criterion.
In particular, we consider points with height residues greater than ±50 m to be untrustworthy, and they were excluded from the CBA compilation process. The following graphs and maps are compiled without these excluded points. There are options to verify calculated MC values and estimate their error. For some databases, we had the original MC or TC values, which allows us to compare and control different approaches. Figure  D3  shows graphs and statistical comparisons for some countries. The maximum differences are at the level of several milligals, and the RMS error in most cases is below 1 mGal. Note that the graphs do not show excluded points (above ±50 m height criterion), where significant differences in MC may be obtained. Another possibility.
• N, 2 • E and the upper right at coordinates 51 • N, 23 • E. 6. 1. 2 Relevant specifications Pan-Alpine_Gravity_ database_2020. datThisfilecontains all results organized into seven columns -Lon, Lat, EH, CBA, FA, MC, BC -which respectively correspond to Lon = longitude (decimal degrees, ETRS89), https://doi. org/10. 5194/essd-13-2165-2021EarthSyst. Sci. Data, 13, 2165Data, 13,  -2209Data, 13,  , 2021.
Volume 1, Number 1 and cross-lingual benchmarks, we evaluate and analyze a wide array of recent state-of-the-art monolingual and cross-lingual representation models, including static and contextualized word embeddings (such as fastText, M-BERT and XLM), externally informed lexical representations, as well as fully unsupervised and (weakly) supervised cross-lingual word embeddings. We also present a step-by-step dataset creation protocol for creating consistent, Multi-Simlex -style resources for additional languages. We make these contributions -the public release of Multi-SimLex datasets, their creation protocol, strong baseline results, and in-depth analyses which can be be helpful in guiding future developments in multilingual lexical semantics and representation learning -available via a website which will encourage community effort in further expansion of Multi-Simlex to many more languages.
Vulić et al. Multi-SimLex low-resource languages typically lack similar evaluation data. Even if some resources do exist, they are limited in their size (e. g. , 500 pairs in Turkish (Ercan and Yıldız 2018), 500 in Farsi (Camacho-Collados et al. 2017), or 300 in Finnish (Venekoski and Vankka 2017)) and coverage (e. g. , all datasets which originated from the original English SimLex-999 contain only high-frequent concepts, and are dominated by nouns). This is why, as our departure point, we introduce a larger and more comprehensive English word similarity dataset spanning 1,888 concept pairs (see §4). Most importantly, semantic similarity datasets in different languages have been created using heterogeneous construction procedures with different guidelines for translation and annotation, as well as different rating scales.
For instance, some datasets were obtained by directly translating the English SimLex-999 in its entirety  (Leviant and Reichart 2015; Mrkšić et al. 2017)  or in part (Venekoski and Vankka 2017). Other datasets were created from scratch (Ercan and Yıldız 2018) and yet others sampled English concept pairs differently from SimLex-999 and then translated and reannotated them in target languages (Camacho-Collados et al. 2017). This heterogeneity makes these datasets incomparable and precludes systematic cross-linguistic analyses. In this article, consolidating the lessons learned from previous dataset construction paradigms, we propose a carefully designed translation and annotation protocol for developing monolingual Multi-SimLex datasets with aligned concept pairs for typologically diverse languages.
We apply this protocol to a set of 12 languages, including a mixture of major languages (e. g. , Mandarin, Russian, and French) as well as several low-resource ones (e. g. , Kiswahili, Welsh, and Yue Chinese). We demonstrate that our proposed dataset creation procedure yields data with high inter-annotator agreement rates (e. g. , the average mean inter-annotator agreement for Welsh is 0. 742). The unified construction protocol and alignment between concept pairs enables a series of quantitative analyses. Preliminary studies on the influence that polysemy and cross-lingual variation in lexical categories (see §2. 3) have on similarity judgments are provided in §5. Data created according to Multi-SimLex protocol also allow for probing into whether similarity judgments are universal across languages, or rather depend on linguistic affinity (in terms of linguistic features, phylogeny, and geographical location).
We investigate this question in §5. 4. Naturally, Multi-SimLex datasets can be used as an intrinsic evaluation benchmark to assess the quality of lexical representations based on monolingual, joint multilingual, and transfer learning paradigms. We conduct a systematic evaluation of several state-of-the-art representation models in §7, showing that there are large gaps between human and system performance in all languages. The proposed construction paradigm also supports the automatic creation of 66 cross-lingual Multi-SimLex datasets by interleaving the monolingual ones. We outline the construction of the cross-lingual datasets in §6, and then present a quantitative evaluation of a series of cutting-edge cross-lingual representation models on this benchmark in §8. Contributions. We now summarize the main contributions of this work: 1) Building on lessons learned from prior work, we create a more comprehensive lexical semantic similarity dataset for the English language spanning a total of 1,888 concept pairs balanced with respect to similarity, frequency, and concreteness, and covering four word classes: nouns, verbs, adjectives and, for the first time, adverbs.
Besides steering research in multilingual parsing  (Zeman et al. 2018; Kondratyuk and Straka 2019; Doitch et al. 2019 ) and cross-lingual parser transfer  (Rasooli and Collins 2017; Lin et al. 2019; Rotman and Reichart 2019) , the consistent annotations and guidelines have also enabled a range of insightful comparative studies focused on the languages' syntactic (dis)similarities (Bjerva and Augenstein 2018;  Ponti et al. 2018a; Pires, Schlinger, and Garrette 2019). Inspired by the UD work and its substantial impact on research in (multilingual) syntax, in this article we introduce Multi-SimLex, a suite of manually and consistently annotated semantic datasets for 12 different languages, focused on the fundamental lexical relation of semantic similarity  (Budanitsky and Hirst 2006; Hill, Reichart, and Korhonen 2015).
For any pair of words, this relation measures whether their referents share the same (functional) features, as opposed to general cognitive association captured by co-occurrence patterns in texts (i. e. , the distributional information). Datasets that quantify the strength of true semantic similarity between concept pairs such as SimLex-999  (Hill, Reichart, and Korhonen 2015)  or  SimVerb-3500 (Gerz et al. 2016 ) have been instrumental in improving models for distributional semantics and representation learning. Discerning between semantic similarity and relatedness/association is not only crucial for theoretical studies on lexical semantics (see §2), but has also been shown to benefit a range of language understanding tasks in NLP. Examples include dialog state tracking  (Mrkšić et al. 2017; Ren et al.
2018) , spoken language understanding  (Kim et al. 2016; Kim, de Marneffe, and Fosler-Lussier 2016) , text simplification  (Glavaš and Vulić 2018; Ponti et al. 2018b; Lauscher et al. 2019) , dictionary and thesaurus construction  (Cimiano, Hotho, and Staab 2005; Hill et al. 2016). Despite the proven usefulness of semantic similarity datasets, they are available only for a small and typologically narrow sample of resource-rich languages such as German, Italian, and Russian  (Leviant and Reichart 2015) , whereas some language types and some crucial issues with prior efforts focused on the creation of multi-lingual semantic resources, namely: i) limited coverage; ii) heterogeneous annotation guidelines; and iii) concept pairs which are semantically incomparable across different languages. 3) We offer to the community manually annotated evaluation sets of 1,888 concept pairs across 12 typologically diverse languages, and 66 large cross-lingual evaluation sets.
6) We make all of these resources available on a website which facilitates easy creation, submission and sharing of Multi-Simlex-style datasets for a larger number of languages. We hope that this will yield an even larger repository of semantic resources that inspire future advances in NLP within and across languages. In light of the success of Universal Dependencies  (Nivre et al. 2019) , we hope that our initiative will instigate a collaborative public effort with established and clear-cut guidelines that will result in additional Multi-SimLex datasets in a large number of languages in the near future. Moreover, we hope that it will provide means to advance our understanding of distributional and lexical semantics across a large number of languages. All monolingual and cross-lingual Multi-SimLex datasets-along with detailed translation and annotation guidelines-are available online at: https://multisimlex.
Semantic similarity and association overlap to some degree, but do not coincide  (Kiela, Hill, and Clark 2015; Vulić, Kiela, and Korhonen 2017). In fact, there exist plenty of pairs that are intuitively associated but not similar. Pairs where the converse is true can also be encountered, although more rarely. An example are synonyms where a word is common and the other infrequent, such as to seize and to commandeer. Hill, Reichart, and Korhonen (2015)  revealed that while similarity measures based on the WordNet graph  (Wu and Palmer 1994)  and human judgments of association in the University of South Florida Free Association Database  (Nelson, McEvoy, and Schreiber 2004)  do correlate, a number of pairs follow opposite trends. Several studies on human cognition also point in the same direction.
As a consequence, resources providing clean information on semantic similarity are key in mitigating the side effects of the distributional signal. In particular, such databases can be employed for the intrinsic evaluations of specific WE models as a proxy of their reliability for downstream applications  (Collobert and Weston 2008; Baroni and Lenci 2010; Hill, Reichart, and Korhonen 2015) ; intuitively, the more WEs are misaligned with human judgments of similarity, the more their performance on actual tasks is expected to be degraded. Moreover, word representations can be specialized (a. k. a. retrofitted) by disentangling word relations of similarity and association. In particular, linguistic constraints sourced from external databases (such as synonyms from WordNet) can be injected into WEs  (Faruqui et al.
Previous Work and Evaluation Data Word Pair Datasets. Rich expert-created resources such as WordNet  (Miller 1995; Fellbaum 1998) , VerbNet  (Kipper Schuler 2005; Kipper et al. 2008) , or FrameNet  (Baker, Fillmore, and Lowe 1998)  encode a wealth of semantic and syntactic information, but are expensive and time-consuming to create. The scale of this problem gets multiplied by the number of languages in consideration. Therefore, crowd-sourcing with non-expert annotators has been adopted as a quicker alternative to produce smaller and more focused semantic resources and evaluation benchmarks. This alternative practice has had a profound impact on distributional semantics and representation learning  (Hill, Reichart, and Korhonen 2015). While some prominent English word pair datasets such as WordSim-353  (Finkelstein et al.
2002) , MEN  (Bruni, Tran, and Baroni 2014) , or Stanford Rare Words  (Luong, Socher, and Manning 2013)  did not discriminate between similarity and relatedness, the importance of this distinction was established by  Hill, Reichart, and Korhonen (2015 , see again the discussion in §2. 1) through the creation of SimLex-999. This inspired other similar datasets which focused on different lexical properties. For instance,  SimVerb-3500 (Gerz et al. 2016 ) provided similarity ratings for 3,500 English verbs, whereas CARD-660  (Pilehvar et al. 2018 ) aimed at measuring the semantic similarity of infrequent concepts. Semantic Similarity Datasets in Other Languages. Motivated by the impact of datasets such as SimLex-999 and SimVerb-3500 on representation learning in English, a line of related work focused on creating similar resources in other languages.
The dominant approach is translating and reannotating the entire original English SimLex-999 dataset, as done previously for German, Italian, and Russian  (Leviant and Reichart 2015) , Hebrew and Croatian  (Mrkšić et al. 2017) , and Polish  (Mykowiecka, Marciniak, and Rychlik 2018). Venekoski and Vankka (2017)  apply this process only to a subset of 300 concept pairs from the English SimLex-999. On the other hand,  Camacho-Collados et al. (2017)  sampled a new set of 500 English concept pairs to ensure wider topical coverage and balance across similarity spectra, and then translated those pairs to German, Italian, Spanish, and Farsi (SEMEVAL-500). A similar approach was followed by  Ercan and Yıldız (2018)  for Turkish, by  Huang et al. (2019)  for Mandarin Chinese, and by  Sakaizawa and Komachi (2018)  for Japanese.
Netisopakul, Wohlgenannt, and Pulich (2019)  translated the concatenation of SimLex-999, WordSim-353, and the English SEMEVAL-500 into Thai and then reannotated it. Finally,  Barzegar et al. (2018)  translated English SimLex-999 and WordSim-353 to 11 resource-rich target languages (German, French, Russian, Italian, Dutch, Chinese, Portuguese, Swedish, Spanish, Arabic, Farsi), but they did not provide details concerning the translation process and the resolution of translation disagreements. More importantly, they also did not reannotate the translated pairs in the target languages. As we discussed in § 2. 3 and reiterate later in §5, semantic differences among languages can have a profound impact on the annotation scores; particulary, we show in §5. 4 that these differences even roughly define language clusters based on language affinity.
A core issue with the current datasets concerns a lack of one unified procedure that ensures the comparability of resources in different languages. Further, concept pairs for different languages are sourced from different corpora (e. g. , direct translation of the English data versus sampling from scratch in the target language). Moreover, the previous SimLex-based multilingual datasets inherit the main deficiencies of the English original version, such as the focus on nouns and highly frequent concepts. Finally, prior work mostly focused on languages that are widely spoken and do not account for the variety of the world's languages. Our long-term goal is devising a standardized methodology to extend the coverage also to languages that are resource-lean and/or typologically diverse (e. g. , Welsh, Kiswahili as in this work).
Multilingual Datasets for Natural Language Understanding. The Multi-SimLex initiative and corresponding datasets are also aligned with the recent efforts on procuring multilingual benchmarks that can help advance computational modeling of natural language understanding across different languages. For instance, pretrained multilingual language models such as multilingual BERT  (Devlin et al. 2019)  or XLM  (Conneau and Lample 2019)  are typically probed on XNLI test data  (Conneau et al. 2018b ) for cross-lingual natural language inference. XNLI was created by translating examples from the English MultiNLI dataset, and projecting its sentence labels  (Williams, Nangia, and Bowman 2018). Other recent multilingual datasets target the task of question answering based on reading comprehension: i) MLQA  (Lewis et al.
2019)  includes 7 languages ii) XQuAD  (Artetxe, Ruder, and Yogatama 2019)  10 languages; iii) TyDiQA  (Clark et al. 2020)  9 widely spoken typologically diverse languages. While MLQA and XQuAD result from the translation from an English dataset, TyDiQA was built independently in each language. Another multilingual dataset, PAWS-X  (Yang et al. 2019) , focused on the paraphrase identification task and was created translating the original English PAWS (Zhang, Baldridge, and He 2019) into 6 languages. We believe that Multi-SimLex can substantially contribute to this endeavor by offering a comprehensive multilingual benchmark for the fundamental lexical level relation of semantic similarity. In future work, Multi-SimLex also offers an opportunity to investigate the correlations between word-level semantic similarity and performance in downstream tasks such as QA and NLI across different languages.
The Base for Multi-SimLex: Extending English SimLex-999 In this section, we discuss the design principles behind the English (ENG) Multi-SimLex dataset, which is the basis for all the Multi-SimLex datasets in other languages, as detailed in §5. We first argue that a new, more balanced, and more comprehensive evaluation resource for lexical semantic similarity in English is necessary. We then describe how the 1,888 word pairs contained in the ENG Multi-SimLex were selected in such a way as to represent various linguistic phenomena within a single integrated resource. Construction Criteria. The following criteria have to be satisfied by any high-quality semantic evaluation resource, as argued by previous studies focused on the creation of such resources  (Hill, Reichart, and Korhonen 2015; Gerz et al.
2016; Vulić et al. 2017a; Camacho-Collados et al. 2017 , inter alia): (C1) Representative and diverse. The resource must cover the full range of diverse concepts occurring in natural language, including different word classes (e. g. , nouns, verbs, adjectives, adverbs), concrete and abstract concepts, a variety of lexical fields, and different frequency ranges. (C2) Clearly defined. The resource must provide a clear understanding of which semantic relation exactly is annotated and measured, possibly contrasting it with other relations. For instance, the original SimLex-999 and SimVerb-3500 explicitly focus on true semantic similarity and distinguish it from broader relatedness captured by datasets such as MEN  (Bruni, Tran, and Baroni 2014)  or WordSim-353  (Finkelstein et al. 2002). (C3) Consistent and reliable.
The resource must ensure consistent annotations obtained from non-expert native speakers following simple and precise annotation guidelines. In choosing the word pairs and constructing ENG Multi-SimLex, we adhere to these requirements. Moreover, we follow good practices established by the research on related resources. In particular, since the introduction of the original SimLex-999 dataset  (Hill, Reichart, and Korhonen 2015) , follow-up works have improved its construction protocol across several aspects, including: 1) coverage of more lexical fields, e. g. , by relying on a diverse set of Wikipedia categories  (Camacho-Collados et al. 2017) , 2) infrequent/rare words  (Pilehvar et al. 2018) , 3) focus on particular word classes, e. g. , verbs  (Gerz et al. 2016) , 4) annotation quality control  (Pilehvar et al.