dataesr/scientific-paragraphs-categorization huggingface dataset:
Huet et al. [43]  also analyzed the French journal Le Monde archive, a collection of documents from 1944 until 1986 discussing different subjects (e. g. post-war period, end of colonialism, politics, sports, culture). The authors calculated a conditional distribution of the co-occurrence of mentions with their cor-responding entities (Wikipedia article). Then, they linked these Wikipedia articles to YAGO  [44]  to recognize and disambiguate entities in the archive of Le Monde. Heino et al. [20]  investigated EL in a particular domain, the Second World War in Finland, using the reference datasets of WarSampo  8. They proposed a ruled-based approach to disambiguate military units, places, and people in these datasets. Moreover, they investigated problems regarding the analysis and disambiguation of these entities in this kind of data, while they proposed specific rules to overcome these issues.
Regarding the lack of resources in the context of Digital Humanities, a recent study explored the low resource settings with costly annotated data and domain-specific KBs  [45]. The approach proposes a domain-agnostic feedback-based annotation approach based on suggestions from the annotators of potential concepts and adaptive candidate ranking. The method improves the annotation process by 35% compared to annotating without interactive support. EL in historical datasets relies on information such as names or locations that are both non-unique and prone to enumeration and transcription errors. These errors make it impossible to find the correct match with certainty. A recent paper  [46]  brings forward a fully automated probabilistic method for linking historical datasets that enable researchers to create samples at the frontier of minimizing false positives and false negatives, by utilizing the expectationmaximization (EM) algorithm.
Building resources The main component of an EL system is its knowledge base (KB) which allows the storage of the full list of entities used as reference. Modern KBs are rich enough to deal with additional tasks such as extraction of supplementary contexts or surface names, disambiguation of cases, or linking of entities with a particular website entry. A well-known set of publicly available KBs are Wikipedia 9 , Wikidata 10 , and DBpedia  [47]. Here, we briefly describe these KBs. Wikipedia is a multilingual encyclopaedia that includes more than 300 languages, but only near to 70 languages have more than 100,000 articles. It is a widely used KB as a source of information for EL systems but also for building datasets. Multiple research studies, e. g. [29, 31, 48] , make use of the English Wikipedia to train their models and disambiguate entity mentions.
However, it has also been used to study the matching of mentions to Wikipedia articles based exclusively on their cultural heritage as well as for the disambiguation of mentions found in historical documents  [49]. Wikidata is a KB created by the Wikimedia Foundation 11. Its main purpose is to store user-generated data from the various projects supported by Wikimedia. Wikidata is widely used as a standard reference for entities, in the context of digital humanities, Wikidata has been used to annotate CLEF HIPE 2020 and NewsEye, two EL datasets of historical documents. DBpedia is a KB that categorizes data from different Wikimedia projects, like Wikipedia and Wikidata. Furthermore, it associates this information to other KBs such as YAGO  [44]  and GeoNames 12. It has been used in different projects related to EL  [22, 25, 50, 51].
The first element helps us to find the correct page from which to extract the 14 Code available at https://github. com/EMBEDDIA/NEL_Filter. other elements. 15 For instance, the ID Q63832446 redirects automatically to Q4182026. 16 The second element, disam-biguation_page, indicates whether we need to remove the candidate ID as a link cannot refer to an ambiguous entry, such as Moon (Q2432366) 17. If the candidate ID is not a disambiguation page, we request from Wikidata the label (and the alternative labels, if any) associated with the entry in the language of analysis. For instance, the English entry of Namur Gate has as alternative label Naamsepoort. 18 Furthermore, we query Wikidata with the year in which the entry was conceived. For example, in the case of a person, it would be their birth year, while for a book (product), the year in which was published, or for a country (location), their inception date.
Historical datasets While EL on contemporary datasets can take advantage of an abundance of resources and tools  [22, 25, 26, [29] [30] [31] [32] [33] , digitized and historical documents lack annotated resources  [5, 20, 22, 25, 26, 42]. Moreover, contemporary datasets and resources are generally not suitable for building accurate systems to be applied to historical datasets due to several issues, i. e. the variations in orthographic and grammatical rules, word variations, and also the fact that names of persons, organizations, or places could have significantly changed over time  [5, 20]. To the best of our knowledge, there are few publicly available corpora in the literature with entities manually annotated in historical documents  [9, 10, 13]. Most of the EL datasets use contemporary documents  [29] [30] [31]  lacking the distinctive features found in historical documents.
In this paper, we focus on two datasets that contain historical documents in English, Finnish, French, German, and Swedish. The first corpus was produced for the CLEF HIPE 2020 challenge 20   [54]. This corpus is composed of articles published between 1738 and 2019 in Swiss, Luxembourgish, and American newspapers. To build the corpus, the organizers randomly sampled articles from different newspapers according to predefined decades. For each newspaper, articles were randomly sampled among the first years of a set of predefined decades covering the lifespan of the newspa-20 https://impresso. github. io/CLEF-HIPE-2020/. per, with the constraints to both have a title and more than 50 characters in length. The dataset was manually annotated by native speakers according to HIPE annotation guidelines  [54].
The second corpus is the NewsEye dataset 21  [28, 55]  which is composed of a collection of annotated historical newspapers in French, German, Finnish, and Swedish. These newspapers were collected by the national libraries of France 22 (BnF), with documents from 1854 to 1946, Austria 23 (ONB) with documents from 1864 to 1933, and Finland 24 (NLF), with Finnish documents from 1852 and Swedish documents from 1848 to 1918. Tables  3  and 4  describe the number of mentions by time period for the CLEF HIPE 2020 and the NewsEye datasets, respectively. The named entities from both datasets are classified according to their type and, when possible, linked to their Wikidata ID. The entities that do not exist in the Wikidata KB are linked to NIL entries.
Experimental settings For all the languages, we utilize the multilingual pretrained model MUSE 25. Specifically, it is used for the entity embeddings and disambiguation model. The MUSE word embeddings are 300-sized, while the character embeddings are 50-sized. 21 https://zenodo. org/record/4573313#. YH79nnUzY5k. 22 https://www. bnf. fr. 23 https://www. onb. ac. at. 24 https://www. kansalliskirjasto. fi. As CLEF HIPE 2020 does not provide a training dataset for English, we make use of the contemporary corpus AIDA  [37]  for training purposes. Then the generated model is validated on the CLEF HIPE 2020 corpus. Based on the statistical analysis of the training data, we defined the weighted Levenshtein distance ratio of 0. 9, 0. 94, 0. 85, 0. 89, and 0. 82 for the languages German, English, Finnish, French, and Swedish, respectively, to search for other mentions in the probability table if the mention did not have a corresponding entry in the probability table (Fig.
6 ). With respect to the post-processing filter, we query Wikidata, DBpedia, and DBpedia Chapters using their respective SPARQL Query Services. 26. Ten DBpedia Chapters are used: Catalan, Basque, Greek, Indonesian, Dutch, French, German, Japanese, Korean, and Spanish. Furthermore, we explore two edit distance metrics: RapidFuzz Weight Ratio  27  and Weighted Levenshtein Distance  28  with specific costs defined by  [6]. In addition, we explore whether candidates not found in DBpedia should be considered as middle candidates or bottom candidates. In total, we explore 18 different filters and their configuration is presented in Table  5. For evaluating our methods, we compute their strict  29  Fscore (F1) calculated for each language over the full corpus (micro-averaging). 30 Specifically, the F-score is defined as the harmonic mean between precision and recall, where precision is the fraction of correctly linked entity mentions that are generated by a system, and recall is the proportion of all entity mentions correctly linked over all the entity mentions that should be linked.
Results We present in Tables  6  and 7  the F-score obtained by each of the EL approaches detailed in Sect. 5, respectively, for the CLEF HIPE 2020 and NewsEye datasets. Tables  6  and 7  also contain the performance achieved by each post-processing filter applied to every base output generated by the EL systems. From Tables  6  and 7  , we can notice that the match corrections, in general, improved the performance of the base EL candidates. Nonetheless, there are two languages and datasets, English CLEF HIPE 2020 and French NewsEye, where this approach reduced the performance of our EL systems. For the Swedish NewsEye dataset, only one configuration is negatively affected, i. e. when the match correction is coupled with a multilingual probability table. Moreover, we can notice from Tables  6  and 7  that in four cases, CLEF HIPE 2020 English and NewsEye German, Finnish and French, the use of multilingual probability tables p(e|m) reduced the performance of the EL systems.
Filters without re-ordering candidates based on edit distance (filters B), seem to generate the best performances for NewsEye Finnish. Finally, filters based on a Weighted Levenshtein distance (filters C) produce the fewer number of best performances in both corpora, i. e. in CLEF HIPE 2020 German and NewsEye Finnish. We can observe as well in Tables  6  and 7  that, regardless of the corpus, mentions of type person are prone to be linked to entries that correspond to people born after the publication of the newspaper article. This can be clearly observed as filters 3 and 6 (which filter entities of type person) but also filters 1 and 4 (which filter all types of entities) produce the best performance. In addition, we can notice in Tables  6  and 7  that for most datasets it is better not to use middle candidates (filters 1-3).
Probability tables This analysis is based on the gold standard data, specifically on the mentions that are linked to a Wikidata entry, i. e. no NILs. The goal is to improve the understanding of the results and the limitations of the proposed methods. We start the analysis by introducing in Table  10  the number of mentions in the explored corpora that exists in each language and that are found in their respective Wikipedia KBs. As it can be observed in Table  10 , for all the test splits, except for Swedish, at least 90% (91% -96%) of the mentions are associated with an entry in the KBs. In comparison, for the Swedish test dataset, only 84% of mentions have a corresponding entry in the KBs. This means that not all the mentions, in a specific language, have a corresponding article in Wikipedia in the language of analysis.
This phenomenon becomes significant on mentions of type person over the Finnish NewsEye dataset, where only 3% of the mentions can be matched in the probability tables. We can notice in Table  11  that applying a match correction approach (c. f. Sect. 7. 3) increases the number of entities that can be found in the probability tables. For instance, in NewsEye Finnish, the word "Berliiniin" (To Berlin) was spelled incorrectly as "Berliniin", however, the match correction module found the correct entry in the KB, "Berliini" (Berlin). In some languages, like Finnish and Swedish, the matching increment is around 60%. Furthermore, we can increase the match of mentions of type person on the Finnish NewsEye dataset from 3% to 25%. Finally, it is important to highlight that Table  11  allows us determining the maximum number of mentions that can be linked in a dataset if the disambiguation module would be perfect.
Multilingualism Unlike previous recent literature  [40, 41] , MELHISSA combined the probability tables generated by different languages, in order to create a unique multilingual probability table. While this solution clearly brought large performance improvements, several issues should be discussed. As presented in Table  11 , the use of multilingual probability tables increased the number of mentions that match with an entry in the KBs. However, in multiple cases, the number of new men- Bold means the best performance on each configuration. Bold and italics mean best performance on each language.
This makes the filter set on the first position "Uralfloden" instead of the correct entry "Uralbergen". In the case of Finnish, for the mention "Englannin" (England; in genitive singular form), the edit distances considered closer the entry "Englannin kuningaskunta" (Q179876, Kingdom of England) rather than "Englanti" (Q21, England). Based on the fact that only two mentions were affected by this aspect, we consider that, in real applications, it should always use an edit distance metric to reorder the candidates. Regarding the filtering of entries by date, it is clear that it should always be done for mentions of type person. The reason is that most of the Wikipedia entries related to people contain a year of birth. For the other types of mentions, i. e. location, organization, and product, the performance of the filter by date, seems to depend mostly on the dataset and how well the annotation was done or could be done.
For instance, we noticed that some locations were affected by the date filter due to ambiguities in the gold standard annotation. In Fig. 5 , we presented the case of the mention 34 https://github. com/maxbachmann/rapidfuzz. 35 https://www. wikidata. org/wiki/Q80240. "Great Britain" in a press article of 1868  36. The gold standard annotation indicated that the correct entry is Q145, i. e. United Kingdom (of Great Britain and Northern Ireland). However, because the article was published in 1868, the correct entry should have been Q174193, i. e. United Kingdom of Great Britain and Ireland, which refers to the country that existed before the 1921 Anglo-Irish Treaty. The filter managed to propose the actual correct entry in second place, while removed the entry that matched the gold standard.
Some other annotation errors are due to the ambiguity of the entry in Wikidata or the impossibility of finding a better candidate. For example, in the French CLEF HIPE 2020, the mention "Val-de-Travers" in a 1798 document is associated in the gold standard to Q70526  37. Nevertheless, despite the fact that the entry has for label "Val-de-Travers", it refers to a municipality created in 2009 (field "inception"). Thus, the filter removes it from the candidates. Nonetheless, in Wikipedia, it does not seem to exist a better candidate to annotate the entry. Some of the other entries, such as "Valde-Travers District" or "Region of Val-de-Travers" make reference to relative modern locations too. Although we found errors in the gold standard annotation, as indicated before, it should be mentioned that no adjustments or modifications were introduced in it.
In order to fill this gap, we presented MELHISSA, a Multilingual Entity Linking architecture for HIstorical preSS Articles. The main objective of this tool is to link mentions, such as names of people, organizations, and products, to entries in knowledge bases, such as Wikidata. Specifically, we created an end-to-end neural entity linking system that manages multiple languages and has been designed to surpass common errors found in historical documents. The presented system was tested over two historical datasets, NewsEye and CLEF HIPE 2020, comprising five European languages: English, Finnish, French, German, and Swedish. We explored different configurations, such as the use of edit distances, multilingual probability tables, and post-processing filters, in order to create a reliable entity linking tool.
The obtained outcomes demonstrated that MELHISSA is a competitive tool that is able to get an F-score@1 of up to 0. 681 and an F-score@5 of up to 0. 787. We have observed that the use of multilingual probability tables can be useful in languages such as Swedish, while the use of a matching correction module can improve the pairing of mentions and entries in a knowledge base. Furthermore, the application of a post-processing filter can improve in general the entity linking performances in all languages. To be precise, in MELHISSA, the use of multilingual probability tables allowed us to deal with entities that are either foreign words or found within a knowledge base in a different language than the one being analyzed. With respect to the matching correction module, it provided a way to match entities that had a different spelling, due to either OCR errors or language evolution.
Finally, the use of the post-processing filter increased the performance of the MELHISSA thanks to its capacity of removing unlikely candidates, fixing issues from the knowledge bases and reordering the results. Furthermore, although MELHISSA makes use of modern knowledge bases, i. e. Wikidata and DBpedia, which are based on contemporary sources, it is capable of linking historical entities. The main reason is that these knowledge bases represent, in many cases, not only current and contemporary entities but also historical ones. Nevertheless, we are aware that these knowledge bases lack coverage, either partial (certain languages) or total, regarding some historical entities due to their collaborative approach. Despite this shortcoming, we consider that MELHISSA's outcomes could point out missing historical entities and encourage experts to participate in the enrichment and/or improvement of Wikipedia, and in consequence of Wikidata and DBpedia.
25 https://github. com/facebookresearch/MUSE. 123Table Number of mentions by period of time in the CLEF HIPE datasetSplits German English French 1750-1800 1800-18501850-1900 1900-1950 1750-1800 1800-1850 1850-1900 1900-1950 1950-2000 1750-1800 1800-1850 1850-1900 1900-1950 1950-2000.
Data availability Our code is freely available at https://github. com/New sEye/Named-Entity-Linking/tree/master/multilingual_entity_linking for the EL method and at https://github. com/EMBEDDIA/NEL_Fil ter for the filter module. The evaluation tool can be found at https://github. com/creat89/CLEF-HIPE-2020-scorer. The CLEF HIPE 2020 dataset is available at https://impresso. github. io/CLEF-HIPE-2020/ and the NewsEye dataset at https://zenodo. org/record/4573313#. YH79nnUzY5k.
3 3 25 https://github. com/facebookresearch/MUSE. 123Table Number of mentions by period of time in the CLEF HIPE datasetSplits German English French 1750-1800 1800-18501850-1900 1900-1950 1750-1800 1800-1850 1850-1900 1900-1950 1950-2000 1750-1800 1800-1850 1850-1900 1900-1950 1950-2000.
Object Detection In order to remove the need to construct labelled datasets for each environment, we only assume that we are given a small set of images of each safety-critical object and a set of background images (in practice, we use 1 image per object and 1 background). We generate synthetic images by pasting the objects onto a background with different locations, rotations, and other augmentations. We then train a CenterNet-style object detector  (Zhou et al. , 2019)  which performs multi-way classification for whether each pixel is the center of an object. For speed and due to the visual simplicity of the environments, the feature extraction CNN is a truncated ResNet18  (He et al. , 2016)  which only keeps the first residual block. The loss function is the modified focal loss  (Lin et al.
Interestingly, it is possible to combine various scales of observation from large landscape view to very fine observation on the outcrop and offer a good interactivity with the students. Moreover, a specialist of the outcrop or of https://doi. org/10. 5194/se-12-2789-2021 Solid Earth, 12, 2789-2802, 2021 the thematic can be virtually present providing that they are equipped with headset and controllers and have the Minerve software locally installed. In the future, students skills should be evaluated in a systematic way before and after the VR lab session to measure its teaching efficiency. The students surveyed were in general enthusiastic to experiment with virtual fieldwork, and it is therefore a proper tool to increase their empathy with the discipline  (Lonergan and Andresen, 1988) , even if physical discomfort is common.
This could be reduced by shortening the duration of the sessions in immersion in the virtual environment, ensuring breaks, and adding software solutions. Even if it is clear that a virtual observation does not replace a field observation, virtual reality could be a fantastic tool to bring students to remote or even inaccessible places such as submarine fault scarp or planetary bodies. It therefore opens new perspectives for teaching Earth sciences, and we plan to use the Minerve software in other contexts in the coming years. Code and data availability. The Minerve software is developed by Jérémy Billant under an open-source under an open-source license and is available on request. The centimetric DEM and DOM are not open-source but can be made available on request and is described in  Billant et al.
(2019). Data used during the QGIS mapping exercise are available in the Supplement, including a subset of the different 3D textured models acquired along the Roseau Fault https://doi. org/ 10. 17882/84249 (last access: 3 December 2021). Video supplement. A video of a group of students during the together in the field strategy is available from the following link: https://mediacenter. univ-lyon1. fr/videos/?video= MEDIA211029153532899 (last access: 29 October 2021,  Billant and Métois, 2021). We share the view of the teacher J. Billant located in his office in Nice, while the students are all together in the virtual lab at Univ. Lyon 1. English subtitles are provided (T: teacher; S1: student 1; S2: student 2). Supplement. The supplement related to this article is available online at: https://doi.
org/10. 5194/se-12-2789-2021-supplement. Team list. Nora Van Reeth, Félix Moquet, and Amandine Minjollet (iCAP, Innovation, Conception, Accompagnement pour la Pédagogie, University Lyon 1 43 Bd du 11 novembre, 69622 Villeurbanne, France) Author contributions. JB developed the Minerve software. JB, MM, and JEM designed the teaching experiment and actually taught. AM helped during the lab session and the class conception. JE and FL processed and gave access to the scientific data, provided an expert external view, and were beta testers for the Minerve software. The ICAP team took care of the technical issues and helped with the use of the virtual lab. MM and JB led the writing process of this article. All authors contributed and approved its final version. Competing interests. The contact author has declared that neither they nor their co-authors have any competing interests.