dataesr/scientific-paragraphs-categorization huggingface dataset:
Gold standards overview For entity normalization in specialized domains, we present popular normalization datasets whose main features are summarized in Table  1. Entity normalization is not addressed as much as entity recognition or relation extraction. Maybe partly because it implies detailed and specialized references, which implies a great deal of effort on the part of experts in the field. As a consequence, there are not many gold standard datasets, and most of them are in English. In some cases, the normalization is not evaluated as a separate task but jointly with an entity recognition task  [34] , which limits the estimation of performance specifically on it (i. e. the errors on recognition are propagated on normalization). For this reason, we limited our study to datasets for the normalization task alone.
BB-norm dataset of bacteria biotope 4 from BioNLP OST 2020 The BioNLP Open-Shared Tasks 1 proposed several datasets, including 6 tasks on the Bacteria Biotope 4 corpus  [2] , of which the "BB-norm" normalization task. This task can be separated into three distinct normalization subtasks, which are evaluated separately: normalization of microorganisms by the NCBI Taxonomy  [36] , normalization of bacterial habitats by the biotopes part of the OntoBiotope ontology 2  [37] , and normalization of phenotypes by the phenotypes part of the same OntoBiotope ontology. The dataset is separated into training/development/test sets, but the test annotations are not provided. The evaluation is done by an online platform 3 by uploading the predictions. This dataset is based on the entity "categorization dataset" of Bacteria Biotope 3 from BioNLP Shared Task 2016  [24].
The CSIRO adverse drug event corpora (CADEC) There are nowadays three CADEC datasets. The initial corpus  [39]  is the result of the annotation of posts from the medical forum AskaPatient. 6 The annotated mentions are drugs, adverse effects, symptoms and diseases. Two ontologies are used: the non-hierarchical Australian Medicines Terminology (AMT) for drug names and the "Clinical Finding" subpart of the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) for all the rest. Both ontologies have licency issues. 7 Another version uses the Medical Dictionary for Regulatory Activities 8 (MedDRA). To compare approaches, Limsopatham and Collier  [26]  proposed a new dataset separated in 10 folds train/dev/test, now frequently named "Random CADEC" 9 by the community (sometimes also named AskaPatient).
The dataset does not come along with the ontologies, and researchers evaluate their methods on this corpus by using only the list of the 1036 concepts used to annotate the corpus. Tutubalina et al. [6]  finds that 2 https:// tinyu rl. com/ OntoB iotop e2019. 3 http:// bibli ome. jouy. inra. fr/ demo/ BioNLP-OST-2019-Evalu ation/ index. html. 4 https:// www. nlm. nih. gov/ mesh/ meshh ome. html. 5 https:// www. ncbi. nlm. nih. gov/ omim. 6 https:// www. askap atient. com/. 7 A NCTS license is needed to obtain the ontologies. Contact the support team to obtain the former versions used to annotate the corpus (australianhelp@digitalhealth. gov. au). 8 https:// www. meddra. org/. 9 https:// zenodo. org/ record/ 55013. approximately 60% of mentions from each test set were also in the training set, suggesting an easy benchmark.
Consequently, it proposes a third dataset named "Custom CADEC", more popularized in  [32] , which is a fivefold cross-validation dataset (no development sets) avoiding redundancy between training and test sets. The mentions without an associated concept in the ontology as well as ambiguous mentions have also been removed. The Corpus of Online Medical EnTities (COMETA): The COMETA corpus  [40]  consists of 20 K English biomedical mentions in layman's language from an internet forum, and shares some similarities with all the CADEC datasets. A version of the whole SNOMED-CT is used as ontology, and almost half of the mentions are annotated by a concept from the "Clinical Finding". There are two splits of this corpus: a "stratified split" in which, for each concept appearing in the test/dev sets, this concept appears at least once in the training set, and a "zero-hot split" in which there is no overlap between the concepts appearing in the test/dev sets and the train set.
Task 4 of adverse drug reaction extraction from drug labels (ADR) dataset from TAC 2017 The Text Analysis Conference 10 (TAC) proposed a track to extract adverse reactions from drug labels  [3]. Task 4 was an independent normalization task which uses the MedDRA ontology, which contains around 20,000 concepts. Originally, the task did not use accuracy measure, but nowadays, authors that evaluate their methods on this dataset use it preferably.
The chemical disease relation track of the BioCreative V challenge This track was originally designed for joint entity recognition and normalization  [41]. Two types of entities should be extracted and normalized: disease and chemical. The disease mentions are to be normalized with the MEDIC ontology and the chemical mentions by the non-hierarchical Comparative Toxicogenomics Database 11 (CTD). Authors use now separated corpora, one for each entity type: the disease dataset (BC5CDR-D) and the chemical dataset (BC5CDR-C). The accuracy is also the preferred evaluation metric. Earlier, the BioCreative challenges have shared some others normalization datasets, such as the Gene Normalization datasets from BioCreative I  [40] , BioCreative II  [42]  and BioCreative III  [43] , but to our knowledge, they are more rarely used nowadays as gold standards.
The psychiatric treatment adverse reactions corpus (PsyTAR) This annotated corpus  [44]  is also based on the AskaPatient forum and on annotations from the SNOMED-CT and the UMLS Thesaurus. It deals with effectiveness and adverse reactions of psychiatric medications. For evaluation purposes, Miftahutdinov and Tutubalina  [32]  proposed two datasets (each being a k-fold cross-validation dataset): a "Random PsyTAR " and a "Custom PsyTAR ", the latter avoiding the train/test redundancy.
Task 3 "normalization of adverse drug reaction mentions" dataset from SMM4H The 2nd Social Media Mining for Health Applications 12 (SMM4H) proposed an independent 10 https:// tac. nist. gov/. 11 http:// ctdba se. org/. 12 https:// healt hlang uagep roces sing. org/ share dtask2/. normalization task of adverse drug reactions in social media texts. The MedDRA ontology is used for annotations. All data sets are available. 13   Task 1 dataset from the ShARe/CLEF eHealth evaluation lab 2013 The annotated mentions in this corpus are disorders. The original task is a joint entity recognition and normalization tasks  [45]. A disorder is defined as any span of text which can be mapped to a concept in SNOMED-CT and which belongs to the Disorder Semantic Group from the UMLS. Although the clinical data in the corpus has been anonymized, the corpus is not open and requires some registration procedures to be manipulated.
The medical concept normalization (MCN) corpus The fourth i2b2/VA shared task  [46]  proposed a corpus for entity recognition evaluation. Luo et al. [47]  start from the full SNOMED CT and the RxNorm  [48]  ontologies to annotate the recognized mentions and build the Medical Concept Normalization (MCN) corpus. The mentions are disorders, as well as other medical problems, treatments and tests. TwADR-L This dataset 14   [26]  is composed of tweets that have been annotated by concepts from the SIDER 4 database of drug profiles. 15 The used version of this database contains 2220 concepts. The corpus is separated into 10 folds for cross-validation.
Comparative evaluation In specialized domains, there is no initiative to share entity normalization gold standards in a single place that would facilitate multiple evaluations and comparisons of proposed methods. Comparatively in the open domain, there are some interesting initiatives to compare entity disambiguation gold standards, such as the "Disambiguate to Knowledge Base" (D2KB) experiment from the GERBIL benchmarking system  [33]. Rosales-Méndez et al. [49]  suggests focusing on multilingual EL evaluation through a benchmarking set. van Erp et al. [50]  shows some strengths and weaknesses of EL datasets, some are also interesting to EN (e. g. overlapping mentions between train and test sets). Sevgili et al. [51]  compares some deep learning results of EL methods. One reason for this difference between specialized and open domains is that the access to clinical data (i.
e. documents or knowledge bases) is often restricted  [52]. Moreover, there are few detailed comparisons of features of normalization datasets in specialized domains. Corpus publications focus mainly on intrinsic information about the corpus used: the number of documents, sentences, words, mentions or used concepts in the full corpus. In particular, information on the separation into different folds, when this separation is done, is rarely provided. Most of the existing inter-analysis of datasets were done in publications which describe a new method evaluated on some datasets. Sometimes, the initial annotated corpus is not separated into different folds, and it is the authors of a method who publish a new dataset with specific sets for evaluation purpose. For instance, the "Random CADEC" dataset  [26]  is the result 14 https:// zenodo.
org/ record/ 55013. 15 http:// sidee ffects. embl. de/. 13 https:// data. mende ley. com/ datas ets/ rxwfb 3tysd/1. of the separation of the all-in-one CADEC corpus  [39]. Note that during this process, some modifications can be made, such as simplifications (e. g. removing NIL or ambiguous mentions). In  [26] , the average number of mentions that a concept annotates (considering only used concepts from the whole ontology) are presented for three datasets. Although not discussed, this information was possibly intended to give an idea of the difficulty of the benchmark: if low, the task will fall within more challenging few-shot or one-shot learning. In  [53]  or  [11] , in addition to the usual information but decoupled on the training set and the test set, the number of NIL mentions are also presented between the ShARe/ CLEF, the NCBI-DC and the TAC-ADR datasets.
For example, 32. 7% of the test mentions in the ShARe/CLEF corpus are NIL mentions. A method that does not have a conceptless class in its prediction categories would therefore be at a disadvantage on this dataset. In  [22] , two distinct folds were created from a unique dataset where each one contains annotating concepts which are not in the other. One is chosen to train their method, the other for evaluation. This represents an analysis of the Zero-Shot Learning (ZSL) cases, that is, the mentions' concept to predict have never been seen during training. In  [54] , the authors have clarified that some datasets do not use the whole ontology initially considered, but a subset built from the concepts contained in the training and test sets. They also present some ZSL indicators and the proportion of ambiguous mentions per dataset.
The ZSL information is interesting because it gives an idea of the performance of a method on a real corpus. Indeed, in specialized domains, since the ontologies used for annotation contain a large number of concepts, it is futile to hope to have even one example for each concept in a gold standard. Tutubalina et al. [55]  describes a cross-domain approach to extrinsically evaluate the ability to address zeroshot: by training a method on a dataset, then using it on another dataset and at inference time to estimate the nearest label for each mention. To our knowledge, the first and unique work which directly concerns entity normalization comparative evaluation in specialized domains is  [55]. The authors have analyzed five datasets (NCBI-DC, BC5CDR-D, BC5CDR-C, TAC-ADR and SMM4H) and have compared the NCBI-DC state-of-the-art method BioSyn  [23]  to a baseline based on BERT  [56]  on them.
In particular, they observe in some datasets that a large number of mentions appear several times in the test set, and many are also seen at training time. This is also true for the Random CADEC dataset, and on refined test sets intended to reduce these biases. In fact, Tutubalina et al. [6]  shows that the distribution of mentions through the folds have a significant impact on performances, leading to an average of 15% loss between initial and refined test sets.
Contributions • We propose indicators for characterizing EN datasets. We hypothesize that these indicators could help to categorize dataset for future work, regardless of language type or domain, but rather in terms of difficulties for current adaptable approaches. • We propose a new mixed EN baseline method, which runs in a few minutes on a standard laptop (CPU). While not a strong baseline, it is a generic and easy-to-use one, whose performance well serves as an indicator of the difficulty of an EN task. • We formalize an accuracy measure to evaluate EN methods, which takes into account the multi-normalization cases, which are often overlooked. • We identify some potential biases of entity normalization evaluation: (1) Preprocessing may not be well described in the reference article of a method, while it may be domainspecific, and be responsible for the superior performance of a method.
Entity normalization methods We report the leaderboards of the BB4 bacterial habitat task and the NCBI Disease Corpus in Tables  2  and 3  respectively. The main evaluation metric used for the BB4 task is a non-strict similarity score  [19] , but an accuracy metric is also provided by the task organizers through an evaluation web platform. For fair comparisons between datasets and methods, we only report here the results with accuracy as provided by BB4 task organizers and in  [57]  for the last published methods. This accuracy does not change the ranking of the first three methods compared to the similarity score, but has an impact on the next ones. To our knowledge, these two leaderboards represent the state-of-the-art in mid-2022. To support our purposes, we have chosen the state-of-the-art method on each corpus at the time we started our experiments in 2020, that is C-Norm  [57]  for BB4 and BioSyn  [23]  for NCBI-DC.
To our knowledge, no previous publication has shown evaluation of C-Norm on the NCBI Disease Corpus, or of BioSyn on the BB4 dataset. Bio-Syn and C-Norm have the advantage of being at least publicly shared, but it is not yet a widespread effort in the entity normalization community. Our experience shows that the published results are mainly reproducible. Nevertheless, we have not conducted additional experiments to study reproducibility issues (e. g. non-significant scores). We also created a baseline that basically combines the four strategies (i. e. DictB, DistR, SymR and ML), each in their naive version.
BioSyn BioSyn  [23]  was the state-of-the-art method on the NCBI Disease Corpus dataset at the time we started our experiments in 2020. It's a ML-based method using TF-IDF bagof-word representations  [71]  (i. e. DictB approach) as well as embeddings (i. e. DistR approach) of mentions computed by BioBERT  [72]. The method uses both representations to predict top-k candidates by weighting the influence of one or the other in a final score, that is, by weighting the influence of morphological or distributional information. During training, BioBERT embeddings are fine-tuned to modify this score. Since concepts regularly have multiple labels, training is done by seeking to maximize the probabilities of predicting all the correct concept labels in the top-k candidates for each mention, not only the top-1 as most methods do.
During the training, the system learns to rank k candidates containing also some negative examples with high probability of prediction. The authors claim that this kind of integration of negative sampling allows to improve the BioBERT embeddings for the task. Even if not described in detail in  [23] , the method includes some preprocessing: lowercasing and punctuation removal of mentions and labels, an acronyms resolution with Ab3P, 16 some rules to resolve composite mentions and a list of corrections for bad spelling of some words. We used the official available code 17 for preprocessing and the core method.
C-Norm C-Norm  [57]  is the current state-of-the-art method on the BB4 dataset. It's a DistR-ML which uses a weak supervision strategy (i. e. use of all label-concept couples from the ontology as positive examples) and integrates some ontological information. The method does not represent each concept through their labels, but by a unique semantic encoding vector which is mainly based on the subsumption information available in ontologies. The method learns a projection from an embedding space to an ontological space defined by the semantic vectors concepts. The objective function tries to globally maximize the cosine similarities between projected vectors of mentions and their concept vector(s). C-Norm uses as input Word2Vec embeddings which should be fine-tuned in a specific way, including preprocessing (lowercasing, lemmatization and stopword removal) and a training corpus selection.
We have encountered difficulties in running the method on corpora with larger ontologies and/or datasets. We have optimized the code so that it can more easily run on a Google Colab CPU server 18 (limited to 12Go RAM). One of the hyperparameters of the method is the number of convolutional filters. The authors state that it should be set to the number of concepts in the used ontology. On the NCBI Disease Corpus, we fixed that number to 3000 as on the BB4 dataset, rather than 9664 as it should be (i. e. there are 9664 concepts in the MEDIC ontology). Finally, we don't exactly use the scripts pointed by the authors, but instead reproduce the preprocessing steps. Indeed, these preprocessing should be performed through the corpus processing engine AlvisNLP/ML, 19 which made it difficult to use on our server.
Baseline As we mentioned, baseline methods are often absent from normalization studies. D'Souza and Ng  [12]  describes the first sieve approach for entity normalization. This approach is based on several methods which are sorted by expected precision, and inversely, by expected recall. The most precise method is used first on a dataset, and the mentions that do not have predictions are then passed to the second most precise method, and so on. The first sieve method was based on DictB-SymR methods, but Ferré et al. [58]  proposed a 2-sieves method where the second sub-method is based on DistR-ML and trained on the whole training set. Inspired by this approach, we construct a simple baseline based on two submethods: a first DictB-SymR method, and a second DistR-ML one. This combined 18 https:// colab.
resea rch. google. com/. 19 https:// bibli ome. github. io/ alvis nlp/. 16 https:// github. com/ ncbi-nlp/ Ab3P. 17 https:// github. com/ dmis-lab/ BioSyn. approach aims to leverage the strengths of each type of classical approach, and, rather than being a strong baseline, it is expected to be robust to domain or task changes. The DictB-SymR method starts by adding mentions of a training dataset as labels of the associated concept, as it is also done by BioSyn. Unlike BioSyn, if several mentions with the same surface form are annotated by different concepts, the mention is added as a label only for the concept that was the most frequent. Then for prediction, the method lowercases and stems all mentions and all labels and seeks for exact match. In a very simplified way, by adding training examples, this method is also ML-based.
Dataset description For this work, we particularly focus on two datasets, which have in common to be frequently used to evaluate methods: the BB4 and the NCBI-DC datasets. Both are publicly available and use a relatively small and hierarchically organized ontology. For BB4, the bacterial habitat hierarchy of the OntoBiotope ontology owns 3172 concepts. For the NCBI-DC, the MEDIC ontology owns 9664 concepts. Due to the hierarchical nature of these ontologies, there are no NIL mentions (i. e. each mention of interest can be at least normalized by the root concept). The full BB4 dataset has three entity types (habitat, phenotype and microorganism), but as observed in  [57] : "the Phenotype part of the dataset is much smaller than the Habitat part and thus it is harder to observe clear tendencies on this dataset".
Moreover, the microorganism part is not really challenging regarding terminological variations, and the ontology used for this part is relatively large. Thus, as others do, we only use the habitat part of the dataset. Nevertheless, for some experiments, we use the full OntoBiotope ontology, which also owns a phenotype hierarchy with 429 concepts. The NCBI-DC has a particularity: the presence of unresolved composite mentions (e. g. the composite mention "breast or ovarian cancer" is given as it is, rather than resolved in two mentions: "ovarian cancer" and a discontinuous mention "breast cancer"). The objective of the task is then to normalize these mentions by all the concepts that should normalize each of their component entities. We choose to consider these mentions as any other case of multi-normalization.
20 the "PubMed-w2v. bin" file from http:// bio. nlplab. org/. For comparison purposes, we analyzed some indicators on the "Custom CADEC" dataset. This dataset is different from NCBI-DC and BB4: it is separated into fivefolds for cross-validation, their annotating ontologies are relatively big and it has been already reworked from its previous version "random CADEC" to correct some observed biases. Moreover, the corpus is publicly shared, but not the two ontologies. The Clinical Finding hierarchy of the SNOMED-CT owns 99,814 concepts, and it is unclear what is the part of the non-hierarchical AMT terminology which has been used (the trade and medicinal product types mainly used are 7188).
Our proposal for a quality measure As discussed, there is no standard evaluation system for the NCBI Disease Corpus. Authors typically resort to their own scripts for computing accuracy, without describing it upon mentioning "we used accuracy". Thus, the evaluation of multi-normalization cases is then not explicit. For instance with BioSyn, the personal accuracy evaluates multi-normalization cases, but differently depending on whether they are composite mentions (all concepts are needed to validate correct prediction) or multi-labeled mentions (just one of the correct concepts is needed to validate correct prediction). To our knowledge, no formalization of an accuracy measure taking into account the multi-normalization cases has yet been proposed (although we believe that such a measure may have already been implicitly used in some tasks).
The measure is detailed in Eq. 3: where strict(m, c) = 1 if c is a correct concept of m, 0 otherwise Equation 3: Equation of accuracy generalized to multi-normalization cases. Here p i is the number of predicted distinct concepts c j i for the mention m i , and n i is the correct number of distinct concepts to find. As the NCBI-DC dataset has the specificity of containing some unprocessed composite mentions, which are evaluated as in Eq. 3, in addition to the cases of multi-labeled mentions, the BioSyn method proposes a slightly different evaluation script: a method only needs to predict one of the correct concepts to get the full point for a multi-labeled mention. This seems to us as relatively mild compared to Eq. 3. As a contribution, we therefore propose a generalized accuracy in Eq. 4.
Here p i is the number of predicted dis- tinct concepts c j i for the mention m i , and n i is the correct number of distinct concepts to find. We also use this accuracy function on the BB4 corpus, which seems to give the same results as the one used on the BB4 online evaluation platform. 21 Note that if a method predicts a single concept per mention in a multi-normalization dataset, then Eq. 4 reduces to Eq. 3.