dataesr/scientific-paragraphs-categorization huggingface dataset:
In this article, we introduce ARIEX, which is a method to rank web information extractors that overcomes the deficiencies that we have found in the literature. It is automated, so that researchers can bias the conclusions as little as possible, open, so that it can easily accommodate new performance measures, and agnostic, so that it can be applied to as many different kinds of proposals as possible. Furthermore, it addresses the following questions: how to set up the experimental environment, how to create appropriate evaluation splits, how to compute the experimental data, how to cook them, how to compute the rankings, and how to report on the results. We have also analysed the performance measures that have typically been used in the literature and we have concluded that they are skewed in contexts in which the datasets are unbalanced, which are common in our context.
Unfortunately, that experience also revealed that there are two obstacles in practice: the lack of public datasets and the unavailability of public implementations. Previously, we have listed the repositories of which we are aware. They provide a large collection of datasets on very different topics and a sufficiently large number of documents; the problem is that we found that most of them are outdated, that is, they provide web documents that are not representative enough of the kind of web documents that typical web sites provide nowadays. Our experience also proves that there are very few proposals with a public implementation; contacting the authors does not usually help since it is very unlikely to get an answer from them; furthermore, most of the implementations one can find are research prototypes that are very difficult to set up and get running; implementing other proposals is not commonly sensible because their details are subtle and sometimes not described in complete detail, not to mention the effort that such a re-implementation requires.
Unfortunately, there is little we can do regarding the previous problems, except for pushing other authors to contribute to the research community with their datasets and implementations. We think that the method that we present in this article shall definitely help in this task since it provides a solid guideline to evaluate, compare, and rank different proposals; we expect that researchers are not so reluctant to publish their datasets and implementations if they know that they are going to be compared using a solid method that guarantees that the results cannot be biased and are then not subject to (mis)interpretations. We think that this is very appealing for both the authors of new information extractors, who can use ARIEX to systematically prove that their proposals outperform others in the literature, and practitioners, who can use it to make informed decisions regarding which of the proposals in the literature is the most appropriate to tackle a particular information extraction problem.
A. 2. Efficiency measures Unfortunately, efficiency measures have not been paid much attention in the literature. Almost no author reports on them, but we think that they are very important to provide an actual overall pic-ture of how a proposal performs in practice. They are of uttermost importance to practitioners who have to make a decision regarding which the most appropriate proposal is regarding a particular problem. We suggest using the following ones: learning time (LT) and learning memory (LM), which refer to the time taken and the memory required to learn a rule set, respectively; and extraction time (ET) and extraction memory (EM), which refer to the time taken and the memory required to extract information from a document. If a proposal is based on heuristics, then its learning time and its learning memory can be trivially set to zero, since it does not learn any rules, but extracts information directly from a dataset.
A. 3. A note on global error-related measures Previously, we have reported on the error-related measures that we have found in the literature. Obviously, we recommend using the global ones since they are the only that report on how good a proposal is at both extracting the information in which we are interested and ignoring the rest. The standard is to use the F 1 score, which combines precision and recall. Unfortunately, our study reveals that this measure is not appropriate in the context of ARIEX. The reason is that the F 1 score is skewed in the context of unbalanced datasets. A dataset is said to be unbalanced when the number of instances of a slot deviates from the number of instances of the remaining slots. In our context, the datasets are naturally unbalanced because the amount of information to be ignored in a web document typically exceeds the amount of information to be extracted; furthermore, some slots are optional and some others are multi-valued, which also contributes to making the datasets naturally unbalanced.
To understand the reason why using the F 1 score in the context of unbalanced datasets is problematic, we use the following example: assume that a proposal is evaluated on a dataset that has 15 documents that provide a total of 15 instances of a given slot; assume, too, that the resulting confusion matrix is (t p 1 , tn 1 , f p 1 , f n 1 ) = (15, 2371, 98, 0), which implies that precision is 0. 13, recall is 1. 00, and the F 1 score is 0. 23. In other words, it does not seem to be a good proposal because precision is very low, but note that this proposal is actually very good because it makes very few mistakes. Assume that another proposal is evaluated on a dataset that provides 15 documents, but only 13 instances of the slot being considered; assume, too, that the corresponding confusion matrix is (t p 2 , tn 2 , f p 2 , f n 2 ) =  (13, 960, 40, 0).
That is, its precision is 0. 25, its recall is 1. 00, and its F 1 score is 0. 39. Neither seems this proposal to be excellent, but a little better than the previous one. Note however, that a deeper analysis can easily reveal that both proposals behave very similarly because they successfully extract every instance of the slot being considered and roughly 4% of the examples to be ignored are mistakenly extracted as belonging to that slot. In other words, they behave very similarly regarding their ability to extract or ignore information. The problem is that the F 1 score provides a distorted view of these proposals because they have been evaluated on testing sets that have different skews. A good global error-related measure must depend only on the proposal being evaluated, not on the dataset used to evaluate it being balanced or unbalanced.
That is, it should be possible to maximise the measure by improving the techniques that lie at the core of a proposal, not by changing the proportion of information to be ignored in a dataset. To find out which of the global error-related measures that we have presented before is not skewed in the presence of unbalanced datasets, we have re-written their formulations in terms of the following measures: the true positive rate (T PR = t p/(t p + f n)), which measures the proportion of instances of a slot that are extracted as belonging to that slot with regard to the total number of actual instances of that slot, the false positive rate (FPR = f p/( f p + tn)), which measures the proportion of information that is extracted as belonging to a given slot with regard to the information that must be ignored, and the skew of the dataset (S = (tn + f p)/(t p + f n)), which measures the proportion of information to be ignored with regard to the information to be extracted as belonging to a given slot.
Note that we have selected the true positive rate and the false positive rate because these measures provide a clear picture of how good a proposal is at extracting or ignoring information and they have been proven not to be skewed in the context of unbalanced datasets  [30]. Next, we present the results of re-writing the global error-related measures in terms of the previous measures: (1 + T PR -FPR) Realise that the area under the ROC curve is the only measure that does not depend on S when it is re-written, which analytically proves that it is the only measure that is not skewed in the context of unbalanced datasets. In other words, it is the only that we can recommend in the context of ARIEX. Regarding our previous examples, the area under the ROC curve is 0. 98 in both cases, which reflects that the corresponding proposals were not that bad and that the dataset being unbalanced does not have an impact on the results.
A. 4. A note on computing per-extractor measures To compute a ranking, we need to compute the performance measures on a per-extractor level. It is very easy to compute per-extractor efficiency measures because we just need to measure the time that elapses since an experiment starts running until it finishes or to probe the maximum amount of memory requested. Regarding effectiveness measures, the problem is a little more involved because we can compute per-slot measures and we then have to combine them in a manner that makes sense in the context of unbalanced datasets. Generally speaking, the problem has been addressed using macro-, micro-, or weighted averages. Macro averages are calculated by computing the effectiveness measures in a per-slot basis and then computing their unweighted averages; micro averages are computed from a global confusion matrix that is, in turn, computed by adding the confusion matrices that correspond to each slot; weighted averages are computed like macro averages, but the measures are weighted by the number of actual instances of each slot.
Our general recommendation is to use macro averages because micro-and weighted averages have been proven to be skewed in the context of unbalanced datasets  [64]. We have also found some specific research results regarding the area under the ROC curve, which is the most appropriate effectiveness measure that we have found so far  [30, 65, 70]. The only one that seems both effective and computationally tractable is the one by Hand and Till  [65] , which computes it as follows: AUC-ROC = i, j∈S,i≺ j AUC-ROC i, j (|S| 2 -|S|)/2 where AUC-ROC i, j refers to a new pairwise measure that combines every two slots, ≺ denotes an arbitrary ordering of the slots, e. g. , a lexicographic ordering, and S denotes the set of slots to be extracted. Note that there are (|S| 2 -|S|)/2 pairs of slots if their order is not taken into account.
Acknowledgements We are grateful to  Dr. José C. Riquelme , from the  University of Sevilla, Spain , for the many fruitful discussions regarding evaluating proposals and comparing experimental results. We would also like to thank  Dr. Francisco Herrera , from the  University of Granada , for sharing his statistic analysis software with the research community. Our work was supported by  FEDER  funds that were awarded by the  European Commission , the  Spanish Government , and the  Andalusian government  through contracts  TIN2007-64119 ,  P07-TIC-2602 ,  P08-TIC-4100 ,  TIN2008-04718-E ,  TIN2010-21744 ,  TIN2010-09809-E ,  TIN2010-10811-E ,  TIN2010-09988-E ,  TIN2011-15497-E , and  TIN2013-40848-R. The work by  Patricia Jiménez  was also partially supported by the  University of Southern California  during a research visit that she paid to the  Information Sciences Institute.
It is worth mentioning that the other package RandomFields allows the simulation of a large class of random fields such as Gaussian random fields, Poisson fields, binary fields, Chi2 fields, t fields and max-stable fields (see  Schlather et al. (2015) ). These simulations are obtained essentially by construction and not from the covariance structure of the process. In RandomFields, conditional random fields (what is the purpose of the present paper) are given for specific spatial and spatio-temporal random fields. It may be noted that the FieldSim package can simulate any conditional Gaussian process as long as you know its covariance function. We propose here to adapt the FieldSim package to conditional simulations. Definitions and notation will be introduced in the following section with the process class, the setProcess procedure and the fieldsim procedure.
The FieldSim package In the new version 3. 2 of the package FieldSim, new features have been added. The most important add is the process class and the setProcess function. An object of class process has different slots: • the name of the process. Several names are reserved for classical fractional Gaussian processes: see Table  1  for details. "cond" is used for all kind of conditional simulations (see further). • the slot values stores the values of the process on the simulation (and visualisation) grid. • an object of class manifold which is the Riemannian manifold on which the process is lying ; an object of the class manifold has four slots: name which is the name of the manifold we consider. The name "line", "plane", "sphere" and "hyperboloid" are taken for the eponymous manifolds. atlas which is the union of discretized domains that cover the manifold (must be a matrix where the number of rows is the dimension of the space where the manifold lives); distance which is the distance considered on the manifold ; origin which is the origin considered on the manifold (must be a point on the manifold).
In particular, NLP methods have been developed to face the limitations of the retrieval systems of clinical trial registries such as clinicaltrials. gov. [12, 13]  For instance, clinical trial records have been notably analyzed using NLP to provide formal representations of eligibility criteria, or to enrich eligibility criteria with meta-data to improve the retrieval of relevant clinical trials for patients  [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26]. However, none of these studies have analyzed the performance of retrieval of clinical trials across diseases, but rather across features of eligibility criteria (e. g. age, BMI 1 or more complex features) for specific diseases. Moreover, the health conditions studied in registered clinical trials must be classified by using a taxonomy of diseases that allows for comparisons between the numbers of clinical trials and the actual burden of diseases.
Clinical trial records In the WHO Trial Registration Dataset, the "Health Condition(s) or Problem(s) studied" field contains a natural language description of the primary condition or problem studied in any clinical trial. Figure  1  shows an example for which the health condition field is "Knee Osteoarthritis" and "Hip Osteoarthritis". This description is not captured by a coded field, with a standardized taxonomy of diseases, but is rather described in a free-text field. Moreover, the analysis of this free-text field alone may not be sufficient to identify the GBD categories of interest. Numerous health condition fields are empty, have entry errors, correspond to "Healthy volunteers", or the relevant GBD category may be difficult to identify because of synonymy. Thus, we also considered the "Public Title" and "Scientific Title" fields, which are most likely to bring additional information about the condition studied in the clinical trial and to enrich the mapping.
Musculoskeletal disorders Fig. 1  Example of classification of a clinical trial record towards the GBD categories. The classification process is based on text extraction from the trial record, text annotation using UMLS concepts, projection of UMLS concepts to ICD10 codes, projection of ICD10 codes to candidate GBD categories among the 28 GBD categories, and GBD classification based on the candidate GBD categories. In this example, the text annotation involved use of the WSD server for MetaMap, and no expert-based enrichment was needed The Unified Medical Language System® (UMLS®), developed at the US National Library of Medicine (NLM), is the most comprehensive metathesaurus to analyze biomedical text in English to date  [32]. We based our classifier on established methods using the UMLS knowledge source to automatically annotate trial records with ICD10 codes.
Free text annotation with concepts from the unified medical language system We first annotated the text fields (health condition, public title and scientific title) with concepts from the UMLS metathesaurus  [32]. The annotation involved use of Meta-Map, a tool from the NLM for recognizing UMLS concepts in text  [33]. We considered only UMLS concepts corresponding to diseases or injuries (MetaMap implementation in Additional file 1). A Word Sense Disambiguation (WSD) server can be used to select a single UMLS concept when a text is annotated with several UMLS concepts. We developed the classifier with and without using the WSD server. In Fig. 1 , the health condition field was annotated with the concepts "Osteoarthritis, Knee" (C0409959) and "Osteoarthritis of hip" (C0029410).
Mapping of UMLS concepts to ICD10 codes Each UMLS concept was then projected to one or several ICD10 codes. The projection involved a semantic-based approach to connect different terminologies present in the UMLS database, namely the Restrict-to-ICD10 algorithm, as implemented in the IntraMap program (IntraMap implementation in Additional file 1)  [34]. In the example from Fig. 1 , the concept "Osteoarthritis, Knee" was projected to the ICD10 codes "Coxarthrosis [arthrosis of hip]" and "Coxarthrosis, unspecified".
Expert-based enrichment Some UMLS concepts were not mapped to any candidate GBD category. We manually reviewed those UMLS concepts appearing in more than 10 clinical trials registered at the WHO ICTRP database by February 2014 and projected them to candidate GBD categories when relevant. We manually reviewed 503 UMLS concepts, among which 62 could be projected to candidate GBD categories (Additional file 1: Datasets S1 and S2). We developed the classifier with and without the expert-based enrichment.
Clinical trial data used in our study The test set included data from 3 different sources. First, we used data from the Epidemiological Study of Randomized Trials, which selected all primary publications of clinical trials published in December 2012 and indexed in PubMed by November 2013  [35]. Among the 1,351 publications, we identified 519 trials registered at the WHO ICTRP. Two independent physicians manually classified Fig. 2  Methodological stages for classification. The classification of clinical trial records has 5 stages. The 4 initial stages allow for deriving pathways from the clinical trial record to candidate GBD categories: annotation of text from the trial record with UMLS concepts by using MetaMap, projection of UMLS concepts to ICD10 codes with IntraMap, projection of ICD10 codes to candidate GBD categories, and expert-based enrichment when automatic pathways are not possible.
The fifth stage allows for deriving the GBD classification of the trial based on prioritization rules over the pathways found each publication according to GBD categories. Second, we used data from a WHO study that extracted a random 5 % sample of clinical trials of interventions registered in the ICTRP by August 2012  [36]. One physician classified 2,381 trial records with GBD categories according to Table  C3  in  [37] , with consensus with a second physician in case of ambiguity. We identified 1,271 trial records for which the classification could be unambiguously mapped to our grouping of GBD categories. Finally, we used data from an ongoing study from our team that involves 973 clinical trials of cancer registered at ICTRP before June 2015. One physician classified each record according to GBD categories, with consensus with a second physician in case of doubt.
Research reproducibility The classifier was coded by using R 3. 2. 2 (R Development Core Team, Vienna, Austria). The programs of the classifier is publicly available for the research community to use at the open source platform github (github. com/iatal/ trial_gbd). It includes all the codes underlying the classification of clinical trial records downloaded at the WHO ICTRP or at clinicaltrials. gov websites towards the 28-or 171-class grouping of GBD categories. In addition, an online interface to optimize manual classification of clinical trials records registered at the WHO ICTRP is available at (http://www. clinicalepidemio. fr/gbd_study_who/). Finally, the classification using the best-performing version of the classifier is provided for all interventional trials registered at WHO ICTRP (N = 109,603 trials by February 2014, Additional file 2).
Exact-matching corresponds to the proportion (in %) of trials for which the automatic GBD classification is correct. Exact-matching was estimated over all trials (N = 2,763), trials concerning a unique GBD category (N = 2,328), trials concerning 2 or more GBD categories (N = 28), and trials not relevant for the GBD (N = 407). The weighted averaged sensitivity and specificity corresponds to the weighted average across GBD categories of the sensitivities and specificities for each GBD category plus the "No GBD" category (in %). The 8 versions correspond to the combinations of the use or not of the Word Sense Disambiguation server during the text annotation, the expert-based enrichment database, and the priority to the health condition field as a prioritization rule. The baseline did not used the UMLS knowledge source, but a clinical trial record was classified to a GBD category if at least one of the disease names defining that GBD category appeared verbatim in the condition field, the public or scientific titles, separately, or in at least one of these three text fields The lowest performance was for the "Injuries" and "Maternal disorders" categories.
Previous studies of automatic indexing used health topics in medical research. The Medical Text Indexer (MTI), developed at the NLM, is used for providing indexing recommendations for data sources such as MEDLINE, PubMed and ClinicalTrials. gov. [29, 44]  MTI produces Medical Subject Headings (MeSH) recommendations by combining a statistical method and a natural language processing method based on MetaMap and the Restrictto-MeSH implemented in IntraMap. This algorithm was shown to be successful for automatically assigning ICD9 Exact-matching and weighted averaged sensitivities and specificities for the classifier to the 28 GBD categories for each source of data. The version of the classifier used was: using the Word Sense Disambiguation server, the expert-based enrichment database and the priority to the health condition field.
Exact-matching corresponds to the proportion (in %) of trials for which the automatic GBD classification is correct. Exact-matching was estimated over all trials, trials concerning a unique GBD category, trials concerning 2 or more GBD categories, and trials not relevant for the GBD. The weighted averaged sensitivity and specificity corresponds to the weighted average across GBD categories of the sensitivities and specificities for each GBD category plus the "No GBD" category (in %) codes to radiology reports  [45]. To our knowledge no previous work has used the knowledge-based sequence MetaMap -IntraMap to assign GBD categories to clinical trials. The Aggregate Analysis of ClinicalTrials. gov project used indexing with MeSH terms to group trials by medical specialty  [30]. However, the medical specialties cannot be connected to the burden of disease.
Evans et al. projected all articles indexed in MEDLINE to GBD categories based on indexing publications with MeSH terms from the MTI  [46]. The authors linked MeSH terms to ICD9 codes by using the UMLS database. In our work, we directly targeted a classification of texts from trial records by using ICD10 codes because GBD categories are defined with that terminology. Instead of using MeSH terms as an intermediate for projection, which may increase the error rate, we chose to develop our method for classifying automatically health topics according to GBD categories based on ICD10. In addition, we mapped ICD10 codes to GBD categories because the GBD 2010 study provides a burden estimate for each GBD category, and not for each ICD10 code. Moreover, these previous studies focused on the curation of health topics of clinical trials records registered at ClinicalTrials.