dataesr/scientific-paragraphs-categorization huggingface dataset:
The LR algorithm obtained the best performance on many metrics, although the SVM algorithm obtained a higher recall using the titles and titles and abstracts, and Random Forests obtained the highest AUC on the same sets. The Neural Networks algorithm obtained the highest precision using the abstracts, titles and abstracts, and titles and metadata sets. The balance between precision and recall is an important topic to take into consideration in this type of approach. We included both the balanced F1 scores and the F2 scores, as both have been used in other document curation studies. However, we cannot say for sure the balance that is expected by a curator. While ideally no relevant document should be excluded by a classifier (False Negatives), it will make the curators task harder if many irrelevant documents are presented (False Positives).
Error Analysis In order to interpret the gap of results between the training set and the predictions obtained from the classifiers on each cross-validation iteration, the LR classifier built with the titles was analysed. This classifier had a similar recall score to the abstracts but, as the titles are shorter, they make the interpretation easier. One interesting pattern we noticed was that almost all titles that had the words "food frequency questionnaire" were classified as relevant. As previously mentioned in section 2. 2, this related to how articles about dietary biomarkers were chosen to be included in the database. From a total of 82 titles containing these words, only 2 were classified as irrelevant (both had words such as "calcium", "water" and "energy" that were mostly found on irrelevant articles); 29 were TP and the remaining 51 were being wrongly labelled as relevant.
The title "Toenail selenium as an indicator of selenium intake among middle-aged men in an area with low soil selenium" was classified as negative, when it was in fact used in the database (FN). 39 out of 40 titles with the word "selenium" were not used in the database and thus labelled irrelevant: this over-represented feature may be the reason why the classifier failed to classify this article as relevant although selenium was considered of interest by the annotators. It is also important to highlight that papers inserted in the database have been analysed considering the full-texts. This means that papers tagged as "relevant" either by the classifier and/or manually, could subsequently be rejected by the annotators, for a variety of reasons including "the paper is notavailable online," or "the data in the paper is not presented in a way acceptable for the database.
" These papers would then be considered false positive by the classifier, because they are present in the corpus of citations but absent from the database. Restricting the analysis to the dietary biomarker citations provided much better metrics than when using all the data from the database (dietary, pollutants, and reproducibility values) (Table  6 ). When restricting the analysis to citations describing the different classes of biomarkers of pollution, the performance of the models was even lower (preliminary results not shown). This difference in performance could be explained by the difference of nature of the data searched by the annotators for the different sets of biomarkers. For dietary biomarkers, the focus was made on publications providing correlation values between dietary intakes and biomarkers measured in human biospecimens, and mostly describing validation studies of dietary questionnaires with biomarkers.
For the pollutant biomarkers, the focus was made on papers describing concentration values of pollutant biomarkers in human biospecimens. Moreover, by lack of time and human resources, not all potentially relevant publications on pollution biomarkers were inserted in the Exposome-Explorer, while the dietary biomarkers were handled more attentively. As a consequence, the dietary biomarkers account for almost half of the entries of the database. All of this could explain why the model seems to perform better for dietary biomarkers. Having a closer look at false positives obtained by the classifier on pollutants could be a good way to check if the model developed on dietary biomarkers could also be applied to pollutants, and identify new relevant papers from the corpus of pollutants. This also means that as we obtain a more comprehensive corpus for other classes of biomarkers, the performance of our machine learning solution will also improve.
CONCLUSION The Exposome-Explorer database is being manually curated, without any assistance from machine learning tools. As the number of scientific papers continues to grow, text-mining tools could be a great help to assist the triage of documents containing information about biomarkers of exposure and keep the database updated. To this end, several machine learning models were created using different combinations of preprocessing parameters and algorithms. These classifiers were trained using the publications' abstracts, titles and metadata. The model with the highest F2-score (70. 1%) was built with the LR algorithm and used the titles and abstracts to predict a paper's relevance. We also extracted named-entities from the abstracts, obtaining a total of 45 candidate biomarkers. To apply this methodology to the database curation pipeline, the IR task will consist of two steps.
In the first one, articles will be retrieved using the query search on WOS, to target domainspecific publications. Then, the classifier could be used to narrow down the publications even more, and a named-entity recognition tool can be used to provide candidate entries to the database. Manual curation will still be needed, to extract information about biomarkers from full-text articles. In the future, we will work on improving the results from the classifiers that use the metadata set. For example, by assigning different weights to the authors, according to the position they appear in, or by creating new features that result from the combinations between all authors within the same article. We will also study the impact of the recognized biomarkers in the retrieval classification. When analysing why the model misclassified some publications, a few chemicals, like "calcium" and "selenium" were strongly associated with irrelevant articles.
An idea to explore is to replace chemical tokens by a category they belong, such as "chemical," and see if it improves the precision and recall of the classifiers. This should avoid over-fitting on the training set. Furthermore, we will improve this gold standard by adding more types of biomarkers, that can also classify non-dietary biomarkers. Another idea to explore is to train deep learning models for document classification, even though this would require more training data and will take longer to train than the algorithms used in this paper. Finally, the performance of classifiers trained on this dataset when applied to the results of other search queries will be explored.
Integrating datasets from different disciplines is hard because the data are often qualitatively different in meaning, scale, and reliability. When two datasets describe the same entities, many scientific questions can be phrased around whether the (dis)similarities between entities are conserved across such different data. Our method, CLARITY, quantifies consistency across datasets, identifies where inconsistencies arise, and aids in their interpretation. We illustrate this using three diverse comparisons: gene methylation vs expression, evolution of language sounds vs word use, and country-level economic metrics vs cultural beliefs. The non-parametric approach is robust to noise and differences in scaling, and makes only weak assumptions about how the data were generated. It operates by decomposing similarities into two components: a 'structural' component analogous to a clustering, and an underlying 'relationship' between those structures.
Introduction The need to compare different sources of information about the same subjects arises in most quantitative sciences. With sufficient effort, it is always possible to construct a model that accounts for data of arbitrary complexity. But without this time-consuming work, can we visualise the data to determine whether the different sources describe the same qualitative phenomena? Many datasets are best expressed in terms of similarities or differences between subjects, and are frequently compared by plotting the resulting matrices side by side. Examples include the co-evolution of language and culture  [Sokal, 1988] , as well as genetics and phenotype  [Creanza et al. , 2015] , which are all linked through their geographical constraints and shared history. Further uses include identifying brain function using neural activity patterns  [Kriegeskorte et al.
The purpose of CLARITY Figure  1  is a 'graphical abstract' to illustrate what CLARITY is designed to detect. Rather than learning a model, CLARITY identifies features of one dataset that are anomalous in the second -marginalising out structures present in both. It does this from information on subjects, i. e. labelled entities on which we have data such as countries, languages, genes, etc. -which are matched across datasets. It looks for structuresthat can be loosely thought of as clusters or shared variance components -that are present in the second dataset, but were not in the first. It reports by identifying subjects with large residuals that 'persist', i. e. can't be predicted from the first dataset, for a wide range of complexity of representation. CLARITY works with dissimilarities (and equivalently, similarities), which until they are formally defined (see Section 4.
Overview of comparison approaches How different is the information provided about the same subjects in two datasets? For what follows, we are interested in the relationship between the subjects, rather than particular features in the datasets, and we assume that we have enough information to build a meaningful similarity matrix between the subjects. The gold standard approach involves generative modelling, in which the joint model for both datasets is specified. Examples include host-parasite coevolution  [Brooks, 1979]  and comparing linguistic and genetic data  [Amorim et al. , 2013]. However, each analysis is bespoke, requiring an expert modeller able to specify a joint model for the two datasets. If the datasets take a matrix form then testing whether two matrices are statistically equivalent is another natural starting point.
For this, Mantel's test  [Mantel, 1967]  and related approaches  [Smouse et al. , 1986] , can be used. However, for the sort of scientific investigation that we are considering here, the null hypothesis that the two datasets have 'the same' distribution, or, in the case of shared historical processes, are 'independent' of each other, can often be rejected a-priori. We are thus interested in richer comparisons, able to highlight specific subjects that behave differently in two datasets. Data can be directly compared by transforming one to look like the other. When applied to matrices, an important class are Procrustes transformations  [Hurley and Cattell, 1962] , which use rotation, translation and scaling  [Schneider and Borlund, 2007]  to perform the maximally achievable matching. Procrustes transformations have been used for testing matrix equality under transformation  [Jackson, 1995] , and are often combined with initial rank reduction via Spectral decomposition for matrix comparison, e.
g. [Peres-Neto and Jackson, 2001 ]. If we are not constructing an explicit model of both datasets, nor testing whether they are identical, then the remaining options revolve around constructing summaries that can be compared. Many methods exist to compare covariance matrices. Testing  [Steiger, 1980]  is again straightforward. Metrics comparing Residuals persist when structure changes! Figure  1 : What is CLARITY For? CLARITY compares a reference dataset (i) to a target dataset (ii-iii) to perform a structural comparison. (a) Structure is defined in terms of the similarity between subjects (here letters a-p) that in this example fall into clusters. These clusters have a relationship, here the distance between clusters. (b) The data are quantified as a similarity matrix between all subjects.
Another important class of summary are tree-based methods that represent each dataset as a tree, which can be compared using standard metrics. These include topological distance  [Penny and Hendy, 1985, Bille, 2005] , and tree-space  [Nye et al. , 2017] , and the approach is implemented in popular packages such as 'phangorn'  [Schliep, 2011]  in R. The downside is that handling model uncertainty is difficult, with only some types of tree being stable to small changes in the data  [Carlsson and Mémoli, 2010]. Often the data are not completely hierarchical -for example, tree-based methods can be misleading when the data have a mixture element to them  [Mossel, 2005]. Conversely, whilst mixtures might be compared using fixed-dimensional mixture based methods  [Tipping, 1999, Mahalanabis and Štefankovič, 2009] , this can be misleading when the data have an hierarchical element to them  [Lawson et al.
, 2018]. With CLARITY we are addressing scientific questions that relate to which similarity structures are present in two datasets. There are other scientific questions that might be asked. For example, Canonical Correlation Analysis (CCA)  [Hotelling, 1936 , Seber, 2009]  and related approaches can be applied on datasets with matched features, as in e. g. ecology  [Ter Braak, 1987]  and machine learning  [Hardoon et al. , 2004 , Raghu et al. , 2017]. CCA addresses the question of which features in one dataset are important for understanding another. Because of this focus on features, CCA cannot be used directly in any of the simulations or real datasets that we consider below. Qualitatively this is because the datasets can match perfectly if the number of features is higher than the number of subjects.
High level view of CLARITY for comparing data from different sources This section contains a high-level mathematical description of the sort of comparison CLARITY is useful for. Technical mathematics is left for the Methods, Section 4. 2, but at a high level, the key concepts are given in Box 1. CLARITY allows comparison of arbitrary datasets for which the same set of d subjects are observed. It represents the similarity of a reference dataset Y 1 non-parametrically using increasingly rich representations of complexity k ≤ d. At each k we learn a structure A k , which is a d × k matrix, and a relationship X (k)  between the structures, which is a k × k matrix. Learning is by minimising the squared error of the estimate Ŷ1 , which is the sum of the squared residuals. Example: In Figure  1b  Y 1 is a similarity.
, d clusters. We also learn the relationship X (k) 1 i. e. similarity between clusters in that dataset. We then predict Y 2 from each set of clusters A k , but learn a completely new relationship between clusters X (k) 2. With few clusters, we have a bad representation of Y 1 whilst with many clusters we overfit it. We are therefore interested in subjects that are poorly explained for many intermediate clusterings, i. e. those with persistent residuals. The CLARITY model uses a range of complexities k to represent data. The full set of structures therefore quantifies a rich range of models. For example, it can be constrained to a hierarchical clustering (i. e. a tree), and in Section 2. 4 we show that if the data are tree-like, the structure can be interpreted in terms of a 'soft tree' (Figure  2 ), which can capture deviations from a strict tree model.
Structural comparison: The structure learned from the reference Y 1 at each complexity is used to predict the target Y 2. Y 2 may be numerically quite different if the relationships are different. However, as long as the datasets can be predicted in this sense then we say that the matrices are defined to be structurally similar ; this will happen if the same clusters, mixtures or eigenvalues are important in both datasets and describe the same subjects. Residual persistence charts: We use graphical summaries to present useful scientific insights, focussing on structures that persist over a range of model complexity. These are inspired by the concept of persistent homology from Topological Data Analysis  [Wasserman, 2018]. When the complexity k is sufficiently high, every (full rank) dataset can predict every other, so the focus is on which structures in Y 2 are explained late in the sequence defined by Y 1 , i.
e. persist. This is captured by the residual persistence P ik , a matrix whose entries for data subject i at a complexity k, are the sum (over j) of the squared residuals. first dataset. In this condition, persistences remain high until much higher complexity, especially so for the anomalous subject i. Key decisions when using CLARITY: There are just two substantive choices to make when running CLARITY. First, how to construct the similarity matrices Y 1 and Y 2 , which may be determined by the data. Second, the choice of structure A k , for which the two main approaches we provide are SVD, which is computationally convenient, or a mixture model, which is more interpretable. We have not found a case for using clusters in practice, as they are not robust. There is one final technical choice, which is the statistical procedure for assessing significance, which we address in Section 4.
This mixture model is interpretable, as we demonstrate in the simulation study. Specifically, a 'structural difference' at complexity k means that the latent clusters of Y 2 are not in the k most important latent clusters of Y 1. Further, the subjects that are poorly predicted, i. e. whose cluster membership is not captured, can be read off from the residual matrix. Persistences and residuals decrease with model complexity and are affected by correlations between similarities. Despite the complexities of working with a similarity matrix, in Theorem 2 (Methods Section 4. 8) we prove that the model is stable in the presence of noise, so that if two datasets were resampled then their residuals based on our notion of structure are not expected to change by a large amount. The theoretical and simulation results together demonstrate that the CLARITY paradigm is performing a meaningful comparison.
Comparing Simulated Gene Methylation to Gene Expression This example focusses on identifying anomalies: subjects (here loci) that behave differently in one dataset compared to another. The CLARITY framing using a similarity matrix means that the datasets need not have features in common. It is very common in epigenetics to wish to compare different measurements on genes. These are often measured on different scales -methylation is a proportion whilst expression is a positive number -and writing a formal model is hard. Further, the raw data may not be available to each researcher as they are identifiable and sensitive. It is therefore very helpful if summaries, such as those based on similarities, can be compared. To emphasise the utility of CLARITY we focus on an epigenetic simulation model from the literature  [Gu et al.
This variability adds to signal from causative factors of interest. The simulation therefore assumes only a -5% average correlation. The simulation is for a case-control study in which some people were controls, and others had a tumor. Methylation is simulated in different classes  ('hyper' and 'hypo' methylated loci)  which interact with casecontrol status. Expression is simulated conditional on methylation, with large and random noise. Finally, anomalous loci are chosen in which the relationship between methylation and expression is reversed. See Methods 4. 5 for full details. The important feature of this setup is that the set of people considered to quantify methylation and expression are independent. The only thing they share in common is the set of loci on which data are reported. Figure  4  illustrates this dataset and highlights the ability of CLARITY to extract the anomalous loci, despite the considerable structure in this dataset.
a) Methylation Data In the linguistic literature, it is often argued qualitatively, that phonetic and lexical change can be unevenly favoured by different social situations of language transmission and contact, see e. g. [Thomason and Kaufman, 1988]. Here, we use CLARITY to provide a quantitative test, the first such experiment known to us. Figure  7a-b , demonstrates that the two change types induce closely aligned similarities between language. While this is not surprising, given that the same factors influence both types of linguistic change, we show that, despite this high degree of correlation, there are still detectable 'structural' differences between lexical and phonetic changes. Two components of our comparison are crucial: (i) we use a large dataset that enables us to apply CLARITY significance testing and thus derive a statistically credible result; (ii) allowing the CLARITY 'relationship' to differ for phonetic and lexical, we effectively permit the rates of both types of change to differ considerably without the matrices becoming structurally dissimilar; in other words, the null hypothesis of CLARITY is broad enough that its rejection would be informative.
The true history of both phonetic and lexical change is unknown, but can be conceptualised as a graph that consists of a "vertical" inheritance via a tree that captures independent change, and "horizontal" edges that capture change through language contact. Such a graph induces a similarity matrix between languages. Many graphs may induce the same matrix, making direct inference of the history graph impossible in the general case, Figure  5a-b. But the similarity matrix does allow us to distinguish between different classes of graphs, cf. Figure  5a -b and Figure  5c. With CLARITY, we can infer changes to the structure of the underlying graph without explicitly learning that graph. With two matrices representing similarity due to phonetic vs. lexical change, we can use CLARITY to find out whether phonetic and lexical change go hand in hand.
This way, we can be confident that a rejection of the null hypothesis would correspond to a true real-world difference in 'structure'. Finally, automatic cognate detection cannot distinguish between cognates inherited from the last common ancestor and words that have been borrowed between sister branches after divergence. For recovering the true tree part of the language-family history, this is a drawback; for us, this is a benefit, as it captures language contact relations in the lexical-change data, just as it affects the phonetic-change data. Our data come from one of the largest existing historical-linguistic datasets, NorthEuraLex v0. 9  [Dellert and Jäger, 2017] , which stores phonetic transcriptions of words expressing 1016 different meanings in over a hundred languages. We focus on the 36 Indo-European languages in NorthEuraLex, for which we computed measures of both phonetic and lexical dissimilarity using a state-of-the-art method  [Dellert, 2018] , as discussed in more detail in Sec.
Despite high correlation, CLARITY allowed us to discover a clear difference between the two processes of language change. How should we interpret this finding? In terms of the informal representation above, our results about Lex and P hon indicates that the real-world quantities T rueLex on the one hand, and unrelated. phon. sim and cognate. phon. sim on the other, are affected by subtly different historical processes. Given that P hon depends on a superset of real-world quantities that Lex depends on, it is not surprising that we only find the effect in the direction of predicting P hon from Lex. The difference between the two is considerable, but is only discovered with statistical significance for groups of languages closely related to each other, such as the Slavic and Scandinavian languages.