dataesr/scientific-paragraphs-categorization huggingface dataset:
In the case of missing information, a TEI-compliant local index of places (<listPlace>) could also be generated and used to integrate information. Ad hoc visualization could then be automatically generated, deriving the geographic coordinates of places both from the local index and from the LOD sources. It is well known that the adoption of standards is promoted by providing freely available tools that support such standards. In this case the tool would allow researchers to generate cartographic projections of their corpora by using TEI. As a related issue, it would be interesting to find ways to connect local indexes of places found in digital TEI editions to the main geo-data sets such as DBpedia or Geonames so that the additions made by individual researchers can benefit the whole community.
Secondly, the semantics of place mentions could be made more complex, especially for fiction, differentiating between places that are just mentioned, and places where the action actually takes place. A complex taxonomy is proposed in  Piatti et al. (2013) , allowing for the annotation of dreamt, longed for or remembered places. In such cases as these, an extension of TEI would be required, as this information is clearly mention-specific, and belongs in the text. An interesting move in this direction is found in  Ciotti et al. (2014) , who propose an Open Annotation Data Model (OA) that can be used to make more complex annotations of entities in TEI texts, including toponyms. 25 hirdly, it should be possible to annotate and collect indirect geographical information also from textual elements that, though not toponyms, bear a relationship to places.
Acknowledgments This work has been done within the LABEX  OBVIL  project, and received financial state aid managed by the  Agence Nationale de la Recherche , as part of the  programme "Investissements d'avenir"  under the reference  ANR-11-IDEX-0004-02 ; it was also partly supported by a  "Frenand Braudel" incoming scholarship  from the  Fondation Maison Sciences de l'Homme, Paris.
20  Other similar LOD databases contain temporal information for a specific geographic area only, such as for instance the New York City Chronology of Place, a Linked Open Data Gazetteer ("NEH Grant Details: NYC Chronology of Place, a Linked Open Data Gazetteer. " 2015. Accessed January 22, 2016. https://securegrants. neh. gov/publicquery/main. aspx?f=1&gn=HD-51618-12.
Acknowledgments We would like to thank  Frans Cornelissen  for providing the eye-tracker;  Jop Luberti  for providing the pictures of the objects used in the eye-tracking study;  Paolo Toffanin  and  Frits Leemhuis  for technical and research support;  Ria Woldhuis  for general support; and  Annemieke ter Harmsel ,  Nico Leenstra ,  Marije Sleurink ,  Floor Burgerhof ,  Maraike Coenen ,  Esme´e van der Veen ,  Charlotte de Blecourt ,  Wydineke Boels ,  Margriet Bekendam , and  Lisette Stuifzand  for help with collecting data and transcribing participant responses. We also greatly appreciate the efforts of our participants.
Funding The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work was supported by a  VIDI Grant  (Grant No. 016. 093. 397 )from the  Netherlands Organization for Scientific Research (NWO) , the  Netherlands Organization for Health Research and Development (ZonMw) , a  Marie Curie Intra-European Fellowship  ( FP7-PEOPLE-2012-IEF 332402 ),  Rosalind Franklin Fellowship  from  UMCG , and funds from  Heinsius Houbolt Foundation ,  Cochlear Ltd. ,  Pento Speech  and  Hearing Center Zwolle ,  Doorhout Mees ,  Stichting Steun Gehoorgestoorde Kind  as well as through the  LabEx CeLyA (''Centre Lyonnais d'Acoustique'' ,  ANR-10-LABX-0060/ ANR-11-IDEX-0007 ) operated by the  French National Research Agency.
We wrapped our biosigner algorithm around three reference binary classifiers (Partial Least Squares-Discriminant Analysis, Random Forest, and Support Vector Machines) which have been shown to achieve specific performances depending on the structure of the dataset. By using three real biological and clinical metabolomics and transcriptomics datasets (containing up to 7000 features), complementary signatures were obtained in a few minutes, generally providing higher prediction accuracies than the initial full model. Comparison with alternative feature selection approaches further indicated that our method provides signatures of restricted size and high stability. Finally, by using our methodology to seek metabolites discriminating type 1 from type 2 diabetic patients, several features were selected, including a fragment from the taurochenodeoxycholic bile acid.
, 2011; Armitage and Barbas, 2014) , and neurodegenerative diseases  (Graham et al. , 2013; Mapstone et al. , 2014). Powerful statistical and data mining approaches have been developed to learn classification rules from omics datasets despite the high feature over sample ratio and the large proportion of correlated features  (Trygg et al. , 2007; Scott et al. , 2013; Tarca et al. , 2013). Such approaches include Support Vector Machines (SVM;  Boser et al. , 1992) , Partial Least Square-Discriminant Analysis (PLS-DA;  Wold et al. , 2001; Barker and Rayens, 2003) , and Random Forest  (Breiman, 2001) , which have been widely used in transcriptomics, proteomics, and metabolomics  (Brown et al. , 2000; Diaz-Uriarte and Alvarez de Andres, 2006; Madsen et al. , 2010; Robotti et al. , 2014). Although these models can achieve good predictions accuracies, the excess of features in the training dataset increases both the risk of overfitting and the prediction variability.
In addition, in the context of biomarker discovery and clinical diagnostic, selection of a restricted list of candidate markers is mandatory before entering the subsequent qualification/verification phases  (Baker, 2005; Rifai et al. , 2006; Keating and Cambrosio, 2012). Since the comprehensive analysis of all 2 p combinations of p features is not computationally tractable for large omics datasets, several statistical and data mining techniques for feature selection have been described with the common goal of extracting a restricted list of variables (i. e. , a molecular signature) still providing high performance of the classifier  (Guyon and Elisseeff, 2003; Saeys et al. , 2007). One strategy consists in filtering the features before building the classifier  (Golub et al. , 1999). In such filter techniques, features are ranked according to a univariate (e.
Such strategies are computationally efficient but the signature may be large and subject to substantial variation upon repetition of the algorithm (instability). Moreover, only one type of classifier is used, whereas several studies have shown that best classification performances are obtained by distinct models depending on the structure of the dataset  (Guo et al. , 2010; Tarca et al. , 2013; Determan, 2015). Therefore, a third category of approaches, called wrapper methods, are of interest because they can be applied to any classifier, and take into account the specificities of the classifier in the process of feature selection  (Kohavi and John, 1997). The wrapper feature selection methods (e. g. , Recursive Feature Elimination, RFE, applied to SVM;  Guyon et al. , 2002)  iteratively (i) select groups of features which still provide a good classification accuracy, and (ii) re-build the model on the data subset.
Several heuristics have been described to find optimal combination of features (either deterministic, such as forward and backward selection of individual or groups of variables, or stochastic, such as genetic algorithms or simulated annealing;  Kuhn and Johnson, 2013). A limitation of current wrapper methods is that the selection criterion is based on the classifier performance only: the added-value of including a particular group of features instead of noise into the model (which we call the feature subset significance hereafter) is not evaluated. Here, we therefore propose a new wrapper algorithm based on random permutation of feature intensities in test subsets obtained by resampling, to assess the significance of the features on the model performance. We wrapped our algorithm around three classifiers, namely PLS-DA, Random Forest, and SVM, and applied our feature selection approach to four real metabolomics and transcriptomics datasets, including one unpublished clinical LC-HRMS analysis of plasma samples from diabetic patients.
THEORY The objective of our method is to find the significant feature subset necessary for a classifier to optimally discriminate between two classes. Given a machine learning methodology, our algorithm thus provides both the molecular signature (i. e. , the significant feature subset) and the trained classifier, which can subsequently be used for prediction on new datasets. Feature selection is based on a backward procedure in which significance of each feature subset is estimated by random permutation of the intensities. The dataset is then restricted to the significant feature subset, and the whole procedure is performed iteratively until, for a given round, all candidate features are found significant (in this case the signature consists of these features), or until there is no feature left to be tested (in this case the signature is empty).
The algorithm thus consists of three steps (Algorithm 1 and Figure  1   Guyon et al. , 2002). Finally, the rank k are aggregated by computing the median to obtain the final ranking: rank = RANK(MEDIAN 1≤k≤boot (rank k )) (1) where RANK and MEDIAN are the usual ranking and median functions. 3. Selection of significant features. The objective of this step is to discard all non-significant features from the dataset. The method consists in finding the largest non-significant feature subset S f = {g|rank(g) ≥ rank(f )} (or, equivalently, the feature f ns of lowest rank such that S f ns is not significant). A half-interval search algorithm is used to find f ns : for a given f , the significance of S f is estimated by randomly permuting all S f feature intensities in the test k subsets (Figure  2 ), and computing the predictions accuracies of the model k on these permuted subsets (accuracy k,perm ).
If the proportion of accuracy k,perm ≥ accuracy k over all boot comparisons is above a defined threshold (5% by default), S f is declared not significant, and the next candidate feature f ′ has the rank closest to the mean of rank(h) -1 and rank(f ) + 1, where h is the last significant feature detected. Otherwise S f is significant, and the next f ′ is the feature with the rank closest to the mean of rank(f )-1 and rank(l)+1 where l is the last non-significant feature detected. At the end of the half-interval search, the dataset is restricted to the features of ranks < rank(f ns ). If no feature has been found significant, the dataset is restricted to the half of features with lowest ranks, but these features are not registered as significant in this round. 4. Building the final model. Steps 1-3 are repeated until, for a given round, candidate features are all found significant (these features then correspond to the signature), or until there is no feature left to be tested (the signature is empty).
Datasets • LC-HRMS metabolomics • sacurine: Urine from human adults The metabolomics analysis of urine samples from a cohort of employees from the CEA Saclay research institute by liquid chromatography coupled to high-resolution mass spectrometry (LC-HRMS) has been described previously  (Roux et al. , 2012; Thevenot et al. , 2015). Briefly, the samples were analyzed by ultra-high performance liquid chromatography (Hypersil GOLD C18 column, Thermo Fisher) coupled to a high-resolution mass spectrometer   Repeat the while procedure above until either X is constant (i. e. , the S subset from the last round was empty), or until X does not contain any feature left [step 4] 14: If X contains features, return the model trained on X 15: end function (LTQ-Orbitrap Discovery, Thermo Fisher). The raw files were preprocessed by using the XCMS  (Smith et al.
, 2006)  and the CAMERA R packages  (Kuhl et al. , 2012). Annotation at levels 1 and 2 from the metabolomics standard initiative (MSI;  Sumner et al. , 2007)  was performed by in-house and public databases query in addition to MS/MS experiments. Finally, the intensities of the annotated metabolites were validated by using the Quan Browser module from the Xcalibur software (Thermo Fisher), corrected for signal drift and batch-effect, and log 10 transformed  (Thevenot et al. , 2015). Here, we used the sacurine subset corresponding to the negative ionization mode, which consists of 183 samples and 109 annotated metabolites, and is available from the ropls R Bioconductor package. • spikedApples: Apples spiked with known compounds One control group of 10 apples and several spiked sets of the same size have been analyzed by LC-HRMS (SYNAPT Q-TOF, Waters;  Franceschi et al.
, 2012). The spiked mixtures consisted of 2 compounds which were not naturally present in the matrix, and 7 compounds aimed at achieving a final increase of 20-100% of the endogeneous concentrations. The dataset is included in the BioMark R Bioconductor package  (Franceschi et al. , 2012). The control and the first spiked groups (i. e. , a total of 20 samples) were used in this study. • diaplasma: Plasma from diabetic patients Collection of plasma samples from type 1 and type 2 diabetic patients (Hôtel-Dieu, Paris, France) was performed with informed consent of the subjects, in accordance with the 1964 Helsinki declaration and its later amendments. Samples were analyzed by ultra-high performance liquid chromatography (Nexera, Shimadzu) coupled to a high resolution mass spectrometry operating in the negative ionization mode (Orbitrap Exactive, Thermo Fisher).
Raw data were processed with XCMS  (Smith et al. , 2006)  and CAMERA  (Kuhl et al. , 2012) , and the resulting peak table was annotated by matching the measured m/z and retention times against an in-house database from MS spectra of pure compounds. Signal drift was corrected by using a loess fit of the intensities from quality control (pool) samples injected periodically  (Dunn et al. , 2011; Thevenot et al. , 2015). Features which did not meet the following quality control criteria were discarded: (i) ratio of mean intensity in samples over mean intensity in blanks (mobile phase only) >2, (ii) significant correlation between the intensities of the diluted pool samples and the dilution factor, and (iii) coefficient of variation of pool intensities <30%. Five samples with p-values < 0. 001 by using either the Hotellings' T 2 outlier test, or the Z-score corresponding to the highest deviation of intensity quantiles  (Alonso et al.
• Microarray transcriptomics • leukemia: Bone marrow from acute leukemia patients Samples from patients with acute lymphoblastic (ALL, 47 patients) or myeloid (AML, 25 patients) leukemia have been analyzed by using Aymetrix Hgu6800 chips. The resulting dataset contains expression data from 7129 gene probes  (Golub et al. , 1999) , and is available from the golubEsets R Bioconductor package.
• Partial Least Squares Discriminant Analysis (PLS-DA) The PLS-DA implementation from the ropls bioconductor package  (Thevenot et al. , 2015 ; version 1. 2. 2) was used after mean-centering and unit-variance scaling of the features. Briefly, the binary response is converted to a numeric vector y of values in {-0. 5; 0. 5}, and a PLS regression is performed with the NIPALS algorithm  (Wold et al. , 2001; Barker and Rayens, 2003). The number of components is determined automatically as follows  (Eriksson et al. , 2001) : a new component h is added to the model if : FIGURE 2 | Determination of the largest non-significant subset of features (step 3 of the biosigner algorithm). During the two first steps of the algorithm, the dataset has been split into train and test subsets by resampling, and features have been ranked according to their aggregated importance for the classifier built on the train subsets (the lower the rank, the higher the importance).
Finally, the predictive performance of the full model is assessed by the cumulative Q2Y metric: Q2Y = 1 - r h=1 (1 -Q2Y h ). We have Q2Y ∈ [0, 1], and the higher the Q2Y, the better the performance. However, models trained on datasets with a larger number of features compared with the number of samples can be prone to overfitting: in that case, high Q2Y values are obtained by chance only. To estimate the significance of Q2Y (and R2Y) values,  Szymanska et al. (2012)  therefore proposed to perform permutation testing: models are built after random permutation of the y values, and Q2Y perm are computed. The p-value is equal to the proportion of Q2Y perm above Q2Y (in this study, the number of random permutations was set to 1000).
Quality of the Feature Selection The (balanced) prediction accuracy of a classifier is the mean of sensitivity and specificity. The stability of the signature was determined as follows: the dataset was split into 10 subsets, each containing 90% of the samples, and the feature selection approach was applied to each subset, resulting in 10 signatures. The stability was the average similarity over all pairs of signatures. We used the similarity measure proposed by Lustgarten and colleagues, since (i) it is adjusted for the commonality of subsets obtained by chance only, and (ii) it allows to compare signatures of different sizes  (Lustgarten et al. , 2009). The performancerobustness trade-off (hereafter performance) was computed as the harmonic mean of accuracy and stability  (Determan, 2015).
Software The biosigner package was written in R (R Core Development Team, 2015; version 3. 2. 2) and is available (http://bioconductor. org/packages/biosigner) from the Bioconductor repository  (Gentleman et al. , 2004). It includes the diaplasma LC-HRMS metabolomics dataset. The package was run on a laptop computer (Windows 7; Intel Core i5 2. 6 GHz processor; 8 GB RAM). The biosigner algorithm is also available with a graphical interface, as a Galaxy module within the Workflow4Metabolomics. org online resource for computational metabolomics (http://workflow4metabolomics. org; Giacomoni et al. , 2015).
Development of the biosigner Algorithm We developed a new wrapper algorithm to select features which significantly improve the prediction of any binary classifier (see Section 2 and Figure  1 ). A feature subset S f is declared significant if the predictions on test subsets (generated by resampling) are less accurate after randomly permuting the intensities of all features in S f (Figure  2 ). The dataset is then restricted to the significant features and the procedure is iterated until the set of significant features remains unchanged (i. e. , corresponds to the final signature), or until there is no feature left in the dataset to be tested (Theory section and Figure  1 ). The algorithm was wrapped around three machine learning approaches, namely Partial Least Squares Discriminant Analysis (PLS-DA), Random Forest, and Support Vector Machine (SVM), which rely on specific mathematical backgrounds (latent variables, decision trees, and kernel methods, respectively).
Evaluation on Published Metabolomics and Transcriptomics Datasets We addressed the performance of the biosigner algorithm by analyzing the signatures obtained on two metabolomics and one transcriptomics real datasets. We started with a well-annotated human metabolomics dataset, in which the concentrations of 109 metabolites have been measured in urine samples from a cohort of 183 adult volunteers by using liquid chromatography coupled to high-resolution mass spectrometry (LC-HRMS,  Thevenot et al. , 2015). A previous study of the physiological concentration differences between males and females using this sacurine dataset  (Thevenot et al. , 2015)  has shown that: (i) no specific gender clusters were observed by Principal Component Analysis (PCA), (ii) 45 metabolites had a significant difference of medians between genders (with Mann-Whitney U-tests and a False Discorvery Rate threshold of 5%), and (iii) PLS-DA modeling of gender had a significant Q2Y value of 0.
58 (metric between 0 and 1 estimating the prediction performance by cross-validation; see Section 3). By applying the biosigner algorithm, signatures consisting of 2 (Random Forest) or 3 (PLS-DA, SVM) metabolites (i. e. , less than 3% of the initial features) were identified (Figure  3A ). Testosterone glucuronide was common to all 3 signatures, and oxoglutaric acid and p-anisic acid to 2 of them. All selected metabolites had a clearcut difference of intensities between males and females (Figure  4A ). Prediction accuracies of the models restricted to the signatures were all superior or equal to the models trained on the full dataset (Table  1 ). We then studied a recently published metabolomics dataset spiked with known compounds  (Franceschi et al. , 2012). The spikedApples dataset results from the LC-HRMS analysis of groups of apples which have been spiked with various concentrations of 7 endogenous metabolites and 2 exogenous compounds.
The peak table used in this study consists of two groups of 10 apple samples (control and spiked), and 1632 features (among which 22 were identified by the authors as originating from the spiked molecules;  Franceschi et al. , 2012). Preliminary modeling of the control vs. spiked response by PLS-DA with the FIGURE 3 | biosigner discovery of molecular signatures from biological and clinical omics datasets. (A,B,D) The sacurine, spikedApples, and diaplasma metabolomics datasets result from the LC-HRMS analysis of, respectively, urines from male or female adult volunteers  (Thevenot et al. , 2015) , control apples or apples spiked with a mixture of known compounds  (Franceschi et al. , 2012) , and plasma from type 1 or 2 diabetic patients (see Section 3). (C) The leukemia transcriptomics dataset has been obtained by microarray analysis of bone marrow from patients with acute lymphoblastic or myeolid leukemia  (Golub et al.
, 1999). The biosigner signatures of gender (sacurine dataset), spiking (spikeApples), ALL/AML leukemia (leukemia), and diabetes type (diaplasma) obtained independantly by each of the 3 classifiers are shown (for diaplasma, the extended signatures up to tier A are displayed). The S tier corresponds to the final signature, i. e. , metabolites which passed all the selection iterations determining whether the candidate variables significantly improve the model prediction. In contrast, metabolites from the other tiers were discarded during all but the last (A) or previous (B to E) iterations. RF, Random full dataset indicated that the Q2Y was not significant: feature selection was therefore mandatory to avoid overfitting (Table  1 ). The biosigner algorithm identified complementary signatures of 1 or 2 features (Figure  3B ).
Classifiers trained on the dataset restricted to the signatures outperformed the models trained on the full dataset. Interestingly, the single features selected by PLS-DA and Random Forest corresponded to cyanidin-3galactoside and a potassium adduct of phloridzin, respectively, which both belong to the list of expected discriminating metabolites: the former is absent from the natural matrix (control group), and the concentration of the latter is increased up to 80% in the spiked group  (Franceschi et al. , 2012 ; Figure  4B ). Surprisingly, the two features selected by SVM are less concentrated in the spiked group (Figure  4B ). It should be noted, however, that such a decrease in concentrations was found significant by univariate hypothesis testing for a total of 9 features (including 722.