dataesr/scientific-paragraphs-categorization huggingface dataset:
Workflow The steps leading to the establishment of a rule network are described in Fig 2. Users can upload their data via the web interface as tab-delimited text files that contain samples (rows) and attributes (columns). Each file is then parsed and turned into a table of a database. RQL is coupled with the standard SQL language; therefore, users can skip the file upload step by connecting directly to a database by specifying the corresponding parameters. The advantage is that the data can be queried directly from where they are stored without any complex preprocessing task. The second benefit is that it takes advantage of DBMS technologies for the optimization of query performance and hence computation time. The second step consists of specifying queries that define the semantics of the rules to generate.
Computational performance In order to assess the computational performance of RulNet, the two predefined queries were run on the platform using datasets consisting of randomized numerical data. Computing times and the total number of rules discovered (exact and approximate rules) were recorded for datasets of 10 to 4,000 samples (with 100 attributes) and 10 to 4,000 attributes (with 100 samples; Fig  3 ). The number of rules given is only indicative as it could vary depending on the dataset used. For Q1, computing time is low (< 3 s) and does not increase significantly with the number of samples (Fig  3A ). For Q2, computing time is substantially higher than for Q1 (Fig  3C ), because two tuple variables are defined in Q2, which increases with the number of samples. The number of rules discovered increases with the number of samples and is maximum for 500 and 100 samples for Q1 and Q2, respectively.
Grains were sampled every 3 to 5 days between 10 (i. e. two thirds through the lag phase of grain development) and 34 (i. e. physiological maturity) days after flowering for each of the three main treatments. Grains from the six treatments were also sampled at 0, 6, 9, 12, 18, 24, 48 and 72 h after the shift in N and S supply. Three independent replicates were used and the following measurements were made for each sampling date. Gene expression was monitored using a custom 40k Nimblegen wheat microarray  [46]. Transcriptomic data are available from ArrayExpress repository under the experiment names "N and S regulation of wheat storage protein accumulation" (accessions E-MTAB-1782) and "short-term effect of N and S supply shifts on gene expression in wheat grain" (accessions E-MTAB-1920). The main grain storage protein (GSP) fractions contributing to wheat bread making quality are glutenins and gliadins.
Pearson RNs were generated using a distance matrix computed using Pearson's correlations with a cutoff threshold of 0. 8. In RulNet, we used the Q1 query from the query builder with Confidence and Support thresholds of 0. 90 and 0. 10, respectively. WGCNA RNs were inferred using the R package WGCNA version 1. 34  [51]  with power adjacency set to 9. The three methods show common features as well as topological differences (Fig  4 , Table  1 ). For the three methods, the scaling exponent of the power law distribution is lower than 2 (Table  1 ), indicating a high importance of the hubs in the networks  [19]. The number of nodes is similar for Pearson and RulNet but is 38% lower for WGCNA, while the number of edges is similar for RulNet and WGCNA but is 60% higher for Pearson. The network connectivity assessed by the average number of neighbors (i.
Evaluation of RulNet using the DREAM5 network inference challenge datasets The DREAM5 challenge was a community competition to evaluate network inference methods that predict genome-scale transcriptional regulatory networks from gene-expression microarray datasets. Three standard datasets were used for the comparison taken from a prokaryotic model organism (E. coli), a eukaryotic model organism (S. cerevisiae) and an in silico generated network  [19]. Each of these three datasets comprises a wide range of experimental conditions: genetic perturbations (e. g. gene deletions), drug and environmental perturbations, some microarrays are part of time-series, others are not. Twenty nine teams participated to the DREAM5 challenge and thirty five methods were compared. For each method, participants were asked to provide a ranked list of 100 000 interactions per dataset which was used to assess the performance of the methods.
These metrics were calculated using only the interactions for which experimentally supported interactions exists (gold standard interactions). Network predictions from individual teams were integrated to form community networks by rescoring interactions according to their average rank across all methods  [19]. We benchmarked RulNet using the three datasets and methodology used in the DREAM5 challenge and compared our results with those from the 35 methods that participated in this challenge  [19]. Two queries called QD1 and QD2 were written to take into account the diversity of the experimental conditions (S3 File). QD1 allows the discovery of rules between highly and weakly expressed genes, while QD2 allows the discovery of rules between knockout transcription factors and their putative target genes.
QD2 is more reliable since it is based on TF deletion experiments, however it does not concern all transcription factors but only those for which such experiments are available. For each of these two queries, interactions were ranked by descending leverage and confidence. The best rank obtained with QD1 and QD2 were kept for each interaction, which defines a third method named QD1+2. This new method makes the best out of the two queries QD1 and QD2 that respond to two different questions and will expectedly give better results than each query used alone. The ranked interactions for each dataset and each method are given in Supplementary S1 Dataset to S9 Dataset. QD2 gives higher AUPR scores than QD1. This result is related to the biological meaning of the queries and to the fact that the AUROC reflects the quality of the results obtained throughout the full list of interactions while the AUPR is strongly affected by the performance at the top of the prediction list (i.
Programming knowledge RN inference is an effective tool to predict and analyze RNs using high throughput data. Powerful algorithms and tools have been made available to infer RNs. Unlike many similar tools, programming skills are not required by RulNet users to perform complete network inference, analysis and visualization. Writing simple queries is made easy using the inbuilt query builder, though prior knowledge of the SQL query language would make it easier for users to understand the RQL language for more advanced queries that would exploit all the features in RulNet.
Taking advantage of the complementarity The main advantage of RulNet stems from its adaptability to different biological problems. This is achieved through its capability of performing different queries in the same software framework. In previous comparative analyses (e. g. DREAM5) it has been shown that most approaches used to infer RNs from the same datasets show relatively distinct results  [19]. The differences observed between the inferred RNs does not necessarily demonstrate any poor performance from a particular approach but rather highlights their complementarity. The Rul-Net approach supported the development of multiple queries to discover rules defining RNs in a unifying framework. These rules were used to evaluate RulNet using the DREAM5 datasets. We showed that the evaluation of the different DREAM5 tests could be improved by exploring different queries.
This work was supported in the context of a Ph. D. grant from the  French Ministry for Higher Education and Research  to JV and received funding from the  French Government  managed by the  Research National Agency (ANR)  in the framework of the Investments for the  Future  ( ANR-10-BTBR-03 ),  France AgriMer  and the  French Fund  to support Plant Breeding (FSOV). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
7, 1] interval. This led to 1000 simulated gene trees (G True ) on which the numbers of speciations, duplications, transfers, and losses in average are respectively 29. 75, 3. 87, 0. 88, and 8. 26. The mean number of genes per family in this dataset is 28. 9, the largest family has 67 genes and the smallest 11 genes. Each family is found in 20. 88 species in average, with a minimum of 5 and a maximum of 33 species. These gene trees were used to generate DNA sequences with the Seq-Gen program  [14]. From these sequences, RAxML  [15]  was used to infer 1000 maximum likelihood gene trees (G ML ). For more details, we refer to  [8] ,  [12]  Results section. We used the so-obtained 1000 pairs of dated species tree/gene trees to compare the performances of the two approaches presented in this paper (strategies s3-s5) with the ones presented in  [7, 8]  and  [10]  (strategies s1, s2, s6 and s7 respectively, more details below).
We did not test the method presented in  [9]  as the software is not yet available. For testing the method in  [10] , we used the tool evenscape of the software Xscape (version used in  [10] ). Note that Xscape considers the species tree as undated. Several options to compute event supports are proposed by this program; we tested them all, and present here the ones that give the best accuracy: supports computed using option 'I' , based on the number of regions, without taking into account the area of the regions. A list of the options and their description is given in the Additional file 1. Meaningful cost vectors can be obtained from the real simulated frequencies of each event type as described in  [12,  see Equation  1 of the Experiments on simulated datasets section]: if a real gene history encompasses n tot events (duplications, transfers and losses) among which n e are duplications (respectively transfers and losses), then the cost of a duplication (respectively of a transfers and of a loss) is log(n tot /n e ).
s6 Region-based: This strategy uses the tool eventscape of Xscape, which is an implementation of the method described in  [10]. Since Xscape does not generate reconciliations, this software is used to assign supports to our best median reconciliations -namely the ones of strategy s3 with cost vector (2, 3, 1). We do this instead of using the full list of events returned by Xscape because the full list provides too many events leading to a large number of false positives; moreover, many events in the list cannot happen in a same reconciliation. This choice gives indeed substantially better results than using the Xscape list (data not shown). The input cost range was chosen as the one used in  [10] : λ/δ in [0. 1; 5], and τ/δ in [0. 1; 5]. Hence,we have r m = (0. 02, 0. 2, 0. 1) and r M = (50, 10, 5).
Further, asymmetry indexes were calculated for both datasets. Accuracy of discrimination by a Support Vector Machine (SVM) and the AAL VOIs was tested against a validated method (PALZ). At the voxel level SMP8 showed a relative hypometabolism in the bilateral precuneus, and posterior cingulate, temporo-parietal and frontal cortices. Discriminant analysis classified subjects with an accuracy ranging between. 91 and. 83 as a function of data organization. The best values were obtained from a subset of 6 meta-VOIs plus 6 asymmetry values reaching an area under the ROC curve of. 947, significantly larger than the one obtained by the PALZ score. High accuracy in discriminating MCI converters from healthy controls was reached by a non-linear classifier based on SVM applied on predefined anatomo-functional regions and inter-hemispheric asymmetries.
, 2007; Patterson et al. , 2010)  particularly at the early stages of the disease (i. e. MCI) or when expert readers are not available on site. For this reason, some automated software, either free on the web (such as Statistical Parametric Mapping, SPM, and 3D-Stereotactic Surface Projections, 3D-SSP) or traded on the market (such as the T-sum computation and the PALZ score embedded in PMOD®) been applied to analyze patients3 scans. Such as for T-sum, accuracy may vary between patients with ADdementia and patients with MCI, as it has been shown for MRI  (Chincarini et al. , 2014). Machine learning and pattern recognition algorithms have also been developed to aid in neuroimage analysesfor a review see  Lemm et al. (2011). Recently, using various automated image-based classification methods, efforts have been made to discriminate AD and MCI patients from healthy controls by MRI  (Cuingnet et al.
, 2011) , FDG-PET  (Arbizu et al. , 2013; Gray et al. , 2012; Illan et al. , 2011; Toussaint et al. , 2012) , in a multimodal fashion  (Zhang et al. , 2011)  or implementing univariate and multivariate analyses  (Toussaint et al. , 2012) , cross-sectional and longitudinal information  (Gray et al. , 2012) , image projection onto a feature space  (Illan et al. , 2011)  and atlas-  (Cuingnet et al. , 2011)  and manually-  (Zhang et al. , 2011)  based segmentation methods. Most of these studies were conducted in large cohorts of patients from the Alzheimer3s Disease Neuromaging Initiative (ADNI) database and reached a statistical accuracy in discriminating AD patients from controls of 93% when using three biomarkers (MRI, FDG-PET and CSF)  (Zhang et al. , 2011)  and higher than 90% with FDG-PET only  (Arbizu et al.
, 2012). After template editing, all brain PET scans were processed by affine and nonlinear spatial normalization into the stereotaxic space of the Montreal Neurological Institute through the study-customized FDG template using SPM. The spatially normalized sets of images were then smoothed with an 8-mm isotropic Gaussian filter to blur individual variations in gyral anatomy and to increase the signal-to-noise ratio. Brain PET from MCI-converter patients were compared on a voxel-by-voxel basis to those from the normal controls using a "two-sample t-test" design of SPM8  (Friston et al. , 1994)  implemented in Matlab R2014a (MathWorks, Natick, Massachusetts, USA). The significance threshold was set at p b 0. 05, corrected for multiple comparisons with family wise error (FWE). Only clusters containing more than 100 voxels were considered to be significant.
Region of interest identification and data preprocessing Mean FDG uptake values were computed in 45 anatomical volumes of interest (VOIs) in each hemisphere (90 VOIs) as defined by the AAL Atlas  (Tzourio-Mazoyer et al. , 2002). The dataset to be analyzed was obtained by an in-house created Matlab-based script that automatically processed mean FDG uptake signal intensity from each of the 90 AAL VOIs. The mean signal intensities computed for each VOI were normalized within each subject to the average intensity of the cerebellar VOIs included in AAL as defined by  Schmahmann et al. (1999). This choice was based on the knowledge that the cerebellum is poorly affected by the AD pathological process and on the evidence that, when using the cerebellum instead of whole brain counts as the reference region, accuracy in distinguishing AD patients from controls increases  (Soonawala et al.
, 2002). Next, the number of VOIs was further reduced by merging regions with similar anatomo-functional characteristics into meta-VOIs in order to decrease the number of variables for statistical analysis and to verify if the analysis of specific and functionally meaningful meta-VOIs might contribute to characterize the pathological process. All these calculations were performed by the Matlab script in a single step allowing a substantial spare of time and simplifying the whole process. Twelve meta-VOIs were constructed in each hemisphere: 1. Occipital Cortex (Calcarine/Lingual/Inferior Occipital/Middle Occipital/Superior Occipital Gyri); 2. Thalamus/Putamen/Pallidum/Caudate; 3. Parahippocampal gyrus/Amygdala/ Hippocampus/Insula; 4. Orbito-frontal Cortex (Inferior Frontal/Medial Frontal/Superior-Orbital Frontal Gyri); 5.
Discriminant analysis was performed by a non-linear classifier based on the Support Vector Machine (SVM) method with Radial Basis Functions  (Cortes and Vapnik, 1995). The procedure was independently applied to the different datasets extracted from the same groups of PET images in order to evaluate the influence of data preprocessing and to identify the regions with the highest statistical impact. Each dataset included cerebellumnormalized FDG-PET values relevant to a particular set of VOIs. Since asymmetric metabolism has often been found in pathological condition  (Kovalev et al. , 2006; Rodriguez et al. , 1993; Zahn et al. , 2004) , absolute values of inter-hemispheric asymmetries were also considered and evaluated, for each VOI, as: asy voi ¼ abs nv voi;l -nv voi;r nv voi;l þ nv voi;r where nv voi,x is the normalized FDG value relevant to the right (r) or left (l) hemisphere and abs() stands for the absolute value.
The two basic datasets were made up by the normalized values relevant to the 90 (45 bilateral) AAL regions and by the normalized values relevant to the 24 (12 bilateral) anatomo-functional homogeneous meta-VOIs. Two other datasets were obtained by adding hemispheric values to each of the previous ones: 90 VOIs plus 45 asymmetry values for the AAL regions (135 values) and 24 VOIs plus 12 asymmetry values for the meta-VOIs. This last dataset (36 values) was then the starting point to explore the accuracy of a smaller set of variables by the application of a step-wise backward selection procedure (by removing the less influential variables, one at a time). In order to evaluate our VOI-based classifier with reference to an already validated system, the PMOD software (PMOD Technologies, http://www.
pmod. com) was used for automatic, voxel-based evaluation of scans with the 'Alzheimer' option computing the 'Probability of ALZheimer' or PALZ score. The PALZ score  (Herholz et al. , 2002 ) is a voxel-based parametric mapping method yielding diagnostic information on brain regions that are typically affected in AD: it aims at discriminating AD from healthy controls above 50 years of age. Individual FDG-PET images are compared to a fixed database of normal elderly scans through a voxel-wise t-test, including age as confounding variable. The PALZ score is computed as the sum of t-scores in a pre-defined AD-pattern mask and is compared with a threshold for abnormality drawn from the normal elderly database: PALZ-score higher than threshold (11,089) indicating abnormal FDG-PET. As some centers provided DICOM files and some others provided Analyze files, DICOM files were converted into Analyze format.
Statistics The reliability of the discrimination was evaluated by a series of conventional parameters. For each classification procedure (PALZ-score and VOI-based with different datasets) we evaluated accuracy, sensitivity, specificity, positive and negative likelihood ratios, odds ratio and the area under the Receiver Operating Characteristic (ROC) curve (AUC). All these parameters were evaluated by a cross-validation procedure which entails the separation between the model fitting and testing to enable generalization of results and to prevent overfitting (the adaptation to features peculiar only to the study sample). Cross-validation was implemented by the 'leave-one-out' technique in which each subject was classified by a model fitted to all remaining ones, so creating a virtually-independent testing set with the same size of the original sample.
Parietal Cortex (right), 5. Anterior Cingulate (left), and 6. Posterior Cingulate (left). In the same subset, the most relevant AAL regions for asymmetry values were: 1. Occipital Cortex, 2. Pallidus/Caudate nucleus/Thalamus, 3. Parahippocampal Gyrus/Amygdala/Hippocampus/ Insula, 4 Orbito-frontal Cortex, 5. Frontal Cortex, and 6. Cuneus/ Fusiform/Precuneus. The high rate of correct classification, particularly for the datasets 90R and 6R + 6A, was emphasized by the high positive likelihood ratios (the ratio between the probabilities of positive test in patients with respect to controls) and low negative likelihood ratios (the ratio between the probabilities of negative test in patients with respect to controls) and by the high diagnostic odds ratios (the ratio between the odds of positivity in patients with respect to controls).
The values of these parameters are reported in Table  1  with relevant CIs showing a large overlapping among the different datasets. The same measures (with relevant confidence intervals) are reported for the classification based on the PALZ-score. We first considered the classification based on the standard fixed threshold (PALZ-score = 11,089) which yielded a high specificity (. 92) but a   e1. relatively low sensitivity for MCI-converter patients. A different thresh-(  8116 ) drawn from the point on the ROC curve closer to the optimal one (sensitivity = specificity = 1) yielded a higher sensitivity (. 77) but lower specificity (details in Table  1 ). The ROC curves relevant to the PALZ-score and 3 of the analyzed datasets (90R, 24R and 6R + 6A) are showed in Fig. 2. Pairwise comparisons between the VOI-based procedures and PALZ-score showed that the procedure based on the best fitted meta-regions (6 selected regions + 6 asymmetries) performed significantly better than the PALZscore (greater AUC: z = 2.
15 (29. 17-237. 0) 30. 22 (12. 79-71. 42) 21. 23 (9. 43-47. 81) 42. 30 (16. 84-106. 2) 112. 9 (36. 75-346. 6) 20. 20 (8. 57-47. 64 The classifier was applied to different datasets obtained by different pre-processing of the same set of images. 90R: 90 AAL regions; 24R: 24 anatomo-functional meta-VOIs; 90R + 45A: same as 90R plus inter-hemispheric asymmetries; 24R + 12A: same as 24R plus inter-hemispheric asymmetries; 6R + 6A: 6 meta-regions and 6 inter-hemispheric asymmetries drawn from step-wise backward selection. All parameters, but the area under ROC curve, are dependent on the threshold which was chosen looking for the best point along the ROC curve: considering the PALZ tool the value of the parameter T-sum associated to this point was 8116 (last column) while the standard threshold (implemented in the commercially available software package by PMOD Technologies, Switzerland) is set at 11,090 (second to the last column).
On the other hand, the sensitivity of the PALZ score is based on AD patients reaching 83% in very mild  (Herholz et al. , 2002)  and 85% in mild AD patients  (Herholz et al. , 2011) , but it has not been specifically trained and validated in prodromal AD patients. Besides the high accuracy achieved by our classification method, its potential interest in everyday clinical routine lies in the implementation of a readily available anatomical atlas and in the use of an in-house Matlab-based script able to assess and sort fully automatically in a few minutes the uptake values in more than hundred segmented VOIs and meta-VOIs making them available for further statistical analyses. The processing time and the use of the script are easier than similar VOI and meta-VOI analyses in which computation time is longer and automation procedures are more complex  (Caroli et al.
Indeed, in the present study only subjects (controls and patients) in the last decades were included. This may have accounted for the lack of a statistically significant effect. A limitation of the study is that MCI-converters were discriminated only respect to healthy controls. Further studies are needed to test if the capability of the used automatic classifier to identify MCI patients about 2 years before conversion will also apply to cohorts of MCInon-converting to AD or to patients bearer of other confounding pathological conditions. In this respect to properly select the latter groups for statistical analyses a follow-up time as long as 7 years is required  (Bennett et al. , 2002)  and these analyses will be the next step to validate the present methodology. A further limitation of this study relies on the fact that we did not evaluate the possible confounding effect of apolipoprotein E genotype, as these data were available for just some of the patients.