dataesr/scientific-paragraphs-categorization huggingface dataset:
Some basic information is needed to start the program such as: input dataset(s), class label name, problem type (regression or classification). BioDiscML pipeline presented in Figure  1  works as follows: It starts with the preprocessing section. After merging the input datasets when many are submitted, a first sampling step separates the data in a train and a test set (2/3 and 1/3, respectively, by default), this latter will be used after model creation to assess nonoverfitting. Then, a feature ranking algorithm sorts the features based on their predictive power with respect to the class. Only the first best 1,000 s features are kept by default. Then, in the feature selection section, for each machine learning algorithm defined in BioDiscML (i. e. , the classifiers), and for each optimization evaluation criterion (i.
e. , a chosen evaluation metric), two types {d 1 , d 2 ,. , d q }with q ≥ 1containing m q features. In each dataset the first column is used to create the joining of all datasets and consists of instances unique identifiers. If an identifier does not exist in all datasets, it will be ignored. The class label column Y is required and must be specified by the user. In addition to the class label, the dataset d 1 contains a set of m 1 features noted A 1 = {A 1,1 ,A 1,2 ,. , A 1,m 1 }where A 1,m 1 ,the m 1 -th feature of d 1 , is a vector denoted {a 1,m 1 ,1 ,a 1,m 1 ,2 ,. , a 1,m 1 ,n }. Hence the feature vector of the n-th instance of the dataset d 1 is noted x 1,n = {y n , a 1,1,n ,a 1,2,n ,. , a 1,m 1 ,n }. In case of multiple datasets (q ≥ 2), the feature vector of the n-th instance of the dataset d r is noted then x r,n = {a r,1,n ,a r,2,n ,.
, a r,m r ,n }, where m r is its number of features. The resulting set of merged datasets is called D. Due to experimental errors or partially answered forms by patients, missing data may be present in the dataset. If one wants to conserve the features with missing data, the ML library used by BioDiscML will replace all missing values for nominal and numeric features with the modes (i. e. , value that occurs most often) and means from the training data, respectively. Also, manipulating large files is painful and one would exclude specific features without editing the input files. Thus, we implemented in BioDiscML features exclusion capabilities, where it simply ignores columns entered by the user. Finally, a stratified sampling, which preserve the initial classes balancing, is applied to generate a test set for further evaluation to assess non-overfitting.
Both Information Gain and ReliefF are used in conjunction with a ranker search algorithm, which ranks features by their individual evaluations. By default, and to reduce the dimension of the dataset, BioDiscML will only keep informative features (Information Gain >0. 01 or |ReliefF| >0. 01) or the first 1,000 best features, ordered by their absolute value of their score (ReliefF provides positive and negative correlation scoring with continuous class) (see Algorithm 1).
Optimal feature subset search methods For each ML algorithm listed in Supplementary Table  S1 , and for each selected criteria selected in Supplementary Table  S2 , from the ranked features S obtained in Algorithm 1, models are trained using several feature search approaches, including: Forward stepwise selection (FSS), Backward stepwise selection (BSS), Forward stepwise selection and Backward stepwise elimination (FSSBSE), Backward stepwise selection and Forward stepwise elimination (BSSFSE), and "top k" features. In the stepwise procedures, features having an equal predictive power to the outcome (i. e. , distributions similar among classes) and retained in the model may be selected randomly or by order of appearance in the dataset. Forward stepwise selection (FSS). Also called sequential forward selection  (Reunanen, 2003) , where features are added one by one to the model.
Average of metric scores are computed over all tested instances. The holdout method is the simplest kind of cross validation where the dataset is randomly separated into two sets generated at sampling procedure (see Figure  1 ), called the training set and the testing set. The model is trained using the training set only, then is used to predict the class for the data in the testing set as evaluation. However, this type of evaluation can have a high variance since it depends heavily on which instances end up in the training and test sets. Thus, a repeated holdout is also performed 100 times (by default) with random sampling without replacement. Repeated Holdout consists of randomly select and hold out a 1/3 of the training sample for testing, build model with only the remaining samples, retrieve its performances, and repeat the process many times.
Otherwise, by default, BioDiscML best model selection procedure aims to identify the model having a high agreement between the various evaluation methods, hence assessing stability and low overfitting of the model. To this purpose, select the model having the best average MCC with a standard deviation lower than 0. 1 (or another adjusted threshold set by the user). The user can change the best model selection strategy at ease in the program configuration file. For example, one would select a trained model on train set having the best.
Correlated Features Search The identified signatures by stepwise search methods will tend to ignore all redundant/correlated features. To use the models as "black box" for pure prediction, this may be optimal, but not for biological interpretation because one would understand why the selected features have a link with the predicted class. To this purpose, from the features in the signature, BioDiscML retrieves all other correlated features from the original dataset using Pearson and Spearman correlations. BioDiscML also identifies all neighbor features discovered during feature ranking procedure by Information Gain and ReliefF methods. Both provide feature ranking scores that are used to detect the features having the same predictive power, i. e. , similar behavior among instances. With these techniques, redundant information lost during the feature selection process are recovered, hence helping for further interpretation of the signature.
RESULTS We compared BioDiscML to various recent approaches dedicated to biomarker discovery and modeling, including MINT  (Rohart et al. , 2017a) , AucPR  (Yu and Park, 2014) , and RGIFE  (Swan et al. , 2015)  to demonstrate the better predictive performances that BioDiscML offers on various omics datasets. In all cases, BioDiscML outperform these state-of-the-art tools.
BioDiscML vs. Mint MINT implements a multivariate integrative method able to integrate independent datasets, reduce batch effect, classify instances and identify key discriminant variables. In their study, they performed a feature selection and classification evaluation of a stem cell dataset. According to their published results, they identified a signature of 17 genes which predicted the test and train sets with a BER of 9. 4 and 7. 1% resp. Using the exact same train set, BioDiscML identified a signature of 19 genes by optimizing the AUC of a Random Forest model with 100 iterations and using the FSSBSE feature search method. The measured BER on the test set was 7%, and on the train set 3. 5, 3. 6, 6. 8, and 7. 2% using 10 CV, LOOCV, and repeated holdout and bootstraping resp. To select this model among the 4,710 successfully generated models, we simply retrieved the one having the lowest BER on the holdout method.
Using this long signature, only immune system process was added compared to the short signature, which also exists in the MINT signature. Moreover, this long signature provided perfect predictions on all instances of the test set. We also compared both signatures GSEA (see Methods). MINT signature did not show any significantly enriched ontologies, literature co-citation, co-expression etc. At the opposite, the short signature of BioDiscML found about 20 hits related to stem cells in co-expression databases (GeneSigDB and MSigDB) and co-expression Atlas. Also, about 20 other hits were found in literature co-citation about cognitive diseases (Alzheimer, Parkinson, Schizophrenia). The long signature provided even more hits, in many other categories.
BioDiscML vs. AucPR In their study, authors of AucPR, an AUC-based approach using penalized regression, have evaluated the performance of their tool against four datasets. While AucPR showed a very good prediction performance on three of four tested datasets, the average AUC on ColonCA dataset was about 90% using both best penalization regression approach modes of the tool (Lasso and ElasticNet). Considering AucPR had the lowest performance on this dataset, we tried the performance of BioDiscML on it. In their paper, authors report the boxplots of 100 AUCs obtained by repeated holdout (random separation of 2/3 of the data for training and the remaining for testing) without sampling step. Using the same data and same evaluation method without sampling before training, two models identified by BioDiscML, on the 3,967 successfully generated models, shared the same best average AUC score.
We chose the one having the best MCC on repeated holdout, a model based on a Hoeffding Tree (parameters: infogain split, Naive Bayes adaptive leaf prediction strategy, grace period of 200, tie threshold of 0. 05) optimized by AUC. This model provided an average AUC of 99. 3% (0. 632+ rule at 0. 047) using 10 genes discovered by FSSBSE. This is an improvement of AUC of about 11%. Both AucPR modes AucL and AucEN selected in comparison 30 and 22 genes resp. The benchmark comparison of AUCs is reported in Figure  4. The model identified by BioDiscML has a much better performance in terms of average AUC and variance over bootstrapping. GSEA was not performed since this dataset didn't provided gene identifiers.
BioDiscML vs. RGIFE RGIFE is an heuristic method intending to identify reduced panels of biomarkers with highly predictive performance. It first ranks features by their contribution to the generated models, and dynamically removes blocks of features. It also introduces a concept called soft-fail, which considers an iteration successful despite a performance drop within a tolerance level and specific circumstances. We evaluated the performance of BioDiscML on three datasets tested in RGIFE, including Central Nervous System (CNS), DLBCL, and Prostate Cancer datasets. On the 10 tested datasets by RGIFE, the three selected datasets showed accuracies around 60-70% for 10 CV, while BioDiscML identified models and signatures providing prediction performance close to perfection (100% accuracy) with lower number of features.
Performances are reported in Table  2 , where, for each dataset, we identified two models found by BioDiscML. To provide a fair comparison with the RGIFE manuscript we selected models having the best 10 CV accuracy (with best bootstrapping accuracy and lowest number of features in case of models' performance equality), which ended with 100% accurate models. But since this typical measure approach tends to be over-optimistic on the real performance of the models and because overfitting was suspected, we also reported models having the best bootstrapping accuracy. Obtained models show accuracies between 10CV and Bootstrapping more consistent, hence showing models are stable. In any case, 10CV accuracy was always better with BioDiscML results. The two signatures found for CNS dataset presented an overlap of five genes, and a merged list of the signatures show several GSEA significant hits related medulloblastoma and other cancers.
For BLBCL dataset, no genes overlapped the two signatures, and we found significant hits related to dehydrogenase activity in the GSEA analysis on the merged list of the signatures, which has a link with follicular lymphoma to diffuse large B-cell lymphoma  (Montoto et al. , 2007). Finally, the prostate cancer signatures showed no overlap either, but GSEA analysis on the merge lists show several hits related to this cancer. In terms of computing performances, on a same server containing four Intel(R) Xeon(R) CPU E5-2695 v2 @ 2. 40GHz (48 threads), BioDiscML runtime was 28, 387, and 393 min on CNS, DLBCL, and Prostate Cancer datasets resp. , and generated 5,751, 6,479, and 6,408 models resp. , without exceeding 16 GB memory usage. In comparison, computation time reported by RGIFE in their Supplementary Data show ranges about 180-400 min.
DISCUSSION A Simplified but Customizable Automated ML Tool BioDiscML tool has been developed to enhance biomarker discovery using an exhaustive ML approach and propose automation of ML steps to perform such task. A large variety of algorithms is available and combinations of strategies are countless if we consider the hyperparameters of all classifiers and feature selection algorithms. This complexity is a barrier to non-expert users attempting to use ML to analyze their data. Thus, we designed BioDiscML to simplify ML steps without penalizing the performance, such as using fast and optimal feature ranking algorithms and feature search methods, limit the number of features after feature ranking, and establish predefined classifiers hyperparameters to reduce computing time. Although editable in BioDiscML configuration file, these intentional limitations provide researchers a program that generate results without intervention within a few hours of calculation on a recent computer.
A Sampling Procedure to Avoid Overtfitted Models BioDiscML implements a sampling step to assess the nonoverfitting and the good performance of identified models and signatures, where it splits the dataset into two stratified (class balancing is preserved) random parts. The program also accepts a second input file as a test dataset, as long as it is in the same format as the train set. In case of very limited instances, it is possible to skip the sampling operation, although not recommended because of the risk to not detect overfitted models. A reasonable number of instances (i. e. , samples) should be provided to BioDiscML, else it is expected to obtain models with low performances. For example, we estimate that a highly heterogeneous dataset, such as prostate or breast cancer data, should contain at least half-hundred patients per class, while a dataset based on a study involving cloned living species could be limited to half a dozen individuals per class.
BioDiscML Uses All Available Classifiers From a Widely Accepted and Efficient ML Library There is a plethora of ML algorithms specialized in classification (i. e. , categorical class) and regression (i. e. , continuous class). BioDiscML covers many of them but can also be manually limited to the most known and widely applied in biomedical research for the development of predictive models such as Random Forest, Decision Trees, Rules, Naive Bayes, Artificial Neural Networks, Bayesian Networks and Support Vector Machines. They all resulted in effective and accurate decision-making  (Jagga and Gupta, 2015). But the final models created with these classifiers in various studies were all delivered after an exhaustive search work. BioDiscML aims to reduce this search time by providing the models adapted to user datasets.
All ML algorithms are provided by an advanced freely available ML library toolkit, called Weka. Besides this library, various ML libraries exist, such as SciKit-Learn  (Nelli, 2015)  (written in Python) and packages in R  (Lesmeister, 2017). BioDiscML implements Weka library for various reasons, including its wide usage in computational biology  (Gewehr et al. , 2007; Bendl et al. , 2014; Bernardi et al. , 2015; Arganda-Carreras et al. , 2017; Chicco, 2017; Alves et al. , 2018) , its high citation rate (at August 2018) and its highly versatile object-oriented language JAVA (e. g. , easy to parallelize, multi-platform compatibility, GUI integration, generally already installed on clients, etc. ), which is much faster  (Fourment and Gillings, 2008)   (Sutter and Kalivas, 1993)  for variables selection since it would be computationally expensive here.
Signature Interpretation Is Still a Challenge A biologist will want to interpret and validate in silico the signature, since there is an obvious relation between the identified biomarkers in a signature and the predicted class (e. g. , outcome). To perform such task, there exist many Gene Set Enrichment Analysis (GSEA) tools, such as ToppGene suite  (Chen et al. , 2009)  or Enrichr  (Kuleshov et al. , 2016). These GSEA tools will provide a characterization of signature and confirm to the biologist if the signature has a biological meaning with the original study from which the dataset have been generated. Some more extensive literature searches may provide more insights and help linking the signatures' features with the predicted class. Moreover, in some cases, the biologist, based on its experience and knowledge, may not find the biomarkers he expects in the signatures.
However, to overcome this lack, BioDiscML generates a subset of the input dataset containing only the sample values of the signature' features. This subset in comma-separated values format can be loaded easily in other visualization software such as Microsoft Excel, Orange  (Demšar et al. , 2013) , RapidMiner  (Hofmann, 2016) , or R  (Gardener, 2012)  to generate heatmaps or boxplots.
BioDiscML Exhaustive Approach Outperforms Recently Published Tools We benchmarked BioDiscML against recent tools proposing different approaches to discover biomarker signatures. Benchmarks showed that BioDiscML outperforms these stateof-art methods using same datasets. Because of its exhaustive approach, it was able to identify one or more models with smaller signatures providing much better prediction performances. We also demonstrated in the case of the stem cell dataset that BioDiscML signature contained different genes but similar ontologies than the MINT signature, with a better prediction performance. A GSEA also showed that the BioDiscML signature had much more biological evidence, denoted by the occurrence of stem cells topics in the co-expression databases. The genes in the BioDiscML signatures were also present in neurodegenerative diseases, highlighting the link of these genes with the neuronal system, supported by evidence of efficient stem cell-based therapies for neural repair  (Volkman and Offen, 2017).
For the other benchmarked datasets which contained gene references, the GSEA analyses also showed supporting evidences assessing the biological relation between the genes found in the signatures and the biological experiment from where they were produced. It is important to note that short but still very predictive model' signatures can be extended as an "enriched" signature which include the correlated genes. These enriched signatures may increase the accuracy of the signature, but more importantly they can help to better understand the biological meaning of the model. On the MINT dataset, BioDiscML showed a perfect prediction on the test set with the enriched signature and retrieved more ontologies. Finally, in this paper we benchmarked BioDiscML only on transcriptomics datasets from microarray data provided by the tools we tested.
Performant Models Identified in Minutes BioDiscML computing performances are highly dependent on the size of the input dataset and the available processors. To generate all models implemented in the software, it requires a few hours of computation. However, it is possible to restrict BioDiscML to a specific list of algorithms, hence reducing the computation time to seconds or minutes. It is also possible to extract the best signatures and models produced since the beginning of BioDiscML execution at any time. We have prioritized the training of the most common and fastest classifiers to propose a large number of computed models shortly after starting BioDiscML. More complex models, such as Multilayer perceptrons, are set in low priority. More running time will simply increase the probability to obtain a better model.
CONCLUSIONS This paper introduces BioDiscML, dedicated to identify optimal combination of biomarkers (i. e. , features) and machine learning models to predict measured outcomes. It provides a userfriendly and powerful solution to researchers in the medical field looking to identify predictive features, essential to the development of personalized medicine approaches and research of new therapeutic targets. This software has the benefit to exploit a large number of machine learning classifiers within a fully automated process combined with data pre-processing, hence facilitating the work of a non-machine learning experts audience. Expert users have also the possibility to configure advanced options. BioDiscML is a great opportunity to reduce biomarkers search time, by revealing the most adapted classifiers to a given dataset and even proposes new algorithms poorly explored in the literature that could have a great potential to classify biological data.
Dimension reduction by Information Gain and ReliefF Input: train instances of D (merged datasets), classifierType (classification or regression) Output: Dataset with ranked best features S for each feature array A do if classifierType = classification then meritScore a = Compute information gain value of A with respect to classes Y else meritScore a = Compute ReliefF value of a with respect to classes Y end if if meritScore a = 0 then add | meritScore a |to meritScores end if end for SortedFeatures = Sort meritScores from largest to smallest values.
BioDiscML accepts as input one ({d 1 } only) or many ({d 1 ,. ,d q }) symbol-separated table-like structured datasets containing samples in row and features in columns. Start Input Data Sampling Train set Feature ranking 1000 best features Test set Set of ML For each For each For each classifiers * classifier criterion ** feature Feature selection methods evaluated by Top k FSS BSS FSSBSE BSSFSE cross validation*** FIGURE 2 | Features-optimized models Evaluate models using 10CV, LOOCV, Holdout, Repeated Holdout and Best model selection Compute correlated features Full report and output files Bootstrapping Root Mean Squared Error, etc. (see Evaluation Criterion). ***Feature selection methods include forward stepwise selection (FSS), backward stepwise selection (BSS), forward stepwise selection and backward stepwise elimination (FSSBSE), backward stepwise selection, and Forward stepwise elimination (BSSFSE), and "top k" features (see Optimal Feature Subset Search Methods).
of feature search selection are performed: top k features and Correlated Features Search) and compiled in a tabular-separated stepwise (see Optimal Feature Subset Search Methods). Top k text file report. Depending computing performances and dataset simply select the best k elements from the ordered feature set size, a few hours may be needed for BioDiscML pipeline to finish. to create a model. In the stepwise approaches, for each element Before the end of BioDiscML execution, a user can execute at any in the ordered set, features are added and/or removed one by time BioDiscML from the checkpoint in parallel to perform the one depending on the feature search method. At each iteration, best model selection process, which will retrieve models from the the created model is evaluated by 10-fold cross validation (10 feature-optimized model list generated and updated in real-time.