dataesr/scientific-paragraphs-categorization huggingface dataset:
Recall is the ratio of mutants correctly predicted as subsuming among all the subsuming mutants. F-measure indicates the weighted harmonic mean of Precision and Recall. Matthews Correlation 11. statementComplexity, expressionComplexity, MutantType, BlockDepth, CfgDepth, CfgPredNum, CfgSuccNum, NumInBlock, NumOutDataDeps, NumInDataDeps, NumOutCtrlDeps, NumInCtrl-Deps, AstNodeParentType, NumberOfAstParents, AstNodeType, NumberOfAstChildren Coefficient (MCC)  [40]  is a reliable metric of the quality of prediction models  [55] , that in contrast to the previous metrics, also takes into account the True Negatives (correctly predicted non-subsuming mutants). It is generally regarded as a balanced measure that can be used even when the dataset is unbalanced, i. e. , the classes are of very different sizes, e.
THREATS TO VALIDITY External Validity: Threats may relate to the subjects we used. Although our evaluation expands to both C and Java projects  of different sizes, the results may not generalize to other projects or programming languages. We consider this threat as low since we have a large sample of programs, i. e. , we perform one of the largest mutation testing studies to date. Other external threat lies in the operators we used, since our prediction approach might not work for different types of mutants. To reduce this threat, we employ modern mutation tools, for both C and Java that implement a large variety of mutation operators. For the C-Benchmark, taken from  [12] , 816 simple operators across 18 categories were considered; while for creating our Java-Benchmark, we consider the group "ALL" of mutation operators provided by Pitest  [17] , resulting in 112 simple operators across 29 categories.
e. , on average, 2. 15% of the mutants do not get a valid label (4. 2% in C and 0. 1% in Java). Threats may also relate to the features we implemented for training the Decision Trees baseline. We follow the guidelines provided in  [11] , to extract the 16 features for our Java dataset. Unfortunately, many of the 28 features for C programs presented in  [11]  depend on the semantic of the C language, that we found unfeasible to be replicated in Java. However, the prediction performance of Decision Trees in Java are in line with the results obtained for C, indicating that the impact of this threat is low. Other internal validity threats could be related to the test suites we used and the mutants considered as subsuming and equivalent. To deal with this issue, we used well-tested programs and state-of-the-art tool to generate extensive pools of tests (KLEE  [10] , SEMu  [12] , and EvoSuite  [23] ).
The approach relies on a set of both static and dynamic features (relying on coverage and code attributes) and achieves relatively good results (on average with 10% error). Though, PMT mainly targets intra-project predictions, while Cerebro targets inter-project. Nevertheless, PMT is incomparable to Cerebro since it aims at evaluating test execution results, while we do mutant selection prior to any test execution. In other words, we aim at identifying the mutants to be used for test design/generation, while PMT to verify whether mutants are killed by some tests. Therefore, the two methods target different but complementary problems. Evolutionary Mutation Testing (EMT)  [21]  utilises dynamic features (execution traces) in order to identify interesting locations and mutant types. As such, EMT requires tests and user feedback, which make it different but complementary to ours; Cerebro can set a starting point for EMT or integrate its predictions within EMT's fitness function.
Recently, it has become increasingly common to pre-train the entire model on a data-rich task, which causes the model to develop general-purpose abilities and knowledge that can then be transferred to downstream tasks  [52]. In this practice aka Transfer Learning and its applications to computer vision  [44] ,  [63] , pre-training is typically done via supervised learning on a large labeled data set like Image-Net  [53]. In contrast, modern techniques for transfer learning in Natural Language Processing (NLP) often pre-train using unsupervised learning on unlabeled data  [20] ,  [36]. The resulting pre-trained models are further trained on specialized datasets to accomplish the desired tasks. Unsupervised pre-training for NLP is attractive and seems a good fit for neural networks as it have been shown to exhibit remarkable scalability, i.
e. , it is often possible to achieve better performance simply by training a larger model on a larger data set  [26] ,  [28] ,  [37] ,  [54]. It will be worthwhile to explore such available pre-trained models  [22] ,  [39]  and if these can be further refined to address our specific prediction task. On the other hand, as we have shown that Cerebro is proficient in capturing the silent features and patterns of the code context, it is promising to explore Cerebro in securityspecific task such as prediction of zero-day vulnerabilities, which pose a very high risk  [66]. Vulnerabilities are fewer in comparison to defects, limiting the information one can learn from. Also, their identification requires an attacker's mindset  [42] , which developers or code reviewers may not possess. Lastly, the continuous growth of codebases makes it difficult to investigate them entirely and track all code changes.
com/apache/ 26a308f commons-text. git commons-csv https://github. com/apache/ 865872e commons-csv. git commons- https://github. com/apache/ 2c0429a lang commons-lang. git commons-io https://github. com/apache/ c126bdd commons-io. git commons-net https://github. com/apache/ 33df028 commons-net. git commons- https://github. com/apache/ 475910a codec commons-codec. git jsoup https://github. com/jhy/jsoup. git 528ba55 joda-time https://github. com/JodaOrg/joda- 767c94e time. git 7. https://commons. apache. org 8. https://github. com/JodaOrg/joda-time/ 9. https://github. com/jhy/jsoup 10. LINE:BRANCH:MUTATION:OUTPUT:METHOD:CBRANCH.
High-content omic technologies coupled with sparsity-promoting regularization methods (SRM) have transformed the biomarker discovery process. However, the translation of computational results into a clinical use-case scenario remains challenging. A rate-limiting step is the rigorous selection of reliable biomarker candidates among a host of biological features included in multivariate models. We propose Stabl, a machine learning framework that unifies the biomarker discovery process with multivariate predictive modeling of clinical outcomes by selecting a sparse and reliable set of biomarkers. Evaluation of Stabl on synthetic datasets and four independent clinical studies demonstrates improved biomarker sparsity and reliability compared to commonly used SRMs at similar predictive performance.
Introduction High-content omic technologies, such as transcriptomics, metabolomics, or cytometric immunoassays, are increasingly employed in biomarker discovery studies. 1, 2  The ability to measure thousands of molecular features in each biological specimen provides unprecedented opportunities for development of precision medicine tools across the spectrum of health and disease. Omic technologies have also dictated a shift in statistical analysis of biological data. The traditional univariate statistical framework is maladapted to large omic datasets characterized by a high number of molecular features 𝑝 relative to the number of samples 𝑛. The 𝑝 ≫ 𝑛 scenario drastically reduces the statistical power of univariate analyses, a problem that cannot be easily overcome by increasing the value of 𝑛 due to cost or sample availability constraints.
3, 4  Statistical analysis in biomarker discovery research comprises three related yet distinct tasks, all of which are necessary for translation into clinical use and impacted by the 𝑝 ≫ 𝑛 problem: 1) prediction of the clinical endpoint via identification of a multivariate model with high predictive performance (predictivity), 2) selection of a limited number of features as candidate clinical biomarkers (sparsity), and 3) confidence that the selected features are among the true set of features (i. e. , truly related to the outcome, reliability). Several machine-learning methods, including sparsity-promoting regularization methods (SRMs), such as least absolute shrinkage and selection operator (Lasso)  5  or elastic net (EN),  6  provide predictive modeling frameworks adapted to 𝑝 ≫ 𝑛 omic datasets, but the selection of a sparse and reliable set of candidate biomarkers remains an important challenge.
12, 13  However, in these methods, the threshold for feature selection or the target false discovery rate (FDR) are defined a priori, which uncouples the feature selection from the multivariate modeling process. Without prior knowledge on the data, these methods can lead to suboptimal feature selection, requiring multiple iterations to identify a desired threshold. This limitation also precludes optimal integration of multiple omic datasets into a unique predictive model, as a single fixed selection threshold may not be suited to the specificities of each dataset. Here we introduce Stabl, a supervised machine learning framework that bridges the gap between multivariate predictive modeling of high-dimensional omic data and the sparsity and reliability requirements of an effective biomarker discovery process.
Stabl combines the injection of knockoffmodeled noise or random permutations into the original data, a data-driven signal-to-noise threshold, and integration of selected features into a predictive model. Systematic benchmarking of Stabl against Lasso, EN, and SS using synthetic datasets, three existing real-world omic datasets, and a newly generated multi-omic clinical dataset demonstrates that Stabl overcomes the shortcomings of state-of-the-art SRMs: Stabl yields highly reliable and sparse predictive models while identifying biologically plausible features amenable to further development into diagnostic or prognostic precision medicine assays. The complete package for Stabl is available online at https://github. com/gregbellan/Stabl.
Selection of reliable predictive features using estimated false discovery proportion (FDP) When applied to a single cohort drawn at random from the population, SRMs will select informative features (i. e. , truly related to the outcome) with a higher probability, on average, than uninformative features (i. e. , unrelated to the outcome). 5, 12  However, as uninformative features typically outnumber informative features in high-dimensional omic datasets,  1, 2, 11  the fit of an SRM model on a single cohort can lead to selection of many uninformative features despite a low probability of selection. 12, 14  To address this issue, Stabl implements the following strategy (Fig. 1  and methods): 1. Stabl fits SRM models (e. g. , Lasso or EN) on subsamples of the data using a procedure similar to SS.
12  Subsampling mimics the availability of multiple random cohorts and estimates each feature's frequency of selection across all iterations. However, this procedure does not provide an optimal frequency threshold to discriminate between informative and uninformative features objectively. 2. To define the optimal frequency threshold, Stabl creates artificial features unrelated to the outcome (noise injection) via random permutations  [1] [2] [3]  or knockoff sampling,  13, 15, 16  which we assume behave similarly to uninformative features in the original dataset  17  (see theoretical guarantees in methods). The artificial features are used to construct a surrogate of the false discovery proportion (FDP+). We define the "reliability threshold", 𝜃, as the frequency threshold yielding the minimum FDP+ across all possible thresholds.
This method for determining 𝜃 is objective, in that it minimizes a proxy for the FDP. It is also data-driven, as it is tailored to individual omic datasets. As a result, Stabl provides a unifying procedure that selects features above the reliability threshold while building a multivariate predictive model. Stabl is amenable to classification and regression tasks and extends to integration of multiple datasets of different dimensions and from different omic modalities.
Stabl improves sparsity and reliability while maintaining predictivity: synthetic modeling We benchmarked Stabl against Lasso and EN using synthetically generated training and validation datasets containing known informative and uninformative features (Fig. 2a ). Simulations representative of real-world scenarios were performed, including variations in the sample size (n), total features (p), and informative features (S). Models were evaluated using three performance metrics (Fig. 2b ): 1. Sparsity: the average number of features selected compared to the number of informative features. 2. Reliability: overlap between the features selected by the algorithm and the true set of informative features (Jaccard Index).
Predictivity: mean square error (MSE). Before performing benchmark comparisons, we tested whether the FDP+ defined by Stabl experimentally controls the FDR at the reliability threshold 𝜃, as the true value of the FDR is known for the synthetic dataset. We observed that FDP+ (𝜃) was indeed greater than the true FDR value (Fig. 2c  and S1 ). These observations experimentally confirmed the validity of Stabl in optimizing the frequency threshold for feature selection. Furthermore, under the assumption that the uninformative features and the artificial features are interchangeable, we bound the probability that FDP exceeds a multiple of the proximity to FDP+ (𝜃), thus providing a theoretical validation of our experimental observations (see theroretical guarantee in methods). Stabl was tested using a random permutation method (Fig.
2g -i and S7-9). The performance of models built using a fixed frequency threshold varied greatly depending on the simulation conditions. For example, for a small sample size (n < 75), the 30% threshold had the best sparsity and reliability. However, for a large sample size (n > 500), the 80% threshold resulted in greater performances. In contrast, Stabl models systematically reached optimal sparsity, reliability, and predictivity performances. Further, we show that 𝜃 varied greatly with the sample size (Fig. 2j  and S10 ), illustrating how Stabl adapts to datasets of different dimensions to identify an optimal frequency threshold solution. In sum, synthetic modeling results show that Stabl achieves better sparsity and reliability compared to Lasso or EN while preserving predictivity and that the set of features chosen by Stabl is closer to the true set of informative features.
Stabl enables effective biomarker discovery in clinical omic studies We evaluated Stabl's performance on four independent clinical omic datasets. Three were previously published with standard SRM analyses, while the fourth is a newly generated dataset introduced and analyzed for the first time here. Because clinical omic datasets can vary greatly with respect to dimensionality, signal-to-noise ratio, and technology-specific data preprocessing, we tested Stabl on datasets representing a range of bulk and single-cell omics technologies, including RNA sequencing (RNA-Seq), high-content proteomics (SomaLogic and Olink platforms), untargeted metabolomics, and single-cell mass cytometry. For each dataset, Stabl was compared to Lasso and EN on single-omic data or to early fusion and late fusion on multi-omic data over 50 random repetitions using a repeated five-fold cross-validation (CV) strategy.
As the true set of informative features is not known for real-world datasets, the performance metrics differed from those used for the synthetic datasets: 1. Sparsity: determined by the average number of features selected throughout the CV procedure. 2. Reliability: assessed using univariate statistics in the absence of a known true set of features. 3. Predictivity: the area under the receiver operator characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC) for classification tasks or the MSE for regression tasks.
Identification of sparse, reliable, and predictive candidate biomarkers from single-omic clinical datasets Stabl was first applied to two single-omic clinical datasets featuring a robust biological signal with significant diagnostic potential. The first example is a large-scale plasma cell-free RNA dataset (p = 37,184 cfRNA features) isolated from pregnant patients with the aim of classifying normotensive or preeclamptic (PE) pregnancies (Fig. 3a, b ). 18, 19  The second example is a high-plex proteomic dataset (p = 1,463 proteomic features, Olink) collected from two independent cohorts (a training and a validation cohort) of SARS-CoV-2-positive patients to classify COVID-19 disease severity (Fig. 3c, d ). 20, 21  In these two examples, although both Lasso and EN models achieved very good predictive performance (AUROC > 0.
80, Fig. 3 , S11-12), the lack of sparsity or reliability hindered the identification of a manageable number of candidate biomarkers, necessitating additional feature selection methods that were decoupled from the predictive modeling process. [18] [19] [20] [21]  Consistent with the results obtained using synthetic data, Stabl achieved comparable predictivity to Lasso (Fig. 3e, f ) and EN (Fig. S11a, b ) when applied to the single-omic datasets. However, Stabl identified sparser models. For the PE dataset, the average number of features selected by Stabl was reduced over 20-fold compared to Lasso (Fig. 3g ) or EN (Fig. S11c ) respectively. For the classification of patients with mild or severe COVID-19, the number of features selected by Stabl was reduced by a factor of 2. 7 compared to Lasso (Fig.
3h ) and 4. 5 compared to EN (Fig. S11d ). Stabl's reliability performance was also improved compared to Lasso and EN. The univariate p-values (Mann-Whitney test) for the features selected by Stabl were lower than for those selected by Lasso (Fig. 3i, j ) or EN (Fig. S11e, f ). Independent evaluation of the COVID-19 validation dataset confirmed these results (Table  S1 ): 100% of features selected by Stabl passed a 5% FDR threshold (Benjamini-Hochberg correction) on the COVID-19 validation dataset (mean -log[p-value] = 9. 0), compared to 91% for Lasso (mean -log[p-value] = 6. 7, Fig. 3k ) and 85% for EN (mean -log[p-value] = 6. 2, Fig. S11g ). Stabl was also compared to SS using 30%, 50%, and 80% fixed frequency thresholds (Table  S2 ). Consistent with the synthetic modeling analyses, the predictivity and sparsity performances of SS varied greatly with the choice of threshold, while Stabl provided a solution that optimized sparsity while maintaining predictive performance.
For example, using SS with a 30% compared to a 50% threshold resulted in a 42% decrease in predictivity for the COVID-19 dataset (AUROC30% = 0. 85 vs. AUROC50% = 0. 49), with a model selecting no features. Conversely, for the PE dataset, fixing the frequency threshold at 30% vs. 50% resulted in a 5. 3 fold improvement in sparsity with only a 6% decrease in predictivity (AUROC30% = 0. 83 vs. AUROC50% = 0. 78). Identification of fewer and more reliable features using Stabl facilitated the biomarker discovery process, pinpointing the most informative biological features associated with the clinical outcome. For example, three out of thirteen (23%) cfRNA features (CDK10,  22   S3 ). For the COVID-19 dataset, several features identified by Stabl echoed key pathobiological mechanisms of the host inflammatory response to COVID-19.
For example, CCL20 is a known element of the COVID-19 cytokine storm,  25, 26  CRTAC1 is a newly identified marker of lung function,  [27] [28] [29]  PON3 is a known biomarker decreased during acute COVID-19 infection,  30  and MZB1 is a protein associated with high neutralization antibody titers after COVID-19 infection (Fig. 3j ). 20  The Stabl model also selected MDGA1, a previously unknown biomarker candidate of COVID-19 severity (Table  S4 ). Together, the results show that Stabl improves the reliability and sparsity of biomarker discovery in two single-omic datasets of widely different dimensionality while maintaining predictivity performance.
Stabl successfully extends to multi-omic data integration We extended the assessment of Stabl to complex clinical datasets combining multiple omic technologies. In this case, the algorithm first selects a reliable set of features at the single-omic level, then integrates the features selected for each omic dataset in a final learner algorithm, such as linear or logistic regression. We compared Stabl to early and late fusion Lasso, two commonly employed strategies for multi-omic modeling, on the prediction of a continuous outcome variable from a triple-omic dataset. The analysis leveraged a unique longitudinal biological dataset collected in independent training and validation cohorts of pregnant individuals, together with curated clinical information (Fig. 4a ). 31  The study aimed to predict the difference in days between the time of blood sample collection and spontaneous labor onset (i.
e. , time to labor). The study addresses an important clinical need for improved prediction of labor onset in term and preterm pregnancies as standard predictive methods are inaccurate. 32, 33  The triple-omic dataset contained a proteomic dataset (p = 1,317 features, Somalogic), a metabolomic dataset (p = 3,529 untargeted mass spectrometry features), and a single-cell mass cytometry dataset (p = 1,502 immune cell features, see methods). When compared to early and late fusion Lasso, Stabl estimated the time to labor with comparable predictivity (Fig. 4b  training and validation cohorts), while selecting fewer and more reliable features (Fig. 4c ). Importantly, Stabl calculated a different reliability threshold for each omic sublayer (𝜃[Proteomics] = 36%, 𝜃[Metabolomics] = 35%, 𝜃[mass cytometry] = 52%, Fig.
4g-i ). On the validation dataset, available for the proteomic and mass cytometry data only, 26% of features selected by Stabl passed a 5% FDR threshold (Benjamini-Hochberg correction), compared to 4% for early fusion Lasso and 5% for late fusion Lasso, showing that Stabl selected more reliable features (Table  S5 ). These results emphasize the advantage of the data-driven threshold, as fixing a common frequency threshold across all omic layers would have been suboptimal, risking over-or under-selecting features in each omic dataset to be integrated into the final predictive model. From a biological standpoint, Stabl streamlined the interpretation of our prior multivariate analyses,  31  honing in on sentinel elements of a systemic biological signature predicting the onset of labor that could be leveraged for development of a blood-based diagnostic test.
Stabl identifies promising candidate biomarkers from a newly generated multi-omic dataset Application of Stabl to the three existing omic datasets demonstrated the algorithm's performance in the context of biomarker discovery studies with a known biological signal. To complete its systematic evaluation, Stabl was applied to our newly generated multi-omic clinical study performing an unbiased biomarker discovery task. The aim of the study was to develop a model to predict which patients will develop a postoperative surgical site infection (SSI) from analysis of pre-operative blood samples (Fig. 5a ). A cohort of 274 patients undergoing major abdominal surgery were enrolled and preoperative blood samples were collected. Using a matched, nested case-control design, 93 patients were selected from the larger cohort to minimize the effect of clinical or demographic confounders on identified predictive models (Table  S7 ).