dataesr/scientific-paragraphs-categorization huggingface dataset:
Marginal The best-performing model used a random survival forest algorithm trained by combining the clinical covariates with a single additional aggregate feature, called the genomic instability index (GII), calculated as the proportion of amplified or deleted sites based on the copy number data. This result highlights the importance of evaluating models using a controlled experimental design, as the best-performing method in the uncontrolled experiment combined clinical variables with GII in addition to selected gene expression features (clinical variables plus only GII was not evaluated), and the controlled experiment pointed to isolating GII as the modeling insight associated with high prediction accuracy. The random survival forest trained using clinical covariates and GII was significantly better than a random survival forest trained using clinical covariates alone (P = 2e-12 by paired Wilcoxon signed rank test based on 100 bootstrap samples with replacement from the test dataset).
Consistency of results in independent datasets In the first round of the competition, we did not restrict the number of models a participant could submit. This raises the possibility of model overfitting to the test set used to provide realtime feedback. We therefore used 2 additional datasets to evaluate   S4 ). The methods and features sets are arranged according to the mean concordance index score. The ensemble method (cyan curve) infers survival predictions based on the average rank of samples from each of the four other learning algorithms, and the ensemble feature set uses the average rank of samples based on models trained using all of the other feature sets. Results for the METABRIC2 and MicMa datasets are show in Figure  S1. (B) The concordance index of models from the controlled phase by type.
The ensemble method again utilizes the average rank for models in each category. doi:10. 1371/journal. pcbi. 1003047. g003 the consistency of our findings. The first dataset, which we called METABRIC2, consisted of the 988 samples (excluding those with missing survival data) from the METABRIC cohort that were not used in either the training dataset or the test dataset used for realtime evaluation. The second dataset, called MicMa, consisted of 102 samples with gene expression, clinical covariates, and survival data available  [4, 28]  and copy number data presented in the current study (see Methods). We used the models from our controlled experiment, which were trained on the original 500 METABRIC samples, and evaluated the concordance index of the survival predictions of each model compared to observed survival in both METABRIC2 and MicMa.
It is thus difficult to reproduce exactly the predictions provided by these assays or to perform a fair comparison to the present methods on a dataset that includes samples from the whole spectrum of breast tumors. The actual Oncotype DX score is calculated from RT-PCR measurements of the mRNA levels of 21 genes. Using z-score normalized gene expression values from METABRIC2 and MicMa datasets, together with their published weights, we recalculated Oncotype DX scores in an attempt to reproduce the actual scores as closely as possible. We then scored the resulting predictions against the two datasets and obtained concordance indices of 0. 6064 for METABRIC2 and 0. 5828 for MicMa, corresponding to the 81 st ranked model based on average concordance index out of all 97 models tested, including ensemble models and Oncotype DX and MammaPrint feature sets incorporated in all learning algorithms (see Table  S5 ).
Similarly, the actual MammaPrint score is calculated based on microarray gene expression measurements, with each patient's score determined by the correlation of the expression of 70 specific genes to the average expression of these genes in patients with good prognosis (defined as those who have no distant metastases for more than five years, ER+ tumors, age less than 55 years old, tumor size less than 5 cm, and are lymph node negative). Because of limitations in the data, we were not able to compute this score in exactly the same manner as the original assay (we did not have the metastases free survival time, and some of the other clinical features were not present in the validation datasets). We estimated the average gene expression profile for the 70 MammaPrint genes based on all patients who lived longer than five years (with standardized gene expression data), then computed each patient's score as their correlation to this average good prognosis profile.
We scored the predictions against the two validation datasets and observed concordance indices of 0. 602 in METABRIC2 and 0. 598 in MicMa, corresponding to the 78 th ranked out of 97 models based on average concordance index. We were able to significantly improve the scores associated with both MammaPrint and Oncotype DX by incorporating the gene expression features utilized by each assay as feature selection criteria in our prediction pipelines. We trained each of the 4 machine learning algorithms with clinical features in addition to gene lists from MammaPrint and Oncotype DX. The bestperforming models would have achieved the 8 th and 26 th best scores, respectively, based on average concordance index in METABRIC2 and MicMa. We note that using the ensemble strategy of combining the 4 algorithms, the model trained using Mammaprint genes and clinical data performed better than clinical data alone, and achieved the 5 th highest average model score, including the top score in METABRIC2, slightly (.
005 concordance index difference) better than the random forest model using clinical data combined with GII, though only the 17 st ranked score in MicMa. This result suggests that incorporating the gene expression features identified by these clinically implemented assays into the prediction pipeline described here may improve prediction accuracy compared to current analysis protocols. An ensemble method, aggregating results across all learning algorithms and feature sets, performed better than 71 of the 76 models (93%) that constituted the ensemble, consistent with our finding that the ensemble strategy achieves performance among the top individual approaches. For the 19 feature selection strategies used in the METABRIC2 and MicMa evaluations, an ensemble model combining the results of the 4 learning algorithms performed better than the average of the 4 learning algorithms in 36 out of 38 cases (95%).
Also consistent with our previous result, for both algorithms that did not use ensemble strategies themselves (elastic net and lasso), an ensemble model aggregating results across the 19 feature sets performed better than each of the individual 19 feature sets for both METABRIC2 and MicMa. Taken together, the independent evaluations in 2 additional datasets are consistent with the conclusions drawn from the original real-time feedback phase of the completion, regarding improvements gained from ensemble strategies and the relative performance of models.
The primary challenge of any competition framework is to ensure that mechanisms are in place to prevent overfitting and fairly assess model performance, since performance is only meaningful if models are ranked based on their ability to capture some underlying signal in the data. For example, such an approach requires datasets affording sufficient sample sizes and statistical power to make meaningful comparisons of many models across multiple training and testing data subsets. We propose several strategies for assessing if the results obtained from a collaborative competition are likely to generalize to future applications and improve on state-of-the art methodologies that would be employed by an expert analyst. First, baseline methods should be provided as examples of approaches an experienced analyst may apply to the problem.
In our study, we employed a number of such methods for comparison, including methodologies used in clinical diagnostic tests and multiple state-of-the-art machine learning methods trained using only clinical covariates. Second, performance of models should be evaluated in multiple rounds of independent validation. In this study, we employed a multi-phase strategy suggested by previous researchers  [50]  in which a portion of the dataset is held back to provide real-time feedback to participants on model performance and another portion of the dataset is held back and used to score the performance of all models, such that participants cannot overfit their models to the test set. If possible, we recommend an additional round of validation using a dataset different from the one used in previous rounds, in order to test against the possibility that good performance is due to modeling confounding variables in the original dataset.
This experimental design provides 3 independent rounds of model performance assessment, and consistent results across these multiple evaluations provides strong evidence that performance of the best approaches discovered in this experimental design are likely to generalize in additional datasets. Finally, statistical permutation tests can provide useful safeguards against the possibility that improved model performance is attributable to random fluctuations based on evaluation of many models. Such tests should be designed carefully based on the appropriate null hypothesis. A useful, though often insufficient, test is to utilize a negative control null model, for example by permuting the sample labels of the response variable. We suggest that additional tests may be employed as post-hoc procedures designed specifically to provide falsifiable hypotheses that may provide alternative explanations of model performance.
Within several months we developed and evaluated several hundred modeling approaches. Our research group consisted of experienced analysts trained as both data scientists and clinicians, resulting in models representing state-of-the art approaches employed in both machine learning and clinical cancer research (Table  3 ). By conducting detailed post-hoc analysis of approaches developed by this group, we were able to design a controlled experiment to isolate the performance improvements attributable to different strategies, and to potentially combine aspects of different approaches into a new method with improved performance. The design of our controlled experiment builds off pioneering work by the MAQC-II consortium, which compiled 6 microarray datasets from the public domain and assessed modeling factors related to the ability to predict 13 different phenotypic endpoints.
MAQC-II classified each model based on several factors (type of algorithm, normalization procedure, etc), allowing analysis of the effect of each modeling factor on performance. Our controlled experiment follows this general strategy, and extends it in several ways. First, MAQC-II, and most competition-base studies  [20, 22, 26] , accept submissions in the form of prediction vectors. We developed a computational system that accepts models as rerunnable source code implementing a simple train and predict API. Source code for all submitted models are stored in the Synapse compute system  [51]  and are freely available to the community. Thus researchers may reproduce reported results, verify fair play and lack of cheating, learn from the bestperforming models, reuse submitted models in related applications (e.
g. building prognostic models in other datasets), build ensemble models by combining results of submitted models, and combine and extend innovative ideas to develop novel approaches. Moreover, storing models as re-runnable source code is important in assessing the generalizability and robustness of models, as we are able to re-train models using different splits or subsets of the data to evaluate robustness, and we (or any researcher) can evaluate generalizability by assessing the accuracy of a model's predictions in an independent dataset, such as existing related studies  [5]  or emerging clinical trial data  [52]. We believe this software system will serve as a general resource that is extended and re-used in many future competition-based studies. Second, MAQC-II conducted analysis across multiple phenotypic endpoints, which allowed models to be re-evaluated in the context of many prediction problems.
Finally, whereas MAQC-II employed training and testing splits of datasets for model evaluation, our study provides an additional level of evaluation in a separate, independent dataset generated on a different cohort and using different gene expression and copy number profiling technology. Consistent with findings reported by MAQC-II, our study demonstrates strong consistency of model performance across independent evaluations and provides an important additional test of model generalizability that more closely simulates real-world clinical applications, in which data is generated separately from the data used to construct models. More generally, whereas MAQC-II evaluated multiple prediction problems in numerous datasets with gene expression data and samples numbers from 70 to 340, our study went deeper into a evaluating a single prediction problem, utilizing copy number and clinical information in addition to gene expression, and with a dataset of 2,000 samples in addition to an independentlygenerated dataset with 102 samples.
The model achieving top performance in both the initial evaluation phase and the evaluation in additional datasets combined a state-of-the-art machine learning approach (random survival forest) with a clinically motivated feature selection strategy that used all clinical features together with an aggregate genomic instability index. Interestingly, this specific model was not tested in the uncontrolled phase, and was the result of the attempt to isolate and combine aspects of different modeling approaches in a controlled experiment. The genomic instability index measure may serve as a proxy for the degree to which DNA damage repair pathways (including, for instance, housekeeping genes like p53 and RB) have become dysregulated  [37]. Beyond the specifics of the top performing models, we believe the more significant contribution of this work is as a building block, providing a set of baseline findings, computational infrastructure, and proposed research methodologies used to assess breast cancer prognosis models, and extending in the future to additional phenotype prediction problems.
Breast Cancer Prognosis Competition Design and Software Our competition was designed to assess the accuracy of predicting patient survival (using the overall survival metric, median 10 year follow-up) based on feature data measured in the METABRIC cohort of 980 patients, including gene expression and copy number profiles and 16 clinical covariates (Table  1 ). Participants were given a training dataset consisting of data from 500 samples, and data from the remaining 480 were hidden from participants and used as a validation dataset to evaluate submitted models. We developed the computational infrastructure to support the competition within the open-source Sage Synapse software platform. Detailed documentation is available on the public competition website: https://sagebionetworks. jira. com/wiki/ display/BCC/Home.
The system is designed to generalize to support additional community-based competitions and consists of the following components (Figure  5 ): 1. The ability for participants to access training data stored in the Sage Synapse software system through programmatic APIs, with initial support built for the R programming language. 2. A programmatic API for training and testing predictive models. To date, we have developed support for models developed in the R programming language conforming to a simple interface implementing methods named customTrain and customPredict. Any model conforming to this interface can be plugged-in to the competition infrastructure, trained on the training dataset, and evaluated for prediction accuracy in the validation dataset, as well as using various cross-validation statistics.
3. The ability to upload models, including re-runnable source code, in Synapse, allowing models to be shared with the community in a fully transparent, reproducible environment. 4. An automated model evaluation system for assessing the performance of submitted models and outputting the scores to a web-based real-time leaderboard. We stress this aspect of the framework, based on the findings from previous competitions that rapid feedback is critical to motivating its participants to improve their model beyond the baseline  [36]. 5. Communication and social networking tools, such as wikis and discussion forums (http://support. sagebase. org). All models are available with downloadable source code using the Synapse IDs displayed in Table  S1  and Table  S4. An automated script continuously monitored for new submissions, which were sent to worker nodes in a computational cluster for scoring.
Each worker node ran an evaluation script, which called the submitted model's customPredict method with arguments corresponding to the gene expression, copy number, and clinical covariate values in the held-out validation dataset. This function returns a vector of predicted survival times in the validation dataset, which were used to calculate the concordance index as a measure of accuracy compared to the measured survival times for the same samples. Concordance index scores were shown in a realtime leaderboard, similar to the leaderboards displaying the models scores shown in Table  S1  and Table  S4. Concordance index (c-index) is the standard metric for evaluation of survival models  [53]. The concordance index ranges from 0 in the case of perfect anti-correlation between the rank of predictions and the rank of actual survival time through 0.
Study timeline Data on the original 980 samples were obtained for this study in early January, 2012. Study design and computational infrastructure were developed from then until March 14 th , at which point participants were given access to the 500 training samples and given 1 month to develop models in the ''uncontrolled experiment'' phase. During this time, participants were given real-time feedback on model performance evaluated against the held-out test set of 480 samples. After this 1-month model development phase, all models were frozen and inspected by the group to conduct post-hoc model evaluation and identify modeling strategies used to design the controlled evaluation. All models in the controlled evaluation were re-trained on the 500 training samples and re-evaluated on the 480 test samples.
After all evaluation was completed based on the original 980 samples, the METABRIC2 and MicMa datasets became available, and were used to perform additional evaluations of all models, which was conducted between January 2013-March 2013. For the new evaluation, all data was renormalized to the gene level, as described below, in order to allow comparison of models across datasets performed on different platforms. Models were retrained using the re-normalized data for the same 500 samples in the original training set.
Model source code All model source code is available in the subfolders of Synapse ID syn160764, and specific Synapse IDs for each model are listed in Table  S1  and Table  S4. Data stored in Synapse may be accessed using the Synapse R client (https://sagebionetworks. jira. com/wiki/display/SYNR/Home) or by clicking the download icon on the web page corresponding to each model, allowing the user to download a Zip archive containing the source files contained in the submission. regression) in chunks of 50 randomly selected gene expression samples. This process was repeated 100 times which resulted in an overall survival association score s i ~{ P j log p ij where p ij is the p-value associated with the Cox regression on the expression of gene i in sample set j. All genes were sorted in descending order by their survival association score and the top 50 oncogenes and transcription factors were kept.
A list of human transcription factors was obtained from  [59]  and a list of oncogenes was compiled by searching for relevant keywords against the Entrez gene database. GII is a measure of the proportion of amplified or deleted genomic loci, calculated from the copy number data. Copy number values are presented as segmented log-ratios c ij with respect to normal controls. Amplifications and deletions are thus counted when c ij w1 or c ij v{1and devided by the total number of loci N. GII~1 N X N i~1 1 c ij w1 z X N i~1 1 c ij v{1 !.
Figure1. Gene expression subclass analysis. (A) Comparison of hierarchical clustering of METABRIC data (left panel) and Perou data (right panel). Hierarchical clustering on the gene expression data of the PAM50 genes in both datasets reveals a similar gene expression pattern that separates into several subclasses. Although several classes are apparent, they are consistent with sample assignment into basal-like, Her2-enriched and luminal subclasses in the Perou data. Similarly, in the METABRIC data the subclasses are consistent with the available clinical data for triplenegative, ER and PR status, and HER2 positive. (B) Kaplan-Meier plot for subclasses. The METABRIC test dataset was separated into 3 major subclasses according to clinical features. The subclasses were determined by the clinical features: triple negative (red); ER or PR positive status (blue); and HER2 positive with ER and PR negative status (green).
The survival curve was estimated using a standard Kaplan-Meier curve, and shows the expected differences in overall survival between the subclasses. (C,D) Kaplan-Meier curve by grade and histology. The test dataset was separated by tumor grade (subplot C; grade 1 -red, grade 2 -green, grade 3-blue), or by histology (subplot D; Infilitrating Lobular -red, Infiltrating Ductal -yellow, Medullary -green, Mixed Histology -blue, or Mucinous -purple). The survival curves were estimated using a standard Kaplan-Meier curve, and show the expected differences in overall survival for the clinical features. doi:10. 1371/journal. pcbi. 1003047. g001.
Association 1000 molecular features (gene expression and/or copy number) most predictive of survival in a univariate Cox regression analysis on the training set. Top-Varying 1000 molecular features (gene expression and/or copy number) with the greatest variance in the training set. Cancer Census 1526 gene expression and copy number features corresponding to 487 genes from the Cancer Gene Census database [60]. Higgins 1000 gene expression and copy-number features with the greatest variance among oncogenes identified by Higgins et al. [61]. Metabric Clustering 754 gene expression and copy number features used to define the clusters in the study by Curtis et al. [29]. MASP: Marginal Association with Subsampling and Prior KnowledgeGene expression of 50 known oncogenes and transcription factors selected by computing univariate Cox regression models on random subsets of the training set and aggregating the resulting p-values (see Methods).
In this paper we present the design and implementation of a generic GA-based optimization framework iMASKO (iNL@MATLAB Genetic Algorithm-based Sensor NetworK Optimizer) to optimize the performance metrics of wireless sensor networks. Due to the global search property of genetic algorithms, the framework is able to automatically and quickly fine tune hundreds of possible solutions for the given task to find the best suitable tradeoff. We test and evaluate the framework by using it to explore a SystemC-based simulation process to tune the configuration of the unslotted CSMA/CA algorithm of IEEE 802. 15. 4, aiming to discover the most available tradeoff solutions for the required performance metrics. In particular, in the test cases different sensor node platforms are under investigation. A weighted sum based cost function is used to measure the optimization effectiveness and capability of the framework.