dataesr/scientific-paragraphs-categorization huggingface dataset:
opendata. bio. ens. psl. eu/finsurf/. The scores for individual bases, the regulatory elements they overlap and the target genes of the latter can be visualised on a UCSC track: https://genome. ucsc. edu/s/alouis72/hg19_finsurf. The code necessary to install FINSURF locally, to produce figures 2, 3 and 4 and to train the model is available at https:// github. com/DyogenIBENS/FINSURF/. suffer from three main limitations. First, specificity and sensitivity are typically evaluated using data similar to the data used for model training, and it is unclear how models generalise to new variants. Second, scores are over-simplified numerical values that do not capture the rich and heterogeneous set of annotations contributing to variant selection. Third, most methods do not assign candidate regulatory variants to a predicted target gene, or resort to a naive "nearest gene" approach to do so, even though this is incomplete in many cases  [16, 22, 23].
Training datasets As positive controls, we selected all variants labelled as Damaging Mutations from the Human Gene Mutation Database  [5]  (version Pro 2017. 2). To focus on non-coding variants, we excluded variants that were annotated as protein-impacting and/or located within CDS according to the HGMD gene annotations. We additionally used the GENCODE  [24]  genome annotation (version 29 lift-over hg19) to further exclude variants located within CDS, start codon, stop codon, or splice sites. This lead to the identification of 880 non-coding damaging mutations. This entire set was included for the "Random" model, but reduced to 878 for the "Adjusted" model and to 877 in the "Local" model, as some variants did not have any appropriate negative controls (see hereafter for the description of these models and sampling schemes).
Additionally, we downloaded two datasets related to transcription factor binding sites from the UCSC Genome Browser: TFBS identified as conserved between mouse, rat, and human; and clusters of TFBS identified in ChIP-seq peaks from 161 experiments across 91 cell types from the ENCODE project. A complementary dataset of TFBS from JAS-PAR, identified within ChIP-seq peaks, was obtained from Ensembl  [28]. Finally, a dataset of DNaseI hypersensitive regions was obtained from the UCSC, corresponding to clusters identified in 125 cell types from ENCODE. A third group of features describe sequence properties related to variant locations. Three features were retained: CpG dinucleotides and CpG islands, downloaded from the UCSC, and variant type (transition, transversion, or INDEL). The last group gathers annotations from recently published datasets predicting regulatory regions in the genome together with their gene targets.
We extracted predicted regulatory regions from the following datasets: GeneHancer  [29]  (version 4. 4, accessed 2018-12-17), PEGASUS  [20]  (2018-12-17), FANTOM5 co-expressed regions  [16]  (2018-11-19), FOCS FANTOM5, FOCS ROADMAP DHS, and FOCS Gro-seq co-expression  [17]  (2018-. We also included promoters (2kb upstream of transcription start sites) and UTR regions from all coding and noncoding genes from the GENCODE  [24]  dataset as potential non-coding regions of interest (v29lifthg19). These datasets were integrated by merging overlapping elements into regions with one or multiple evidences of predicted associations. This integration was not performed for GeneHancer, which was handled separately, as it does not provide a single score for each regulatory region/predicted target pair. A list of predicted targets was derived for each regulatory region by merging all predicted target genes across methodologies.
In addition to these four groups of annotations, which are used for classification by the FINSURF model, other annotations were included for filtering and characterizing variants. Notably, we used the GEN-CODE (v29liftHg19) annotations for the locations of biotypes such as CDS, introns, etc. A total of 471,099,210 genomic positions were thus annotated with the set of descriptors, and evaluated for functional potential with FINSURF. For each position, 2 predictions are reported: one for transitions and one for transversions. Tabix-indexed tabular files for all chromosomes were generated, allowing the fast interrogation of these files for millions of variants of interest. Functional profiles are reported for all variants lying within annotated regulatory regions. For indels, all positions affected by the variant (as well as the one preceding and following positions) are evaluated, but only the highest score is reported.
Model training and performance evaluation We trained three Random Forest models (Random, Adjusted and Local, using different negative control sets) using the Scikit-Learn Python library v. 0. 20. 2, with the following parameters: 1,000 trees, maximum depth = 15 nodes, minimum number of samples per leaf = 1. These parameters were optimized using a nested-cross validation evaluation of 400 models, exploring different range of values. Class size imbalance issues in the training set were solved by sampling with replacement a set of n random variants for each class, where n is the size of the smallest class (in this case, HGMD-DM non-coding variants; n = 880). Each tree in the Random Forest was then built from a different, balanced set of positive and negative controls. Model performance was evaluated using 10-fold cross-validation.
This second curve is of particular interest in the context of imbalanced learning, as it better captures how the proportion of true positives against false positives changes with increasingly lenient thresholds on the prediction score. For the Adjusted model, we maximized the F1-score, defined as the harmonic mean of the precision and recall, and obtained an optimal prediction score threshold of 0. 51. This threshold was used to calculate the confusion matrix. Eight other methods were also evaluated using 10-fold cross-validations on the control dataset. Variants missing a score for any of the methods were excluded from the evaluation (average drop-out rate = 53%). Indels were scored as with FINSURF: all bases covered by the indel were scored, but only the highest scoring position was retained.
Finally, we applied each of the three FINSURF models (Random, Adjusted, Local) on the training datasets of the two other models, in order to evaluate the performance of each model across different genomic contexts. We re-used the 10-fold cross-validation scheme used for model training, but evaluated the 10 partial models using negative test sets sampled with either of the other two models. In order to make the performance curves comparable, an additional random sub-sampling was performed on the negative controls, so that the proportion of positives in the test-subset of each k-fold was as close as the one from the "Local" selection scheme (12. 5%).
Independent evaluation We downloaded 448 disease-causing non-coding variants used to train the ReMM-Genomiser model  [25]. Of these, we excluded 11 variants found within coding regions according to GEN-CODE v29. We then intersected these variants with our training set of regulatory HGMD-DM variants and excluded overlaps. This resulted in a set of 92 non-coding regulatory variants independent from our training positive controls, which were used to compute ROC and PRC curves. Of these, 30 were dropped during the comparison against other methods, as some models did not provide scores for these positions, leaving 62 variants for the comparison. Negative controls (N = 17,122) were sampled from the ClinVar dataset, using the Adjusted sampling protocol. For analysis of a realistic genome-wide VCF, we further excluded diseasecausing non-coding variants located within 1,000 bp of an HGMD-DM variant from the FIN-SURF training dataset, and specifically focused on 49 variants that represent a set of fully independent regulatory variants with respect to our model.
Feature contributions and clustering Feature contributions were calculated using a more computationally efficient, in-house reimplementation of the treeinterpreter package (https://github. com/andosa/treeinterpreter; https://arxiv. org/abs/1906. 10845), which calculates the average decrease in Gini impurity index across all trees for each feature/variant combination. To avoid overfitting, feature contributions for a particular variant were calculated only from trees where this variant was not used for training. K-means clustering was performed on the feature contributions vectors for the positive control variants to explore structure in the training set. Different K values from 2 to 19 were explored, and the optimal K was selected using maximization of the silhouette score and inertia minimization with Scikit-Learn v.
Disease association analysis We collected sets of genes from OMIM  [31] , associated with 30 diseases caused by 49 fullyindependent non-coding variants from the ReMM-Genomiser  [25]  training dataset (see 'Independent evaluation' above). Each disease was associated with a single gene, except for "Cerebral Amyloid Angiopathy, APP-related" (OMIM:605714) for which 3 genes were retrieved from the OMIM webpage. Variants within regulatory regions with predicted targets that contained the identified disease-gene were selected and ranked based on their FINSURF score.
FINSURF accurately distinguishes true regulatory variants from a variety of negative controls We trained random forest classifiers  [32]  using 880 experimentally validated, non-coding regulatory variants as a positive training set, identified in the HGMD  [5]  database as Damaging Mutations (hereafter named "HGMD-DM" variants). We sampled control variants from 31 million non-coding variants with no clinical significance in the ClinVar  [33]  database as negative training sets. Importantly, we observed that negative and positive variants are unequally distributed with regard to genomic features such as introns, intergenes or promoters (Fig  1A ). This probably reflects a mixture of underlying biology of regulatory variants and bias in the HGMD database. We therefore defined three negative training sets to explore the impact of genomic variant distribution on the model ability to discriminate non-coding regulatory variants.
These results confirm that FINSURF is highly efficient to identify disease-causing, penetrant non-coding genetic variants of the type found in the HGMD-DM resource. To verify that FINSURF's performances extend to an independent set of non-coding regulatory mutations, we collected non-coding variants used for training ReMM-Genomiser but absent from the HGMD-DM resource. This small but independent dataset comprises 92 mutations, of which 62 can be scored by all eight methods, including 41 SNV and 21 INDELS. For the negative set, 31,564 variants (of which 17,122 can be scored by all methods) were selected from the non-coding and clinically non-significative ClinVar set, following the Adjusted.
Finally, clusters 6 and 7 are two small clusters where FINSURF displays low accuracy (42 and 85 variants; false negative rate > 60%). Variants in these clusters display characteristics typical of variants from the negative set and have nearly no features contributing positively to their classification. This observation could be caused by insufficient coverage of their regulatory features in the collection used by FINSURF, possibly because they are condition-dependent, or that despite manual curation of the HGMD database, these variants are in fact not regulatory. FINSURF additionally provides a graphical display tool to investigate the functional properties of specific variants. To illustrate this functionality, we explored the functional profiles of two example variants located in the vicinity of SERPINC1, one correctly classified as a regulatory variant (Fig 3C ) and one non-functional but misclassified  (Fig 3D).
The first example case was identified by FINSURF as functional based on its strong evolutionary conservation, as highlighted by the Feature Contribution values. This variant also overlaps a promoter region, a TFBS cluster, and is quiescent in many tissues, but these features only weakly contributed to the classification. The second variant shows no evidence of medical relevance in the ClinVar database, but FINSURF mis-identified it as functional. The functional profile reveals that this variant is located within a promoter, overlaps a TFBS cluster and corresponds to an open chromatin region in many tissues as revealed by DNAse sensitivity, which jointly contributed to its misclassification.
27 10 -23 ), but still among a vast excess of false positives. Then, we relied on the OMIM database  [31]  to establish lists of potentially deregulated genes for each of the 30 diseases, and only retained variants predicted by FINSURF to interact with those genes. For example, cerebral amyloid angiopathy is caused by the β-amyloid precursor protein (APP)  [43] , whose gene is predicted to interact with 428 candidate variants in this realistic setting. FINSURF correctly ranks the pathogenic variant in first position (Fig  4B ). Overall, across the 30 diseases included in the study, between 3 and 495 variants from the synthetic genome are predicted to interact with known disease genes (average: 115). FINSURF ranks the causative pathogenic variants in first position in 11 instances. For 8 other diseases, FINSURF ranked the causative mutation in the top 5 variants, and in 4 additional cases, the causative mutation was in the top 10 variants.
Discussion We developed FINSURF to score variants in the human genome in the context of medical genetics, delivering an efficient strategy to identify regulatory non-coding mutations likely to cause a diagnosed disease. While a growing number of machine-learning models have been developed in recent years to identify functional non-coding variants  [47, 48] , few if any are applicable in practical clinical contexts, because of their design and control choices. A classical problem when designing an experiment, choices of positive and negative controls are not only paramount for model accuracy and precision, but they must also be tailored to the question that the model aims to solve. We know that a typical WGS dataset contains thousands of benign variants that occur in conserved or epigenetically active regions with characteristics of regulatory sequences  [8].
Our goal is to identify a unique mutation among those that likely causes a disease. While developing FINSURF, we paid a high degree of attention to the design of a negative control dataset, and its effects on model performances and generalization. Positive controls, e. g. validated non-coding regulatory variants, remain few and far between in the literature. As a result, most models including FINSURF are trained on similarly limited datasets. However, we show here that the non-coding regulatory variants in the HGMD database are quite heterogeneous in terms of genomic location and biological features (Fig  3 ), suggesting that FINSURF can capture a varied range of functional variants. Possibly less appreciated, negative controls are also crucial to developing a successful model. Randomly chosen benign human variants, or solely based on population frequency, result in models that successfully discriminate negatives from positives but lack precision within broad regulatory regions.
Combining information from FINSURF scores, target predictions and disease aetiology, we are able to correctly identify the causative regulatory mutation as a top candidate in an array of 22 pathologies representative of the state-of-the-art on non-coding pathogenic mutations. This strategy allows users to integrate the discovery power of machine learning models with prior knowledge on gene-to-phenotype associations to restrict and refine searches for candidate genetic variants. Although FINSURF can be used as an agnostic approach to explore regulatory variants associated to any gene, we show that predicted target genes aggregated from molecular and evolutionary evidence by FINSURF can be decisive in identifying variants that interact with genes of interest. This procedure effectively makes it possible to leverage the vast knowledge accumulated from gene functional exploration, knock-outs and perturbation experiments in human and animal models, which can be extracted from the literature or from dedicated databases  [50] , and extends it to the more obscure non-coding fraction of the genome.
The cross-performance of the models is evaluated by comparison of the ROC and precision-recall curves, recalculated from the 10 fold cross validation step of each model (which correspond to the pairs of curves on the diagonal). A given column corresponds to the application of a model on the 10-fold subsets of variants obtained under the different samples methods (with filtering of overlapping variants between training and validation subsets, as described in the Material and Methods). Note that to allow the comparison of performances, a second undersampling of negative control variants was performed in the validation subset for the Random and Adjusted models, in order to match the observed proportion of 12. 5% of positive controls in the Local sampling dataset. The area under the curves of the average curves is reported in the two tables for each combination of model and validation dataset.
b. Results from a 10-fold "Stratified" cross-validation, where only the imbalance of positive and negative variants was taken care of by the stratification procedure. The much better performance and cleaner separation between classes highlights data leakage from separating at random into train and validation sets variants that might share common genomic context. Fig C. Comparison between FINSURF (red lines) and eight other methods also designed to functionally interpret non-coding regulatory variants. a. Comparison of performances on the training dataset of FINSURF "Adjusted". The subsets of variants generated during the 10-fold cross-validation were re-used, by scoring the positive and negative control variants with the other methods. At no time was FINSURF tested on variants used during training.
Variants for which a score could not be computed by any one method (e. g. lack of required annotations) were discarded for all methods. The average performance curve of the 10-folds are reported. We note the high precision of the ReMM Genomiser model, which can be explained by the overlap between the training variants of this model and the HGMD variants used by FINSURF. b. Comparison of performances on the subset of 92 Genomiser variants, of which 30 are discarded due to missing scores by at least one method. Negative control variants were re-sampled following the Adjusted sampling procedure, and those found in the training dataset of the FINSURF model were removed. Note that the ROC curve is the one presented in the Fig  2d. ) to one of the latter (left), the remaining 49 are sufficiently far (> 1kb) to be considered to overlap independent annotations.
51. c. Average functional profile of the 7 clusters. These profiles highlight how feature contributions relate to the raw values of the features, as processed by the random forests. For each cluster, the raw values were normalized, and averaged from the variants within the cluster (colored bars). Reference values (dark grey bars) correspond to the average from the rest of the training dataset (i. e. positive controls in other clusters and negative controls). The color of the bars is based on the average contributions of the features in each cluster. Fig F. Predicting highly pathogenic disease variant in genomic context. a. Box-plots representation and individual FINSURF score values for 92 known pathogenic variants from the Genomiser dataset but absent from the FINSURF training set. While 43 variants are very close (� 1kb.
This work was supported by the  French Government  and implemented by  ANR  ( ANR-10-LABX-54   MEMOLIFE  and  ANR-10-IDEX-0001-02 PSL � Research University ) to HRC, by the  French Minister  for research and Education and by the  Fondation pour la Recherche Medicale , grant number  FDT201805005782 , to LM. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Given the width of possible code-removal patches, we qualify the problem domain as follows. We consider faults revealed by either one failing or one crashing test case only, since it is a situation commonly encountered in practice. For instance, the Bears benchmark  [29] , which is a benchmark of 251 reproducible bugs from 72 different projects, has 71. 32% of builds with a single failing (38. 65% of the total) or crashing (32. 67% of the total) test case. We take the bug data from the Repairnator-Experiments repository  [49] , which is a repository that includes more than 4 thousands builds with a single failing or crashing test case. On those 4k+ faults, we perform a deep qualitative analysis, and identify major contributions for the understanding of code-removal patches. In a nutshell, the contributions of this paper are: -An analysis of nearly 2k failed builds from 674 real Java software projects and their corresponding code-removal patches, that make possible to have a clear view about the effectiveness of code-removal patches on a wide range of programs; -The definition of a comprehensive taxonomy of code-removal patches that can be exploited to better understand the current limitations of program repair techniques, showing the potential issues that may cause the acceptance of wrong code-removal patches; -A comparative analysis of human patches and code-removal patches on the same bug, showing the possibility to exploit automatic code-removal patches to give valuable information to developers about the cause of the problem, e.
Choice of jKali Implementation In addition to the original Kali  [39]  for C programs, we are aware of two other implementations of the approach, Astor's jKali  [32]  and Arja-Kali  [58] , both for Java programs. To choose the implementation to use for our experiments, we analyzed the publicly available data. In Table  1 , we report the number of code-removal patches generated by Arja-Kali and Astor jKali in previous published research. The first column (Experiment ) indicates the reference to the experiment from which the data has been extracted from. The second column (Benchmark ) indicates the benchmark and the number of total bugs on which the repair tools have been tested, The third column (Arja-Kali) and the fourth column (Astor jKali) report the number of code-removal patches generated by Arja-Kali and Astor jKali, respectively.
[11]. This aspect is confirmed also by the results of the experiment conducted by Liu et al. [26]  that compares some program repair tools on the Defects4J benchmark to evaluate the efficiency of test-based automated program repair. In this experiment, it is reported that Arja-Kali is not able to generate patches for bugs related to Apache Commons Lang project. Moreover, the experiment reports that the correctness rate of patches generated by Arja-Kali is 4. 6%, while the correctness rate of patches generated by Astor jKali is 16%. We manually tested the patches generated by Arja-Kali and Astor jKali for the bugs of Defects4J reported in the experiment conducted by Durieux et al. [11]. In that experiment, it is reported that Arja-Kali is able to generate 72 patches, while Astor jKali 27. However, applying the patches to the faulty programs, we observed that only 23 out of 72 patches generated by Arja-Kali work.