dataesr/scientific-paragraphs-categorization huggingface dataset:
Highlights The beamline has been open since September 2014 and to date has processed over 10000 samples. This throughput has now started to result in publications based on diffraction data collected on the endstation  (Koromyslova et al. , 2015). A recent example highlights the advantages of flexible booking and automated data collection  (Kharde et al. , 2015). The Rpf2-  Rrs1 complex is thought to have a role in ribosome biogenesis but no structural information has been available for the complex. Initially, 70 crystals of the native complex were obtained and rapidly screened on MASSIF-1. This resulted in ten data sets from crystal forms with different space groups and resolution limits (Table  2 ). As molecular replacement failed to produce a successful structure solution, a second session was booked with 40 crystals of the selenomethionine derivative screened.
This resulted in ten data sets mostly in a centred orthorhombic space group with the best diffraction to 1. 6 A ˚(Table  2 ). These data sets were processed automatically and this resulted in nine of the data sets being phased by SAD protocols  (Monaco et al. , 2013). The automatically determined model and the final refined model are shown in Fig. 8. The best data set has excellent processing, phasing and refinement statistics (Table  3 ;  Kharde et al. , 2015). The automatic service on MASSIF-1 allowed the researchers to screen a large number of crystals when they became available, through flexible booking of beam time, and rapidly feed results back into the project to improve crystal quality and phase the structure by single-wavelength anomalous diffraction within five weeks. flux into account when calculating data collection strategies.
Flexibility is introduced into the system by allowing sample specific parameters to be specified in the database ISPyB. Additional features will soon be added to the workflows, such as allowing the collection from multiple crystals on the same sample support or multiple positions within the same crystal and high-exposure mesh scans in order to screen for weakly diffracting samples. While measurements are currently limited to variations around classic experiments, it is hoped that more complex strategies such as helical data collections  (Flot et al. , 2010)  and goniometer realignment  (Brockhauser et al. , 2013)  can be included soon. It is also foreseen to include dehydration experiments in the automated pipeline  (Bowler et al. , 2015).
Acknowledgements We thank all the  ESRF  support groups without whom the beamline would not be possible as well as past and present  ESRF Directors of Research (Sine Larsen ,  Harald Reichert ,  Jean Susini ) and  EMBL Grenoble Outstation management (S. Cusack)  for supporting this project. We are very grateful to  Fabiola R. Calvin ˜o ,  Satyavati Kharde ,  Klemens Wild  and  Irmgard Sinning  ( BZH, Heidelberg ) for sharing results on the Rpf2-Rrs1 complex before publication.
Prioritizing missense variants for further experimental investigation is a key challenge in current sequencing studies for exploring complex and Mendelian diseases. A large number of in silico tools have been employed for the task of pathogenicity prediction, including PolyPhen-2, SIFT, FatHMM, MutationTaster-2, Muta-tionAssessor, Combined Annotation Dependent Depletion, LRT, phyloP, and GERP++, as well as optimized methods of combining tool scores, such as Condel and Logit. Due to the wealth of these methods, an important practical question to answer is which of these tools generalize best, that is, correctly predict the pathogenic character of new variants. We here demonstrate in a study of 10 tools on five datasets that such a comparative evaluation of these tools is hindered by two types of circularity: they arise due to (1) the same variants or (2) different variants from the same protein occurring both in the datasets used for training and for evaluation of these tools, which may lead to overly optimistic results.
, 2010] ) or to quantify the overall pathogenic potential of a variant based on diverse types of genomic information (e. g. , CADD  [Kircher et al. 2014] ). Note that SIFT is both a measure of sequence conservation and provides an analytically derived threshold for predicting whether or not protein function will be affected  [Ng and Henikoff, 2003]. Furthermore, popular benchmark datasets for pathogenicity prediction differ in the way they define the pathogenic or neutral character of a variant (see Table  2 ). For instance, neutral variants are supposed to have a minor allele frequency larger than 1% in HumVar  [Adzhubei et al. , 2010] , of less than 1% in ExoVar  [Li et al. , 2013] , and of more than 40% in VariBench  [Thusberg et al. , 2011; Nair and Vihinen, 2013]. Given this wealth of different methods and benchmarks that can be used for pathogenicity prediction, an important practical question to answer is whether one or several tools systematically outperform all others in prediction accuracy.
To address this question, we comprehensively assess the performance of 10 tools that are widely used for pathogenicity prediction: MT2  [Schwarz et al. , 2014] , LRT  [Chun and Fay, 2009] , PP2  [Adzhubei et al. , 2010] , SIFT  [Ng and Henikoff, 2003] , MASS  [Reva et al. , 2011] , FatHMM-W and FatHMM-U  [Shihab et al. , 2013] , CADD  [Kircher et al. , 2014] , phy-loP  [Cooper and Shendure, 2011] , and GERP++  [Davydov et al. , 2010]. We evaluate performance across major public databases previously used to test these tools  [Adzhubei et al. , 2010; Mottaz et al. , 2010; Thusberg et al. , 2011; Li et al. , 2013; Nair and Vihinen, 2013; Bendl et al. , 2014]  and show that two types of circularity severely affect the interpretation of the results. Here, we use the term "circularity" to describe the phenomenon that predictors are evaluated on variants or proteins that were used to train their prediction models.
While a number of authors have acknowledged the existence of one particular form of circularity before (stemming specifically from overlap between data used to develop the tools and data upon which those tools are tested)  [Adzhubei et al. , 2010; Thusberg et al. , 2011; Nair and Vihinen, 2013; Vihinen, 2013] , our study is the first to provide a clear picture of the extent and impact of this phenomenon in pathogenicity prediction. The first type of circularity we encounter is due to overlaps between datasets that were used for training and evaluation of the models. Tools such as MT2  [Schwarz et al. , 2014] , PP2  [Adzhubei et al. , 2010] , MASS  [Reva et al. , 2011] , and CADD  [Kircher et al. , 2014] , which require a training dataset to determine the parameters of the model, run the risk of capturing idiosyncratic characteristics of their training set, leading to poor generalization when applied on new data.
514 HUMAN  MUTATION, Vol. 36, No. 5, 513-523, 2015   et al. , 2013; Li et al. , 2013; Bendl et al. , 2014]. Notably, this type of circularity, which we refer to as type 1 circularity, could cause spurious increases in prediction accuracy for both single tools and combinations of tool scores. The second type of circularity, which we refer to as type 2 circularity, is closely linked to a statistical property of current variant databases: often, all variants from the same gene are jointly labeled as being pathogenic or neutral. As a consequence, a classifier that predicts pathogenicity based on known information about specific variants in the same gene will achieve excellent results, while being unable to detect novel risk genes, for which no variants have been annotated before. Further, it will not be able to perform another critical function: discrimination of pathogenic variants from neutral ones within a given protein.
Datasets and Data Preprocessing In this study, we used five different datasets to assess the performance of available prediction tools and their combinations. We used publicly available and commonly used benchmark datasets: HumVar  [Adzhubei et al. , 2010] , ExoVar  [Li et al. , 2013] , VariBench  [Thusberg et al. , 2011; Nair and Vihinen, 2013] , predictSNP  [Bendl et al. , 2014] , and the latest SwissVar (December 2014) database  [Mottaz et al. 2010 ] (Table  3 ). As these tools can require either nucleotide or amino acid substitutions as input, we used Variant Effect Predictor (VEP)  [McLaren et al. , 2010]  to convert between both formats. We excluded all variants for which we could not determine an unambiguous nucleotide or amino acid change. Note that by contrast, analyses such as that of  Thusberg et al.
(2011)  only assess tools that require amino acid changes as input. As the intersection of the training data from the tool CADD  [Kircher et al. , 2014]  and that of all other datasets is small (fewer than a hundred variants), we systematically excluded all variants overlapping with the CADD training data from all other data sets. The VariBench dataset (benchmark database for variations) was created  [Thusberg et al. , 2011; Nair and Vihinenm 2013]  to address the problem of type 1 circularity. However, while the pathogenic variants of this dataset were new, its neutral variants may have been present in the training data of other tools. HUMAN  MUTATION, Vol. 36, No. 5, 513-523, 2015   VariBench has an overlap of approximately 50% with both HumVar and ExoVar (Supp. Fig. S1 ). We kept the nonoverlapping variants to build an independent evaluation dataset, which we called VariBench-Selected and make available along with this manuscript (Supp.
Data S1). From the predictSNP benchmark dataset, we systematically excluded all variants that overlap with HumVar, ExoVar, and VariBench and called the resulting dataset predictSNPSelected. Eventually, we created a fifth dataset, SwissVarSelected. Here, we excluded from the latest SwissVar database (December 2014) all variants overlapping with the other four datasets -HumVar, ExoVar, VariBench, and predictSNP. Thus, SwissVarSelected should be the dataset containing the newest variants across all datasets. With one possible exception, none of the prediction tools or conservation scores we investigated in this manuscript were trained on VariBenchSelected, predictSNPSelected, or SwissVarSelected. The exception is that some variants in the selected datasets may overlap partially with variants used to train MT2  [Schwarz et al.
Data Availability and Reproducibility of Results For each dataset, we compiled comma-separated files containing all obtained tool scores and predicted labels as well as information about the variant (true label, nucleotide and amino acid changes, minor allele frequencies, UniProt accession IDs  [Magrane and Consortium, 2011] , Ensembl gene, transcript and protein IDs  [Flicek et al. 2014] , and dbSNP IDs (rs#) if available  [Sherry et al. , 2001] ). All these datasets including the tool predictions can be found at the VariBench Website (http://structure. bmc. lu. se/VariBench/GrimmDatasets. php) as well as a single Excel file at the journals Website (Supp. Data S2). Further, we provide all Python scripts used to generate all tables and figures in this study along with all datasets, compressed as a single ZIP file (Supp.
Performance Evaluation To evaluate the performance of all the tools in this study, we used a collection of statistics derived from a confusion matrix. To this end, we counted a correctly classified test point as a true positive (TP) if and only if the test point corresponds to the positive class (pathogenic or damaging) and as a true negative (TN) if and only if the test point corresponded to the negative class (neutral or benign). Accordingly, a false positive (FP) is a negative test point that is classified to be positive and a false negative (FN) a positive test point classified as a negative one. Since a few datasets are slightly unbalanced, we assessed the performance of the single tools by computing receiver operating characteristic curves (ROC-curves). Furthermore, we computed Precision-Recall curves (ROC-PR-curves)  [Davis and Goadrich 2006].
We performed a 10-fold cross-validation on the five datasets HumVar  [Adzhubei et al. , 2010] , ExoVar  [Li et al. , 2013] , VariBenchSelected  [Thusberg et al. , 2011; Nair and Vihinen, 2013] , predictSNPSelected  [Bendl et al. , 2014] , and SwissVarSelected  [Mottaz et al. , 2010]  (see Table  3  and the Methods section). For this purpose, we randomly split the dataset of interest into 10 subsets of equal size (to the extent possible). Simultaneously, we kept the ratio of neutral to pathogenic variants the same across all subsets. This avoids generating a biased subset containing only variants of the same kind. We then combined nine subsets to train the model and used the remaining one to assess the performance (testing). We repeated this cross-validation procedure 10 times. Because we were using a regularization term, we had to find a trade-off between the matching model on the training set and the best generalization.
Protein Majority Vote To analyze how protein-related features can influence the performance of the prediction tools, we performed a protein majority vote (MV). For each of the five evaluation datasets, we split the benchmark into 10 subsets, and for each of the subsets, used the union of the nine other subsets as training data. Within that framework, we scored a variant by the pathogenic-to-neutral ratio, in the training data, of the protein that variant belongs to. If the protein did not appear in the training data, we assigned a score of 0. 5. This strategy, while statistically effective on the currently existing databases, is not appropriate, as it cannot discriminate between neutral and pathogenic variants within the same protein.
Evaluation of 10 Pathogenicity Prediction Tools on Five Variant Datasets We evaluated the performance of eight different prediction tools: MT2  [Schwarz et al. , 2014] , LRT  [Chun and Fay, 2009] , PP2  [Adzhubei et al. , 2010] , SIFT  [Ng and Henikoff, 2003] , MASS  [Reva et al. , 2011] , FatHMM-W and FatHMM-U  [Shihab et al. , 2013] , and CADD  [Kircher et al. , 2014]  as well as two conservation scores: phyloP  [Cooper and Shendure, 2011]  and GERP++  [Davydov et al. , 2010 ]. An overview of these tools and conservation scores can be found in Table  1. Details on how these scores were obtained can be found in the Methods section. We evaluated these tools using a range of preprocessed public datasets and subsets of VariBench, predictSNP, and SwissVar (see Methods), resulting in five evaluation datasets  [Adzhubei et al.
, 2010; Mottaz et al. , 2010; Thusberg et al. , 2011; Li et al. , 2013; Nair and Vihinen, 2013; Bendl et al. , 2014]  (see Methods; Table  3 ; Supp. Fig. S1 ). Importantly, two of these datasets (HumVar  [Adzhubei et al. , 2010]  and ExoVar  [Li et al. , 2013] ) overlap with at least one of the training sets used to train the individual tools FatHMM-W, MT2, MASS, and PP2 (Supp. Fig. S1 ; Table  3 ). The selected datasets can be considered to be truly independent evaluation datasets, which are free of type 1 circularity (see Methods). We report AUC values per tool and per dataset in Figure  1  and Supp. Table  S1  (corresponding ROC, PR curves, AUC/AUC-PR values as well as other evaluation metrics can be found in Supp. Figs. S2-S6 and Supp. Table  S1 ). Hatched bars in Figure  1  indicate that the evaluation data were used in part or entirely to train the corresponding tool; these results may suffer from overfitting.
Dotted bars indicate that the tools are biased, due to type 2 circularity (see section "Type 2 Circularity as an Explanation of the Good Performance of FatHMM Weighted"). Five central observations can be made in Figure  1 : first, on the two benchmarks HumVar and ExoVar, the four best performing methods were fully or partly trained on these datasets. Second, while MT2, PP2, and MASS outperform CADD and SIFT on benchmarks that include some of their training data (HumVar, ExoVar), this is not the case on the independent VariBenchSelected and predictSNPSelected datasets. A potential explanation is that type 1 circularity -that is overlap between training and evaluation sets -might lead to overly optimistic results on the first two datasets. Third, across the first four datasets, FatHMM-W outperforms all other tools (Supp.
Type 2 Circularity as an Explanation of the Good Performance of FatHMM Weighted The superiority of FatHMM-W's  [Shihab et al. , 2013]    performance on SwissVarSelected made us investigate its underlying model to find the reason for its superior performance on all but one dataset. FatHMM-W's weighting scheme attributes higher scores to amino acid substitutions in proteins, related proteins, or domains that already include a high fraction of pathogenic variants (see Supp. Text S1). A key element of this weighting scheme is the use of the two parameters Wn and Wd, which represent the relative frequency of neutral variants (Wn) and pathogenic variants (Wd) in the relevant protein family, defined through SUPERFAMILY  [Gough et al. , 2001]  or Pfam  [Sonnhammer et al. , 1997]. To further evaluate the role of this weighting in the performance of FatHMM-W, we compared the original FatHMM-W with a logistic regression over the features (ln(Wn) and ln(Wd)) in a 10-fold cross-validation on the selected datasets (see Methods and Supp.
Text S1). The use of these features alone was sufficient to achieve approximately the same predictive performance as FatHMM-W (see Supp. Table  S2  and Fig. 1 ). Given that the ratio of neutral and pathogenic variants in the same protein family is the key feature used by FatHMM-W, we further analyzed how an even simpler statistic -the fraction of pathogenic variants in the same protein -performs as a predictor. We refer to this predictor as a protein MV (see Methods). MV systematically outperforms FatHMM-W (Supp. Table  S2 ; Fig. 1 ). The pathogenicity of neighboring variants within the same protein is therefore the best predictor of pathogenicity across these datasets. This strategy, while statistically effective on the currently existing databases, is not appropriate. Indeed, it assigns the same label to all variants in the same protein, based on information likely obtained at the protein level (i.
e. , that it is associated with a disease), and cannot distinguish between pathogenic and neutral variants within the same protein. To better understand the outstanding performance of FatHMM-W and the protein-based MV, we examined the relative frequency of pathogenic variants across proteins in all our datasets. In the independent evaluation dataset VariBenchSelected, we found that more than 98% of all proteins (4,425 out of 4,490; Table  4 ) contain variants from a single class, that is, either "pathogenic" or "neutral" (Fig. 2A ). For the remainder of the manuscript, we shall refer to proteins with only one class of variant as "pure" proteins (divided in "pathogenic-only" proteins and "neutral-only" proteins). The existence of such "pure" proteins -while theoretically pos-sible -should not be interpreted as a biological phenomenon.
Rather, these designations are based on current knowledge, and are at least partially an artifact of how these particular datasets are populated. Nearly all (94. 8%) variants in VariBenchSelected are located in pure proteins with 57. 2% in neutral-only proteins and 37. 6% in pathogenic-only proteins (Fig. 2B ). On such a dataset, excellent accuracies can be achieved by predicting the status of a variant based on the other variants in the same protein. We refer to this phenomenon as type 2 circularity. The remaining 5. 2% of VariBench-Selected variants are located in "mixed" proteins (Fig. 2B  and C ), which contain both pathogenic and neutral variants (pathogenicto-neutral ratio in the open interval ]0. 0, 1. 0[ (in Fig. 2C ). While the MV approach will necessarily misclassify some of these variants, it will still perform well on proteins containing primarily neutral or primarily pathogenic variants, and overall, only 0.
7% of all variants are in proteins containing an almost balanced ratio of pathogenic and neutral variants (pathogenic-to-neutral ratio in the interval [0. 4, 0. 6] in Fig. 2C ). Similar dataset compositions can be observed in the other three datasets HumVar, ExoVar, and predictSNPSelected (Supp. Figs. S7-S9 ). A striking property of SwissVarSelected is its much larger fraction of proteins with almost balanced pathogenicto-neutral ratio: 6. 5% of all variants (832 out of 12,729) can be found in the most balanced category of mixed proteins [0. 4, 0. 6] (see Supp. Fig. S10 ), compared with an average of 1. 5% in the other four datasets. To further understand FatHMM-W's performance, we evaluated it separately on mixed proteins. As shown in Figure  3 , Supp. Figures  S11  and S12 , FatHMM-W performs well on pure proteins but loses much of its predictive power on the mixed proteins, as it is misled by its weighting scheme.
On almost-balanced proteins, FatHMM-W is therefore outperformed by all other tools but phyloP (Fig. 3 ). This may also be the first reason why FatHMM-W performs worse on SwissVarSelected than on all other datasets: SwissVarSelected contains many more variants in the most mixed categories (Fig. S10 ). FatHMM-W even performs poorly on mixed proteins from its own training dataset (Supp. Fig. S11 ). We observed that PolyPhen-2 outperforms all other tools in the mixed categories for the datasets predictSNPSelected and SwissVarSelected (Supp. Fig. S12 ). For the VariBenchSelected dataset, no clear winner can be determined (Supp. Fig. S12 ).
518 HUMAN  MUTATION, Vol. 36, No. 5, 513-523, 2015   Overview about the total number of proteins per dataset and the composition of these datasets. The second reason for the drop in performance is the presence of "new" proteins in SwissVarSelected that are unknown to the FatHMM-W weighting database. To show this, we used the HumVar and ExoVar datasets as a proxy for the training data among all our tools (FatHMM's training data is not fully publicly available). We observed that approximately 91% of all pathogenic and approximately 68% of all neutral variants in VariBenchSelected are located in proteins that also occur in HumVar/ExoVar (Supp. Fig. S13  and Supp. Table  S3 ). As FatHMM-W makes use of information from protein families, we computed pairwise BLASTP  [Camacho et al. , 2009]  alignments between all proteins in our selected datasets and proteins in HumVar/ExoVar.
Approximately 99% of all pathogenic variants in VariBenchSelected are located in proteins from Hum-Var/Exovar or proteins with more than 70% sequence similarity to a protein in HumVar/Exovar. Similar statistics can be observed for predictSNPSelected (Supp. Fig. S13  and Supp. Table  S4 ). However, for SwissVarSelected, we observed that only approximately 61% of all pathogenic and approximately 56% of all neutral variants belong to proteins from HumVar/ExoVar (Supp. Fig. S13  and Supp. Table  S5 ). Approximately 78% of all pathogenic and approximately 77% of all neutral variants in SwissVarSelected are located in proteins from HumVar/ExoVar or in proteins with high-sequence similarity (70% sequence similarity) to a protein from HumVar/ExoVar (Supp. Fig. S13  and Supp. Table  S5 ). Hence, a significant proportion of SwissVarSelected variants cannot be found in proteins from the proxy training dataset or proteins with high-sequence similarity.
Evaluation of Two Combined Predictors After studying in silico tools for pathogenicity prediction, we compared the performance of two methods that combine individual tools into a joint prediction, the metapredictors Condel [González-Pérez and L ópez-Bigas, 2011] and Logit  [Li et al. , 2013]. Based on our previous findings, we were interested in how their performance compares when evaluated on datasets that avoid type 1 circularity. Furthermore, we wanted to compare the performance of metapredictors that include FatHMM-W and may suffer from type 2 circularity, to metapredictors that do not include FatHMM-W. Condel's Web server provides two combinations of tool scores: the older (original) version combines PP2  [Adzhubei et al. , 2010] , SIFT  [Ng and Henikoff, 2003] , and MASS  [Reva et al.