dataesr/scientific-paragraphs-categorization huggingface dataset:
"Pure" indicates variants belonging to proteins containing only one class of variant. (x and y) indicate variants belonging to mixed proteins, containing a ratio of pathogenic-to-neutral variants between x and y. ]0. 0, 1. 0[ therefore indicate all mixed proteins (the ratios of 0. 0 and 1. 0 being excluded by the reversed brackets). While FatHMM-W performs well or excellently on variants belonging to pure proteins (VariBenchSelected and predictSNPSelected), it performs poorly on those belonging to mixed proteins. To avoid type 1 circularity, we chose the selected datasets as our evaluation datasets, as they do not overlap with the training dataset of any individual tool or metapredictor. Our results using Logit on all selected datasets confirm those reported by  Li et al. (2013) : Logit outperforms all individual tools and Condel in terms of AUC (Fig.
4 ; Supp. Figs. S14-S18 ). Condel's performance (AUC = 0. 70) is on par with SIFT (AUC = 0. 70) for VariBenchSelected, the best performing of the tools it combines. We then evaluated the combination of tools on the pure and mixed proteins on VariBenchSelected, predictSNPSelected, and SwissVarSelected (Supp. Fig. S19 ). While Logit performs well on the pure proteins, Condel performs at least as well as Logit on variants in mixed proteins. The evaluation of Condel+ and Logit+ may be optimistically biased by type 2 circularity, given the inclusion of FatHMM-W. Across all datasets, we observed that adding FatHMM-W to either tool score combination (Condel+ or Logit+) led to a performance boost (Fig. 4 ; Supp. Figs. S14-S18 ). However, this did not hold for mixed proteins, providing strong evidence for type 2 circularity.
Discussion The wealth of pathogenicity prediction tools proposed in the literature raises the question whether there are systematic differences in the quality of their predictions when evaluated on a large number of variant databases. In an attempt to answer this question, we performed a comparative evaluation of pathogenicity prediction tools and demonstrated the existence of two types of circularity that meaningfully impair comparison of in silico pathogenicity prediction tools. We showed how ignoring these effects could lead to overly optimistic assessments of tool performance. One severe consequence of this phenomenon is that it may hinder the discovery of novel disease risk genes, as these tools are widely used to choose variants for further functional investigation. In this manuscript, we have described and demonstrated "type 1" and "type 2" circularity.
Type 1 circularity occurs because of an overlap between training and evaluation datasets, possibly resulting in overfitting  [Hastie et al. , 2009] , meaning that a tool is highly tailored to a given dataset, but will perform worse on novel variants. Type 1 circularity is a well-known and studied phenomenon in machine learning  [Hastie et al. , 2009]  and there are guidelines on how it can be avoided  [Vihinen, 2013]. To avoid type 1 circularity, we built the Selected datasets, in which none of the variants have previously been seen by any of the tools (with the possible and unavoidable exception of MT2). This makes them the most appropriate datasets on which to draw conclusions regarding the relative performance of the tools. Figure  1  and Supp. Table  S1  suggest that MT2, PP2, and MASS overfit on their training data and have rather weak generalization abilities.
Our efforts to understand the outstanding performance of FatHMM-W on four out of five datasets led to additional insights about type 2 circularity. Our findings about type 2 circularity demonstrate that predicting the pathogenicity of a variantbased on the pathogenicity of all other known variants in the same protein -is a statistically successful, but ultimately inappropriate strategy. It is inappropriate because it will often fail to correctly classify variants in proteins that contain both pathogenic and neutral variants. Further, it will often fail to discover pathogenic variants in unannotated proteins. In the "Selected" datasets, Logit provides the best unbiased performance. SIFT has the lowest performance in the HumVar and ExoVar datasets, but it is also the only predictor that is unbiased in these two datasets.
520 The apparent success of this strategy is due to the fact that, in variant databases, it is frequently the case that all the variants of the same protein are annotated with the same status. Furthermore, pathogenic-only proteins contain many more labeled variants than neutral-only proteins. In these databases, pathogenic amino acid substitutions are heavily concentrated in a few key genes (Fig. 2 ; Supp. Figs. S7-S9 ). These observations regarding the distribution of variants in our datasets likely result -in part -from research practices relevant to the way variant databases are populated. Often, an initial discovery of a pathogenic variant in a gene (for a given disease) leads to additional discoveries of pathogenic variants in the same gene, in part because a given gene will be more heavily investigated once it has been identified as harboring pathogenic variants.
These properties of variant databases explain why we observe that pathogenicity can be predicted from the annotation of variants within the same protein by a MV with excellent accuracies (Fig. 1 ; Supp. Table  S2 ). They also explain why FatHMM-W, whose predictive power is driven by the pathogenic-to-neutral ratio of variants in the same protein, performs so well on VariBenchSelected and predict-SNPSelected (Fig. 1 ; Supp. Table  S2 ). This approach, however, will often fail to correctly classify amino acid substitutions in proteins that contain both pathogenic and neutral variants (Fig. 3 ; Supp. Figs. S11  and S12 ). This partially explains why FatHMM-W performs worse on SwissVarSelected than on all other datasets, as it contains many more variants in the most mixed categories (Fig. 3 ; Fig.
S10 ). The same phenomenon also occurs when building metapredictors: the performance of Logit+ and Condel+ are similarly optimistically biased because they contain FathHMM-W. The pervasiveness of circularity makes it difficult to draw definitive conclusions regarding the relative performance of these prediction tools. We do nevertheless observe a reassuring trend for tools trained for the purpose of predicting pathogenicity to outperform conservation scores. In addition, the Logit combination of SIFT, PP2, and MASS is a slightly better predictor of pathogenicity than any of these tools taken individually. Also, for the mixed proteins in particular, Condel performs better than, or is on par with, Logit across the datasets (Supp. Fig. S19 ). It is important to note that the drop in performance that we observe when applying a method to a dataset (that it was not trained on) could also be due to the different definitions of pathogenicity and neutrality used in the different benchmark datasets (see Table  2 ).
It should be an important goal of future studies to quantify this impact. However, as long as circularity exists in a comparative study, it will mask the effect of these differences in the definition of pathogenicity: in Figure  1 , FatHMM-W seems to provide excellent prediction results across four different benchmark datasets, irrespective of the different definitions of pathogenicity and neutrality. However, our analysis shows that the true origin of this superior performance is type 2 circularity and not robustness to different definitions of pathogenicity and neutrality. Therefore, a key step in future studies examining this problem will be to avoid any type of circularity. The existence of these types of circularity has immediate implications for the further development and evaluation of pathogenicity prediction tools: at the very least, and as recommended previously  [Vihinen, 2013] , prediction tools should only be compared on benchmarks that do not overlap with any of the datasets used to train the tool.
We provide such datasets, VariBenchSelected, predictSNPSelected, and SwissVarSelected (see Supp. Data S1) for this kind of independent evaluation. All prediction tools should make their training datasets public, as type 1 circularity cannot be excluded if any portion of a training dataset is kept private. A more rigorous strategy would be to retrain all predictors on the same dataset, in order to truly evaluate the predictors and not the quality of their training datasets. However, this is only possible if the raw variant descriptors (variant features) -from which the tools derive their predictions -are made available. We investigated all tools presented here and only for PP2 was it straightforward to obtain these descriptors  [Adzhubei et al. , 2010]. To address the problem of type 2 circularity, it is imperative that future studies report prediction accuracy as a function of the  HUMAN MUTATION, Vol.
36, No. 5, 513-523, 2015  pathogenic-to-neutral ratio, as in Figure  3. An even better strategy would be to stratify training and test datasets such that variants from the same protein only occur in either the training or the test dataset, completely removing the possibility of classification within the same protein  [Adzhubei et al. , 2010]. Furthermore, one could construct predictors for different classes of variants, defined by the pathogenic-to-neutral ratio in the proteins, to address the problem that none of the existing tools achieves constantly good results across these classes on VariBenchSelected. Both these alternative approaches, however, rely critically on the availability of raw descriptors used by the corresponding tools. Finally, there is another potential source of circularity to beware of in the future: The novel variants entered in databases may be annotated using existing pathogenicity prediction tools.
negative control variants. " "Other rare variants" b VariBench "Variation datasets affecting "The pathogenic dataset of 19,335 missense mutations "This is the neutral dataset or nonsynonymous coding SNP protein tolerance" c obtained from the PhenCode database downloaded in dataset comprising 21,170 human nonsynonymous June 2009), IDbases and from 18 individual LSDBs. coding SNPs with allele frequency 40. 01 and chromosome For this dataset, the variations along with the variant sample count 449 from the dbSNP database build 131. position mappings to RefSeq protein (> = 99% match), This dataset was filtered for the disease-associated SNPs. RefSeq mRNA, and RefSeq genomic sequences are The variant position mapping for this dataset was available for download. " c extracted from dbSNP database.
" c predictSNP "Benchmark dataset used for the Disease-causing and deleterious variants from SwissProt, Neutral variants from SwissProt, HGMD, HumVar, evaluation of. prediction tools HGMD, HumVar, Humsavar, dbSNP, PhenCode, Humsavar, dbSNP, PhenCode, IDbases, and 16 individual and training of consensus IDbases, and 16 individual locus-specific databases. locus-specific databases. classifier PredictSNP" d SwissVar "Comprehensive collection of "A variant is classified as disease when it is found in "A variant is classified as polymorphism if no disease single amino acid patients and disease association is reported in association has been reported" f polymorphisms (SAPs) and literature. However, this classification is not a diseases in the definitive assessment of pathogenicity" f UniProtKB/Swiss-Prot knowledgebase" e For each dataset, the first column shows the general purpose.
The last two columns describe the positive and negative control categories of variants. a http://genetics. bwh. harvard. edu/pph2/dokuwiki/overview b http://www. plosgenetics. org/article/info%3Adoi%2F10. 1371%2Fjournal. pgen. 1003143 c http://structure. bmc. lu. se/VariBench/tolerance_dataset1. php d http://www. ploscompbiol. org/article/info%3Adoi%2F10. 1371%2Fjournal. pcbi. 1003440 e http://bioinformatics. oxfordjournals. org/content/26/6/851. long f http://swissvar. expasy. org/cgi-bin/swissvar/documentation different tools have been trained on different datasets, increasing the likelihood that variants in the evaluation set appear in at least one of these datasets [González-Pérez and L ópez-Bigas, 2011; Capriotti.
• It may be expressed in radian, from -π to π; • It may be expressed in [-1, 1), in which case the function being computed is 1 π atan2(y, x). Throughout this article, we will refer to this option as "binary angles" There are several reasons for preferring binary angles. The first is that it fully uses the representation space. An output on [-π, π] will be coded on a fixed-point format This work is partly supported by the MetaLibm project (ANR-13-INSE-0007) of the French Agence Nationale de la Recherche. that may represent the interval  [-4, 4). Therefore, all the codes between π and 4 will never be used. The second is that it slightly simplifies implementation: the modulo-2π periodicity of radian angles becomes a modulo-2 periodicity that comes for free in two's complement binary arithmetic.
QD44 Quaternary Retrieving literature with existing MeSH descriptors can be sometimes a big challenge for the above incisive concepts in the field of GP/FM. 5 Hence, we have decided to explore the literature about those domains (Table  1 ) and report the most popular words in Family Medicine, echoing the research in the same domain by epidemiologists and hospital based specialists. 6 The help of the Rouen Institute for Research and Innovation in Biomedicine team 7 has been seminal in developing the relationships of those concepts with the appropriate methods, paving the way of health care semantics. The impact of computer science on terminological issues is striking and family doctors have to understand that mastering the production and management of knowledge lies at their doorstep. They have to become familiar with semantic web technologies and related language processing as the computer will become omnipresent in their daily lives in a near future.
The interface allows the user to build a terminological record by entering an ID (identification number) following by the Preferred label of a term, the synonyms and acronyms, if any, the most appropriate definition with bibliographic citation, the links to Babelnet. org and to Dbpedia or to other relevant semantic links. Internal links to already existing terminologies in hetop. eu can be chosen manually (in grey in the Figure  1 ) or automatically proposed by the interface which allows corrections (in red in the Figure  1 ). All the gathered data can be expressed in Web Ontology Language (OWL),  19  the computer language used in the semantic web for Health Care and Life Sciences. 20 ealthcare delivery systems, investigators and healthcare providers may be denying their patients opportunities to make informed decisions about their health and healthcare.
Results The ten terms related to quaternary prevention are presented here in four languages with the links, definition(s) and their source(s) and bibliographic citation(s). Table  2  is completed by the corresponding MeSH, if any, with the MeSH definition, the count of the term in Pubmed (January 2015) and the date of first citation. Lastly, the links to Babelnet. org and Dbpedia are shown as the main connection to lay languages.
SDM is an important part of patient-centered care; education is often through the use of decision aids such as pamphlets, videos, and computerized tools. (Cochrane, Légaré, 2010). Bib. Citations Légaré F, Ratté S, Stacey D, Kryworuchko J, Gravel K, Graham ID, et al. Interventions for improving the adoption of shared decision making by healthcare professionals. Cochrane database Syst Rev. 2010; (5):CD006732. Available from: http://www. ncbi. nlm. nih. gov/pubmed/20464744.
The effective conductivity is computed using the voxel based software Geodict [14] (c). In parallel, providing that the microstructure has been generated numerically, the effective conductivity can be computed by a resistance network method (d). The numerical microstructure is generated using 1000 randomly placed particles with an average diameter of 1 lm ± 5%. This loose packing is compressed to a 0. 5 relative density, which is a typical value for green powders before sintering  [11]. The relative density d is defined as the ratio between the total volume of particles and the volume of the simulation box: d ¼ P npart V part V box ð1Þ where n part is the number of particles, V part the volume of a particle and V box the volume of the simulation box. During jamming, contacts between particles appear and at 0.
Microstructure characteristics Numerical microstructures are read as 3D images and analyzed using image analysis software (ImageJ) from which several microstructural properties can be extracted (volume fractions, interfacial area density, connectivity and TPB length). The composition of a material is generally known before 3D reconstruction. Therefore, the volume fraction of the different phases may be used to verify that the analyzed volume has been segmented correctly and is representative of the larger sample. Volume fractions are simply calculated by counting the voxels that correspond to each phase. In a composite material, reactions often occur along interfaces. Thus a larger surface area per unit volume is generally associated with higher activity. The interfacial area density is expressed as the ratio between surface area and the volume of the sample (lm À1 ).
Effective conductivity The GeoDict software package  [17]  was used to compute the effective conductivity from our microstructures. GeoDict is based on Fast Fourier Transform and the Bi-CGSTAB method  [28]. Each voxel is assigned an intrinsic conductivity and Ohm's law is applied. This law derived from Fourier's law states that the amount of species (i. e. electrons or ions) that flow through a unit area per unit time is proportional to the negative local potential gradient.
One hundred microliter aliquots of these extracts were injected onto the Waters Alliance HPLC system (Saint-Quentinen-Yvelines, France) for the quantitative determination of the patulin concentration. The patulin was detected with a Waters 2998 Photodiode Array Detector, using a 25 cm × 4. 6 mm Supelco 5 μm Discovery C18 HPLC Column (Sigma-Aldrich) at a flow rate of 1 mL/min. A gradient program was used with water (Eluent A) and acetonitrile HPLC grade (eluent B) and the following elution conditions: 0 min 5% B, 16 min 2% B, 20 min 60% B, 32 min 5% B. The presence of patulin was monitored at a 277 nm wavelength. A calibration curve was constructed with patulin standard (Sigma-Aldrich) at concentrations ranging from 0. 05 to 10 μg/mL. Accordingly, the patulin concentrations were determined and results were expressed in ppm.
Model development Growth rate experimental data were implemented in a home developed C++ language program that is able to interpolate between various points in different or multiple dimensions. Effects of the different parameters (temperature, pH, and a W ) on the P. expansum growth rate were taken into account according to the experimental points already obtained. Thus, the effect of each of these parameters was considered on its own calculating the growth ratio factor effect obtained from the experimental data. The program proceeds as a simultaneous interpolator between the different data points and the effect ratios of each parameter on the growth rate. Therefore, the program allows us to estimate the growth rates (expressed in cm/ day) for fixed temperature, a W, and pH values depending on the variations defined by the input data.
Mathematical modeling of P. expansum growth The growth data modeled in this work comprised the latency phase and the growth curves of P. expansum strain NRRL 3565 at five temperatures, four a W , and three pH. Mycelial extension of colonies against time was almost invariable showing a straight line, after an initial lag period. The growth rates expressed in cm per day were calculated as the slope of the regression curves. The growth rates recorded under the different conditions (data presented above) were used as inputs to calculate the design parameters of equations (3-7). The presented calculation approach has undergone a first mathematical validation, confirming that the differences between the theoretical values predicted by the model and the data obtained under the conditions used to build the model are not significant.
Wherever it is not possible to have human invigilators, a software running on the student computer is used, e. g. , ProctorU  10. However, such measures are insufficient, as the trustworthiness and the reliability of exams are today threatened not only by candidates. Indeed, threats and errors may come from the use of information technology, as well as, from bribed examiners and dishonest exam authorities which are willing to tamper with exams as recent scandals have shown. For example, in the Atlanta scandal, school authorities colluded in changing student marks to improve their institution's rankings and get more public funds  [2]. The BBC revealed another scandal where ETS was shown to be vulnerable to a fraud perpetrated by official invigilators in collusion with the candidates who were there to get their visas: the invigilators dictated the correct answers during the test  [3].
Our formalization also allows us for some properties to detect the cause of the potential failures and the party responsible for them. Note, formalizing the above properties entailed to add features to QEAs. Then, we implement the monitors using MarQ 11   [6] , a Java tool designed to support QEA specification language. Finally, we perform off-line monitoring, based on the available data logs, for an e-exam organized by UJF; and reveal both students that violate the requirements, and discrepancies between the specification and the implementation. Outline. In Sec. 2, we define the events and a protocol for e-exams. We specify the properties and propose the corresponding monitors in Sec. 3. Then, in Sec. 4, we analyze two actual e-exams organized by UJF. We discuss related work in Sec. 5. Finally, we conclude in Sec.
Events Involved in an E-exam Events flag important steps in the execution of the exam. We consider parametric events of the form e(p 1 ,. , p n ), where e is the event name, and p 1 ,. , p n is the list of symbolic parameters that take some data values at runtime. We define the following events that are assumed to be recorded during the exam or built from data logs. -Event register (i ) is emitted when candidate i registers to the exam. -Event get(i , q) is emitted when candidate i gets question q. -Event change(i , q, a) is emitted when candidate i changes on his computer the answer field of question q to a. -Event submit(i , q, a) is emitted when candidate i submits answer a to question q. -Event accept(i , q, a) is emitted when the exam authority receives and accepts answer a to question q from candidate i.
Case Study: UJF E-exam In June 2014, the pharmacy faculty at UJF organized a first e-exam, as a part of Epreuves Classantes Nationales informatisées project which aims to realize all medicine exams electronically by 2016. The project is lead by UJF and the e-exam software is developed by the company THEIA  12  specialized in e-formation platforms. This software is currently used by 39 french universities. Since then, 1,047 e-exams have been organized and 147,686 students have used the e-exam software. We validate our framework by verifying two real e-exams passed with this system. All the logs received from the e-exam organizer are anonymized; nevertheless we were not authorized to disclose them. We use MarQ  13   [6]  (Monitoring At Runtime with QEA) to model the QEAs and perform the verification.