dataesr/scientific-paragraphs-categorization huggingface dataset:
average performances across the runs are reported along with their standard deviation. Next, we compared our results to previously published ones that we have selected because they were non-biased (Methods: Experimental Protocol (Non-biased)). The comparative results are reported in Table  4. For the colon cancer dataset, our predictive modeling was only outperformed by the random forest (accuracy ∼89% vs. ∼85%) but the sizes of these predictors were five times larger (∼50 vs. ∼10 probesets). For the leukemia dataset, our predictive modeling was not outperformed (accuracy ∼98% with ∼3 variables). Regarding the prostate cancer dataset, our modeling was not outperformed. GLMPath method had the same performances (accuracy ∼94%) and was twice smaller (∼2 vs. ∼4 variables). Finally, our model was not outperformed in both the brain cancer (accuracy ∼68% with ∼9 variables) and the lymphoma datasets (∼88%, ∼5 variables).
Hence, the performances of our predictors were at the level of, or higher than those of the previous studies. Moreover, our signatures were most of the time significantly smaller. A noticeable exception is GLMPath  13  whose performances, even though slightly lower than ours, were obtained with remarkably small signature sizes. Our predictive modeling appeared to be among the two best methods, if not the best one. Beside this result, one should stress that our predictive modeling is very simple, quick to compute on commonly available personal computers, and fully automated (no parameter to tune). Application in breast cancer: molecular signature designed for prediction of preoperative chemotherapy treatments. After having assessed the performances of the predictive modeling we propose, we applied it on two breast cancer datasets: Datasets VI & VII (Methods: Datasets).
Dataset VI comes from a clinical trial conducted at the MD Anderson Cancer Center (Houston, Texas, USA). One of the purposes was to provide data for designing predictors of the response to preoperative chemotherapy treatments in breast cancer. We used the same protocol as in  8   According to our predictive modeling, the probesets were ranked by decreasing contributions to the interclass distance measured on the learning set of 82 patient cases  8  and the signatures were computed in this ranking. In Table  5 , we compared the performances of our predictive modeling to two other recently published classifiers. We computed two signatures containing respectively 30 (δ-DLDA-30) and 11 (δ-DLDA-11) probesets. The δ-DLDA-30 signature was created by taking the 30 top ranked genes according to their δ score, and was built for comparing its performance to the two published classifiers of same size.
The δ-DLDA-11 signature was built by the automated procedure which chooses the optimal size of the signature on the training dataset without human supervision (see Methods). The results of our predictive modeling were compared to two published studies on the same datasets: 1) a predictor using the same DLDA classifier (DLDA-30) designed on the 30 probesets of smallest P-value to a t-test,  8  and 2) a signature unveiled by a majority voting predictor (bi-majority-30) designed on the 30 probesets of highest bi-informative values 14 (Terminology and Abbreviations). The δ-DLDA-11 signature had the best performances on the test dataset, and they were even slightly higher than the best performances ever published for this dataset. The main point was that these performances were achieved with three times less probesets (11 vs.
30). Figure  1  shows heatmaps for each of the four signatures cited above, applied on the testing Dataset VI (51 test samples). We can observe that the signatures unveiled by our method (Panels A&B) clearly dichotomize the responder patients. The up-regulated and down-regulated genes are clearly visible, while this seems less apparent for the two other signatures found in the literature (Panels C&D): the classifier rules would probably be more complicated to analyze/interpret. Interestingly, we found that patients PERU11, PERU14, M120, M353 and M503 were misclassified by all the methods. This can suggest that these five patients form one or several sub-clusters regarding the response to therapy. In order to put more focus on the biological relevancy of the genes unveiled in the δ-DLDA-11 signature, we detailed the characteristics of the 11 selected probesets in Table  6  and plotted the boxplots of their expression levels in Figure  2.
the second column (δ -DLDa-11) contains the results of our predictive modeling obtained without fixing the number of probesets of the signature. the optimal non-singular predictor found by our method contained 11 probesets. the third (DLDa-30  8  ) and fourth (Bi-majority-30  14  ) columns report results found in the literature with the same data/protocol. the DLDA-30 signature nor in the bi-majority-30 one: AGR2, SOX11, and TBC1D9 (see Discussion section). Dataset VII is a second independent dataset of 91 patient cases  16  coming from a cohort subject to the same clinical trial and protocol as Dataset VI. 8  Measurements of the expression levels were conducted with the same microarray (Affymetrix U133A). With this dataset we aimed at validating the performances of the predictors we computed on Dataset VI (82 training data samples).
Hence, we only used this new data-set as a test dataset. The performances of our predictors are reported in Table  7  together with those of the DLDA-30  8  and bi-majority-30  14  signatures. The performances of the δ-DLDApredictors were close to those of the two other predictors. More specifically, the δ-DLDA-11 predictor's specificity was slightly lower than that of the DLDA-30 but the two predictors had the same sensitivity value, and almost the same high negative predictive values, ie, the probability of a partial response given that the prediction is a partial response to the treatment. notes: the four heatmaps represent, for each of the 51 patients of the breast cancer testing set (in columns), the different genes (in rows) of each of the four molecular signatures detailed in this paper.
Detailed caracteristics of the δ -DLDa-11 signature. this table contains descriptions of the 11 probesets of highest contributions to the interclass distance unveiled by our predictive modeling on Dataset VI (rank of the probesets following our prioritization by δ scores, names of the targeted genes, affymetrix references of the probesets, values of their contributions to the interclass distance, and P-values to a t-test). In bold are the genes of the 11 δ -signature that were neither member of DLDa-30 nor bi-majority-30 signatures. Finally, we used the two test datasets to assess more deeply the robustness of the δ-DLDA-11 predictor. In this last experiment, the signature was kept constant while the parameters of the DLDA classifier were tuned at each run of the 3-fold cross-validation procedure (Methods: Cross-Validation).
The results of these two distinct cross-validations are reported in Table  8. Neither DLDA-30  8  nor bi-majority-30  14  signatures reported cross-validation assessment. Both results found on Dataset VI (51 test samples) and Dataset VII (91 test samples) are concordant with (and thus reinforce) the ones found without cross-validation. global assessment of the robustness of the method. We statistically assessed the robustness of our predictive modeling on the 133 cases dataset 8 by means of permutation resampling procedures. We first assessed that our predictor was better than a random predictor. To this end, we performed 1,000 random subsampling cross-validations (Methods: Cross-Validation), and conducted our predictive modeling twice at each run: once on the learning set of samples, leading to a first predictor, and once on the same learning set where the samples were randomly assigned to the classes.
The null hypothesis was "The performances of the two predictors are equal". It was strongly rejected in each of the measures of performance (sensitivity, specificity, accuracy, PPV, NPV; see Terminology and Abbreviations; data not shown). This result demonstrated that the predictive modeling comprising both the signature selection and the classifier model (DLDA), was not random guesses. Next, we assessed that this result was not the consequence of the classifier model alone or, otherwise stated, that the signatures were relevant for the predictions. To this end, we have assessed the optimal signatures by designing another random subsampling cross-validation procedure (1000 runs; Methods: Cross-Validation) conducted on the same dataset. At each run the predictive modeling was conducted, together with the design of a predictor with a random signature of same size.
discussion In this study we proposed a feature selection method by which the molecular signatures were the optima of a bi-objective function expressing the tradeoff between the class discrimination (interclass distance) and the size of the signature. We demonstrated that this optimization was reducible to ranking the probesets by their contributions to the distance between the class centroids. This method of signature selection is non parametric because the contribution to the interclass distance makes no assumption on the distribution of the expression levels. Moreover, we have proposed a fully automated method for computing the optimal sized signature on a training dataset without supervision. In this study we used the DLDA classifier method, following the conclusions of an in-depth study that compared several predictive modelings.
10 where the P-values of some probesets are not higher than those at δ = 0. 30 and over). Regarding molecular signatures, the motivation for selecting subsets of genes whose contributions to the interclass distance are the highest (implying that they are of small In the first column (δ -DLDa-30) are the performances of our predictive modeling with the 30 probesets signature predicted on Dataset VI (for direct comparison with other methods). In the second column (δ -DLDa-11) are the results of our predictive modeling obtained with the optimal non-singular DLDa predictor (11 probesets signature) predicted on Dataset VI. the third (DLDa-30  8  ) and fourth (Bi-majority-30  14  ) columns report the performances of two predictors whose signatures were the 30 probesets of smallest P-values to the student t-test (DLDa-30) and the 30 probesets of highest bi-informativeness (Bi-majority-30).
P-values) is now established: they are the optima of the biobjective function. To our best knowledge no reason was ever given for selecting subsets of genes of smallest P-values with no regard to their contribution the interclass distance. Concerning the breast cancer datasets and the issue of predicting the responses to preoperative chemotherapy treatments, our predictive modeling was at the level of, or outperformed the previous studies, with three times smaller signatures. The highest ranked gene of our signatures was ESR1 (estrogen receptor protein coding gene 1, cf. Table  6 ). Although this gene is known to be one of the most determinant marker of the response to the chemotherapy (eg,  15  ), it was neither selected in 8 nor. 14 In the latter study the probesets were ranked according to their "bi-informativeness".
In its whole generality the problem of feature/gene selection is the optimization of multi-objective functions whose objectives are non-separable. An example of such a multi-objective function is the following three objective function: minimize the signature size, maximize the interclass distance, and minimize the signature's gene correlations. Because of the third criterion this three-objective function is not separable. Addressing the issue of searching optimal molecular signatures in non-separable multi-objective functions might bring significant improvements in oncology prediction. Programs availability: A Web application and scripts written in R language that implement the feature selection procedure can be freely accessed or downloaded from http:// gardeux-vincent. eu/DeltaRanking. php.
conclusions Defining molecular signatures as optima of a bi-objective function as a tradeoff between the interclass distance and the signature size, is relevant. Computing the signatures in the ranking of genes' contributions to the interclass distance is well founded and effective. Our benchmarking showed that the performances of the linear discriminant predictors designed on these optimal signatures were at the level of the best reported studies and that the signatures were significantly smaller than almost all of them. Applying the method on two breast cancer datasets brought very small and biologically relevant molecular signatures. They further helped designing robust and efficient predictors for the response to the chemotherapy.
three-fold cross-validation average performances of the δ -DLDa-11 signature, applied on the two external test datasets. the table contains the results of our predictive modeling obtained with the 11 probesets signature (δ -DLDa-11) predicted on Dataset VI (82 training samples). the predictor was applied on both test datasets following a 3-fold cross-validation experimental protocol (methods: 3-fold cross-validation). average performances and standard deviations are reported.
Software and apparatus Eye position and pupil size were recorded monocularly with an EyeLink 1,000 (SR Research, Mississauga, Ontario, Canada), a video-based eye tracker (1,000 Hz; ellipse mode). Stimuli were presented on a gamma-calibrated 21 ′′ ViewSonic p227f CRT monitor (1,024 × 768, 150 Hz). Testing took take place in a dimly lit room. The experiment was implemented with OpenSesame  (Mathôt, Schreij & Theeuwes, 2012)  using the PsychoPy back-end  (Peirce, 2007)  for display control, and PyGaze  (Dalmaijer, Mathôt & Van der Stigchel, 2014)  for eye tracking.
DATA PREPROCESSING AND STATISTICAL ANALYSES Preprocessing of pupil and eye-position signal All signals were locked to the moment that the eyes crossed the vertical (for horizontal saccades) or horizontal (for vertical saccades) meridian (from now on: mid-saccade point). Mean pupil size from 105 to 95 ms before the mid-saccade point was takes as a baseline, and all pupil size measures are reported in area (i. e. , not diameter) relative to this baseline (cf. Mathôt et al. , 2015). We analyzed pupil size from 300 ms before until 1,200 ms after the mid-saccade point. Pupil size during blinks was reconstructed using cubic-spline interpolation  (Mathôt, 2013). Pupil size was not smoothed. To obtain an acceptable noise level in eye-velocity profiles, eye position was smoothed using an 11 ms Hanning window.
Statistics In this study, we used a hybrid algorithm, called H2PC, to learn the Bayesian network (BN) structure among the 15 candidate variables discussed above. The source code of H2PC and the Bnlearn package in R  [16]  are publicly available. The set of causal assumptions used in this study is described in Table  1. The resulting DAG is interpreted as causal BN. The bootstrapping process has been repeated 200 times and an averaged DAG representing the final Bayesian network was drawn, containing only arcs appearing in at least 25% of the DAGs constructed earlier from the bootstrapped datasets (Fig  1 ). Analysis was performed using R software (v. 2. 13. 0) and Netica Software (v. 4. 16, Norsys Software Corp, Vancouver, Canada). A more comprehensive description of the Bayesian network modelling approach can be found in many published work  [17, 18].
A test of Hosmer and Lemeshow goodness-of-fit was performed  [20] , with a retained threshold for statistical significancy retained at p = 0. 05. Analysis was performed using SAS software (v. 9. 3, SAS Institute Inc. , CARY, NC, USA). A receiver operating characteristic (ROC) curve was plotted to evaluate each model predictive performances  [21]. The area under the ROC curve (AUROC) was then calculated for both logistic regression and BN to assess their overall performances regarding fracture prediction. Each ROC curve was compared using a contrast test  [22]. Sensibility, specificity, positive and negative likelihood ratios and positive and negative predicted value were computed, considering the respective Youden index  [23]  for each model.
Results 7547 women were included in this analysis (51 patients were excluded due to missing values). Characteristics of the population are shown in Table  2. The mean age was 80. 5 years (SD = 3. 8). A total of 289 patients sustained a fracture during the follow-up. Relationships between potential risk factors and fracture were assessed independently by the two approaches (i. e logistic regression and causal Bayesian network) and results are displayed in Table  3. The most probable Bayesian network structure given our data was computed according to the observations and some a priori causal assumptions. These assumptions were encoded in a constraint matrix used in the learning process by the algorithmic approach and are shown in Table  1. In the causal pathways proposed by the final model (Fig  1 ), Age and BMI were found to influence directly the BMD and BMD was found to influence directly the probability of hip fracture occurrence.
TFCE (loc,time) = h(loc,time) h=h0 extent (h) E height (h) H dh (1) The TFCE value at a given location (loc) and time point (time) is the integral of all cluster-extents × cluster-heights from h0 (typically the minimum value in the data) to h (typically the maximum value in the data). Parameters E and H are set to 0. 5 and 2 respectively in LIMO EEG (these choices are discussed below). In practice the integral is estimated as a sum, using finite dh (here dh = 0. 1). As discussed by  Smith and Nichols (2009) , TFCE is a generalization of the cluster-mass statistic (E = 1, H = 0), and can be related to Random Field Theory cluster p-values. In the present study, using data driven simulations, we evaluated the ability of cluster-based computational methods to control the type 1 FWER. The first goal of the study was to establish the equivalence of permutation and bootstrap procedures to control the type 1 FWER, in the context of cluster-mass for ERP.
Methods Codes used to generate and analyze the data are available on FigShare at http://figshare. com/articles/Type 1 error rate using clustering for ERP/1008311. The data generated to compare clustering approaches are too large to be shared (∼32 GB) but the code could be used on any data and similar results are expected. Intermediate results (i. e. FWER data per subject) are nevertheless available. For the TFCE simulations, codes and intermediate data are also available at http://figshare. com/articles/Type 1 error rate using clustering for ERP/1008325.
Cluster-attributes using bootstrap and permutation Whilst simulations aimed at comparing bootstrap and permutation techniques in the context of cluster-mass, we also computed cluster-extent and cluster-height to enquire potential differences. Simulations were performed in the context of a within subject two samples Student's t-test, comparing two hypothetical conditions, but results apply in principle to other within subject cases. Ten subjects of the LIMO EEG dataset  (Rousselet et al. , 2009 ; Q3  Rousselet, 2010)  were randomly chosen as representative ERP data. Data were from a 128 electrodes Biosemi system, 250 Hz sampling rate, with 201 time point epochs ranging from -300 ms to 500 ms. During recording, subjects discriminated between two faces with various levels of noise (see references for details) and performed over 1000 trials.
One thousand Monte Carlo (MC) simulations were performed per subject (10,000 MC in total). For each Monte Carlo and each subject, samples of 10, 25, 50, 100, 300, 500, 900 trials per groups were obtained by increment, e. g. when N = 25, the same 10 trials as with N = 10 were present. For each MC, data were randomly assigned to a condition (say face A versus face B) and a two-sample t-test computed for every electrode and time point. Three techniques were used to estimate the null hypothesis (H0) for the sample considered: (i) a permutation t-test in which all the trials from the two conditions are permuted randomly between conditions and a t-test is computed for each permutation, (ii) a modified percentile bootstrap in which all the trials from the two conditions are first pooled together, then sampled with replacement and randomly assigned to the two conditions and a t-test computed for each bootstrap, (iii) a bootstrap-t in which each condition is first mean centered, then sampled with replacement and a t-test is computed for each bootstrap.
For all three techniques, 1000 iterations were computed and maximal values recorded using cluster-forming thresholds of p = 0. 05 and p = 0. 01. In total 7000 draws of data were computed per subject (1000 MC × 7 sample are now based on cluster attributes, effect sizes can differ substantially from the original effects: (i) with cluster extent, effect-sizes are reversed with the sustained effect being stronger than the transient effect because it has a large support in space and time; (ii) cluster-height preserves effect-sizes but discards spatiotemporal information; (iii) with cluster-mass, effect-sizes are reversed but the difference between the sustained effect and the transient effect is attenuated compared to cluster-extend because cluster-mass accounts for height; (iv) with TFCE effect-sizes are preserved, and in contrast to cluster attributes, the shape of each effect is also preserved.
TFCE validation A different set of data driven Monte Carlo simulations were performed on the same 10 subjects as above. For each subject, 1000 MC samples of 200 trials, 100 trials per condition, were drawn randomly and with replacement from a pool of over 1000 trials, thus mimicking a series of draws from the same population. For each MC sample, a two-sample Student's t-test was computed. For each of these t-tests, the null hypothesis was evaluated using a bootstrap-t technique with 1000 iterations. The observed t values were thresholded using cluster-mass (cluster-forming threshold p = 0. 05) and using TFCE. Since TFCE integrates multiple thresholded maps, the cluster extent and height can have different power leading to different enhanced values (Eq. (  1 )). Here we tested 4 possible combinations of TFCE parameters: extent 0.
In this type of simulation, only the variance-covariance structure has an influence on the results. Although the large scale structure of the covariance matrix (i. e. which electrodes correlate with which, when) depends on the experimental set-up, the smaller scale structure of the covariance matrix (i. e. how trials correlate) is expected to be similar across datasets. We can therefore expect to obtain similar results using other datasets, and we do remind interested readers that the code is available to try on their own data. In contrast to the type 1 FWER tested here, many parameters will affect power. The ability to detect an effect and declare this effect as significant is likely to depend on the SNR, the spatiotemporal sampling, and the statistical method used. In particular, cluster extent and cluster height will be affected by differences in source depth and SNR and it is recommended to use cluster-mass or TFCE in most situations.
Forward primers hybridize to their complementary strand, which is the noncoding strand of template DNA, whereas reverse primers hybridize to the coding strand of template DNA. The degree of specificity of a primer is directly linked to how closely its sequence matches its target sequence. The last five bases at the primer 39 end are particularly crucial for specificity. If the sequence of the targeted gene is known in the species of interest, primer design can be directly performed using software such as Primer3 (http://www. bioinformatics. nl/cgi-bin/primer3plus/primer3plus. cgi). Software programs such as these design and rank primers based on their stability and specificity, taking into account the PCR conditions and input data, such as expected primer size range, expected amplicon size, and possibility for the primers to hybridize together or outside of the targeted region.