dataesr/scientific-paragraphs-categorization huggingface dataset:
Various extensions to the purposeful variable selection procedure are suggested. We propose to use backward elimination augmented with a standardized change-in-estimate criterion on the quantity of interest usually reported and interpreted in a model for variable selection. Augmented backward elimination has been implemented in a SAS macro for linear, logistic and Cox proportional hazards regression. The algorithm and its implementation were evaluated by means of a simulation study. Augmented backward elimination tends to select larger models than backward elimination and approximates the unselected model up to negligible differences in point estimates of the regression coefficients. On average, regression coefficients obtained after applying augmented backward elimination were less biased relative to the coefficients of correctly specified models than after backward elimination.
The remainder of the manuscript is organized as follows: the Methods section will first discuss the change-in-estimate criterion and selection by significance. Later, we will present a new proposal for an efficient algorithm, denoted as augmented backward elimination (ABE), combining both criteria. A SAS macro incorporating the ABE algorithm will be introduced  [13, 14]. The subsequent section summarizes results of a simulation study to evaluate the algorithm. Aspects of application of ABE are discussed by means of a study of progression of chronic kidney disease, including the use of resampling methods for confidence interval estimation and for assessing model stability.
They conclude that re-inclusion after exclusion rarely occurs. Therefore, we consider BE-only with a significance criterion of a~0:2 as the consensus method for significance-based variable selection. There is no statistical justification for univariate screening (  [17, 19]  Chapter 4. 4). Forward selection may sometimes be preferred over BE for practical reasons, e. g. , in very highdimensional variable selection problems. Stepwise selection, e. g. , as implemented in SAS procedures  [13] , is essentially a forward selection with additional backward steps.
Variable Selection Based on Significance and Change-In-Estimate In summary, we propose to use BE augmented with a standardized change-inestimate criterion on the quantity of interest for variable selection. We will denote this algorithm as 'augmented backward elimination' (ABE). The algorithm is briefly outlined in Figure  1. The ABE algorithm has been implemented in a SAS macro  [13] , which is described in more detail in a Technical Report  [14]. The SAS macro can handle continuous, binary and time-to-event outcomes by implicitly applying linear regression using PROC REG, logistic regression using PROC LOGISTIC, or Cox proportional hazards regression using PROC PHREG, respectively. Basically, the ABE macro only needs the following specifications: N Type of model (linear, logistic or Cox) N Name of the outcome variable N Names of the explanatory variables from the initial working set N Roles of explanatory variables from the initial working set ('passive or active', 'only passive', 'only active') N Significance threshold a (default: 0:2) N Change-in-estimate threshold t (default: 0:05) Setting t~? (i.
Simulation Study We evaluated the proposed ABE procedure and compared it to BE, no selection and variable selection based on background knowledge in the setting of an etiologic study. Analyses comprised continuous, binary and time-to-event outcomes and were carried out using our SAS macro ABE. We simulated seven normally distributed potential explanatory variables X 1 ,. ,X 7 among which X 1 was the exposure variable of main interest. A latent outcome variable was defined as Y Ã ~b1 X 1 zX 2 zX 4 zX 7. The covariance structure of X 1 ,. ,X 7 was defined such that omission of X 2 ,X 4 or X 7 , or false inclusion of X 3 could induce bias into the estimate of b 1 , and that a pre-specified variance inflation factor (VIF) of X 1 given X 2 ,. ,X 7 was attained. From Y Ã we generated continuous, binary and time-to-event variables to simulate linear, logistic and Cox regression, respectively.
We have demonstrated how to use resampling-based methods to obtain more reliable interval estimates and to evaluate model stability. We have written a SAS macro ABE, which implements augmented backward elimination for linear, logistic and Cox regression. By means of a simulation and an analysis of a biomedical study, we evaluated the ABE algorithm and its implementation in SAS. Depending on the settings of the parameters of ABE (significance threshold a, change-in-estimate threshold t and roles of candidate explanatory variables), the number of variables in the final model selected by ABE will be between the number of variables selected by BE and the total number of variables. Based on our simulations and practical experiences with ABE, we suggest to use a significance threshold of a~0:2 and a change-in-estimate threshold of t~0:05.
DD, KL and GH were supported by the  European Community 's  Seventh Framework Programme , grant number  241544 , URLs http://ec. europa. eu/research/fp7/index_en. cfm and http:// www. syskid. eu. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. All resources needed to complete this study (SAS software, computers, office programs, IT facilities) were provided by the  Center for Medical Statistics, Informatics  and  Intelligent Systems, Medical University of Vienna  ( GH9s  and  DD9s  department), using the Center's regular research budget.
Different programs are complementary for repeatome detection We constructed libraries of consensus sequences by employing the programs called RepeatScout  [14] , RepeatModeler  [15]  (hereafter referred to as RS and RM, respectively), and TEdenovo from the REPET package  [16]  on the A. thaliana Col-0 genome. Each library was used separately as input for genome annotation using the TEannot pipeline which is also included in the REPET package (see Methods). In addition, we used the Tallymer and Tandem Repeat Finder (TRF) programs in order to identify more specifically the sets of short perfect repeats and tandem (simple) repeats, respectively. Interestingly, while compared to other tools the genome annotation using the consensus sequences from TEdenovo yields the highest genome coverage (Figure  1A ), the combination of all the annotation sets covers ,32% of the A.
thaliana genome, i. e. a significant increase as compared to the well accepted approx. 23% repeat content in A. thaliana  [16]. More specifically, while TEdenovo alone enables the most extensive (,90%) coverage of the reference annotations (see Methods), the combination of all tools attains 95% coverage. When not taking into account the contribution of the annotations from TRF and Tallymer, the coverage of the reference annotations holds at 95% with 29. 4% (35 Mb) genome coverage (Figure  1B ). In contrast when comparing the ability of RS, RM, and TEdenovo to detect the annotations generated by Tallymer, we found that RS performed best, showing that consensus sequences generated by the latter program are better representatives of the perfect repeats detected by Tallymer (Figure  1C ). Nevertheless, the combination of the RS, RM, and TEdenovo annotations covers over 94% of the Tallymer annotations, further reinforcing the idea that the combination of these tools enables to detect a wider diversity of the A.
thaliana repeatome. As another type of indicator of sensitivity we used the mapping coordinates of a set of 24-nucleotide (nt) small RNA (sRNA)  [20]. Indeed, in plants,  24 -nt sRNAs are known to map almost exclusively to repeated sequences as they play an important role in epigenetic silencing  [21]  [22]  [23]. Their mapping positions on the genome can therefore serve as natural repeat-associated cartography. This set of 24-nt sRNA covers 9. 7 Mb of the A. thaliana genome, 5% of this amount mapping to host protein-coding sequences (CDS, data not shown). While we found that TEdenovo performs best as lone program by covering 80% of the 24-nt sRNA loci, the combination of RS, RM, and TEdenovo enables to cover 86% of the sRNA map (Figure  1D ) which is a substantial enhancement as compared to the 73% coverage with the reference annotations.
thaliana genome (Maumus and Quesneville, unpublished data). In more details though, the RS annotations are relatively enriched in highly similar matches (over 90% identity) especially as compared to the RM annotations, while the TEdenovo ones are relatively enriched in values around 75-80%. Finally, at the chromosome level, the densities of the RS, RM, and TEdenovo annotations seem largely similar, while local variations are also apparent (Figure  S1 ). Overall, while our results suggest the higher quality and sensitivity of the annotations obtained when using the consensus sequences built by TEdenovo, they also highlight the complementarity of the libraries generated by each program. Remarkably, combining the non-redundant annotations from the three programs yields at least 21% increase of genome coverage compared to each program alone (Figure  2D ).
While the default filters for HSP detection are set to 90% identity and Evalue , 1e-300, we performed two runs: one called ''TEdeno-vo_cool'' with these parameters set at 85% and 1e-50, and the second called ''TEdenovo_soft'' with parameters set at 80% and 1e-20, respectively. These latter values mean that we look for HSPs that are 80% identical with a score expected to be observed fewer than 1e-20 times in a random dataset. The HSPs detected were further processed with the TEdenovo pipeline. These relaxed parameters were found to have a dramatic impact on the size of the library with the TEdenovo_cool and TEdenovo_soft libraries comprising 2,466 and 4,775 consensus sequences, respectively, as compared to 884 with regular TEdenovo. Qualitatively, the size distribution of the consensus from the TEdenovo_cool library shows a significant increase in the number of smaller sequences, while the number of large sequences remains stable as compared to regular TEdenovo (Figure  6A ).
thaliana genome by comparing it to libraries of consensus sequences using local alignment programs embedded in the REPET package. However, because repeated and repeat-derived sequences become more and more mutated and fragmented over time, the alignment-based approaches may be little efficient to detect the oldest repeat remnants. Nevertheless, the decay of ancient repeated sequences would have produced a significant amount of short, (nearly) identical repeats. The physical clustering of such high frequency oligos in consecutive or near consecutive regions on the chromosomes is then likely to testify the presence of repeat remnants. The P-clouds package was conceived following this postulate and enabled to identify that the human repeatome may contribute two-thirds of the human genome as compared to the 45-50% detected using alignment-based programs  [18]   [38].
Genome annotation Some programs such as Tallymer and TRF perform both repeat detection and genome annotation. In contrast, the consensus sequences that are built by RepeatScout, RepeatModeler, and TEdenovo must be provided as inputs to homology-based genome annotation programs such as RepeatMasker  [44] , CENSOR  [45] , and BLASTER  [46]. The TEannot pipeline from the REPET package uses a wrapper that launches and computes results from these three programs. Using a library of A. thaliana reference repeats (see above) we found that, as previously shown  [47] , TEannot provides increased sensitivity compared to the results obtained using RepeatMasker alone (Figure  S10A ). Importantly, the annotations obtained with TEannot were not skewed towards shorter sizes as compared to those obtained with RepeatMasker (Figure  S10B ) showing that the supplementary annotations cannot be accounted by a bias towards detecting shorter segments.
De novo identification of repeated sequences Tallymer was run with parameters searching for 20-mers with at least 4 occurrences. TRF was run with the following set of parameters: 2, 10, 10, 80, 10, 24, 2000. RepeatScout was launched using default settings except extension parameter (stopafter) which was set at 500. The seed size as determined by the program was 15 bp for the A. thaliana genome. RepeatModeler was employed with default settings except that, again, the RepeatScout extension parameter (stopafter) was set at 500. TEdenovo from the REPET pipeline (v2. 0) was run with default parameters. Consensus sequences derived from LTR Harvest predictions were retained only when they presented pfam domains typical of LTR retrotransposon. The parameters for HSP detection with TEdenovo were modified to perform the ''cool'' run (Evalue: 1e-50 and Identity = 85) and the ''soft'' run (Evalue: 1e-20 and Identity = 80).
Data processing and filtering The annotations covering less than 100 bp were discarded in order to draw the plots showing the frequency distribution of identity between consensus sequences and copies, the plots showing identity values between consensus sequences and copies along genome sequences, and the plots showing the distribution of copy sizes. The TEannot pipeline from the REPET package performs a ''long join'' procedure in order to connect fragments that may have been interrupted by a relatively recent insertion event. The sizes of the annotations presented in this work correspond to the results prior this post processing reconstruction step whereas the annotation files loaded on the genome browser correspond to results that benefit the long join procedure. To generate the Bundle library, redundancy in the combined set of consensus sequences from TEdenovo, RepeatScout, RepeatModeler, and reference sequences was filtered using all-by-all blast comparisons with thresholds of 95% identity and 98% length, resulting in 4,295 selected sequences.
Statistical machine learning methods are increasingly used for neuroimaging data analysis. Their main virtue is their ability to model high-dimensional datasets, e. g. multivariate analysis of activation images or resting-state time series. Supervised learning is typically used in decoding or encoding settings to relate brain images to behavioral or clinical observations, while unsupervised learning can uncover hidden structures in sets of images (e. g. resting state functional MRI) or find sub-populations in large cohorts. By considering different functional neuroimaging applications, we illustrate how scikit-learn, a Python machine learning library, can be used to perform some key analysis steps. Scikit-learn contains a very large set of statistical learning algorithms, both supervised and unsupervised, and its application to neuroimaging data provides a versatile tool to study the brain.
INTRODUCTION Interest in applying statistical machine learning to neuroimaging data analysis is growing. Neuroscientists use it as a powerful, albeit complex, tool for statistical inference. The tools are developed by computer scientists who may lack a deep understanding of the neuroscience questions. This paper aims to fill the gap between machine learning and neuroimaging by demonstrating how a general-purpose machine-learning toolbox, scikit-learn, can provide state-of-the-art methods for neuroimaging analysis while keeping the code simple and understandable by both worlds. Here, we focus on software; for a more conceptual introduction to machine learning methods in fMRI analysis, see  Pereira et al. (2009)  or  Mur et al. (2009) , while  Hastie et al. (2001)  provides a good reference on machine learning.
We discuss the use of the scikit-learn toolkit as it is a reference machine learning tool and has and a variety of algorithms that is matched by few packages, but also because it is implemented in Python, and thus dovetails nicely in the rich neuroimaging Python ecosystem. This paper explores a few applications of statistical learning to resolve common neuroimaging needs, detailing the corresponding code, the choice of the methods, and the underlying assumptions. We discuss • NumPy. Provides the ndarray data type to python, an efcient n-dimensional data representation for array-based numerical computation, similar to that used in Matlab  (Van Der Walt et al. , 2011). It handles efficient array persistance (input and output) and provides basic operations such as dot product. Most scientific Python libraries, including scikit-learn, use NumPy arrays as input and output data type.
• SciPy: higher level mathematical functions that operate on ndarrays for a variety of domains including linear algebra, optimization and signal processing. SciPy is linked to compiled libraries to ensure high performances (BLAS, Arpack and MKL for linear algebra and mathematical operations). Together, NumPy and SciPy provide a robust scientific environment for numerical computing and they are the elementary bricks that we use in all our algorithms. • Matplotlib, a plotting library tightly integrated into the scientific Python stack  (Hunter, 2007). It offers publication-quality figures in different formats and is used to generate the figures in this paper. • Nibabel, to access data in neuroimaging file formats. We use it at the beginning of all our scripts.
SCIKIT-LEARN AND THE MACHINE LEARNING ECOSYSTEM Scikit-learn  (Pedregosa et al. , 2011)  is a general purpose machine learning library written in Python. It provides efficient implementations of state-of-the-art algorithms, accessible to non-machine learning experts, and reusable across scientific disciplines and application fields. It also takes advantage of Python interactivity and modularity to supply fast and easy prototyping. There is a variety of other learning packages. For instance, in Python, PyBrain  (Schaul et al. , 2010 ) is best at neural networks and reinforcement learning approaches, but its models are fairly black box, and do not match our need to interpret the results. Beyond Python, Weka  (Hall et al. , 2009)  is a rich machine learning framework written in Java, however it is more oriented toward data mining.
Some higher level frameworks provides full pipeline to apply machine learning techniques to neuroimaging. PyMVPA  (Hanke et al. , 2009)  is a Python packaging that does data preparation, loading and analysis, as well as result visualization. It performs multi-variate pattern analysis and can make use of external tools such as R, scikit-learn or Shogun  (Sonnenburg et al. , 2010). PRoNTo  (Schrouff et al. , 2013)  is written in Matlab and can easily interface with SPM but does not propose many machine learning algorithms. Here, rather than full-blown neuroimaging analysis pipelines, we discuss lower-level patterns that break down how neuroimaging data is input to scikit-learn and processed with it. Indeed, the breadth of machine learning techniques in scikit-learn and the variety of possible applications are too wide to be fully exposed in a high-level interface.
Using the same data for training and testing is not acceptable because it leads to overly confident model performance, a phenomenon also known as overfitting. Cross-validation is a technique that allows one to reliably evaluate an estimator on a given dataset. It consists in iteratively fitting the estimator on a fraction of the data, called training set, and testing it on the left-out unseen data, called test set. Several strategies exists to partition the data. For example, k-fold cross-validation consists in dividing (randomly or not) the samples in k subsets: each subset is then used once as testing set while the others k -1 subsets are used to train the estimator. This is one of the simplest and most widely used crossvalidation strategies. The parameter k is commonly set to 5 or 10. Another strategy, sometimes called Monte-Carlo cross-validation, uses many random partitions in the data.
DATA PREPARATION: FROM MR VOLUMES TO A DATA MATRIX Before applying statistical learning to neuroimaging data, standard preprocessing must be applied. For fMRI, this includes motion correction, slice timing correction, coregistration with an anatomical image and normalization to a common template like the MNI (Montreal Neurologic Institute) one if necessary. Reference softwares for these tasks are SPM  (Friston, 2007)  and FSL  (Smith et al. , 2004). A Python interface to these tools is available in nipype Python library  (Gorgolewski et al. , 2011). Below we discuss shaping preprocessed data into a format that can be fed to scikit-learn. For the machine learning settings, we need a data matrix, that we will denote X, and optionally a target variable to predict, y.
Such a mask is often given along with the datasets or can be computed with software tools such as FSL or SPM. Applying the mask is made easy by NumPy advanced indexing using boolean arrays. Two-dimensional masked data will be referred to as X to follow scikit-learn conventions: mask = nibabel. load( ' mask. n i i '). get_data() func_data = nibabel. load( ' e p i. n i i '). get_data() # Ensure that the mask is boolean mask = mask. astype(bool) # Apply the mask, X = timeseries * voxels n e a r e s t ', cmap= ' h o t ') # Cosmetics: disable axis plt. axis( ' o f f ') plt. show() Note that a background is needed to display partial maps. Overlaying two images can be done thanks to the numpy. ma. masked array data structure. Several options exist to enhance the overall aspect of the plot. Some of them can be found in the full scripts provided with this paper.
Variables with high individual discriminative power are kept. Scikit-learn offers a panel of strategies to select features. In supervised learning, the most popular feature selection method is the F-test. The null hypothesis of this test is that the feature takes the same value independently of the value of y to predict. In scikit-learn, sklearn. feature_selection proposes a panel of feature selection strategies. One can choose to take a percentile of the features (SelectPercentile), or a fixed number of features (SelectKBest). All these objects are implemented as transformers (see section 2. 3). The code below uses the f_classif function (ANOVA F-Test) along with the selection of a fixed number of features. On the reduced feature set, we use a linear SVM classifier, sklearn. svm. SVC, to find the hyperplane that maximally separates the samples belonging to the different classes.
SEARCHLIGHT Searchlight  (Kriegeskorte et al. , 2006 ) is a popular algorithm in the neuroimaging community. It runs a predictive model on a spatial neighborhood of each voxel and tests the out-of-sample prediction performance as proxy measure of the link between the local brain activity and the target behavioral variable. In practice, it entails performing cross-validation of the model, most often an SVM, on voxels contained in balls centered on each voxel of interest. The procedure implies solving a large number of SVMs and is computationally expensive. Detailing an efficient implementation of this algorithm is beyond the scope of this paper. However, code for searchlight and to generate figure 2 is available in the GitHub repository accompanying the paper.
These results match neuroscientific knowledge as they highlight the high level regions of the ventral visual cortex which is known to contain category-specific visual areas. While Searchlight only gives a score to each voxel, the SVC can be used afterward to classify unseen brain scans. Most of the final example script (haxby decoding. py on GitHub) is for data loading and result visualization. Only 5 lines are needed to run a scikit-learn classifier. In addition, thanks to the scikitlearn modularity, the SVC can be easily replaced by any other classifier in this example. As all linear models share the same interface, replacing the SVC by another linear model, such as ElasticNet or LogisticRegression, requires changing only one line. Gaussian Naive Bayes is a non-linear classifier that should perform well in this case, and modifiying display can be done by replacing coef by theta.