dataesr/scientific-paragraphs-categorization huggingface dataset:
g. , number of generated tracks). The inter-subject differences of data size and computation time for a data set with identical acquisition parameters are mainly explained by differences in brain size (connectome extraction depends on the number of seed voxels). All computations described in the following section were done on a high performance computer consisting of multicore CPUs at a clock rate of ~2. 54 GHz per CPU core and 8GB RAM per core. In Table  1 , we provide the generated data amount and computational cost of each individual step. Under the assumption of linear scalability, we calculated execution time of the pipeline based on our test setup (computer consisting of 184 CPU cores at a clock-rate of 2. 54 GHz). The scaling assumes an octa-core workstation with equally fast CPU cores and I/O latencies.
Only the tractography and connectome extraction modules of the pipeline are parallelized. Since preprocessing steps are all executed on a single core their executional times are scale invariant. The accumulated execution time for these scale invariant preprocessing processes is around 10 h. • Fiber Tractography: around 2 h. Hence for a work station as described above: (2 h * 184 Cores)/8 Cores = 46 h. • Computing SC Matrices: 2 h using a maximum of 68 cores in parallel. For a workstation hence it follows: (2 h * 68 cores)/8 cores = 17 h. To process a single subject from the raw DICOM data to the final SC & FC matrices using a workstation containing eight CPU cores, one would need approximately 73 h. Mandatory pipeline inputs are three raw MRI data sets in DICOM format: T1-weighted structural, BOLD and dwMRI with at least 45 distinct gradient directions and one b0 image as first image in the data set.
Optionally, one EEGLAB set file and one data set containing axon tract directionality information, e. g. , derived from CoCoMac database, http://cocomac. org, and adapted to the human brain  (Bezgin et al. , 2012)  can be read. Exemplary pipeline output files compatible with TVB upload requirements are public and can be found via this URL: https://github. com/the-virtual-brain/tvb-data. A detailed description of pipeline output, TVB file formats and the optional EEG source-imaging step can be found in the Supplementary material.
Preprocessing of T1-weighted anatomical MRI data High-resolution anatomical images are used to create a segmentation of white matter, for segmentation and parcellation of cortical and subcortical gray matter (to be used for full-brain simulation within TVB and for the extraction of region-wise averaged BOLD activity) and for the generation of tractography masks. The following major steps are performed by FREESURFER's recon-all function: motion correction, intensity normalization, skull stripping, removal of non-brain tissue, brain mask generation, cortical reconstruction, WM and subcortical segmentation, cortical tessellation generating GM-WM and GM-pia interface surface-triangulations and probabilistic atlas based cortical and subcortical parcellation. Relevant output of these operations are NifTi volumes that contain GM and WM parcellations.
All MNI voxels that have been falsely warped onto subject-space WM voxels are excluded from the final parcellation, while all MNI parcellation voxels that have been assigned to subjectspace GM voxels are assumed to be correct in the final parcellation. Then, the algorithm proceeds by iterating through all unassigned GM voxels and assigning each voxel the label of the majority of GM voxels in its neighborhood. In the case that there are two or more equally sized groups of voxels in the voxel's neighborhood that are assigned to different regions, the search-radius is increased until an unambiguous decision can be made. Optional: Merging connectomes with directionality information derived from other sources Directionality of fiber tracks cannot be derived from dwMRI data. However, databases exist that contain directionality information derived from axonal tract-tracing, such as the CoCoMac database (http:// cocomac.
This approach is justified by the assumption that the coupling strength of a region is proportional to the size of the GWI of that region, assuming a fixed ratio of long-range connections per microcircuit volume. We acknowledge that this approximation is the result of several simplifying assumptions on the nature of longrange connectivity that at the moment cannot be verified empirically due to a lack of empirical data. Pseudo-code given in Supplementary material summarizes the steps of the algorithm. Upon tractography the pipeline computes distinct connections and aggregates them for each region to generate three types of SC matrices (Fig. 3  visualizes details of the concept): -raw counts, contain track counts of all tracks that were found between each pair of regions (symmetric), -distinct connection counts, contain only distinct connections between each pair of regions (symmetric), -weighted distinct connection counts, in which the strength of each distinct connection is divided by the number of all distinct connections leaving the voxel (yielding asymmetric strength matrices).
ICC(3,1) was computed using the MATLAB method Intraclass Correlation Coefficient created by Arash Salarian (www. mathworks. com/matlabcentral/fileexchange/22099). ICC(3,1) estimates the similarity of different strength estimates for the same subject, with a value of 1 indicating perfect agreement between sessions. In addition to ICCs for full connectivity matrices ICCs for node strengths were computed. The node strength ω i of node i is defined as the sum of edge weights (i. e. , strengths) a ij with all connected nodes j over all n nodes, ω i ¼ X n j¼1 a i j : Another type of Intraclass Correlation Coefficient was used to specifically test whether global connectivity properties showed larger variability between subjects than within subjects over scanning sessions: Thereby, between-subject variance σ bs 2 is contrasted with the pooled within-subject variance σ ws 2 of the respective metrics.
Different toolboxes often use different image formats and coordinate systems. The raw output files of MRI scanners are often written in the form of DICOM files while image processing software like FSL or SPM typically process analyze or NIfTI volumes. Regarding data structures, raw imaging data is mostly stored as gray-scale images while brain-surface meshes, like those generated with FREESURFER, exist in the form of surface-triangulations specified by coordinates of endpoints and edges that connect them to triangles. Different imaging modalities acquire and store data in different imaging "spaces", e. g. , standard atlas space (e. g. , Talairach or MNI), structural space (high resolution, small voxelsize), functional space (low to medium resolution, large to medium voxel-size, temporal dimension) or diffusion space (low to medium resolution, large to medium voxel-size, different gradient directions).
While it is true that anisotropy is modulated by physiological aspects like the degree of myelination, fiber density and packing that certainly affect connectivity strength, it is, however, not possible to disentangle structural effects like incoherent axon orientations, which is the rationale for probabilistic tractography (i. e. , spread diffusion profiles as an indicator of crossing fibers). While hardly any ground truth data exists for coupling strength estimation, the accuracy of tractography algorithms to localize existing major white matter tracts can be verified with a certain amount of confidence  (Jbabdi and Johansen-Berg, 2011). Although validation of the topology of extracted connectomes is an active area of research  (Seehaus et al. , 2013; Schreiber et al. , 2014) , there is no dataset or method that is able to directly estimate connection strengths between regions for an entire brain.
Source imaging Several choices can be made during source imaging that require careful consideration with regard to the underlying (M)EEG data and the intended purpose of the estimated source time-series. Brainstorm offers a variety of options and source imaging algorithms that impact the resulting source time-series. Regarding head model computation, three models are available in Brainstorm: The simpler single sphere and overlapping sphere models (approximating the head as spheres) and the more advanced BEM included from the open-source software OpenMEEG. While the simpler models deliver proper results for MEG, for EEG the BEM model should be preferred, especially if subject specific anatomical data is available. Noise covariance estimation is sensitive with regard to the type of EEG experiment done and the intended use of the data.
Background: In applications of supervised statistical learning in the biomedical field it is necessary to assess the prediction error of the respective prediction rules. Often, data preparation steps are performed on the dataset-in its entirety-before training/test set based prediction error estimation by cross-validation (CV)-an approach referred to as "incomplete CV". Whether incomplete CV can result in an optimistically biased error estimate depends on the data preparation step under consideration. Several empirical studies have investigated the extent of bias induced by performing preliminary supervised variable selection before CV. To our knowledge, however, the potential bias induced by other data preparation steps has not yet been examined in the literature. In this paper we investigate this bias for two common data preparation steps: normalization and principal component analysis for dimension reduction of the covariate space (PCA).
Furthermore we obtain preliminary results for the following steps: optimization of tuning parameters, variable filtering by variance and imputation of missing values. Methods: We devise the easily interpretable and general measure CVIIM ("CV Incompleteness Impact Measure") to quantify the extent of bias induced by incomplete CV with respect to a data preparation step of interest. This measure can be used to determine whether a specific data preparation step should, as a general rule, be performed in each CV iteration or whether an incomplete CV procedure would be acceptable in practice. We apply CVIIM to large collections of microarray datasets to answer this question for normalization and PCA. Results: Performing normalization on the entire dataset before CV did not result in a noteworthy optimistic bias in any of the investigated cases.
Background In supervised statistical learning, it is widely recognized that prediction models should not be constructed and evaluated using the same dataset. While the training dataset is used for all steps towards obtaining the prediction rule, the test dataset is used to evaluate its prediction error and, ideally, should not be at all involved in the training phase. Cross-validation and related procedures consist of considering several divisions into training data and test data and averaging the estimated prediction errors of the respective prediction rules constructed in each iteration. In our paper we use K-fold cross-validation ("CV"), but all ideas and procedures can be extended to other resampling techniques used for prediction error estimation. By "incomplete CV"  [1] , we are referring to CV procedures in which some analysis steps are performed beforehand using the whole dataset.
With incomplete CV, at each iteration the excluded fold acting as test data may affect the derived prediction rule, since it was preliminarily used for data preparation-which contradicts the principle of test data requiring perfect separation  [2]. In contrast, if all steps leading to the prediction rules are performed in each CV iteration using only the corresponding training set, the CV procedure is "full CV". The problems resulting from incomplete CV have been extensively studied in the literature with regard to preliminary variable selection for classification based on high-dimensional microarray data  [1, [3] [4] [5]. If performed before splitting the dataset into K folds, supervised variable selection often leads to strongly downwardly biased error estimates. The now widely adopted procedure to avoid this problem consists of conducting the variable selection step in each CV iteration anew using the training dataset only  [1, 3] , i.
e. considering it as part of the classifier construction process. Similarly, it has been suggested that parameter tuning should be performed using the training dataset only  [6] [7] [8]. However, the bias resulting from incomplete CV with respect to parameter tuning has to our knowledge never been investigated in the literature. Variable selection and parameter tuning are-by farnot the only procedures often run in practice before CV. For example, raw data from high-throughput biological experiments such as microarrays have to be normalized before so-called high-level analyses such as predictive modeling can be conducted. The selection of features which exhibit high variability across the observations is another example of a data preparation step often performed when analyzing microarray data.
Further examples relevant to any type of data include imputation of missing values, dichotomization and non-linear transformations of the features. In this paper, all these procedures are designated preparation steps to stress that they are performed before the construction of the prediction rule. Preparation steps are not limited to these few examples. The analysis of growingly complex biomedical data (including, e. g. , imaging or sequencing data) increasingly requires the use of sophisticated preprocessing steps for making raw data analysable. Note, however, that the question of the impact of CV incompleteness is not relevant to those data preparation steps which prepare the observations independently of each other, such as background correction for microarray data. It is an open question whether preparation steps lead to underestimation of the prediction error if performed before splitting the dataset into K folds, as seen with variable selection.
Another example is genotype calling in the context of genetic association studies: it is common practice to use not only the whole dataset of interest, but also further datasets, to improve genotype calling accuracy. In the context of high-dimensional data, two further important preparation steps often performed using the whole dataset are dimension reduction procedures such as Principal Component Analysis (PCA) and normalization-for example normalization using the RMA ("Robust Multi-array Average") method  [12]  for microarray gene expression data. It is not clear whether the resulting prediction error estimate is optimistically biased if one applies these two methods to the whole dataset before splitting the data into K folds. In an effort to answer this question we present a new measure which enables the quantification of the impact of incomplete CV with regard to steps of interest, the "CV Incompleteness Impact Measure" (CVIIM).
It is based on the ratio of the CV prediction error resulting when the investigated preparation steps are applied only once using the whole dataset to the CV prediction error resulting when they are incorporated into CV. By incorporating preparation steps into CV we mean that they are performed in CV on each training dataset anew and subsequently applied to the excluded fold via so-called addon procedures. The goal of this paper is two-fold: (i) to provide a new measure-the "CVIIM"-which is intended to be used by methodological researchers or statisticians working on statistical learning applications to determine whether a particular preparation step should-in general-be trained in each CV iteration successively or whether it can be safely performed as a preliminary step on the whole dataset without generating a relevant optimistic bias; and (ii) to apply this new measure to answer this question and provide guidelines for two important preparation steps, PCA and normalization, in the case of high-throughput molecular data.
The paper is structured as follows: the section Methods first presents the microarray gene expression datasets used in our studies, the concept of addon procedures and the two methods-normalization and PCA. Then we introduce CVIIM and briefly illustrate its use and behavior in the well-investigated case of variable selection, using four example datasets. Concluding this section we describe the designs of the studies on the impact of CV incompleteness with respect to normalization and PCA; the results of these studies are presented in the section Results. In the section Discussion we present preliminary results obtained for other data preparation steps and discuss further issues. The section Conclusions summarizes the main conclusions of the paper.
Data material We used a wide range of publicly available, highdimensional, mostly transcriptomic datasets in our real data analyses. See Table  1  for an overview. With the exception of ProstatecTranscr all datasets were downloaded from the ArrayExpress database  [13]  or the NCBI GEO database  [14]. All datasets feature a binary target variable and are of human origin. Details on the biological background of the datasets may be obtained online via the respective accession numbers available from Table  1  and via the R scripts written for the preparation of the individual datasets for analysis. The latter are available from Additional file 1 and can be used to download and prepare the individual datasets automatically. The dataset ProstatecTranscr appeared in  [15]  and is available in the form of an Rda-file from Additional file 1 as well.
g. [16]. However, in general this approach may lead to a higher prediction error in the case of small test datasets because of the larger variance of the output of preparation steps. Test datasets of size 1 (corresponding to, say, patients examined one at a time) are an extreme case where this approach is completely infeasible. Moreover, for some preparation steps such as variable filtering by variance this naive approach cannot be applied since it would lead to the selection of different variables in the training and test datasets and thus make the application of the prediction rule impossible. Another straightforward idea is to "train" the preparation step on the training data and to use the output of the preparation step to prepare the test data. We refer to such a procedure as an "addon procedure".
Here, the quantiles of the test observations are replaced by the quantiles of the training observations after quantile normalization of the latter. Since background correction and summarization are performed on an array-by-array basis, no addon strategies are necessary for these procedures. In the vignette of the Bioconductor package vsn, Huber  [19]  presents a version of variance stabilization in which no calibration is performed, i. e. only a global variance stabilization transformation is conducted. In contrast to standard VSN this procedure does not involve any observation-specific parameters, so it is possible to determine an addon procedure: the global VSN parameters estimated on the training data are used to transform the test data.
The cross-validation incompleteness impact measure (CVIIM) In the following we present CVIIM, our new measure for the extent of bias induced by incomplete CV with respect to a data preparation step of interest. Let s be the available dataset from which a prediction rule is to be derived. s is assumed to be an i. i. d. sample of size n with observations drawn from the distribution P, where P is the joint distribution of predictors and response variable. Note that the assumption of i. i. d. observations made here is owing to the fact that throughout this paper we are concerned with cross-validation, i. e. dataset internal validation. With external validation this assumption is generally not appropriate. Further, let e full,K (s) be the prediction error estimated by full K-fold CV, i. e. when all steps leading to the prediction rule, including data preparation steps, are performed at each CV iteration anew based only on the training dataset.
Similarly let e incompl,K (s) be the prediction error estimated by incomplete K-fold CV, i. e. when the data preparation step(s) of interest is performed before CV, using the whole dataset. For simplicity of notation, we additionally assume that e full,K (s) and e incompl,K (s) are obtained by averaging over a large number of CV runs, i. e. over a large number of random partitions, and can thus be treated as deterministic. For S ∼ P n , our new measure "CVIIM", "Cross-Validation Incompleteness Impact Measure", is defined as: CVIIM P,n,K := ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ 1 - E[e incompl,K (S)] E[e full,K (S)] if E[e incompl,K (S)] < E[e full,K (S)] and E[e full,K (S)] > 0 0 o t h e r w i s e. ( ) 1 Note that we defined CVIIM P,n,K as a theoretical quantity, not calculable, but estimable from real data. It is simply estimated by replacing the expected CV errors by their empirical counterparts e incompl,K (s) and e full,K (s): CVIIM s,n,K := ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ 1 - e incompl,K (s) e full,K (s) if e incompl,K (s) < e full,K (s) and e full,K (s) > 0 0 o t h e r w i s e.
The discrepancy between e incompl,K (s) and e full,K (s) depends on how strongly the specific preliminary step conducted on the whole dataset increases the homogeneity of the covariate values across observations and (for supervised preparation steps) the empirical association between the covariate values and the values of the target variable. With the interpretation of CVIIM s,n,K in mind and based on real data results and expectations regarding the impact of specific data preparation steps, we define the following tentative rules of thumb for categorizing the computed values in terms of the impact of CV incompleteness with regard to the considered step(s): [ 0, 0. 02] ∼ no influence, ] 0. 02, 0. 1] ∼ weak, ] 0. 1, 0. 2] ∼ medium, ] 0. 2, 0. 4] ∼ strong, ] 0. 4, 1] ∼ very strong. We outline an artificial example to demonstrate, step by step, a possible application of CVIIM P,n,K.
We are interested in measuring the extent of overoptimism connected with performing the quantile normalization step of RMA before CV in gene expression based classification. Suppose we have a dataset with gene expression measurements from 32 patients suffering from breast cancer and from 22 disease-free patients. Per patient we have measurements of the expression of 54,675 genes. As classification method we use Nearest Shrunken Centroids (NSC). The error e incompl,5 (s), as estimated by incomplete 5-fold CV, is computed by conducting the RMA normalization beforehand on the whole dataset and performing 5-fold CV on the normalized dataset. In this procedure only the fitting of NSC is repeated in each CV iteration on the training datasets. The CV is repeated 300 times to obtain more stable results.
The full CV error e full,5 (s) is computed by performing a 5-fold CV in which the quantile normalization step of RMA (as well as the fitting of the NSC) is re-performed in each CV iteration on the respective training set, with addon normalization of the corresponding test set through the addon procedure by Kostka and Spang  [17]. This procedure is again repeated 300 times. Suppose we were to obtain e incompl,5 (s) = 0. 15 and e full,5 (s) = 0. 1503. Then, CVIIM s,n,K = 1 -0. 15/0. 1503 ∼ 0. 002. According to our rules of thumb this would correspond to no influence on the estimated error. This result obtained for a specific dataset and specific classifier, however, may not be representative of all datasets and classifiers in the field of gene expression based classification. Extending this example, we point out that it is necessary to study several datasets and several analysis settings representative of the considered field in order to formulate recommendations regarding incomplete CV for a particular step.