dataesr/scientific-paragraphs-categorization huggingface dataset:
Qualitative observations For our two real datasets, the literature provides clear expectations about brain areas involved in the tasks performed by the participants and the associated decoding question addressed in our experiments. The active finger movements performed by the participants during the acquisition of Dataset1 are known to recruit contralateral primary motor and sensory as well as secondary sensory cortices, ipsilateral dorsal cerebellum as well as the medial supplementary motor area  [25]. In Dataset2, the participants were passively listening to vocal and non-vocal sounds. The contrast, or decoding, of these two types of stimuli is classically used to detect the so-called temporal voice areas, which are located along the superior temporal cortex (see e. g. [29] ). We now describe the results obtained with G-MVPA and ISPA with respect to this priori knowledge.
The searchlight decoding analyses performed at the group level were all able to detect clusters of voxels where the decoding performance was significantly above chance level (p < 0. 05, FWE-corrected using permutation tests) with both G-MVPA and ISPA, for Dataset1 and Dataset2 and with all sizes of the spherical searchlight. The detected clusters were overall consistent across values of the searchlight radius, with an increasing size of each cluster when the radius increases. In Dataset1, both strategies uncovered two large significant clusters located symmetrically in the left and right motor cortex. Additionally, ISPA was able to detect other significant regions located bilaterally in the dorsal part of the cerebellum and the parietal operculum, as well as a medial cluster in the supplementary motor area (note that some of these smaller clusters also become significant with G-MVPA with the larger searchlight radii).
These areas were indeed expected to be involved bilaterally given that button presses were given with both hands in this experiment. In Dataset2, both G-MVPA and ISPA yielded two large significant clusters in the temporal lobe in the left and right hemispheres, which include the primary auditory cortex as well as higher level auditory regions located along the superior temporal cortices, matching the known locations of the temporal voice areas. Figure  3  provides a representative illustration of these results, for a radius of 6 mm.
Quantitative evaluation Our quantitative evaluation focuses on the two largest clusters uncovered in each dataset, i. e. the ones in the motor cortex for Dataset1 and the ones in the temporal lobe for Dataset2. We first examine the size of these clusters, separately for each hemisphere and each of the five values of the searchlight radius. The results are displayed in the left column of Figure  4. In almost all cases, the size of the significant clusters increased with the searchlight radius (left column). Moreover, the cluster located in the right hemisphere is consistently larger than the one on the left. In Dataset1, the cluster detected by ISPA is larger than the one detected by G-MVPA, regardless of the hemisphere, while in Dataset2, it is G-MVPA that yields larger clusters (except for a 4 mm radius where the sizes are similar).
Then, we study the peak value of the t statistic obtained in each cluster (right column of Figure  4 ). In Dataset1, the peak t value is higher for ISPA than G-MVPA, for all values of the radius. In Dataset2, ISPA yields higher peak t values than G-MVPA for the searchlight radii smaller or equal than 8 mm, and lower peak t values for the larger radius values. Then, we quantify the amount of overlap between the clusters found by G-MVPA and ISPA, by splitting the voxels into three sub-regions: voxels uncovered only by ISPA, only by G-MVPA or by both strategies (overlap). Figure  5  provides an illustration of these sub-regions, which shows that the overlap region (in blue) is located at the core of the detected clusters, while the voxels significant for only one strategy are located in the periphery; these peripheral voxels appear to be mostly detected by ISPA for Dataset1 (red voxels) and by G-MVPA for Dataset2 (green voxels).
G-MVPA and ISPA provide different results In this study, we have performed experiments on both real and artificial functional neuroimaging data in order to compare two group-level MVPA schemes that rely on classifier-based decoding analyses: the vastly used G-MVPA, and ISPA. Our results show that both strategies can offer equivalent results in some cases, i. e. that they both detect significant group-level multivariate effects in similar regions of the cortex for our two real fMRI datasets, and in parts of the two-dimensional parameter space browsed using our artificially generated datasets, but that their outcomes can also differ significantly. For instance, in Dataset1, ISPA was the only strategy that detected multivariate group-level effects in several regions such as the supplementary motor area, the bilateral parietal operculum and dorsal cerebellum, for most of the searchlight radii that we tested (see an example on Figure  3  with a 6 mm radius).
Furthermore, when a region is detected by both strategies, it usually differs in its size, extent and/or precise location, resulting in partial overlap; in most cases, the areas of concordance between the two strategies appeared to be centrally located, while the disagreements are located towards the periphery: in some areas, G-MVPA detects a group-level effect while ISPA does not, and inversely in other areas. Note that for Dataset2, our results generalize some of the observations reported in  [13]  with a different, yet comparable, framework of analysis, on a closely related data set. Surprisingly, the peripheral behaviors were not consistent across the two real fMRI datasets: on Dataset1, ISPA only was able to detect effects on the periphery of the core region where both strategies were equally effective, while on Dataset2, it was G-MVPA which provided significant results on the periphery.
The results of the experiments conducted on the artificial datasets can actually shed some light on these results, thanks to the clear dissociation that was observed in the twodimensional parameter space browsed to control the properties of the data. Indeed, ISPA is the only strategy that allows detecting smaller multivariate effects when the inter-individual variability remains moderate, which is the case in the largest regions detected in Dataset1 because they are located in the primary motor cortex, the primary nature of this region limiting the amount of inter-subject variability. On the opposite, the peripheral parts of the temporal region detected in Dataset2 are located anteriorly and posteriorly to the primary auditory cortex, towards higher-level auditory areas where the inter-individual variability is higher, a situation in which G-MVPA revealed more effective in the experiments conducted with our artificial data.
ISPA: larger training sets improve detection power Our experiments revealed a very important feature offered by the ISPA strategy: its ability to detect smaller multivariate effects. On the one hand, this greater detection power was explicitly demonstrated through the simulations performed on artificial data, where the multivariate effect size was one of the two parameters that governed the generation of the data; we showed that with an equal amount of inter-individual variability, ISPA was able to detect effects as small as half of what can be detected by G-MVPA. Furthermore, on both real fMRI datasets, ISPA was able to detect significant voxels that were not detected using G-MVPA, in a large amount in Dataset1, and to a lesser extent in Dataset2. This detection power advantage is of great importance, since detecting weak distributed effects was one of the original motivations for the use of MVPA  [15].
However, here, the increased sample size comes at the price of a larger heterogeneity in the training set, because of the differences that can exist between data points recorded in different subjects. If these differences are too large, they can represent an obstacle for learning, but if more moderate, the inter-subject variability can reveal beneficial by increasing the diversity of the training set. The fact that we observe a higher detection power with ISPA than with G-MVPA suggests that we are in the latter situation.
ISPA offers straightforward interpretation When using the ISPA strategy, obtaining a positive result means that the model has learnt an implicit rule from the training data that provides statistically significant generalization power on data from new subjects. Since a cross-validation of the type leave-one-subject-out or leave-n-subjects-out is performed on the available data to quantitatively assess such results, it allows to draw inference on the full population from which the group of participants was drawn, including individuals for which no data was available. As previously pointed out in  [22] , the interpretation that follows is straightforward: a significant effect detected with ISPA implies that some information has been identified to be consistent throughout the full population. In more details, this means that the modulations of the multivariate patterns according to the experimental conditions that were the object of the decoding analysis are consistent throughout the population, at least at the resolution offered by the modality used for the acquisition.
g. support vector machines  [4]. Assuming we have n examples per subject, the number of training examples available for each classifier is (K-1)n K for G-MVPA and (S -1)n for ISPA. Overall, with linear-time classifiers, the total complexity of a group-level decoding analysis amounts to O(nKS ) for G-MVPA and O(nS 2 ) for ISPA, which makes them almost equivalent if one assumes that K and S are of the same order of magnitude. With quadratic-time classifiers, the total complexity is O(n 2 KS ) for G-MVPA and O(n 2 S 3 ) for ISPA, which makes ISPA significantly more costly. We therefore advice to use linear-time classifiers such as logistic regression to perform ISPA analyses, particularly with searchlight decoding where the computational cost is further multiplied by the number of voxels. Furthermore, note that thanks to its hierarchical nature, G-MVPA can be performed in an incremental manner for a low computational cost as participants get scanned: every time data from a new subject becomes available during the acquisition campaign, it suffices to run within-subject decoding for this new subject, which costs O(nK), plus the statistical test.
This offers more flexibility for the experimenter than with the inter-subject scheme, for which performing IPSA every time a new subject is scanned amounts to re-doing a full analysis on all the subjects. In terms of software implementation, because within-subject analyses have been the standard since the advent of MVPA, all software packages provide well documented examples for such analyses which are the base tool for G-MVPA. Even if it is not the case for ISPA, it is easy to obtain an equivalent implementation because to perform inter-subject decoding, one simply need to i) have access to the data from all subjects, and ii) set up a leave-one-subject-out cross-validation, these two operations being available in all software packages. As an example, we provide the code to perform ISPA searchlight decoding from pre-processed data available online, which allows reproducing the results described in the present paper on Dataset2: http://www.
possible thanks to the infrastructure France Life Imaging (11-INBS-0006) of the French program Investissements d'Avenir, as well as specific grants from the Templeton Foundation (40463) and the Agence Nationale de la Recherche (ANR-15-CE23-0026). Inter-subject pattern analysis A straightforward and powerful scheme for group-level MVPA Qi Wang, Bastien Cagna, Thierry Chaminade, Sylvain Takerkart https://doi. org/10. 1016/j. neuroimage. 2019. 116205.
Supplementary materials 1. Influence of the number of subjects and number of samples per subject Aim. Although the two real fMRI datasets used in our experiments have approximately the same total number of observations (5400 observations in Dataset1, 5616 observations in Dataset2), they differ in the number of subjects that were scanned and the number of trials per subject: Dataset1 includes 360 trials for each of the 15 subjects, while Dataset2 offers 144 trials for 39 subjects. We here attempt to investigate whether this could explain some of the differences we observed in the results of G-MVPA and ISPA on these two datasets, using new artificial datasets. Experiments. We therefore repeat the same set of experiments as in the paper, using new sets of 14300 datasets generated to maintain the size of the training set constant and modulating the ratio between the number of subjects S and the number of samples per subject N.
We used the following parameter values: (S , N) ∈ {(9, 500),  (11, 400) ,  (17, 250) ,  (21, 200) , (51, 80), (101, 40), (201, 20), (401, 10)}, which allows maintaining the size of the group-level dataset approximately constant (and more particularly, the size of the ISPA training set is exactly constant at (S -1) × N = 4000). Results. We present below the results we obtained, under the same summarized form as in Table  3  of the paper. Tables S1 to S8 represent the two-dimensional parameter space spanned by d and Θ, various values of the (S , N) couple. The cells colored in blue are those where G-MVPA and ISPA yielded significant group-level decoding for 50 or more datasets out of the 100, whereas in the green cells it is the case only for G-MVPA and in the red ones it is the case only for ISPA.
The shape of the region where G-MVPA yields 50 or more significant detections, corresponding here to the green and blue cells, remains approximately rectangular for all values of (S , N). But when the number of subject S increases, it is shifted to the right of the table, i. e. G-MVPA is effective only for larger effect sizes. This is a direct consequence of the decreasing value of N, which is critical for within-subject decoding. For ISPA, the shape and location of the colored region (red and blue cells) where it is effective remains fairly constant, showing that ISPA is not or weakly affected by the ratio N/S when the size of the training set is constant. We now try to address the question of whether the difference in the N/S ratio between Dataset1 and Dataset2 can explain the fact that the most peripheral regions of the main clusters are detected by ISPA for Dataset1 and G-MVPA for Dataset2, in the light of the present results where we vary N/S in artificial datasets.
The N/S ratio for Dataset1 is 360/15 = 24, while for Dataset2, it is 144/39 = 3. 7. We reformulate the previous question as follows: if the N/S ratio of Dataset1 were smaller (i. e. closer to the one of Dataset2), could the peripheral voxels -which are red on Figure  5  of the paper, i. e. are detected by ISPAbecome green, i. e. be detected only by G-MVPA? For this, consider a red cell in Table  S3  (for which the N/S ratio is the closest to the one of Dataset1): can it become green when the ratio N/S decreases? We clearly see that it is not possible, i. e. that all red cells in Table  S3  are also red (in almost all cases) in Tables  S4 to S8  where N/S is smaller. We then ask the opposite question: if the N/S ratio of Dataset2 were larger (i. e. closer to the one of Dataset1), could the peripheral voxels -which are green on Figure  5  of the paper, i.
e. are detected by G-MVPAbecome red, i. e. be detected only by ISPA? For this, consider a green cell in Table  S5  (for which the N/S ratio is the closest to the one of Dataset2): can it become red when the ratio N/S increases? We clearly see that it is not possible, i. e. that all green cells in Table  S5  are also green (in almost all cases) in Tables  S1 to S4  where the N/S ratio is larger. This parallel between the real and the artificial datasets therefore suggests that it is not the difference of N/S ratio between Dataset1 and Dataset2 that can explain the different behaviors observed in the periphery of the significant clusters between G-MVPA and ISPA, illustrated on Figure  5  of the paper.
Influence of the within-subject covariance Aim. Group-level decoding is strongly influenced by the ratio of the inter-and within-subject variance. To study the influence of this ratio, we performed experiments in the paper where the within-subject covariance Σ was fixed and the intersubject variability was parametrically controlled, using our generative model to create a large number of artificial datasets. Here, we study whether our results hold when we change the within-subject variance. Experiments. In this section we vary both the within-and inter-subject variability. We keep the same range of values for Θ, which controls the amount of inter-subject variability: Θ ∈ {0. 2π, 0. 25π, 0. 3π, 0. 35π, 0. 4π, 0. 45π, 0. 5π, 0. 55π, 0. 6π, 0. 65π, 0. 7π}. And we generate five new sets of 14300 datasets using five values for the within-subject covariance matrix:  Results.
Our results are shown in Tables  S9 to S13. As previously, G-MVPA and ISPA yield more that 50 detections out of the 100 datasets available in each cell in regions that only partially overlap. Both strategies are strongly influenced by the within-subject covariance Σ, i. e. they prove less effective when going from Σ 1 to Σ 5 , i. e. when the within-subject distinctiveness of the patterns decreases. However, the shape of the different regions in this parameter space remains constant: G-MVPA is effective in a rectangle area (blue + green cells), showing that it is not affected by the amount of inter-subject variability; and ISPA is effective in a triangle-like area (blue + red cells). This shows that the qualitative nature of our results does not seem to be affected by the value of Σ. Σ 1 = 1 0 0 5 , Σ 2 = 3 0 0 5 , Σ 3 = 5.
Influence of spatial smoothing on the results obtained on the real fMRI datasets Aim. The preprocessing steps performed in the paper on the two fMRI datasets did not include any spatial smoothing. Since there is a debate in the literature on the validity and interest of such smoothing when performing MVPA, we here study the influence of the smoothing on our results. Experiments. We replicate the same experiments as in the paper by adding some spatial smoothing on the beta maps that are used to construct the inputs of the classifier. We use the Gaussian kernel implemented in SPM to perform the smoothing, with full-width at half-maximum values of 3 mm and 6 mm. Results. For clarity, we denote as Dataset1-s3 and Dataset2-s3 the versions of the two datasets with the 3 mm smoothing, and Dataset1-s6 and Dataset2-s6 with the 6 mm smoothing.
The results are presented in the same way as in the paper for the unsmoothed data: Figure  S2  and S3  show the thresholded statistical maps, displaying the significant clusters; Figure  S4  and S5 present the analysis of the size of the main cluster and its peak statistic value, for each dataset, each hemisphere and each size of the searchlight radius; finally, Figure  S6  and S7 describe how the main clusters obtained with G-MVPA and ISPA overlap. Overall, the results obtained with smoothing are consistent with what we found with the unsmoothed data: clusters were found in the same regions of the brain, their size increased with the value of the searchlight radius and the patterns of overlap between the clusters found by G-MVPA and ISPA were overall similar. The main effect of smoothing appears to be that the number of significant voxels increases when the size of the smoothing kernel increases, from 0 mm to 3 mm to 6 mm.
This can be explained by the fact that a small amount of smoothing helps reduce noise which can slightly improve decoding accuracies on the one hand and also increase the reproducibility across cross-validation folds on the other hand. This therefore facilitates detection as probed by our statistical test on accuracies. Furthermore, a notable difference observed with 6 mm smoothing is the fact that the size of the activated clusters found by ISPA for Dataset2-s6 is as large as the ones found by G-MVPA, which is not the case with smaller or no smoothing (see bottomleft graph in Figure  S5 , compared with the one in Figure  S4  or with Figure  4  in the paper). Congruently, the pattern of overlapping of the clusters found by the two strategies is modified for Dataset2-s6 when compared with the one found with smaller or no smoothing (see the right graph of Figure  S7 , compared to the right graph on Figure  S6  and Figure  6  in the paper): the number of voxels detected only by G-MVPA (green voxels) decreased in an important proportion.
This is consistent with the putative explanation that was put forward in the paper, that stated that the large amount of green voxels found in Dataset2 could be caused by a large inter-individual variability: indeed, one of the effect of spatial smoothing is often to reduce this interindividual variability. In a complementary manner, this could also mean that the idiosyncrasies that could drive the detection of the green voxels leave at high spatial frequencies. The spatial smoothing, by attenuating these frequencies, would therefore make them vanish as we observe in this experiment.
Bias and variance analyses for G-MVPA and ISPA Aim. In order to further compare G-MVPA and ISPA, we here assess their bias and variance for estimating classifiers. For this, we exploit the generative model used to construct our artificial dataset to define the true classifier, and measure how each strategy deviates from this ground truth. Experiments. In this section we use the same artificial datasets as in Section 2. 3 of the paper. Before inducing some random inter-individual variability with a rotation around the origin, the two classes are separated by the the straight line x = 0. Because the mean variability (i. e. rotation angle) across the population is equal to zero, the true group-level classifier is defined by f : x = 0. Classifiers estimated from the i-th subject in G-MVPA or the i-th cross-validation split in ISPA are denoted as f i G and f i I , respectively.
We use the angle between the estimated and true classifiers to measure the estimation error: the angles between f i G and f , f i I and f are denoted as θ i G and θ i I , respectively. We can then compute: where bias G , var G and bias I , var I are the bias and variance of G-MVPA and ISPA respectively. For each point in the parameter space defined by d and Θ, we compute the average bias and variance of the 100 datasets available in each cell of the tables shown in Section 3. 1 of the main paper. Results. The values of the average bias and variance of G-MVPA and ISPA are shown as the intensities of the images presented on Figure  S8 , which represent the parameter space defined by d and Θ, as in Tables 1-3 of the main manuscript. Overall, the bias and variance of ISPA appears to be more constant than the ones of G-MVPA, which reach much higher values.
Acknowledgments This work was carried out within the  Institut Convergence ILCB  ( ANR-16-CONV-0002 ). It was granted access to the  HPC resources of Aix-Marseille Université  financed by the project  Equip@Meso  ( ANR-10-EQPX-29-01 ) of the  French program Investissements d'Avenir. The acquisition of the data was made.
The system included an infrared (IR) reflective sensor glued together with small 1. 5 mm air-delivering tubing positioned just beneath the superior eyelid of the subject. The EBC device, comprising a portable airpuff and headset sound-delivering unit, controlled the timing, and intensity of both the airpuff and the sound (pure tone). It also converted the analog IR-reflection signal to numeric and sent it to a personal computer. The IR-reflection signal was collected online with the San Diego Instruments Labview software and then analyzed offline with a custom-made routine under Python (Python Software Foundation). During the overall experiment a continuous background white noise was delivered to the subject through the headset in order to provide constant ambient noise. The conditioning stimulus (CS) consisted of a 400 ms-1.