dataesr/scientific-paragraphs-categorization huggingface dataset:
After conducting these transformations, we added the effects of the covariates (adding some value multiplied by the covariate), truncated the resulting values to avoid outliers, and rescaled the data to be like FreeSurfer again. We created many datasets, but we chose some that effectively only showed EoS or only showed true effects and were varied in features and BAC. To know which only showed EoS or only showed true effects, we used a logistic regression model to predict the response and calculated the accuracy using both standard formulas and the "multisite. accuracy " R package, which corrects for the site  ( Solanes et al. , 2021 ). We considered "datasets with only EoS " those with ∼50% mul-  (a)  100% "Simple true effects #1 " 73% 73% 0% "Simple true effects #2 " 82% 82% 0% Average RAC [simple true effects] : (a)  0% Simple EoS-removal effectiveness: (a)  100% MAREoS: Method Aiming to Remove the EoS.
( a) Average RAC s and EoS-removal effectiveness are limited to 0-100%. These numbers may differ slightly from those reported at https://www. imardgroup. com/mareos-benchmark/ because the latter are based on a parallel collection of datasets for which the variable "response " is not public. tisite accuracy -even if they showed high raw accuracy. Similarly, we considered "datasets with only true effects " those that showed similar (high) raw and multisite accuracies  ( Solanes et al. , 2021 ).
Measurement of the EoS-removal effectiveness of a MAREoS for simple EoS As detailed above, each dataset contains baseline multisite MRI data from patients and the subsequent responses to a given treatment. First, separately for each dataset and within a ten-fold cross-validation scheme, the developers must use the training subset to fit and apply a MAREoS to remove the EoS, find and remove the linear effects of two clinical covariates, fit the simple machine-learning algorithm, and use all these models to predict whether patients in the test subset respond to treatment. Second, again separately for each dataset, the developers must calculate the predictions' sensitivity, specificity, and balanced accuracy (BAC). The sensitivity is the percentage of responders correctly predicted to respond.
The specificity is the percentage of non-responders correctly predicted not to respond. The BAC is the average of sensitivity and specificity. Relevantly, the developers must calculate the BAC using these basic formulas. In other words, they cannot correct the site with the "multisite. accuracy " R package  ( Solanes et al. , 2021 )  that we would otherwise recommend. The reason is that we need uncorrected accuracies to measure the EoS-removal effectiveness of a MAREoS. Developers may download the specific R scripts to conduct all these steps from https://www. imardgroup. com/mareos-benchmark/. Afterward, they must perform the following calculations to measure the EoSremoval effectiveness of the MAREoS. The first calculation, also performed separately for each dataset, is the relative accuracy change ( RAC ): 𝑅𝐴𝐶 = [ 𝐵𝐴𝐶 𝑤𝑖𝑡ℎ𝑜𝑢𝑡 𝑀𝐴𝑅𝐸𝑜𝑆 ] -[ 𝐵𝐴𝐶 ] [ 𝐵𝐴𝐶 𝑤𝑖𝑡ℎ𝑜𝑢𝑡 𝑀𝐴𝑅𝐸𝑜𝑆 ] -50% To illustrate the idea, Table  1  shows the RAC calculations in the "Simple EoS #1 " and "Simple true effects #1 " datasets using a standard support vector machine (SVM) algorithm and ComBat (see details later).
In the dataset with simple EoS, BAC was 74% when we fitted the SVM without applying any MAREoS. Using a MAREoS, the (EoS-inflated) BAC decreased to 50%. Then, RAC  [simple EoS]  in this dataset would be 100% (i. e. , the accuracy is 100% closer to 50%). We could naively interpret that the MAREoS has reduced the bias by 100%. However, in the dataset with simple true effects, BAC decreased from 73. 05% to 73. 02% when using a MAREoS due to an undesirable potential side effect: data degra-dation. Then, RAC  [simple true effects]  in this dataset would be 0. 14% (i. e. , due to data degradation, the accuracy is 0. 14% closer to 50%). At a theoretical level, it might be interesting to note that the formula of the RAC would also work for accuracy increases -possible in datasets with both true effects and EoS.
In such datasets, the EoS might prevent the machine-learning algorithm from fully detecting the true effects. For instance, BAC could be 70% before a MAREoS, while 75% after the MAREoS, for what RAC would be -25%, now meaning that the accuracy is now 25% farther from 50%. However, this RAC would be little informative because we would know neither the amount of EoS removed nor whether there was also data degradation. Turning to the measurement of the EoS-removal effectiveness, the second calculation consists of adjusting the average RAC [simple EoS] (i. e. , the naïve bias reduction) with the average RAC [simple true effects] (i. e. , due to data degradation) to derive the EoS-removal effectiveness: EoS -removal ef fectiveness = mean ( 𝑅𝐴𝐶 [ simple EoS ] ) -mean ( 𝑅𝐴𝐶 [ simple true ef fects ] ) 100% -mean ( 𝑅𝐴𝐶 [ simple true ef fects ] ) Limiting the average RAC s and the EoS-removal effectiveness to 0-100% may be sensible.
Going back to Table  1  , if a MAREoS shows RAC [simple EoS] = 100% (a naïve 100% reduction in bias) and RAC [simple true effects] = 0. 14% (a 0. 14% decrease due to data degradation), then the simple EoS-removal effectiveness was 100%. In the datasets with only simple EoS, we may assume that the MAREoS would first remove simple EoS, reducing the accuracy. And afterward, it would degrade the data leading to (minimally) decreasing the remaining accuracy ( Figure  2 , A1  ). Or vice versa, we may assume that the MAREoS would first degrade the data, (minimally) decreasing the accuracy. And afterward, it would remove simple EoS, reducing the remaining accuracy ( Figure  2 , A2  ). Data degradation may seem negligible in this example, but it might be relevant in others.
Measurement of the EoS-removal effectiveness of a MAREoS for complex EoS The overall strategy for measuring how well a MAREoS removes complex EoS is the same as for measuring how well a MAREoS removes simple EoS. However, datasets with complex EoS may also include simple true effects or simple EoS, and the MAREoS may remove both simple and complex EoS. Therefore, we may wish to subtract the part of the BAC attributable to simple effects as follows: 𝐵 𝐴𝐶 [ 𝑐 𝑜𝑚𝑝𝑙𝑒𝑥,𝑐 𝑜𝑟𝑟𝑒𝑐 𝑡𝑒𝑑 ] = 𝐵 𝐴𝐶 [ 𝑐𝑜𝑚𝑝𝑙𝑒𝑥 ] - ( 𝐵 𝐴𝐶 [ 𝑠𝑖𝑚𝑝𝑙𝑒 ] -0. 5 ) where BAC  [complex]  is the BAC obtained with the complex machinelearning algorithm (e. g. , a lasso with first-order interactions), BAC  [simple]  is the BAC obtained with the simple machine-learning algorithm (i. e. , the lasso without interactions), and BAC  [complex,corrected]  is the BAC  [complex]  after "subtracting " the simple effects.
In the dataset with complex EoS, BAC  [complex]  was 87% when we fitted the SVM without applying any MAREoS. However, SVM predictions were likely not exclusively based on complex effects but also simple effects. To quantify these simple effects, we used a simple lasso, which showed a BAC [simple] of 66% ( Table  2  ), i. e. , 16% more than tossing a coin. Removing the part of accuracy due to simple effects with the formula above, we estimated that SVM's BAC  [complex,corrected]  was 87% -16% = 71%. Using a MAREoS, SVM's BAC  [complex]  decreased to 66% and lasso's BAC  [simple]  to 50% (i. e. , 0% over tossing a coin, Table  2  ). We thus estimated that using a MAREoS, SVM's BAC  [complex,corrected]  was 66% -0% = 66%. Finally, we could estimate that RAC [complex EoS] was 21% using the same formula as for simple effects: 𝑅𝐴𝐶 [ 𝑐𝑜𝑚𝑝𝑙𝑒𝑥 ] = [ 𝐵𝐴𝐶 𝑤𝑖𝑡ℎ𝑜𝑢𝑡 𝑀𝐴𝑅𝐸𝑜𝑆 ] [ 𝑐 𝑜𝑚𝑝𝑙𝑒𝑥,𝑐 𝑜𝑟𝑟𝑒𝑐 𝑡𝑒𝑑 ] -[ 𝐵𝐴𝐶 ] [ 𝑐 𝑜𝑚𝑝𝑙𝑒𝑥,𝑐 𝑜𝑟𝑟𝑒𝑐 𝑡𝑒𝑑 ] [ 𝐵𝐴𝐶 𝑤𝑖𝑡ℎ𝑜𝑢𝑡 𝑀𝐴𝑅𝐸𝑜𝑆 ] [ 𝑐 𝑜𝑚𝑝𝑙𝑒𝑥,𝑐 𝑜𝑟𝑟𝑒𝑐 𝑡𝑒𝑑 ] -50% We did the same calculations for the dataset with complex true effects.
Example: ComBat To exemplify how to measure the EoS-removal effectiveness of a MAREoS, we applied ComBat to the public datasets provided. We then conducted the calculations needed to measure the EoS-removal effectiveness. First, we downloaded the public datasets from https://www. imardgroup. com/mareos-benchmark/. Each dataset is a table with the following columns: the identification of the simulated individual, the MRI data (cortical thickness or surface area or subcortical volumes), the site, the values of the two clinical covariates, and the distribution in folds. We also downloaded the Johnson-Fortin-Radua version of the ComBat MAREoS  ( Fortin et al. , 2018 ; Johnson et al. , 2007 ; Radua et al. , 2020 )  from https://enigma. ini. usc. edu/protocols/statistical-protocols/ The following analyses refer to one dataset.
After repeating the same procedure for folds 2-10, we had predicted the response in all individuals. We then proceeded to calculate the sensitivity, specificity, and BAC. Finally, we combined the BACs with and without ComBat to calculate the RAC. We provide the R scripts to conduct such calculations at https://www. imardgroup. com/mareos-benchmark/ After conducting these analyses for each "Simple " dataset ( Table  1  ), we had a RAC for each of the "Simple EoS " datasets (100% and 107%), which we averaged (and limited to 0-100%) to obtain an average RAC [simple EoS] of 100%. Similarly, we had a RAC for each of the "Simple true effects " datasets (0% and 0%), which we averaged to obtain an average RAC [simple true effects] of 0%. Finally, we calculated the EoS-removal effectiveness (100%).
Therefore, we should conclude that ComBat entirely removes the bias related to simple EoS (100%) and has negligible data degradation (0%). The results were nearly identical when we used a lasso algorithm without interactions for the "Interaction " datasets ( Table  2  ). The RAC s for the "Interaction EoS " datasets were 100% and 100% (average RAC [simple EoS] = 100%), and the RAC s for the "Interaction true effects " datasets were -1% and -4% (average RAC [simple true effects] = 0%), leading to EoS-removal effectiveness = 100%. Therefore, we should conclude again that ComBat entirely removes the bias related to simple EoS (100%) and has negligible data degradation (0%). These datasets had complex effects (e. g. , the multiplication of pairs of brain regions differed between groups). However, these effects are not detectable by the lasso algorithm without interactions; thus, they did not influence these calculations.
The results were very different when we analyzed the same "Interaction " datasets using a lasso algorithm with first-order interactions ( Table  3  ). First, all BAC were substantially higher (e. g. , 80% instead of 63% for "Interaction EoS #1 " without MAREoS). This increase is because this machine-learning algorithm could detect the first-order interactions present in these datasets. However, we must highlight here that, as we saw in the previous paragraph, these datasets also included simple effects, which we had to subtract before specifically studying the complex effects. For instance, for "Interaction EoS #1 " without MAREoS, BAC was 80%, but we subtracted 13% (i. e. , the BAC of the simple algorithm, 63%, minus 50%) for what the corrected BAC was 80% -13% = 67%. Once we corrected all BACs, we proceeded as before.
The RAC s for the "Interaction EoS " datasets were -5% and -13% (average RAC [complex EoS] = 0%), and the RAC s for the "Interaction true effects " datasets were 3% and 5% (average RAC [complex true effects] = 4%), leading to EoS-removal effectiveness = 0%. Therefore, we should conclude that ComBat does not remove the bias related to complex EoS (0%) and may show minor data degradation (4%).
Other machine-learning algorithms We repeated the above calculations with standard random forest  ( Liaw and Wiener, 2002 ) , support vector machine  ( Meyer et al. , 2021 ) , and gaussian processes  ( Karatzoglou et al. , 2004 )  algorithms to provide insights on the use of MAREoS with these machine-learning algorithms. We used the default options of the "", "", and "" R packages, which involve radial basis function kernels for support vector machine and gaussian processes. We show again the R code to conduct such calculations at https://www. imardgroup. com/mareos-benchmark/. With the "simple " datasets, BACs, RAC s, and simple EoS-removal effectiveness were similar when using lasso, random forest, support vector machine, or gaussian processes algorithms ( Table  4  ). There were differences between algorithms, but they were small and likely due to chance.
The analysis of the "interaction " datasets showed that the random forest (and, to a lesser extent, the support vector machine) algorithm detects the complex effects in these datasets ( Table  5  ). In contrast, the Gaussian processes algorithm only detects some. We thus repeated the calculations of complex EoS-removal effectiveness for the random forest and support vector machine algorithms. Table  6  shows that random forests and support vector machine algorithms yielded RAC  [complex EoS]  substantially different from the 0% calculated using the lasso algorithm with first-order interactions. As introduced earlier, we assumed that lasso with first-order interactions could detect only one type of complex EoS: those based on first-order interactions. Thus, a RAC [complex EoS] of 0% meant that ComBat does not remove these complex EoS.
Benchmark website For machine-learning researchers, the website (https://www. imardgroup. com/mareos-benchmark/) includes information about the types of effects detectable by different machine-learning algorithms and the effectiveness of MAREoS in removing different types of EoS. See Figure  3  for a diagram of the steps to choose an appropriate MAREoS for a specific study. Note that the numbers in the website may be slightly different from those in the manuscript because the former are based on a parallel collection of datasets for which the variable "response " is not public. We created the latter datasets to keep objective ranks of the effectiveness of the different MAREoS for different types of EoS.
Dataset Balanced accuracy Corrected balanced accuracy (a)   Relative accuracy change (RAC) Average RAC [complex true effects] : (b)  4% "Complex EoS including interactions "-removal effectiveness: (b)  0% MAREoS: Method Aiming to Remove the EoS. ( a) Subtracting the simple effects, estimated as BAC with lasso without interactions minus 50%. ( b) Average RAC s and EoS-removal effectiveness are limited to 0-100%. These numbers may differ slightly from those reported at https://www. imardgroup. com/mareos-benchmark/ because the latter are based on a parallel collection of datasets for which the variable "response " is not public.
Dataset Balanced accuracy Corrected balanced accuracy (a)   Relative accuracy change (RAC) Average RAC [complex true effects] : (b) 1% 0% "Complex EoS including interactions "-removal effectiveness: (b)  51% 34% MAREoS: Method Aiming to Remove the EoS. ( a) the simple effects, estimated as BAC with the lasso algorithm without interactions minus 50%. ( b) Average RAC s and EoS-removal effectiveness are limited to 0-100%. These numbers may differ slightly from those reported at https://www. imardgroup. com/mareos-benchmark/ because the latter are based on a parallel collection of datasets for which the variable "response " is not public. save the resulting data with the set identification "train_fold1"; and c) remove these effects from individuals labeled to be in fold 1 and save the resulting data with the set identification "test_fold1".
Discussion This work presents a strategy to measure the EoS-removal effectiveness of a MAREoS in multisite machine-learning studies. We provide datasets with only simple true effects, datasets with only simple EoS, datasets with complex true effects, datasets with complex EoS, and formulas to measure the EoS-removal effectiveness from the BAC obtained when fitting prediction models in these datasets. We also provide a benchmark website to rank the EoS-removal effectiveness of the different MAREoS and a relationship of the types of EoS that may bias accuracy for different machine-learning algorithms. For instance, we report that ComBat removes all simple EoS as needed for predictions based on simple lasso algorithms while it leaves residual complex EoS that may bias the predictions based on standard support vector machine algorithms.
In other words, the extendable benchmark website provides the types of EoS that researchers should remove for a given machinelearning algorithm and the effectiveness of each MAREoS for removing each type of EoS. The most important limitation of the present work is that it encompasses only one type of complex EoS: those due to first-order interactions between brain regions. Other complex EoS could potentially derive from higher-order interactions or other regional relationships. However, we believe that the investigation of new types of complex EoS, along with the (challenging) development of methods to measure them (e. g. , creating specific datasets), should be the work of future studies. We created an extendable benchmark website for this reason. Another potential limitation of this work is that we only created binary outcomes (response vs.
, 2021 ; Noble et al. , 2017 ; Tanaka et al. , 2021 ; Tong et al. , 2020 ). The data from these individuals, known as "traveling subjects ", have two valuable characteristics. First, they are real data and thus very likely have hidden features that our datasets may not have. Indeed, some data indicate that the traveling-subject outperforms ComBat  ( Maikusa et al. , 2021 ). Second, these data allow an excellent study of the differences between MRI devices. In this scenario, differences between sites unrelated to MRI devices should be negligible. Together, these characteristics enable the development of promising deep learning-based MAREoS  ( Tian et al. , 2022 ). However, the traveling-subject approach also has its drawbacks. For instance, it can only be done prospectively (i. e. , it is not helpful for mega-studies based on previously acquired data, such as those in the ENIGMA consortium  ( Dima et al.
Acknowledgement This work was supported by the  Baszucki Brain Research Fund and Milken Institute  ( "The  ENIGMA Bipolar Medications Initiative: A Global Study of Bipolar Disorder Medication Associations in the Brain  " grant) and the  Spanish Ministry of Science, Innovation and Universities / Economy and Competitiveness / Instituto de Salud Carlos III  ( CPII19/00009 ,  PI19/00394 ,  FI20/00047 ), co-financed by ERDF Funds from the  European Commission  (" A Way of Making Europe ").
Whole-brain spatial dependency We first quantified the spatial dependency between functional connectivity and distance by calculating whole-brain variograms assessing how functional connectivity (Pearson's correlation) varies with distance along the cortical surface for each hemisphere (Figure  1 , lower panel). We used resting state fMRI data from 51 participants from the Human Connectome Project. We took two scans on the same data for each individual allowing us to calculate the reliability of these metrics within an individual. Averaging these vertex-wise variograms across the whole cortex, the global variogram, reveals an initially steep rise (rapidly increasing dissimilarity with distance (Figure  1 ). This is followed by a continuous increase up to the measured limit (all vertices included distances up to 150 mm, which was the maximum distance present for all vertices (see Supplementary Figure  1  for vertex distance distributions and higher upper measurement limits).
Imaging Data The data used in this study are available from the Human Connectome Project (https://www. humanconnectome. org/study/hcp-young-adult/document/extensivelyprocessed-fmri-data-documentation), the PRIMatE Data and Resource Exchange (https://fcon_1000. projects. nitrc. org/indi/indiPRIME. html) and Neurosynth https://neurosynth. org/analyses/topics/). The majority of the analyses were performed on 51 participants' resting state fMRI from the Human Connectome Project's minimally pre-processed dataset (34 female); this involved registration to a common MNI152 template, minimal spatial smoothing and extensive filtering for slow drifts, motion and other nuisance signals estimated using independent components analysis  (38). The 4D fMRI datasets for each participant were projected onto the Conte32k surface and the number of faces reduced resulting in 10,000 remaining vertices (using Matlab's reducepatch command).
Two resting-state runs (with opposite phase encoding direction, left-to-right and right-toleft, from the same scanning session) were taken from each participant. No further pre-processing was performed on the data. Since we were not focused on acrossparticipant or within-participant variability, and for computational efficiency, we focused only on two scans from a subset of the whole Human Connectome Project dataset. Group averaged data from 14 macaque monkeys (two female) was used from the Newcastle cohort. Surface geodesic distance and homologous regions to the human data were taken from  (39). The vertex-wise map of cortical myelin was the group-average map taken from the Human Connectome Project 900-subject release; it is released in the Conte32k surface space and reduced to the same 10,000 vertices as the fMRI data.
Similarly, the Yeo cortical parcellation (4) in Conte32k surface space was taken from the same HCP 900 data release and was also reduced to 10,000 vertices. The 50 Neurosynth data derived topic maps were downloaded in MNI152 2mm space and then projected onto the mid-thickness Conte32k surface using the Connectome Workbench  (40)  and then reduced to the same 10,000 vertices. Topics that were not related to cognitive tasks/states were removed, leaving 24 topics. Geodesic distance Pairwise geodesic distance was calculated along the cortical surface between all vertices (excluding the medial wall) using the Connectome Workbench tools, as implemented through the BrainSmash toolbox  (41). This was done on each hemisphere's mid-thickness Conte32k surface reduced to 10,000 vertices prior to calculating the distances.
Theoretical variogram It is common practice to fit a function to empirical variograms, this is typically used prior to spatial regression; however, in our case, it allows us to compactly summarise the shape of the empirical variogram with a small number of parameters, facilitating comparisons across datasets and vertices, and aggregation across multiple vertices. For the reported analyses we used an exponential function. This is motivated by a range of prior studies suggesting exponential relationships between distance and various neural measures (e. g. ,  (43) ). We also performed a similar fit for three other theoretical models (a Gaussian, a power-law model, and a periodic model which allows for non-monotonic functions), with qualitatively similar results. Empirical variograms were trimmed to bins between 2 and 19 (to remove bins with few sampled distances).
Cognitive tasks From the Neurosynth 50 data-derived topics dataset  (22) , those that did not refer to cognitive or behavioral states were removed, leaving: cognitive, inhibition, motor, numerical, action, conflict, spatial, emotion, empathy, decision, pain, memory, language, semantic, face, imagery, visual, eye movement, motion, attention, auditory, reward, social, working memory. The corresponding map for each topic was thresholded (absolute value z>10, although qualitatively similar results were observed for other thresholds) and binarized, resulting in a vertex-wise mask of values that were strongly implicated for that topic (other thresholds produced qualitatively similar results). For each topic, the range and sill (taken from the theoretical variogram from the group average functional connectivity analysis) for each vertex within each mask were averaged together.
Data availability The data used in this study are available from the Human Connectome Project (https://www. humanconnectome. org/study/hcp-young-adult/document/extensively-processed-fmri-data-documentation), the PRIMatE Data and Resource Exchange (https://fcon_1000. projects. nitrc. org/indi/indiPRIME. html) and Neurosynth https://neurosynth. org/analyses/topics/). Data to recreate the variograms in Figure  3  is available in source data file. Figure Legends.