dataesr/scientific-paragraphs-categorization huggingface dataset:
SPA steps The general method from which the sloppiness of parameters can be quantified requires a number of steps in setting up a model. As noted previously, it first requires parameter optimization. This is different from published versions of CDP, which all had parameter sets that were chosen by hand. Thus, we will first use a parameter optimization algorithm to find the best parameters to explain the largest amount of variance in seven published databases of 'reading aloud' (the term 'naming' is also often used) before the SPA is done. Six of these databases contain words and one contains nonwords. Once optimization has been done, it is possible to examine how systematic changes in parameters affect model performance. By examining how parameters and combinations of them change the performance of the model as they are moved away from the optimum, it is possible to get an idea of how much one parameter (or combinations of them) affects model performance compared to another.
This is because it is easier and less constraining to fit a model to one data point (i. e. , a single empirical phenomenon) compared to many data points (e. g. ,  Adelman et al. , 2014). In this case, the parameter space usable to explain just one effect can only be larger than the parameter space that a model can use when it needs to explain many effects. For example, with models designed to explain many effects, it may be possible to show a good fit on one effect but be completely off on another, hence reducing the possible parameter space compared to a model that does not need to capture more than one effect. This is a clear advantage of using heterogeneous data for model optimisation and testing, and not just a single effect. To get around the problem of testing computational models and their ability to generalize, one common practice has been to examine the performance of a model using a single set of parameters across multiple databases and comparing that to simple regression with multiple predictors thought to be important in the task as a quantitative baseline, as discussed in the introduction.
Optimization specifics To find the optimum parameter values for CDP on any given dataset, we used the particle swarm algorithm of  Kennedy and Eberhart (1995). The code was adapted from the Standard POS 2006 version  (Clerc, 2012). The default parameters were used. There is no inherent reason why this algorithm is preferred over others. However, it is well known to be a powerful search algorithm and can search across non-continuous state-spaces as well state-spaces of high dimensionality  (Esmin, Coelho, & Matwin, 2015). Practically, it was very efficient in optimizing the parameter values of CDP and thus made the problem tractable. For each data set examined, all of which contained the mean reaction times (RTs) of individual items, optimization was done on 27 different parameters (see Table  1 ).
Twenty-two particles were used and searching of the parameter space was allowed to continue for 26 cycles. This meant that there were 572 runs of different parameter sets each time a model was optimized on a dataset before searching was terminated. The search termination criterion was empirically defined based on initial optimization test-runs showing an asymptote in the cost function minimization. Moreover, the entire procedure was repeated 50 times for each data set and the model with the lowest error rate from the 50 models was chosen for further analyses to increase the chance of finding a model as close as possible to the optimum that CDP could achieve. The model parameter values were found using two different cost functions. As is typical in the reading domain, one was a simple correlation between results produced by the model and human reading aloud latencies (i.
e. , all of the parameters investigated), and j represents the number of words the model was optimized on (note that words the model produced an error on were not used in the calculation). The models were independently optimized on the individual items (word RTs) from each of seven databases. Four contained only monosyllables  (Spieler & Balota, 1997 , Balota & Spieler, 1998; Treiman et al. , 1995; Seidenberg & Waters, 1989  (called the Waters database below to avoid confusion with the Seidenberg Nonwords)), two contained disyllabic words  (Chateau & Jared, 2003; Yap & Balota, 2009) , and one contained only nonwords  (Seidenberg et al. , 1994). Finally, one minor difficulty in the area of reading aloud (and an issue that arises here) is determining what the 'correct' responses are for nonwords.
Optimization performance The results of the 14 models (7 datasets × 2 cost functions) and hence 14 parameter sets that were created via optimization are presented in Table  2. As can be seen, the correlations for the 50 runs of each dataset with the actual data where the model used a correlation cost function were surprisingly similar, although more variable with the MSE cost function. In addition, the models with the strongest correlations in the data with the MSE cost function models had r values that were only slightly less than the models using the correlation cost function. Alternatively, the models with the lowest r values were much lower with the MSE cost function models. This suggests that it was harder to fit models using the MSE cost function. This is unsurprising given searching for correlations is likely to be easier because exact values do not need to be found and also because our model testing used r scores.
". Next, we examined the generalization performance of the model with the lowest error score in each dataset by testing it on the other datasets. That is, for each dataset, the model with the lowest error score  was first chosen (i. e. , seven parameter sets for the models using the correlation cost function and seven parameter sets for the models using the MSE cost function). Each of these models was then used to predict the results of all seven datasets, hence leading to 49 correlations (i. e. , 7 × 7) in each set. This included the correlations where the optimized parameters were tested on the items from the experiments they were optimized on. One difficulty there is when selecting models is that whilst choosing the optimum models with the correlation cost function is obvious, it is less obvious which models should be chosen when using the MSE cost function.
Interestingly, the performance of the optimized models on the data sets they were not optimized on (i. e. , generalization performance) also tended to be better with the models chosen based on the highest correlation rather than the lowest MSE score. The results from the models selected based on having the strongest correlation are reported in Table  3 , the results from the models selected based on having the lowest MSE scores are reported in Table  S1  of the Supplementary material, and SDs on each database are reported in Table  S2. As can be seen, differences between both types of model are small. When referring to the models that use a MSE cost function below, we will always refer to those where the optimal ones were selected based on having the strongest correlation with the data. Note that all calculations (i.
[2010b] ). The HAL database was used for word frequency. The regression analyses were done because it has been argued that a computational model should be better or at least as good as the amount of variance accounted for by a linear regression using these four predictors  (Spieler & Balota, 1997). However, historically, most models stayed well below this critical benchmark. Even the latest version of CDP  (Perry et al. , 2013) , which performed better than all previous models, stayed just below this critical benchmark. As can be seen, correlations between model behavior and the different experiments were generally quite high. The optimized models also correlated more strongly than the regression equations (words) apart from on one data set  (Spieler Young)  where the results were of a similar strength.
The results are also slightly higher than the best handpicked parameter sets we have reported elsewhere (e. g. ,  Perry et al. , 2013). In particular, using exactly the same model as here,  Perry et al. (2013)  report r values of. 43,. 49,. 42, and. 29 for the  Spieler and Balota (1997/1998) ,  Treiman et al. (1995) , and  Seidenberg and Waters (1989) , databases, respectively. Rather than use the optimum models as a comparison, it is also possible to examine the generalization performance of each parameter set, which, across those 4 data sets was. 41 in  Perry et al. (2013). With the models here, the results ranged from. 40 to. 42 (excluding the models optimized on the Seidenberg nonwords). This suggests the results are quite comparable, despite no hand tuning of any parameters. In terms of the  Chateau and Jared (2003)  and  Yap and Balota (2009)  databases, these are not so easy to compare because  Perry et al.
(2013)  only reported regression results where 8 factors were used to predict RTs. They then compared the regression results to those from CDP plus an additional factor. The results they found showed that CDP picked up a similar amount of variance to the regression equation. Whilst we did not use the same factors to compare CDP as  Perry et al. (2013)  did, as can be seen from Table  S3 , the optimized CDP models do better than a regression that included 4 factors. In terms of more specific details, the mean correlation from each data set the models were optimized on was. 49/. 48 (Correlation/MSE cost function) compared to. 37/. 41 (Correlation/MSE cost function) for sets they were not optimized on. The model optimized on the Seidenberg nonwords had the worst generalization performance. The Seidenberg nonwords were also comparatively difficult to predict for models optimized on the other databases.
This suggests that, perhaps unsurprisingly, words are better at predicting words than nonwords. The error rates of the different models are also informative (see Table  4 ). The only comparison where high error rates occurred was when the model using the correlation cost function was optimized on the Chateau and Jared words and then used to predict the Seidenberg nonwords. This suggests that models only optimized on words may but will not necessarily have trouble generalizing to nonword pronunciations. It is possible to evaluate CDP further with respect to the regression equations in terms of how well those equations generalize to data sets they were not fit to. To do this, we ran regressions on all of the data sets with only words in them. In a similar way to what we did with CDP, we first took the parameters from the regression equations that were derived from each dataset they were calculated from.
The model thus behaves like most models of real-world systems in this respect (e. g. ,  Gutenkunst et al. , 2007a). There were some differences caused by the type of database. As can be seen, with the Seidenberg nonwords, the parameter range was broad with all orthographic lexical parameters, apart from the letter to orthographic lexicon inhibition. This was unlike the words. This should not be seen as surprising, because words are stored in the lexicon of CDP but nonwords are not, and so parameters relating to lexical (word) retrieval should have less effect with nonwords. The main exception to this is the letter to orthography inhibition parameter. This parameter stops spurious forms (i. e. , words with similar but not the same letters as the word/nonword that is presented) from being incorrectly activated.
Thus, correlations allow more parameters to have weak contributions to the final scores. A final important difference was that the Seidenberg nonwords used a reasonably specific minimum naming criterion with the MSE cost function models (i. e. , how much activation at the phonemic buffer is needed before a word can be read aloud), unlike the word databases. This is likely to be because nonwords create less activation than words in the model because they are not supported by lexical information. Thus, it is harder for them to generate enough activation to reach the minimum naming criterion at the phoneme level. These results suggest that differences between the naming of words and nonwords can be compensated for by using a lower naming criterion before naming occurs with nonwords. Thus, with the MSE cost function model, CDP predicts people read words and nonwords aloud based on different amounts of information being available when the words are presented in pure lists (i.
It is worthwhile noting that the values used for equation (8) need to be close enough together so the approximation can be done reasonably but not so close together that the approximation may be likely to be affected by small fluctuations that are not representative of the immediate state-space they are embedded in. To examine this, we looked at how the cost score changed each time this function was used across all possible intervals in all of the databases (43327 data points). As can be seen in the Supplementary material (Fig. S5 ), the two smallest steps only ever cause the cost function to deviate tiny amounts, and there were no obvious outliers. The third step also causes relatively small differences, although a small number of outliers begin to be produced. Given the size of the changes, our choice of using the first two steps thus seems reasonable.
Hessian eigenvalues and eigenvectors The specific importance and influence of each parameter in the sloppy model can be further quantified via ranking methods designed to make use of the information provided by the eigenvalues and eigenvectors of the Hessian matrix. Eigenvalues and eigenvectors of the Hessian matrix can be determined by solving the following set of linear equations H θ E = λE (9) where E is an eigenvector that satisfies Eq. (  9 ) and λ is its respective eigenvalue. In practice we achieve this by using the Matlab TM eig() command which returns a diagonal NxM matrix of eigenvalues and a NxM matrix of eigenvectors, where each eigenvector is a column of the returned matrix. The leading eigenvalue of the returned quantities, λ i (largest magnitude) points in the stiffest direction -that which corresponds to the thinnest principal axis of the cost ellipsoids (see Fig.
3 ), defined as w i = 1 ̅̅̅̅ λ i √ (10) where w i is the width related to the i-th eigenvalue. The eigenvectors associated with the eigenvalues describe the stiffest/sloppiest combination of parameters. A total of 27 parameters in each of seven data sets are explored within this study. For each data set, the Hessian matrix of the cost function is computed using forward finite difference measures, giving a total of seven 27 × 27 Hessian matrices for each of the correlation and MSE cost function model types. For each matrix the eigenvalues and eigenvectors are calculated and are investigated using the relevant measures of interest which are explored below. We note that we only used positive eigenvalues and applied a lower limit of 10 -10 on the acceptable magnitude. Any values that fell below this were ignored.
Sloppy parameter analyses results We first examined the seven databases that we optimized CDP on. As can be seen in Fig. 4 , the results of the SPA showed that the eigenvalues calculated from each database showed a typical sloppy spectrum, with most of the variability always predicted by a small number of eigenvalues and many that have essentially no effect on the behavior of the model. In particular, in the graph, each blue line represents an eigenvalue, and the Y axis represents how large it is. Because the Y-axis uses a log scale, the fairly linear spacing of the eigenvalues in each database means that there is an exponential increase in how large the value of each eigenvalue is. In addition, because the value of an eigenvalue is directly related to the extent to which the eigenvalue affects the performance of the model (e.
g. , an eigenvalue of 10 means that it accounts for 10 times more of the differences in model performance than a value of 1), it means that a small number of eigenvalues account for most of the variation in the performance of the model in each dataset. Thus, across all of the simulations, there are a small number of dimensions in the data that are responsible for most of the model performance (note that these should not be confused with dimensions of the raw parameter axesi. e. , the parameters that our model uses -for example the 'X' and 'Y' dimensions of a map). This distribution of eigenvalues found is very important because it means that the model can potentially be tractably analysed. If every dimension had a similar effect on performance (i. e. , the eigenvalues were all similar) the results would be much more difficult to interpret.
Alternatively, the correlation cost function eigenvalues spanned over 2-3 orders of magnitude, which is less than the results from other models that have been reported in different areas. This suggests that the state-space of the MSE cost function models have stiffer dimensions than the correlation cost function models. Alternatively, the type of scaling had much less effect on the eigenvalues. Next, we examined which parameters were important in CDP. We did this by calculating parameter rankings for all of the parameters based on the eigenvalues and eigenvectors found for each dataset (see Supplementary material for how this calculation was performed). This makes it possible to analyse how strongly each parameter was aligned with the strongest eigenvectors. High values represent parameters that are stiff parameters and align well with eigenvectors with high values.
The other parameters tended to be important in only a subset of the databases. In summary, the results of the analyses of the parameters across seven large databases provide a number of useful insights. Notably, the number of parameters that affect CDP is much more limited than the number of parameters it has. This suggests CDP's predictions are much more constrained than what might be assumed from the number of parameters it has. The results also show that quite similar experiments can produce different patterns of sensitivity with the model's parameters. A consequence of this is that having accurate intuitions about the key factors in certain experiments according to the model is likely to be difficult, and only testing the differences with the stiff parameters can provide the correct insights.
Conclusion Challenges in understanding how complex models work and the risk of overfitting has been often addressed in cognitive modelling by trading off predictive accuracy with model simplification. Our results show that a cognitive model with a complex architecture and a large number of parameters like CDP does not overfit the empirical data even when using a parameter optimization algorithm. Notably, optimizing the model on one dataset yielded parameters that generalized appropriately to many other datasets. Most importantly, parameter optimization allowed us to introduce SPA, a state-of-the-art mathematical technique, to quantify the effects of different parameters on model performance. This analysis showed that the parameters that do and do not meaningfully affect the performance of a complex cognitive model like CDP can be identified.
Repetition versus change control experiment We performed two sessions of an additional experiment with monkey A during which we showed four different types of sequence chunks containing each 4 images. The types of sequences were XXXX, XXXY, XYYZ and WXYZ, where letters indicate any of 948 grayscale images of objects from the Brainscore database  (Majaj et al. , 2015) , randomly changing with each sequence presentation. Sequence types were uniformly distributed across a recording session so that there was no global context. All stimuli could occur in any position and were presented 1 to 5 times. This experiment served to control for stimulusspecific adaptation that could underly the deviance response to the last stimulus in XXXY trials. As we used a broad range of stimuli, the images used as WXY are expected to be on average as distant from Z than X is from Y in XXXY trials, thereby leading to the same amount of stimulus-specific adaptation for the population responding to Y (in XXXY) as to Z in (WXYZ).
used a random permutation test with cluster-based correction for multiple comparisons(Maris and Oostenveld 2007). After estimating the population axes and projecting single trials onto these axes, we generated 100 surrogate datasets by shuffling the trial conditions of test trials. We then computed the AUROCs for the different sequence variables based on the trajectories with the shuffled trial labels. We averaged the true AUROCs across recording sessions (10 in monkey H, 7 in monkey A) and likewise obtained 100 surrogate session-averages. The true AUROCs per sequence variable were transformed into t-values by subtracting the average over the permutations and dividing by their standard deviation, separately for each time point. Absolute t-scores that passed a threshold of 3 standard deviations were candidates for significant clusters in time.
Acknowledgements We thank  Julien Lemaitre  for providing veterinary care,  Abhilash Dwarakanath  for his help with the data preprocessing pipeline and  Pierre-Louis Bellet  for providing the brain graphic of Figure 1. This project/research has received funding from the  European Union 's  Horizon 2020 Framework Programme for Research and Innovation  under the  Specific Grant  Agreement No. 945539  ( Human Brain  Project  SGA3 ).
Data The dataset used here is part of the MPI-Leipzig Mind-Brain-Body (MPILMBB) database  (Mendes et al. , 2019). The complete dataset consists of a battery of self-reported personality measures, measures of spontaneous thought, task data, and structural and resting-state functional MRI (one hour, divided into four adjacent 15-min sessions) from participants between 20 and 75 years of age. A detailed description of the participants, measures, and data acquisition protocol has been previously published along with the dataset  (Mendes et al. , 2019).
9) and then decomposed it using PCA. We chose the PCA approach for gradient calculation, as Hong and colleagues  (Hong et al. , 2020)  have shown that compared to non-linear decomposition methods, PCA provides better reliability and higher phenotypic predictive value for connectivity gradients. For ease of interpretation and comparability, group gradients were aligned to a subsample of the HCP dataset  (Essen et al. , 2013)  included in BrainSpace. Finally, following  Mckeown et al. (Mckeown et al. , 2020)  10 gradients were calculated in order to maximize the gradient fit for all individuals during alignment. Individual gradients were calculated for each participant, aligned to the group-level gradients, resulting in a 400x10 matrix for each participant. Subsequent regression analyses were limited to the first three gradients, which have been relatively well-characterized in previous work  (Margulies et al.
, 2020). The endpoints of G1 are different in that one end is anchored by the visual network alone, as opposed to visual and somatomotor, while in G2, the somatomotor network is separated from both the visual and default mode networks, as opposed to the visual network alone. For ease of interpretation and comparability with previous studies  (Mckeown, Turnbull) , gradients were aligned to a subsample of the HCP dataset  (Essen et al. , 2013)  included in BrainSpace. Similar to this template, the first two gradients together describe network-level connectivity space, anchored at three ends by the visual, somatomotor and default mode network, respectively (Fig. 3 ). Single gradients tended to be stable over the 4 sessions, with a discriminability index of 0. 964 for Gradient 1, 0. 918 for Gradient 2, and 0.
Acknowledgments: We thank the  Bordeaux Imaging Center  funded by  ANR-10-INBS-04  for imaging,  Motac Neuroscience  for lending the rotarod apparatus,  Céline Ducroix-Crépy  and  Jean-Christophe Helbling  ( INRAE UMR 1286 ) for genotyping and the staff from the animal facility of INRAE UMR 1286 for animal care.