A Spectral Theory of Neural Prediction and Alignment
Authors: Abdulkadir Canatar
Jenelle Feather
Albert Wakhloo
SueYeon Chung
Venue: NeurIPS 2023 spotlight (2023)
Keywords: computational neuroscience
neural manifolds
neuro-AI
statistical physics
representational geometry
TL;DR: We offer a theoretical analysis investigating the factors contributing to the neural prediction error in terms of the spectral properties of deep neural network models and brain responses.

Abstract: The representations of neural networks are often compared to those of biological systems by performing regression between the neural network responses and those measured from biological systems. Many different state-of-the-art deep neural networks yield similar neural predictions, but it remains unclear how to differentiate among models that perform equally well at predicting neural responses. To gain insight into this, we use a recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target. We apply this theory to the case of regression between model activations and neural responses and decompose the neural prediction error in terms of the model eigenspectra, alignment of model eigenvectors and neural responses, and the training set size. Using this decomposition, we introduce geometrical measures to interpret the neural prediction error. We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression. The work demonstrates that carefully decomposing representational metrics can provide interpretability of how models are capturing neural activity and points the way towards improved models of neural activity.

A Spectral Theory of Neural Prediction and Alignment
Abdulkadir Canatar 1.2
Jenelle Feather 1.2+
Albert J. Wakhloo 1.3 SueYeon Chung
Center for Computational Neuroscience, Flatiron Institute
Center for Neural Science, New York University
77uckerman Institute, Columbia Univeristy
Equal contribution
facanatar,jfeather, awkkhooo www.::/wwwaa.ooogogaaaaaaaoaaooe
Abstract
The representations of neural networks are often compared to those of biological systems by performing regression between the neural network responses and those measured from biological systems. Many different state-of-the-art deep neural networks yield similar neural predictions, but it remains unclear how to differentiate among models that perform equally well at predicting neural responses. To gain insight into this, we use recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target. We apply this theory to the case regression between model activations and neural responses and decompose the neural prediction error in terms of the model eigenspectra, alignment of model eigenvectors and neural responses, and the training set size. Using this decomposition, we introduce geometrical measures to interpret the neural prediction error. We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression. The work demonstrates that carefully decomposing representational metries can provide interpretability of how models are capturing neural activity and points the way towards improved models of neural activity.
Introduction
Neuroscience has a rich history of using encoding models to understand sensory systems [1, 2, 3]. These models describe mapping for arbitrary stimuli onto the neural responses measured from the brain and yield predictions for unseen stimuli. Initial encoding models were built from hand-designed parameters model receptive fields in early sensory areas [4, 5], but it was difficult to extend these hand-engineered models to capture responses in higher-order sensory regions. As deep neural networks (DNNs) showed human-level accuracy on complex tasks, they also emerged as the best models for predicting brain responses, particularly in these later regions [6, 7, 8, 9, 10].
Although early work showed that better task performance resulted in better brain predictions [6], in modern DNNs, many different architectures and training procedures lead to similar predictions of neural responses [11, 12, 13]. This is often true even when model representations from each other using other metrics of comparison [14, 15]. For instance, previous lines of work have shown clear differences between models from their behavioral responses corrupted texturized images [16, 17, 18), from their alignment to human similarity judgments [19], from generating stimuli that are "controversial" between two models [20], and from stimuli that are metameric models [15] or to human observers [21]. Other work has focused on directly comparing representations [22, 23] behavioral patterns [24] of one neural network to another, finding that changing the training dataset or objective results in changes to the underlying representations. Given the increasing number of findings that demonstrate large variability among candidate computational models, is an open
notably different
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

question of why current neural prediction benchmarks are less sensitive to model modification, and how to design future experiments and stimulus sets to better test our models.
Metrics of similarity used to evaluate encoding models of neural responses are often condensed into a single scalar number, such the variance explained when predicting held-out images [6. 11]. When many models appear similar with such metrics, uncovering the underlying mechanisms driving high predictivity is challenge [25]. To uncover the key factors behind successful DNN models of neural systems, researchers have proposed various approaches to explore the structural properties of internal representations in DNN models in relation to neural data. One promising approach focuses on analyzing the geometry of neural activities [25]. Several studies have examined the evaluation of geometries and dimensionalities of DNNs [26, 27, 28], highlighting the value of these geometric approaches. They provide mechanistic insights at the population level, which serves as an intermediate level between computational level (e.g., task encoding) and implementation level (e.g., the activities of single neurons or units). Furthermore, the measures derived from representation geometries can be utilized to compare the similarity between models and neural data [7, 25, 29]. Although these approaches have been successful, they do not directly relate the geometry of DNNs to the regression model used to obtain neural predictions, which uniquely captures the utility of a model for applications such as medical interventions
[11] and experimental designs [30]. Recent empirical findings by Elmoznino et al. [31] have suggested a correlation between the high-din -igediimensionalityy of model representations and neural predictivity in certain scenarios, but this work lacks explicit theoretical connections between the spectral properties and predictivity measures. Our work contributes to this line of investigation, by sceking theoretical explanation and suggesting new geometric measures linked to a model's 's neural prediction error for future investigations.
To bridge the gap between DNN geometry and encoding models used for neural prediction, we turn to a recent line of work that derived a predictive theory of generalization error using methods from statistical physics [32, 33J. Specifically, two different spectral properties that affect the regression performance were identified: Spectral bias characterizing how fast the eigenvalues of the data Gram matrix decay, and task-model alignment characterizing how well the target aligns with the eigenvectors of this data Gram matrix [33]. While the former describes how large the learnable subspace is, the latter defines how much of the variance of the target is actually learnable. Note that in general. these two properties may vary independently across representations. Hence, the trends among regression scores across DNNs must be carefully studied by taking into account the joint trends of spectral bias and task-model alignment.
In this work, we explore how encoding models of neural responses are influenced by the geometrical properties of the DNN representations and neural activities. Our analyses primarily concern how and why models achieve the predictivities that they do, rather than offering alternative scalar metric for predictivity. We find that the neural prediction error from ridge regression can be broken down into the radius and dimension of the error mode geometry, characterizing the error along each eigenvector. When investigating models along these error axes, we see a large spread in the eттor mode geometry, suggesting that the correlation values obtained from regression analyses obscure many geometrical properties of the representations.
Our primary contributions are:
We analytically decompose the neural prediction error of ridge regression from a model to a set of brain data in terms of the model eigenspectra, the alignment between the brain data and the eigenvectors of the model, and the training set size.
We introduce two geometric measures that summarize these spectral properties and directly relate to the neural prediction error. We show that these measures distinguish between different models with similar neural prediction errors using a wide variety of network architectures. learning rules, and firing rate datasets from visual cortex.
Using spectral theory, we demonstrate that for networks effective in predicting neural data, we can ascertain if their superior performance stems from the model's spectra or alignment with the neural data. Our findings indicate that (a) trained neural networks predict neural data better than untrained ones due to better alignment with brain response and (b) adversarial training leads to an improved alignment between the eigenvectors of networks and early visual cortex activity.

---

A
Examplc V1/V2 Stimuli (PreemanZiemma2213)
B
Y: Response
Neural Prediction with Regression
neuron
P ssimuli
Pi training set size A(p): regression weights
135 Stimali
(P=135)
Response predictions: (p) = X8(p)
Deurons
Example V4/TT Stimuli (Majajllong2015)
X: Model Activations
P stimmuii
Neural prediction error us measure of fit:
3200 Stimali
Mit DNN Layers
(P=3200)
( YI
Eg(p)
M F
M units
C
Spectral Alignment Theory
Ag
Spectral Decomporition of Activatiin (PCA):
Weight on Eigenvector
(s(p))
Error Modes:
W:
Eigenvalues: (Ae)
W.(p) (p), + x)²
XXT EnMNT
Eigenvectors: [vi)
Neural prediction
Measure of Tadk-Model Alignment projection of
10
error:
v.13
Fg(p)
Σ (1)
Aligument Coefficients: W;-
Y 1
$0
100
eigenvector index
Figure 1: Experimental and theoretical framework. (A) Example stimuli from visual experiments used for neural prediction error measures of V1, V2, V4, and IT. (B) Framework for predicting neural responses from model activations. Responses are measured from a brain region and from a model stage of DNN X. Weights for linear regression are learned to map X to Y. and the empirical neural prediction error (p) is measured from the response predictions. (C) Spectral alignment theory. Neural prediction error can be described as function of the eigenvalues and eigenvectors of the model Gram matrix. The neural response matrix is projected onto the eigenvectors v, with alignment coefficients W, For the number of training samples p, the value of each error mode is given as (p) = (p). The total neural prediction error can be expressed as a sum W; (p) terms, visualized as the area under the W, (p) curve.
2 Problem Setup
2.1 Encoding Models of Neural Responses
We investigated DNNs as encoding models of visual areas V1, V2, V4, and IT. The neural datasets previously collected on 135 texture stimuli for V1 and V2 [34), and on 3200 object stimuli with natural backgrounds for V4 and IT [35] (Fig. 1A). Both datasets were publicly available and obtained from [11]. The neural responses and model activations were extracted for each stimulus (Fig. IB), and each stage of each investigated neural network was treated as a separate encoding model. We examined 32 visual deep neural networks. Model architectures included convolutional neural networks, vision transformers (ViTs), and "biologically inspired" architectures with recurrent connections, and model types spanned a variety of supervised and self-supervised training objectives (see Sec. S1.2 for full list of models). We extracted model activations from several stages of each model. For ViTs, we extracted activations from all intermediate encoder model stages, while in all other models we extracted activations from the ReLU non-linearity after each intermediate convolutional activation. This resulted a total number of 516 analyzed model stages. In each case, we flattened the model activations for the model stage with no subsampling.
2.2 Spectral Theory of Prediction and Alignment
In response to total of P stimuli, we denote model activations with M features (e.g. responses from one stage of a DNN) by X E PxM and neural responses with N neurons (e.g. firing rates) by Y E P Sampling a training set (X1.p> Y1:p) of size p P. ridge regression solves (Fig. IB):
B(p)
arg min
X1p3 Y1pplla areg argll 11118
Y(p) x(((p)
(1)
3

---

( for which we utilize !|
We analyze the normalized neural prediction error, Fly(P)
theoretical tools from learning theory [36, 37, 38], random matrix theory [39, 40, 41] and statistical physics [42, 32, 33] to extract geometrical properties of representations based on spectral properties of data. In particular, the theory introduced in [32, 33] relies on the orthogonal mode decomposition (PCA) of the Gram matrix XX the model activations, and projection of the target neural responses onto its eigenvectors:
| v.ll Y P
xx = Σ A.V. w;=
(vi. = dو.
Here, associated to each mode i, W, denotes the variance neural responses Y in the direction V.. and X; denotes the tth eigenvalue. Then, the neural prediction error is given by:
x² Ex(2) Σ W.E.(P), E(P) = 1-7 (p) + xx)2
where Orey >
Lim1 ph+t must be solved self-consistently, and y Di=t (pActx)² (see Sec. SI.1 for details) [32, 33J. Note that the theory depends not only on the model eigenvalues X; but also on the model eigenvectors v, along with the responses Y, which determine how the variance in neural responses distributed among model's eigenmodes.
pAd
Although the equations are complex, the interpretations of W; and (p) are simple. Wi quantifies the projected variance in neural responses on model eigenvectors (alignment between neural data and model eigenvectors, i.e., task-model alignment [33]). Meanwhile, E, as function of the training set size p. determines the reduction in the neural prediction error at mode and depends only on the eigenvalues, XE (i.e., spectral bias [33]).
In this work, we combine both and introduce error modes (.(p) = W,E,(P):
k²
W
W.(p) = - (pt
Eg(p)
Σ W((p)
(4)
As shown in (Fig. IC), W; associated with large eigenvalues X; will decay faster with increasing than those associated with small eigenvalues.
The generalization performance of model is fully characterized by its error modes, W.. However, due to its vector nature, W; is not ideally suited for model comparisons. To address this limitation, we condense the overall shape of W, into two geometric measures, while preserving their direct relationship to the neural prediction error. This is in contrast to previous such measures, such the effective dimensionality, that only depend on model eigenvalues [31].
Here, we define a set of geometric measures that characterize the distribution of a model's W2 via the error mode geometry (Fig. 2A). Specifically, we rewrite the neural prediction error E, (p) as:
Rem (p) = Σ W,(r),, Dem(p) (E, Wi(p)) Σ, W.(p)²
Fg(P)) = Rem (p) / Dem(p)
(5)
The error mode radius Rem denotes the overall size of the error terms, while the error mode dimension Dem represents how dispersed the total neural prediction error is across the different eigenvectors (Fig. 2A). Note that, the neural prediction error E, (p) above has a degeneracy of error geometries; many different combinations of Rum and Dam may result in the same (p) (Fig. 2B). Given fixed neural prediction error, a higher value of Rem (p) indicates that only a few W; are driving the error, while a higher Dem (p) indicates that many different W, are contributing to (p).
3 Results and Observations
We first confirmed that Eq. 3 accurately predicted the neural prediction error for the encoding models considered here. We performed ridge regression from the model activations of each model stage of

---

A Bg(p) Ronlp)y Dem(p)
B
Example visualization of emorr mode goometry
Model 1:
IP ortor modes
Remip)-
Wilpz²
distributed actoss
Remaining (p)
11
many model
denote equal F. (p)
aigenvecttrs (V)))
Σ. Wi(o)
higher
610 10
Dem(p)
Σ, Wo(p)²
lower R...
Model Remaining E (p) concentraied on
Note: modes in large eigenvalue directions decay faster
com why May
0.04
Rea
W
small number model cigenvectors (v
Model Model same neural prediction evror (ie some (,j)), but diffor in error mode geomerry
lower Dus higher R...
Figure 2: Summary measures for error mode geometry. (A) Error modes (W;) are related to the model cigenspectra (Ae) and alignment coefficients (W₁). Error mode geometry summarized by Rem and Dem characterizes the distribution of сгror modes and directly relates to the neural prediction error (p) (B) Interpreting contour plots. Error mode geometry distinguishes between models with the same prediction error (p)( Neural prediction error the product of Rem and /Dem and is thus easily visualized on contour plot where each contour represents equal (p)..
each trained neural network to the neural recordings for each cortical region using a ridge parameter of Oreg = 10- 14, and also calculated the theoretical neural prediction error for each using Eq. 3. Given that Eq. is only accurate in the large P limit in which both the number of training points and the number of test samples is large (see Sec. SI.1 and Fig, SL.4.2), we used 60/40 train-test split (p =066PPP) in the experiments below. As shown in Fig. 3A, this split yielded pear-perfect agreement between the theoretical and empirical neural prediction errors for areas V4 and IT (R2==0.77 and 0.97, respectively), and very strong agreement for areas VI and V2 (Rº=0.9 and 0.9, respectively). We present further data for this agreement in Sec. SI.4. Furthermore, we found that the ordering of models according to their neural prediction error is very similar to the ordering obtained via the partial least squares (PLS) regression method used in Brain-Score [11] (see Sec. SI.3). We maintained these choices for train-test split sizes and Oseg for all subsequent experiments.
We visualized the spread of error mode geometries for each brain region across different trained models in Fig. 3B. Each point colored by the empirical Eg (p), while the contour lines show the theoretical Eg (p) value. Among trained models, there was range of Rem and Dem values even when many models have similar (p). This demonstrates that our geometrical interpretation error mode geometry can give additional insights into how models achieve low neural prediction error. the following sections, we explore how these geometrical properties of the neural prediction error vary with model stage depth, training scheme, and adversarial robustness.
the
3.1 Error Mode Geometry Varies across Model Stages
We analyzed the neural prediction error obtained from activations at different model stages. In Fig. 4A, we plot the error mode geometry for all analyzed DNN model stages when predicting brain data from VI and IT. Each point is color coded based on its relative model depth. Lower neural prediction errors were achieved (i.e. points lie on lower F. contours) for earlier model stages in carly visual area V1, and for later model stages in downstream visual area IT. This observation is in agreement with previous findings [6. 11, 43] where artificial neural networks have similar hierarchy compared to the visual cortex. Qualitatively, the trends across model stages were more prominent than differences observed from comparing model architectures for supervised models (Fig. S1.5. 1)), or from comparing supervised trained models to self-supervised models of the same architecture (Fig. SI.5.2).
To investigate the source of this geometric difference across model stages, we performed an experi- ment on each model where we took the eigenspectra X; measured from the first (Farly) last (Late) model stage and paired this with the alignment coefficients W; measured from the Early or Late model stage (Fig, 4B). These two spectral properties can be varied independently, as W; depends on
5

---

A
VI
V2
V4
IT
E.(p): Theoretical
0.7
P=0090
0.6
F= 0.90
0.6
009777
==0
0.6
0.54
0.5-
0.4
0.4
0.4
0.3
0.3-
0.3
0.2
0.2
0.4 E, (p): Empirical
0.2
0.4
0.6
(p): Empirical
F (p): Empirical
(.)): Empirical
VI
V2
V4
IT
50
1.0
0.8
10
45
Fight)
0.6
9
40
0.4
8
0.2
0.05
0.025
0,050
0.075
0.005
0.010
0.015
0.015
Ran
Ran
Run
Run
Figure 3: Error mode geometry of model predictions for each brain region. (A) Theoretical values of neural prediction error match empirical values from ridge regression. Each point corresponds to predictions obtained from one of the 516 analyzed model stages. (B) Error mode dimension and radius for each brain region. Each point corresponds to the error mode /Dem and Rem for the region from specific model stage of trained neural network models. Points colored by the empirical error from performing ridge regression between the model activations and neural responses. Rem and /Dem values obtained from theoretical values, and contour lines correspond theoretical values of the neural prediction error, such that points along the contours have the same error.
the model eigenvectors not on the model eigenvalues. We then measured the neural prediction error that would be obtained from such model. This experiment allows us to isolate whether the observed differences neural prediction error are primarily due to the A; or W; terms. the VI data, there was little difference in the error mode geometry between the location of the W; terms, however using early model stage À corresponds to lower neural prediction error (summarized in Fig. 4C). In contrast, for the IT data using the late stage W, terms corresponded lower (p)p) mainly through reduction the eтror mode Rem (i.e. many error modes are better predicted). The spectral properties shown in full for the early and late stages of ResNet50 as measured the V1 and IT datasets (Fig. 4D). These results highlight the need to study the spectral properties of the model together with the brain responses as summarized by W;: Simply studying the properties of the eigenspectra of the model insufficient characterize neural prediction error (see SI.5.5 for more details).
3.2 Trained vs. Untrained
We analyzed how the error mode geometry for neural predictions differed between trained and randomly initialized DNNs (Fig. 5A). In line with previous results [1 14, 44], we found that training yielded improved neural predictions as measured via smaller (p) (Fig. 5B). This improvement was most notable in regions V2, V4, and IT, where there was also characteristic change in the error mode geometry. In these regions, while Rem decreased with training. Dem surprisingly increased. However, the decrease in Rem was large enough to compensate for the higher D leading to overall reduction in (p). VI, we observed only small difference neural prediction error between trained and untrained models, similar to previous results for early sensory regions [45, 14]. As shown in Fig. S1.5.7, we found qualitatively similar results in a different V1 dataset with larger number of tested stimuli [46].
What differed between the trained and random model representations that led to differences in eтror mode geometry? To gain insight into this, we investigated how the eigenspectra (Ax) and the alignment coefficients (W) individually contributed to the observed error mode geometry visual area IT. We performed an experiment the trained and random ResNet50 model activations where

---

Figure 4: Model stage depth and error mode geometry. (A) Each point repre- sents the enror mode geometry obtained from a specific model stage of a trained neural network, where the color of the value represents the depth of the model stage. The color normalized for each model to the [0, 1] range where the car- lier and later model stages have lighter and darker colors, respectively. Rem and VDm values are obtained from theoret- ical values, and contour lines correspond to theoretical values of the neural predic- tion error as in Fig. 2. (B,C) Predicted (p) that would obtained when using the eigenspectra (A₂) from the first (Farly) or last (Late) stage of each of the 32 ama- lyzed models, paired with the alignment coefficients (W;) terms from the Early or Late stage of each model. Full error mode geometry is given (B) where each point corresponds to single model with the cho- sen W. and X terms, and (C) shows box plots summarizing the (p) from each comparison. In V1, using the A from the early stage of the model resulted in better neural prediction error and a lower Rem but higher VDem. regardless of the W; terms used. In IT, using the alignment terms W, from the late model stage re- sulted in smaller neural prediction er- ror and lower Rem- with little change in VDem- (D) Full spectra for A; W; and W, from Early and Late model stages of ResNet50 on the V1 and IT datasets.
A
Latest
11
50.0
Stage Depth (Normalized)
47.5
45.0
42.5
9
40.0
Earliest
0.03
0.04
0.05
0.005
0.010
R
R,,
Early vs. Late
Early vs. Late
50
W (alignment) Early Late
7. (eigenspeetra)
10-
Early
45-
9-
Late
x
8-
0.04
0.06
0.08
0.005
0.010
0.015
R
R_,
VI
IT
D
VI
IT
Late (Last Stage) Early (Fiest Stage)
ResNe150
ResNoc50
0.65
10
E (p)
0.45
Weight on Eigenveein
fo W I pi
0.25
Bacy
late
10
Earty Tang
Late
100
1000
model cigcavactor
we measured X; from one model and paired it with the W; measured from the other model (Fig. 5C). Using W; from the trained model led to much lower Rem than when using W. from the random model. This decrease in Rem when using the W; terms from the trained model of the improvement in (p) when we used the trained model to predict the IT neural data. In contrast, when using the eigenspectra of the random model, the Dem was lower than when using the eigenspectra of the trained model, at the cost of slight increase in Rem- Note that the model eigenspectra clearly decayed at a significantly faster rate the random vs. trained models (Fig. 5D), but it was the properties of the alignment terms W; that drove the change in Eg (p) between trained and untrained models. Thus, the eigenvectors of the trained model were better aligned with the neural data compared to the random model, which resulted in low prediction error regardless of which eigenspectra was used.
the main driver
3.3 Adversarial vs. Standard Training
Recent work has suggested that adversarially trained networks have representations that are more like those of biological systems [47, 48, 15]. We analyzed the neural prediction error and error mode geometry of adversarially trained ("Robust") neural network models [49, 50], and their non- adversarially trained counterparts. The error mode geometry for an Ez(e = 3) ResNet50 is given in Fig. 6A, and similar results are shown for (e 4) ResNet50 in Fig. SI.5.8.
The effect of adversarial training on error mode geometry varied across model stages and cortical regions. In the V1 dataset, lower neural prediction error was observed for carly stages of robust
7

---

A
V2
Random
Trained
0.48
0.48
50
0.75
50
- -alle
Stage Depth (Normalized)
10
10-
45
Early
0.04
0.06
0.04
0.06
0.01
0.02
0.01
0.02
R,,
R,
R_,
R
B
VI
V2
V4
IT
C
IT: ResNet50
D
Trained ResNet50
Random ResNet50 (layer422relu)
W (alignment) trained randons
2.ccl)
10a
0.8
48
TNP
2, (oigenspeet)) nandom tnineed
Weight on Eigemestor
W
10
4044
46
10"
44
0.0
Rundom
Trained
Random-
Trained
Random.
Trained
Random.
Trained
eurly Stage Depth
0,006 0.008
0,010
1000 2000 3000 'n 1000 2no0 3000 model cigenvecter
R_
Figure 5: Error mode geometry changes with model training. (A) Error mode for trained models (blue) and randomly initialized models (green) for each brain region. The same model stage of each network is connected with line, and stage depth is shown with opacity. Contours are given by theoretical values of F.g (p), as in previous plots. (B) Neural prediction error was lower for trained models compared to random models all brain regions, but most notable in V2, V4, and IT. Median is given as black line and box extends from the first quartile to third quartile of the data. Whiskers denote 1.5x the interquartile range. (C) Predicted Ep(p) that would be obtained on IT dataset when using eigenspectra from trained or random ResNet50 models, paired with the alignment W; terms from each of the trained or random ResNet50 models. Size of point denotes model stage depth. Using the W; terms from the trained model resulted in lower (p) due to a lower Rem, while using the A from the trained models increased Dem and did not appreciably change (p)p). (D) Model eigenvalues, alignments, and error modes for the final analyzed model stage of the ResNer50 architecture on IT dataset for both trained and random models. Even though the eigenspectra for the random model was notably different than the eigenspectra for the trained model, we observed in (C) that the driver the decreased Ey (p) for the trained model was a difference in the W; terms.
models (Fig. 6B), in line with previous work [48]. This corresponded to a decrease in Rem with little change Dem. suggesting that the decrease neural prediction error was shared relatively equally across all error modes. In contrast, at late model stages, Dem was smaller for the robust models compared to the standard models, but there was no difference neural prediction error between the robust and standard networks. In regions V2, V4, and IT, we observed that there was little difference in neural prediction error between the standard and robust models, but that error mode geometry in robust models was associated with a lower Dem and a higher Rem This suggests that the difference between robust and standard models in these regions was due to a small number of error modes having higher W; in the robust models. This did not lead to better neural prediction error for regions V2, V4, and IT, but nevertheless, the error mode geometry revealed that the data predicted in different ways.
To better understand the differences between robust and standard networks in V1, of experiments testing the effects adversarial training on model eigenspectra and alignment with neural data. For each model stage of the ResNet50 architecture, we paired the eigenspectra (A₂) of the standard or robust network with the alignment coefficients (W,) of the standard or robust network (Fig. 6C). We then calculated the error mode geometry and neural prediction error for all four possible pairings. This approach allowed us to examine how adversarial training-relatedd changes in the spectrum and alignment coefficients affected the error mode geometry for each model stage.
series
We found that the alignment coefficients of the robust models consistently yielded lower V1 prediction error (Fig. 6D). In early model stages, using the W, from the robust models led to lower neural

---

A
VI
V2
V4
Robust Standard
95
06
Late
00'0
25:0
Stage Depth
10-
46
46
42-
-Early
0.03
0.04
0.03
0.04
0.005
0.010
0.005
0,010
R
cm
B
VI: E (p)
C
VI: Standard vs. Robust
D V1: Standard vs. Robust
0.40
Empirical
Standard Robust
(alignment)
Early Stages
Late Stages
Early Stages
Late Stages
0.36
(eigenspeetra)
Sualord
11-
B(P))
04
0.32
0.28
Robux
10-
*
03
0.24
Stnddnt's Staddot W
Stanand lbbutt W-
Robut Sundard W-
Robut Robas W
Sundint' Stamton W-
Sundied W--
Rubont Sandar W--
Robut Rehes W-
Stuge Depth (Fariy ->i Late)
early Layer
Model Robust Model
Ram
Ren
Figure Error mode geometry and adversarial robustness. (A) Error mode geometry for standard ResNet50 (Blue) and adversarially- trainnd (Robust) ResNet50 (Green) models for each brain region. The same stage of each network is connected with an arrow. and stage depth is shown with opacity. Contours are given by theoretical values of Eg as in previous plots. (B) For standard and robust function stage depth. (C) Predicted F., (p) that would be obtained on V1 dataset for X; of standard or robust ResNet50 networks paired with W; of standard or robust ResNet50 networks. Each point was from single stage of the model, and plots are broken up by the early stages (first 8/16 analyzed stages) and the late stages (last 8/16 analyzed stages) of the model for visualization purposes.
ResNet50 models, (p)()) for VI data
(D) Summarized neural prediction error for the experiment in (C). For all stages, the robust W; led to lower error, but in the late stages, there was an additional effect from the choice of Aş.
prediction error via reduced Rem, and the choice of X had a negligible effect on the prediction error for these stages. In the later model stages, the W; from the robust models also achieved lower Eg. but for these stages, the choice of X; had a non-negligible impact on the resulting prediction error. Overall, these results suggest that adversarial training leads improved alignment between the eigenvectors of the model and early visual cortex activity.
4 Discussion
Many metrics have been proposed to compare model representations those observed in biological systems [51, 52, 22, 23, 11]. However, is typically unclear which aspects the representational geometry lead particular similarity score. In this work, applied the theory of spectral bias and task-model alignment for kernel regression [33] to the generalization error for encoding models of neural activity. This provides theoretical link between a model's ability predict neural responses and the cigenspectra and cigenvectors of model representations. We introduced geometric measures Rem and Dem of the егтоr modes summarize how the spectral properties relate to neural prediction eтror, and showed that models with similar errors may have quite different geometrical properties. We applied this theory to investigate the roles of stage depth, learned representations, and adversarial training in neural prediction eтrо throughout the ventral visual stream.
Dimensionality of neural and model representations
Recent work has investigated spectral properties of neural activity in biological [53, 54) and artificial [31, 28] systems. One empirical study suggested that high nigh-dimensional model representations have

---

lower neural prediction error, where effective dimensionality was defined as the participation ratio of the eigenvalues alone [31]. However, we empirically found that this effective dimensionality of representations was not well correlated with prediction error for our datasets (Sec. S1.5.5). The lack of consistent relationship between effective dimensionality and neural prediction error was expected based on our theory, as the neural prediction error depends both on the eigenvalues and eigenvectors of a model. These quantities co-vary from model to model, and thus model comparisons should include information from both. In this work, we demonstrated how our theory can be used to examine whether differences in observed prediction error are primarily due to the eigenvalues or alignment coefficients (which are dependent only the eigenvectors), and found that in many instances, the alignment coefficients were the main driver of lower prediction error.
Dependence of Neural Prediction Error on Training Set Size
Our spectral theory of neural prediction characterizes the relationship between the training size, the generalization error, and the spectral properties of the model and brain responses. This highlights that model ranking based on neural prediction error may be sensitive to the train/test split ratio. In Eq. 3, we see that, for small p. the eтror modes corresponding to small A generally remain close to their initial value (p) R (O), and hence their contribution to neural prediction error does not decay (Eq. 4). On the other hand, at large sample size p. these small A, error modes change from their initial values and therefore may change the ranking of the models. These observations regarding the role of model eigenspectra and dataset size may be used in the future to design more comprehensive benchmarks for evaluating models.
Limitations
Although our empirical analysis follows what we expect from theory and shows a diversity of error mode geometries, the datasets that we used (particularly for V1 and V2) have limited number of samples. In some cases, such as the large p limit, theoretical results for the neural prediction error may not match empirical results. Additionally, we only investigated one dataset for V2, V4, and IT. and only two datasets for VI. Future work may reveal different trends when using different datasets. Moreover, this theory assumes that the brain responses are deterministic functions of the stimuli. As shown in [33], this assumption may be relaxed, and future work can examine the role of neuronal noise in these contexts.
Future work
The results in this paper demonstrate that our theory can characterize the complex interplay between dataset size, representational geometry, and neural prediction error. provides the basis for numerous future directions relating spectral properties of the model and data to measures of brain-model similarity. Insights from our theory may suggest ways that we can build improved computational models of neural activity.
While we focused on DNN models pre- pre-trained on image datasets, closely related line of work has investigated the properties of DNN models directly trained to predict neural responses [46, 55, 56]. Future work could compare the error mode geometries from end-to-end networks to those obtained from networks pre-trained on image data. Such an analysis could elucidate how these two vastly different training schemes are both able to predict neural data well. Additionally, our experiments focused on the case of the same small regularization coefficient for all regression mappings, but future work could explore the case of optimal regularization. The spectral theory can also be applied to other types of brain-mode comparisons, such Partial Least Squares Canonical Correlation Analysis.
Although we were able to differentiate many error mode geometries based on Rgm and VDem we initially expected to see many more values of (Rem, VDem) for each fixed (p). The investigated vision models seem to be constrained to a particular region in the (Rem: / -spaee (Fig. 3). Future work may shed light on whether this is due to implicit biases in model architectures or due to a fundamental coupling of (Rem and vDem) due to constraints on the (p(p) term.
There are many architectural differences between the tested DNNs and biological systems. Future work may investigate how biologically plausible constraints [9] (e.g., stochasticity [48, 57], divisive normalization [58), power-law spectra [59]) and diverse training objectives [44, 13)) lead to improved neural prediction error through changes in error mode geometry.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Nga Yu Lo, Eero Simoncelli, N Apurva Ratan Murty, Josh McDermott and Greta Tuckute for helpful discussions, and Jim DiCarlo for comments on an early version of this work. We would also like to thank the reviewers for their helpful comments, which strengthened the paper. This work was funded by the Center for Computational Neuroscience at the Flatiron Institute of the Simons Foundation and the Klingenstein-Sing -Sioons Award S.C. All experiments were performed on the Flatiron Institute high-performance computing cluster.
References
[1] Matteo Carandini, Jonathan B Demb, Valerio Mante, David J Tolhurst, Yang Dan, Bruno A Olshausen, Jack Gallant, and Nicole C Rust. Do we know what the early visual system does? Journal of Neuroscience, 25(46): 2005.
[2] Marcel AJ van Gerven. A primer on encoding models in sensory neuroscience. Journal of Mathematical Psychology, 76: 17:1172, 2017.
[3] Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-444 4100 2011.
[4] Christina Saretl-Cugell and John G Robson. The contrast sensitivity of retinal ganglion cells of the cat. The Journal of physiology, 187(3):517-555) 1966.
[5] Edward H Adelson and James R Bergen. Spatiotemporal energy models for the perception of motion. Josa a, 2(2):284-299, 1985.
[6] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance-opimiized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 11 111683):619-822 2014.
[7] Seyed-Mahdi Khaligh-Razze -aziviii and Nikolaus Kriegeskorte. Deep supervised, but not un- supervised, models may explain it cortical representation. PLoS computational biology, 10(11)=e1033115, 2014.
[8] Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman Haimanne and Josh H McDermott. A task lask-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630 644, 2018
[9] Blake A Richards, Timothy P Lillicrap. Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. deep learning framework for neuroscience. Nature neuroscience, 22(11):1761-1770 2019.
[10] Grace W Lindsay. Convolutional neural networks as a model the visual system: Past, present, and future. Journal of cognitive neuroscience, 33(11)):017-2031 2021.
[11] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[12] Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit hierarchical region correspondence. bioRxiv, pages 2022-09, 2022.
[13] Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRxiv, pages 2022-03, 2022.
[14] Andrew Saxe, Stephanie Nelli, and Christopher Summerfield. If deep learning is the answer, what is the question? Nature Reviews Neuroscience, 22(1):55-67, January 2021.
[15] Jenelle Feather, Guillaume Leclerc, Aleksander Madry, and Josh MeDermott. Model metamers reveal divergent invariances between biological and artificial neural networks. Nature Neuroscience, pages 1-18, 2023.
[16] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between
11

---

human and machine vision. Advances in Neural Information Processing Systems, 34:23885- 23899. 2021.
[17] Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 13:19000-19015, 2020.
[18] Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
[19] Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A Vandermeulen, Katherine Her- mann, Andrew K Lampinen, and Simon Korblith. Improving neural network representations using human similarity judgments. arXiv preprint 230.00330. 2023.
[20] Tal Golan, Prashant C Raju, and Nikolaus Kriegeskorte. Controversial stimuli: Pitting neural networks against each other as models of human cognition. Proceedings of the National Academy of Sciences, 117(47):29330-299
2020.
[21] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint arXiv:1412.655 2014.
[22] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
[23] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pages 151999529. PMLR, 2019.
[24] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems, 33:9995-10006, 2020.
[25] Yuo Chung and LF Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology, 70:137-144, 2021.
[26] Uri Cohen, Yuo Chung, Daniel D. Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature Communications, (11:::74,, Feb 2020.
[27] Mehrdad Jazayeri and Srdjan Ostojic. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Current opinion in neurobiology, 10:113-120, 2021.
[28] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[29] Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 1 119(43))e2200800119 2022.
[30] Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 364(6439) 9436. 2019.
[31] Eric Elmoznino and Michael F Bonner. High-performing neural network models of visual cortex benefit from high latent dimensionality. bioRxiv, pages 2022-07, 2022.
[32] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024-1034. PMLR, 2020.
[33] Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1):2914 2021.
[34] Jeremy Freeman, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. A functional and perceptual signature of the second visual area primates. Nature Neuroscience, 16(7))974-981, Jul 2013.
12

---

[35] Najib J Majaj, Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 35(39):13402-113 13402-13418 2015.
[36] Bernhard Schölkopf, Alexander Smola, Francis Bach, et al. Leurning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
[37]
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. volume 2:3. MIT press Cambridge, MA, 2006.
[38] Thomas Hofmann, Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine
learning. The Annals of Statistics, 36(3):1171, 2008.
[39] Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices, volume 20. Springer, 2010.
[40] Arthur Jacot, Berfin Simsek. Francesco Spadaro, Clément Hongler, and Franck Gabriel. Implicit regularization of random feature models. In International Conference on Machine Learning, pages 4631-4640. PMLR, 2020.
[41] Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with teacher-student model. Advances in Neural Information Processing Systems, 34:18137-18151, 2021.
[42] Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A, 45(8) 1992.
[43] Soma Nonaka, Kei Majima, Shuntaro C Aoki, and Yukiyasu Kamitani. Brain hierarchy score: Which deep neural networks are hierarchically brain-like? IScience, 24(9):003013, 2021.
[44] Santiago A Cadena, Konstantin F Willeke, Kelli Restivo, George Denfield, Fabian H Sinz, Matthias Bethge, Andreas S Tolias, and Alexander S Ecker. Diverse task-driven modeling of macaque v4 reveals functional specialization towards semantic tasks. bioRxiv, pages 2022-05, 2022.
[45] Santiago A Cadena, Fabian H Sinz, Taliah Muhammad, Emmanouil Froudarakis, Erick Cobos, Edgar Y Walker, Jake Reimer, Matthias Bethge, Andreas Tolias, and Alexander S Ecker. How well do deep neural networks trained on object recognition characterize the mouse visual system? In Real Neurons {{&} Hidden Units: Future directions at the intersection of neuroscience and artificial intelligence@ NeurIPS 2019, 2019.
[46] Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS computational biology, 15(4)=e1006897, 2019.
[47] Chong Guo, Michael Lee, Guillaume Leclere, Joel Dapello, Yug Rao, Alcksander Madry, and James Dicarlo. Adversarially trained neural representations are already as robust biological neural representations. In International Conference on Machine Learning, pages 8072-8081. PMLR, 2022.
[48] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo. Simulating primary visual cortex at the front of cnns improves robustness image perturbations. Advances in Neural Information Processing Systems, 33: 13073-10877, 2020.
[49] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 3533-3545, 2020.
[50] Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robust- ness (python library), 2019.
[51] Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in newral information processing systems, 14, 2001.
[52] Nikolaus Kriegeskorte, Maricke Mur, and Peter A Bandettini. Representational similarity anonysii the branches of systems neuroscience. Frontiers in systems neuroscience,
page 4, 2008
[53] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Har- ris. Hiigh-dimensional geometry of population responses visual cortex. Nature, 571(7765):361- 365, 2019.
13

---

[54] Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, page 214262, 2017.
[55] David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identifi- cation for large populations separating "what" and "where". Advances neural information processing systems, 30, 2017.
[56] David Zipser and Richard A Andersen. A back prepppropgaation programmed network that simulates response properties of a subset of posterior parietal neurons. Nature, 33166158):6796644 1988.
[57] Joel Dapello, Jenelle Feather, Hang Le, Tiago Marques, David Cox, Josh McDermott, James J DiCarlo, and Yeon Chung. Neural population geometry reveals the role of stochasticity in robust perception. Advances in Neural Information Processing Systems, 34:15595-15607, 2021.
[58] Michelle Miller, Yuon Chung, and Kenneth D Miller. Divisive feature normalization improves image recognition performance in alexnet. In International Conference on Learning Representations, 2022.
[59] Josue Nassar, Piotr Sokol, Yeon Chung, Kenneth D Harris, and Memming Park. On 1/n neural representation and robustness. Advances in Neural Information Processing Systems, 33:6211-6222. 2020.
[60] Harold Widom. Lectures on integral equations. Courier Dover Publications, 2016.
[61] Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Out-of-distribure generalization in kernel regression. Advances in Neural Information Processing Systems, 34:12600-12612,
2021.
[62] James B. Simon, Madeline Dickens, Dhruva Karkada, and Michael R. DeWeese. The eigen- learning framework: A conservation law perspective kernel regression and wide neural networks, 2022.
[63] Noureddine Karoui. The spectrum of kernel random matrices. The Annals of Statistics, 38(1):1 - 50, 2010.
[64] Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity with statistical testing. arXiv preprint (xXiv:200800160 2021.
[65] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[66] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[67] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[68] Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint 1404v 2014.
[69] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[70] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 1r5iv:1605.007466 2016.
[71] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[72] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11976-11986, 2022.
[73] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114, PMLR, 2019.
14

---

[74]
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[75] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[76] Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel LK Yamins, and James J DiCarlo. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv, page 408385, 2018.
[77] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing. pages 12310-12320. PMLR, 2021.
[78] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[79] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[80] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint wrx.xx17060000000 2017.
[81] Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Computer vision with a single (robust) classifier. In ArXiv preprint 19Xiv:1906.0944 2019.
[82] Christopher Williams and Carl Rasmussen. Gaussian processes for regression. Advances in neural information processing systems, 8, 1995.
[83] Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "Ridgeless" regression can generalize. The Annals of Statistics, 48(3):1329 1347, 2020.
[84] Dmitry Kobak, Jonathan Lomond, and Benoit Sanchez. The optimal ridge penalty for rorl-world high-dimensional data can be zero or negative due to the implicit ridge regularization. The Journal of Machine Leurning Research, 21(1)=6863-6877 2020.
[85] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949 986, 2022.
[86] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nelson, Eric Jones, Robert Kern, Eric Larson, C Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272. 2020.
[87] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bethge, Felix A. Wich- mann, and Wieland Brendel. On the surprising similarities between supervised and self- supervised models. In NeurIPS 2020 Workshop SVRIM, 2020.
[88] Talia Konkle and George A Alvarez. A self-supervised domain-general learning framework for human ventral stream representation. Nature communications, 13(1))491, 2022.
[89] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3):e20141661 2021.
15

---

Broader Impacts
Our work provides a novel approach towards comparing the fidelity of neural encoding model predictions, and may provide insight that leads to improved models of neural activity. Such models are applicable for neuroengineering purposes, such as retinal prostheties or cochlear implants. However, training the deep learning models tested in the paper consumes a large amount of energy, contributing to carbon emissions. Although we used pre-trained neural networks here future work may need to train networks with biases specifically for low neural prediction error.
SL11 Spectral Theory of Generalization Error in Regression
The spectral theory of generalization error for regression, as developed in [32, 33), relates the spectral properties of the data and target to the generalization error in an analytical way. Here, we summarize the key results from these works and describe how we use the results in our context.
For generic regression task, begin by considering model x(s) IR RM, which maps D-dimensional stimuli s to a set of M-dimensional features x. Additionally, we assume that the neural responses generated by ground truth function y(s) R - R.,, mapping the stimuli to the activations of a set of biological features y (e.g., firing rates). Finally, we assume that the stimuli are drawn from generic distribution vss).
In regression, the goal is to estimate the ground truth ys) from training set of size p and test on all possible stimuli as drawn from vs). Given training dataset with p-samples Dr = {st, yd y"
where s v(s) and yd = =y(s"), ridge regression returns estimator (s) as:
(ss Dv) = x ((Dx)
P
B(Du) arg min
x TB12 + 113111311.
(SI.1.1)
BERMXN
where Oreg is the ridge parameter introduced to regularize the solution. Here, the regression weights and estimator depend on the particular choice of the training Drr of size p. We define the normalized generalization error as:
f dss(ss)l>(ss) y(s; Dw)l?
(Dg) J (ssv(())y(ss) ds
(SI.1.2)
which also depends on the training set Drr Explicit solution of the objective in Eq. SI.1 gives:
(s; Du) x(s) X (XI.pXIp + areg
i:p.
(SL.13)
where XI.p E RRXM the matrix of the model responses to the stimuli Drr. Y E R is the matrix of the neural responses in Dr.. and I denotes the x p identity matrix. Defining the kernel function K(s,s') ) = x(s) x(s'), the predictor can be written as:
(ss) k(s) T (K + areg 1.p:
(SI.1.4)
where is p X p matrix with elements KY,. K(s", s')) and k(s) is p-dimensional vector with elements k(s), K(s,s"). Hence, the generalization error depends on the kernel K(ss particular draw of a training set Du of size p. and the ridge parameter Oreg.
Note that every time a new training set is drawn from the stimuli distribution v(s). the generalization error may change. We are interested in the dataset averaged generalization error which only depends on the size of the training set rather than the particular instances of each Dr. This is defined as:
(x) = (E, (Dww) D.-v(s)
1SL.
where the average is taken over all possible choices for Drr of size p. Now, this quantity only depends on the training set size p, input distribution v(s), model kernel K (s,s'') and the ground truth function (s). This quantity can also be thought of as the cross-validated generalization error, where the generalization errors for each regressor corresponding to a particular instance of Drr are averaged.
16

---

The theory of generalization error in [32, 33] finds that this dataset averaged (p) depends on the spectral properties of the kernel. Mercer's theorem [60] suggests that the kernel can be decomposed into orthogonal modes as K (s, Σg (s);(s));(s)) where the eigenvalues X; and orthonormal eigenfunctions (s) are obtained by solving the equation
f ds' v(s')(() (ss.)) (s);
[ds ((s))s))s())) -āg.
(SI.1.6)
Here we emphasize that the resulting eigenvalues and eigenfunctions strongly depend on the distribu- tion of the stimuli (s)). For example, for the same model kernel, the spectrum might look completely different depending on the stimuli being textures or natural scenes.
The dependence of the neural response function (()) to the generalization error comes from the mode decomposition of (s) on the eigenfunction (s)::
y() = w.),(s), w, dsv(s))((s),((s)
(SI.17))
where w, is the projection of the neural response function ys) onto the 1th eigenfunction (s).. As described in [33, 32, 61, 62], using techniques from statistical mechanics, the generalization error is given by the self-consistent equations
W
Eg(p)
(SL.18)
(p): + x)2
A
Σ
Oreg
pd
+
(pd: x)²
where & must be solved self-consistently. and we defined the alignment coefficients W
W
|w,|"
W d y(s) Σ, w|y""
(SL.1.9)
which give the ratio of the variance of neural responses along mode i to its total variance. In most cases, it is not possible to find an analytical solution to к, but straightforward to solve numerically.
In this paper, since our data are finite, we take the distribution vss) to be the empirical distribution on the full dataset Dial {s" y ) of P data points:
vss) = P Σ 8(s s*),
(SL.1.1)
which is an approximation to the true stimuli distribution where P + 00.
In this case, the functions xss) and (s) are fully described by the data matrices, X and Y, and the generalization error in Eq. 11.1.2 becomes the neural prediction error,
(Du) = (D.) YIF///
Since the full distribution of stimuli characterized by the empirical distribution v(s), training sets are generated by randomly sampling samples from D Furthermore, the eigenvalue problem in Eq. 11.1.6 reduces to PCA problem for the empirical kernel K XX
Kv, = A, P
(SI.1.1))
and the projections of neural responses Y on the eigenvectors v, reduces
(SI.1.12)
We note that this method may misestimate the neural prediction error, since the true eigenvalues and eigenvectors of the kernel K (s, s') are approximated by the finite empirical kernel matrix K for P samples [63].
17

---

With the spectral data } and {w:). we obtain the dataset averaged neural prediction error (p)
(Eq. 3) as:
n²
By(p)= W.E
E(()
W, < 1-7 (p>: + +x)2²
||
(SI.1.13
Here, we normalized the weights so that W. = and (p 0) 1. The functions E, (p) depend only on the eigenvalues and training set size p. Since each W; is the normalized variance of Y along the cigenvector v,, W,E, gives the contribution to the generalization error along mode i when the training set has size p. In our work, we defined these as error modes (p))) = (p).
To understand the behavior of the functions (p),). we look at their limits. When p = 0, E,(0) = for all modes and variance is explained. As p - oo, we either learn the variance W; entirely
(Ea(p- - oo) + 0) if the corresponding X; > 0, or we do not learn it at all if X = 0 (F; - x)) - 1). It can be further shown that, for all p 0, (p) (2(r) X > > [33], which implies that the
percent reduction in (p) is greater for modes with larger eigenvalues.
Several remarks are in order:
The ordered eigenvalues { characterize the spectral bias of the model: for a fixed training set size p. the reduction in (p) is larger for modes associated with large eigenvalues A; than for modes associated with small A, Therefore, a sharply decaying cigenvalue spectrum implies that there is bias toward certain modes being learned at higher fidelity than others, while a flat eigenvalue spectrum implies that learning will be equally distributed across all modes.
W; characterizes how much of the variance of the neural responses Y lie in the 2th mode. These alignment coefficients {W:} characterize task-mode alignment; because of spectral bias, if most of the neural response variance is associated with eigenvectors with high D; terms, the neural prediction error will decrease faster compared when most neural response variance is on eigenvectors with low A; For example, if W; are largest for the modes for which X; are smallest, we would expect large generalization error when the training set size is small.
We consider both } and {W.) as spectral properties of the model for the measured data (e.g. see Fig. IC), and generalization error determined by the interplay between these two spectral properties, together with the training set size p. For fixed set of neural responses D = (s 1 y J==1 different DNN models have different {A,} and (Wi} since for each model both cigenvalues (spectral bias) and the eigenvectors (task-mod alignment) of the model activations change in a complicated way. Since we must specify both of these aspects of the model, comparison of model activations to neural responses should require at least two measures to be complete. Drawing from this argument, we may also conclude that reducing model/data comparisons to commonly used single scalar measures might be misleading as they may miss some of the factors contributing to the predictivity the others capture [64].
In this work, we considered two geometrized measures Rem and VDe that directly relate to the neural prediction error. The two measures jointly summarize whether the distribution of the error modes, W,, are relatively spread out (higher Dem- lower Rem) tightly concentrated (lower Dem, higher Rem) at a fixed generalization error. We found that these terms were a convenient way to summarize spectral properties of the generalization error, but in principle, other measures could be defined.
SI.2 Experimental Details
Code for analyzing the empirical and theoretical neural prediction error can be found at ht hww.r
We conducted our experiments with various vision models trained with different learning paradigms and training schemes. In total, we used 32 models and picked several model stage activations from each. Except for vision transformers (ViT), chosen model stages were intermediate convolutional model stage activations after passed through ReLU non-linearity. For ViT, we choose intermediate encoder model stages. For untrained models, models were randomly initialized for each iteration
18

---

for each experiment. All experiments were conducted on single Nvidia H100 or A100 GPUs with 80GB RAM using PyTorch [65] and JAX 0.4 [66] libraries. Additional details on model stage selection and model architectures can be found in the associated code. In the remainder of this section, we report the models used in this study and explain step-by-step how we obtain the model stage activations to perform regression.
S1.2.1 Supervised Models
We investigated supervised vision models trained on the ImageNet 1000-way categorization task [67]
with publicly available architectures and checkpoints obtained from the Py Torch library [65]. We included Alexnet [68], several ResNets [69]. WideResNets [70], DenseNets [71], ConvNeXts [72],
EfficientNets [73], MobileNet [74] and various vision transforms (ViT) [75].
We also investigated ANN models that were "biologically inspired" with CorNet architecture [76]. For these models, the pre-trained weights are pulled using the library
com/
dicarlolab/CORnet
In Table S1.2.1, we list all standard supervised models used in this study, together with their Top-1 ImageNet accuracies.
Supervised
ViT
Biologically Inspired
alexnet- Ace: 54.92
vit b 32 Acc: 75.16
cornet Acc: 72.51
resnet18 Acc: 68.98
vit 1it Ace: 85.16
cornet_ r Acc: 53.92
resnet50 Acc: 80.63
vit_L_32 Ace: 76.35
cornet z Ace: 45.67
resnet101 Acc: 81.71
vit_h_14- Acc: 85.74
cornet rt Acc: 53.72
resnet152 Ace: 82.25
wide_resnet50_22 Acc: 78.34
wide_resnett _2 Ace: 78.77
densener121 Ace: 73.71
densenet161 Ace: 76.70
d6nsenet169 Acc: 75.31
densenet201 Ace: 76.20
convnext small Acc: 83.69
convnext_base Acc: 83.92
pewweet_larrge Ace: 84.45
efficientmet Ace: 76.87
efficientnet_b44 Ace: 73.89
mobilenet_v2 Acc: 71.13
Table 1: List supervised models studied this work together with their Top-1 ImageNet accuracies on the validation set.
SI.2.2 Self-Supervised and Adversarially Trained Models
We also considered models trained either with self-supervised training (SSL) [77, 78, 79] or adver- sarial training (referred to as "robust" models) [80, 81]. All analyzed models were ResNet50 or ResNet101 architectures.
For SSL models, we considered Barlowtwins [77]. SimCLR [78] and MoCo [79] mod- els whose pre-trained weights were pulled from https://github. com/facebookreseacch barlostwins, https://github facebookresearch/nocco v3, respectively.
and https://gi thwbb//ttttttttt. com/
For the robust models, we obtained the pre-trate pre-trained weights from htt./ttttt///ghhhh....oa/ahhaaa robustness [50].
In Table SI.2.2, we list all SSL and robust models used in this study, together with their Top-1 ImageNet accuracies.
19

---

SSL
Robust
barlowtwins Acc: 71.68
robust_ reseeteesettt01_ 12 Acc: 56.25
simelr_resnet50wg Ace: 65.71
robust_resnet50 linf Ace: 61.26
simclr_resnet101 - Acc: 69.37
robust_resnet50_linf. Acc: 46.33
moco_ moco_resnett0- Ace: 74.15
Table 2: List of self-supervised (SSL.) and adversarially-traiee (robust) networks studied in this work, together with their Top-1 ImageNet accuracies on the validation set.
SI.2.3 Obtaining Model Stage Activations
We obtained activations from each model stage the stimuli from neural datasets. We employ the image preprocessing steps that were used during model training. For all models, this preprocessing is resizing image stimuli to (224, 224) pixels and normalizing them with the mean and variance of the ImageNet dataset, which are μ = (0.485, 0.456, 0.406) and a = (0.229, 0.224, 0.225), respectively. All activations were flattened before regression.
S1.2.4 Ridge Regression Experiment
After obtaining the model stage activations, we perform ridge regression by computing the predictions as:
()))) --xx__ (=x1pXIp + CregTp
Y YI.p.
(SL.2.1)
where Ip denotes the p identity matrix and Oreg denotes the ridge parameter. In this notation, (p))) is the P x N matrix of the predictions for the entire neural response set that is inferred using only subset of samples the training set. Then, X1:p denotes the p M matrix of the training set that the model activations for randomly sampled p stimuli and Y denotes the x N matrix of the corresponding neural activations.
Note that the least-squares regression ill-defined when the number of data points are more than the number of regression variables (i.e. p > MM since the matrix X XI:p....pp + Oreg is singular for Oreg 0. order to account for this issue, one can use the push-through identity to obtain:
Y (p) (p) ( + aregIM
XI YIP:
(S...2
where now IM is the M x MM identity matrix. The form in Eq. S1.2.2 the conventional form of ridge regression, while the equation Eq. S1.2. is the form obtained via the so-called kernel trick [82, 36]. However, we only use the form in Eq. 11.2.1, since the model activations often are high- sigh-dimensional (~ 1M) and hence M P. where P ~ 10 less.
We repeat the same computation over random choices of training set and report neural prediction error Eq. S1.1.11 averaged over these trials. Furthermore, in Sec. SI.4, we show the agreement between empirically calculated and theoretically predicted errors.
In our experiments, we choose Orreg 10 14 for numerical stability, which roughly corresponds to double-precision. Note that this choice of ridge parameter is to get close least-squares regression, in which case Oreg is identically zero. We considered very small ridge parameter, since even for ridgeless regression, strong spectral biases of overparameterized models avoid overfitting and effectively behave as if there is a non-zero ridge parameter [83, 32, 40, 84, 33, 85].
Here chose single regularization parameter for all experiments, however the choice of regular- ization parameters and the relation to the error modes is critical for fully characterizing the responses of encoding models. Future work could address this by choosing ridge parameters separately for different models and model stages, or allowing flexibility for different regularizations for individual unit responses (rather than one regularization for each neural region).
$1.2.5 Computing Theoretical Error Mode Geometry
After obtaining the model stage activations, we compute the eigenvalues and eigenvectors of each activation using the formula in Eq. 11.1.6. In terms of empirical Gram matrices, this corresponds to
20

---

performing Principal Component Analysis (PCA) on the Gram matrix. We project the neural response data onto the obtained eigenvectors and calculate W.. Then, theoretical values of /Dem and Ram are calculated using the formula given in Eq. 5. For each p. the value of x. is solved self-consistenlly using the JAX auto-differentiation library [66] and SeiPy optimizers [86].
SI.3 Relation to Brain-Score Neural Prediction Metrics
In this work, we considered ridge regression to compare model activations to neural responses. While there are many other choices for model comparisons such canonical correlation analysis (CCA), partial least squares regression (PLS) and kernel-target alignment (KTA) [22, 23, 51], we chose to focus on ridge regression as it is theoretically well studied in terms of spectral properties [32, 33).
To demonstrate that the neural prediction error we obtained from ridge regression is comparable to other widely-used metrics, we directly compared our predicted values to those obtained from PLS using the Brain-Score [11] framework. Brain-Score uses PLS regression and reports the noise- corrected Pearson r² score on test data to rank models. Note that PLS regression finds a new basis that maximizes the correlations between two sets. Then the regression done on new variables by performing dimensionality reduction on both sets using the common basis. If one uses all the components in this new basis, then PI.S equivalent to least squares regression.
To calculate the PLS Pearson r² value, we first extracted model activations as described in S1.2.3. We then fit PLS regression using the standard 25 latent components [11] and calculated the Pearson correlations between PLS-based predictions and actual neural responses on held-out data. We took the median correlation across all neurons and divided it by the median noise ceiling, which was calculated using split-half reliability across trials featuring identical stimuli. Train and test data points were randomly assigned, and the final Pearson y² values were calculated by averaging over 10 random splits. This procedure is described in more detail in [11] and corresponds to the default analysis for public Brain-Score neural prediction benchmarks in the following regions and datasets:
V1: movshon. FreemannZiemma22133gplllll V..-
V2: movshon. mowshon.Freemanzeen Preeeemaann/ieemmm221ppppp.....e V2-pls
V4: dicarlo MaajasHHooogg25555bblll.....9. V4-ps
IT: dicarlo. MajaaooMMMajjooooo Maa.aaHHooogg0005ppbllc.c.......
VI
V2
V4
IT
0.8
0-490073
F-05115761
0.6
78817
Q- p=-0.78100 1022
0.6
e-0
g=00899117
o 0847548
Bgio= 0.66
0.6
Pgin- 0.6P)
Egip- 0.6P)
0.5
Egip 0.6P)
0.6
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.0
0.1
0.2
0.3
0.0
0.2
0.2
0.4
0.6
0.2
0.4
0.6
PI.S Scores BrainSoore corrected Pearson
PI.S Scores BrainSoore corrected Pearson
PI. Scores BrainSSore
PI.S Scores Brinnoore
Noise corrected Pearson p2
Noise corrected Pearson
Figure SI.3.I: Brain-Score PLS regression 72 values vs. ridge regression empirical (p) values. Each point corresponds to the predictions obtained for single model and stage on the corresponding region data. Pearson y values and Spearman p values are given for the correlation between the Brain-Score value and the neural prediction error. In regions, there a general trend where models that do well on one metric also do well on the other metric. A train test split of 60%/40% is used for our ridge regression procedure and the 90%/10% split for Brain-Score PL.S.
In Fig. 1I.3.1, we compare the Brain-Score Pearson 72 from each model stage to the empirical ridge-regression E, (p). The Brain-Score metric and neural prediction error from ridge regression are moderately correlated, despite the fact that we are (1) comparing values between ridge regression and PLS regression and (2) comparing a Pearson x? metric to Ep(p) values, which are not directly comparable (Pearson ,2 measures the angle berween predictions and true labels, while neural prediction error measures the distance between the two).
21

---

To directly address the difference in metrics, in Fig. SI.3.2, we compare the Brain- Score PLS regression Pearson r² on the test data to ridge regression Pearson r² on the test data (using the same regression weights were used to obtain the (p)(p) values above. When using the same metrics on the two different regression methods, there a strong correlation for regions. Combined, these results show that by using the ridge regression formulation we generally preserve the ordering of the commonly used Brain-Score metric.
VI
V2
Y4
IT
0.20
F-0-908093
0.20
F-0 4400
0.
F-O --0.886471
F-O 855466 Q-0
100ge Peason?
0.15
0-0.925348
Peasow
0.15
BXdge Peunze
0.3
Ridge Pauroo
0.3
0.10
0.10
0.2
0.1
0.0
0.5
0.2
0.4
Scoces BruiiSouee corrected Pearson
Socres BriinSoore corrected Peurson
Scoues BraisScore
Scores BruinSooee corrected Peurson
corrected Pearson
Figure S1.3.2: Brain-Score PLS regression Pearson r2 values vs. ridge regression Pearson r² values on test data. Each point corresponds to the predictions obtained for single model and stage on the corresponding region data, Pearson x² values and Spearman values are given for the correlation between the Brain-Score value and the Pearson correlation obtained from the test set in the ridge regression. Note that our ridge regression Pearson r² values are not noise corrected, but we were to do so, it would correspond to dividing by a scalar value for each region (the noise ceiling obtained from the population). The same models and predictions are shown in Fig. SI.3. 1, including the train test split of 60%/40% for our ridge regression procedure and the 40%/10% split for Brain-Score PLS.
SI.4 Empirical vs. Theoretical Neural Prediction Error
Complimentary to the discussion around Fig. 3, we confirmed that the theoretical predictions (see Sec. SI. obtained from the spectral properties of data match with the empirical regression discussed in Sec. SI.2 for various values of p. In Fig. S1.4.1, we plot the learning curves for selected model stages a pre-trained ResNet18 architecture for all brain regions. We see that the theory matches the empirical values for all regions.
Note that particular choices of train/test splits may alter the ranking of which encoding models have the lowest neural prediction error (for instance, see layer4. 1. relu changing from the lowest neural prediction error at small p to the highest error at large p in Fig S IT). Additionally, as we mentioned in the main text, increasing the train/test split yielded worse agreement between empirical and theoretical neural prediction errors. The main reason for this disagreement is that the theoretical values depend highly on the estimation of the true eigenvalue distribution of the empirical Gram matrix of data. However, as discussed in Sec. SI.1, the tail of the true eigenvalues are often underestimated by the empirical eigenvalues and that creates discrepancy between observed Eg (p) and theoretically predicted one [63]. This discrepancy can clearly seen Fig. SI.4.2, where we plot the empirical neural prediction errors against the theoretically predicted ones. As the amount of training data increases (corresponding decrease in amount of test data), the correlation between the two guantities decreases monotonically.
22

---

resse08
V2 rescett8
VL. ressett8
IT reueet
1.01
leda
11.0
kect
Agett.idhh ada
600
Figure S1.4.1: Empirical vs. theoretical learning curves, this figure, we show the learning curves for regression from selected model stages in a trained ResNet18 architecture to all brain regions. The colored dots correspond to the mean empirical error over 5 trials, and the colored dashed lines correspond to the theoretical learning curves obtained by the formula for Eg (p). The vertical black line corresponds to the 60 train test split used for the main analysis in this paper.
Tellest Splic 0.9
Te/fest Split 0.9
Te/less Split. 0.9
Teriest Split 0.9 IT (R 200000.000 0-900))
VI -- 0.8 1.812)
V2. (87-6)
V/4 (R) 0.8.59)
0.20
1,00% theoretteal
0.375
E.jp). theorctial
0.25
1.10) tecorctica
0.70
6:180
0:15
0.125
0.15
0.20
0.35
0:30
0.65
0.06
005
0.10
0:15
0.7
05
cmpirical
Te'Test Splic: VI 0.85
TeTes Splic 0.8 (R3= 838)
Tellsst Splic:
ThTeest Splic 0.8. (R3= 0.933)
V4 (R+= 0.921)
0.35
0.4
E,igh, thcoetical
E,oo, incorctical
incoroticaal
8::00
Increasing Train/ Train/Test Test Split
04
0.25
0.3
0.20
0.90
0.1
0.3
0.3
0.3
0.2 Eytsh: empirical
E,tel, empprical
E,(s), empirical
E,te), empirieal
Tr/Test Split: 0.7
Tr/Test Split: 0.7 (R== O 0.866)
Tr/Test Splitt 0.7 (R3=
Ti/Test Sylii 0.7 0R==0 0.5))
(R==0.880
0.7
E,00), Dicorttial
Ejjor, licoreiicall
tacoretical
0.5
(.) Decorticcll
84
OUs
0.4
0.3
0.1
6.2
02
empiriasii
Eg(p), empiriill
5,09% empirical
5((9).mmiiiin
T1/Test Split: 0.6 VI 0.999
Tiffest Split 0.6 V2. (R'=0,897)
Te/les Split: 0.6 V4 (R== 0. 971)
Toflest Split 0.6 IT (R==0 0.973
5.00) lheorttiil
E, (0). Incorctionl
0.5
tncorctirsl
(.).) Incoritcall
066
0.4
0.4
0.4
0.3
0.4
0.4
06
0.4
0.2
0.6
Figure S1.4.2: Empirical and theoretical predictions worsen with train/test split. We show the empirical values of ridge regression Eg (p) compared to the theoretical values. As the amount of data used for training increases (corresponding to a decrease in test data), the correlation between empirical and theoretical values decreases.
23

---

SI.5
Additional Experimental Results
S1.5.1 Geometry grouped by architecture and training types
For visualization purposes, we show the enror mode geometries grouped by the architecture (Fig. 1I.5.I) and for a subset of models all with ResNet50 backbones but with different training types (Fig. S1.5.2).
50
11
ResNets
Nets
ConvNexts
10
DenseNets
EfficientNets
MobileNets
40
CORNets
40
0.03
0.04
0.03
0.04
0.004
0.008
0.012 0.004
0,008
0.012
R
R
Figure S1.5.1: Error mode geometry for supervised networks grouped by architecture type. All models included were trained in supervised manner on ImageNet (standard training without any ad- versarial perturbations) and were obtained from publicly available checkpoints in the PyTorch Library, with the exception of CORNets which were obtained from Model groups were ResNet architectures (resnet18, resnet50, resnet101, resnet152), Wide-ResNet architectures saadd_resset50_11 2, vide_resnet1111 ConvNext architectures (convnext_bast base, manvnext_smalll convnext__aree DenseNet architectures (densenet121, densenet161, densenet169, demsenet201), EfficientNet architectures (efficientnet_bbl efficientnet_b4) MobileNet architectures (mobilenet_v2), CORNet architectures (cornet_r. cornet rt, cornet_s, cornet_ae and ViT architectures (vit_b_32, vit 14, vit_1_16.
VI
V2
IT
Training Type
(ResNet50)
Late
11-
USC
000
Stage Depth (Normalized)
48
10
10-
46
Farly
46-
SimCLR
McCo
Barlow Tinns
0.03
0.04
0.025
0.030
0.004
0.006
0.004
0.006
0.008
R
R
Figure SI.5.2: Error mode geometry for supervised and self-supervised ResNet50 architectures. Supervised ResNet50 architecture (trained on ImageNet without any adversarial perturbations) was compared to ResNet50 architectures trained with sulf-supervised methods SimCLR [78], MoCo [79], and MarlowTwins [77] There are no clear differences in geometries between models with different training mechanisms. This in line with previous work that found similarities in representations between supervised and self-supervised networks [87, 88, 89].
SI.5.2 Geometry across model stages
In the main text, we focused on data from regions V1 and IT when analyzing the error mode geometry's dependence on model stage depth. Analysis of regions V2 and V4 is included in Fig. SI.5.3. Additionally, to show the error mode geometries from different model stages Fig. 4A and Fig. $1.5.3A, we zoomed in on the parts of the space with the most data points and maintained the same axes limits for regions that shared the same stimulus set. Fig. S1.5.4, we present all model stages, including outliers, colored by model stage depth.

---

A
V4
Latest
11
Stage Depth (Normalized)
45.0
10
Figure S1.5.3: Relation of error mode geometry with model stage depth in V2 and V4. (A) Each point represents mode geometry for a specific model stage of trained neural network models, where the color of the value represents the depth of the model stage. For each model, the color code is normalized in the [0, 1] range where the earlier and later model stages have lighter and darker colors, respectively. Rem and / values are obtained from theoretical values, and contour lines cor- respond to theoretical values of the neu- ral prediction error as in Fig. 2. (B,C) Predicted (p) that would be obtained when using the eigenspectra from the first (Early) or last (Late) stage of each model, paired with the alignment W,, terms from the Early or Late stage of each model. Full enror mode geometry is given in (B), while summarized bar plots of the Eg(p) from each comparison is given in (C). (D) Full spectra for A, W; and W. for Early and Late model stages of V2 and V4 datasets.
42.5
40.0
Earliest
0.03
R.
0.05
0.005
0.010
V2: Early vs. Late
V4: Early vs.
W (alignment) Early
0.75
7. (eigenspeetra)
10-
0.0
45
Early
40
Late
+
x
8-
0,025
0,050
0.075
0.100
0,005
0,010
0.015
R
R_,
V2
D
V2
V4
Early (Finst Stage)
S5
ResNor50
0.65
10
(p)
Eigenvezor
0.45
10
I
0.25
10
Late (1 ast Stage)
Weight
10=
Buly A. Budy
Bady B. Lato
Lato 1062,,100 Farg
Tarly 2, Early
Eady 2. Late
Late Larg
Lite Laie
Fate
0
1000 eigenvector index
VI
V2
V4
IT
50
008
Stage Depth (Normaliasd)
Dem
45
0,4
40
0.2
0.05
0.10
0.025
0.050
0.005
0.010
0.015
0.010 0.015
Rem
Rem
Rem
Figure S1.5.4: Error mode geometry for trained neural networks, colored by model stage depth. Same plots as Fig. 4A and Fig. SI.5.3A, but including the outliers.
We further elaborate on the spectral properties of different model stages and the corresponding error mode geometry by using the lo trained ResNet50 [50] model as an example. We plot the eigenvalues of the model activations from each model stage for all tested visual regions(Fig. SI.5.5). For V1 and V2, we saw that earlier model stages had better predictions, while for V4 and later model stages performed better.
25

---

V1- robust resnut50 B
V2- robust resee550 Inf
13"
Sinta
14-
10
1ayed 0:277
2.l
110
320
30
100
V4- robust resvet5O Fnrt
2220-
IT robust resnet50 Jist
Sinle 3.231
S. 0.273
13"2
layend 2irea 3 300
1
Tayert rds 0:419
-5" 23"1
10m
15
1000
2000
31.01 1140 20.1 3000
1100
2001
3391
10001
2000
3000
Figure SL5..5: Spectral properties of &o trained ResNet50 model on all regions. For each region (V1, V2, V4, ITT,, the left panel shows the eigenvalues the model stages. The legend shows the best and worst model stage in terms of neural prediction error, and the colors correspond the colors associated with that model stage (see Fig. 4). The right panel shows the error mode spectrum for each model stage, with the inset showing the ordered spectrum.
The spectral properties of each model stage depend highly on the input distribution. We notice that, under the input distribution of V1/V2 data, the eigenvalues carlier model stages decayed more slowly than later model stages, while under the input distribution of V4/IT data this pattern is reversed where later model stages decayed more slowly. These differences are not due to the recorded neurons, the eigenspectra only function of the stimulus set, pointing out the importance of considering the stimuli used for the neural predictions. Future work could consider choosing stimulus sets to maximize the difference between model eigenspectra. This would likely lead to greater differences between models in their predictions of neural activity patterns.
SI.5.3 Geometry for Trained vs. Untrained Models
In the main text analysis trained vs. untrained models, we zoomed on the parts of the space with the most data points and maintained the same axes limits for regions that shared the same stimulus set (Fig. 5)A. In Fig. S1.5.6, we show all the points of the trained and untrained networks, including the outliers. We also include analysis of the error mode geometry for trained untrained networks when predicting an additional VI dataset [46] Fig. S1.5.7.
VI
V2
V4
11
025
11
075
50
Random Trained Lale
0.50
10
Stage Depeh (Normalized)
9-
9-
40
40
8-
-Earl
0.05
0.10
0.025
0.050
0.075
0.01
0.02
0.025
0.050
R.n.
R.n
R.n
R..
Figure SI.5.6: Error mode geometry changes with model training, full plots showing outliers. Same plot Fig. 5A, but including the outliers
26

---

VI (Freeman al.): 135 Stim
VI (Cadena et al.): 135 Stim
VI (Cadena al.J: 1250 Stim
Random
Trained
11
075
1.0
11.0
00
0.5
10.5
09
Suug Depth (Nommalood)
10-
0.6
110.0
0.0
30
04
9.5-
0.4
0.2
9.0
0.2
26+
-Faryy
0.0
8.5
0.0
Raudont
26-
0.0
3.0
0.05
0.10
0.050
0,075
0.100
0.00
0.04
R.,
Rg
R,
Figure SL5.7: Error mode geometry for V1 dataset from [46]. Random and Trained error mode geometry for (A) previously analyzed V1 dataset, (B) V1 natural image dataset from [46] matched to stimulus set size of (A), and (C) using full V1 natural image dataset from [46]. In all cases, there was a small decrease in (p) when using the trained neural network activations to predict the neural data (but this difference was less notable than in some of the other regions). Increasing the dataset size from 135 to 1250 stimuli did not change the overall trends.
SL.5.4 Geometry for Robust Standard Models
In this section, we consider the model stage-wise changes error mode geometry for an foc adversarially trained network, opposed to the Eg trained network presented in Fig. 6A. In Fig, SI.5.8, we find that the error mode geometry is very similar for this network, suggesting a consistent effect of adversarial training the mode geometry.
Linf
VI
Robust
Standard
50
50
Late
48
48
Stage Depitt (Normalized))
DDm
10
10
46
46
9
42
42
Early
0.04
0.03
0.04
0.004
0.006
0.008 0.010.0.00
0.006
0.008 0.010
Rem
Rem
Rem
Figure $1.5.8: Error mode geometry for for adversarially trained model. Same plot as Fig. 6A, but shown for foo trained ResNet50.
For visualization purposes, in Fig. S1.5.9, we plot the eigenspectra and error mode weights for standard and adversarially trained model.
S1.5.5 Effective Dimensionality and Neural Prediction Error
Previous work has suggested that the effective dimensionality of a model's responses to visual stimuli correlates strongly with its neural predictivity [31]. Here, we did not find relationship between neural prediction error and the dimensionality of the model activations. We measured the effective dimensionality (ED) of the model stage responses for the stimuli in each neural experiment, extracted as described in SI.2.3, with the participation ratio of the eigenvalues of Gx:
ED Ee Σ
(SI.5.1)
We tested the correlation between ED and both Brain-Score performance and neural prediction error. In Fig. SI.5.10. we show that neural prediction error does not correlate well with model dimensionality across any of the datasets considered here. This to expected based on the theory presented although the spectral properties of the model activations to the stimulus set is one of the factors
27

---

VT
V2
1374
10
13
10"1
16'1
19"4
10.
10"1
16-4
13-5
25
50
100
175
100
100
130
125
V4
1T
13
reanatão
30
10
reanatoo
30-
10-2
13a
10
10.
10"
12
12
2000
2000
Figure $1.5.9: Spectral plots for standard and adversarially robust ResNet50. Eigenvalue spectra and error mode weights are given for all model stages of a standard ResNet50 (blue) and Ez(c 3) Robust ResNet50 (green) for all brain regions.
leading the observed values of W., the direction of the model eigenvectors is also important for determining the error mode geometry. These properties are captured by our proposed geometric quantities Rem and Dem. which are directly related to the neural prediction error. Additionally, in the main text, we investigated the neural prediction error in cases where we modified the eigenvalues or alignment coefficients (which are dependent the model eigenvectors).
0.45
0.45
pr 0.94799779
0.40
1644
E,(p=0.67)
0.40
0.40
0.35
0.35
0.30
0.30
0.25
0.35
100 Effective Dimensionality
0
50
100
1000 Effective Dimensionality
500
500 1000
Effective Dimensionality
Effective Dimensionality
Figure SI.5.10. Effective dimensionality from model activations vs. Ep(p) values for each neural region. When the effective dimensionality and generalization error are measured using the full space of model activations, we do not see a strong correlation with either Pearson correlation (r²) or Spearman correlation (p) between ED and (p).)
We note that the authors of [31] applied either an average-pooling or dimensionality reduction method model activations before carrying their regression analyses. To ensure that we were able to reproduce the results of [31], we performed similar pooling on our model activations, which is complete average-pooling on activations from each convolutional filter. When everage-pooling to our model activations, we found that setting Oreg 10-14 as above yielded several outliers with unusually high Ey values. We therefore set Oreg to the value that yielded the lowest possible Ey for each set of model activations. When pooling model activations, we found a relatively strong relationship between prediction error and model dimensionality in IT similar to that reported in [31] (Fig. 11)5.11). We further replicated the results in [31] by directly comparing effective dimensionalities obtained from the spatially averaged activations to the Brain-Score noise-corrected PISS regression values (Fig. 12... Taken together, these results indicate that pooling discards important variability the model responses, and that when we use the full model activations the model dimensionality does not correlate well with neural prediction error.
28

---

V1
V2
V4
IT
0.2114177
F 1-0.180055
0...06605
P. 0.82205
-0.378106
0.8
0--0.355735
80888446666
0.8
0.7
0,3
0.8
p
0.6
0.7
0.6
0.5
0.6
0.4
0.4
0.3
0.5
10
25
50
75
50
Effeective Dimensionality (Averaged Unit Responses)
Dimeusionality (Averaged Unit Responses)
Dimensionalitt (Averaged Unit Responses)
Dimensiomality
(Averaged Unit Responses)
Figure S1.5.11: Effective dimensionality from spatially averaged unit activations to stimulus set vs. (p) values for each neural region. When the effective dimensionality and regression both performed on the spatially averaged unit activations, we see a stronger correlation between the effective dimensionality and the neural responses, particularly in IT and V4. However, the neural prediction emrors are generally higher when using these pooled responses (note the different y axes compared with Fig. SI.5.10), demonstrating that this type of pooling discards much of the important variability in the model responses
V1
V2
V4
IT
Pearson r (Noixe corrected)
0.6
2 0.01
300074
PLS Scores BrainSoore
0.3
@ -.0044001
004704444
0.5
0.3
0.5
0.2
0.2
0.4
0.3
0.1
0.1
0.3
2 0.133616 0.477885
0.2
0.318168 0.794994
0.0
15
15
50
Efffective Dimenssionlity (Averaged Unit Responses)
Effective Dimensionality (Avoraged Unit Responses)
Effective Dimensionality (Averaged Unit Responses)
Effective Dimensionality
(Averaged Unit Responses)
Figure 12: Effective dimensionality from spatially averaged unit activations vs. Brain-Score values for each region. When the effective dimensionality is calculated on the spatially averaged unit responses, rather than the full space of activations, in [31] for IT responses. In other brain regions, the relationship is less monotonic, appearing as the "joint regime" discussed in [31]. Note that the Brain-Score metric does not rely on taking the spatial average across units, but does perform dimensionality reduction as implemented with PLS.
correlation that mirrors that reported
29


Reviews:
[{"review_id": "jl5c6BcXUu", "reviewer": "Reviewer_Q3vo", "summary": "In recent years, DNNs trained on image recognition tasks have emerged as strong predictive models of neural activity in the visual cortex. Typically, a regression model is trained to map DNN responses onto biological neural activity in response to the same inputs. The standard method for evaluating the predictive capacity of given DNN is to quantify the variance explained by the model on held-out images. This compresses the structure of each model into a single scalar value. While this can be used to rank models, it provides little insight into the structure of the model's representations and how they align with the structure of the biological neural representations. Another approach analyzes the similarities in representational geometry between models and neural data. However, the authors, state, these approaches do not relate the geometry of the DNN to the regression model used for the neural predictions.\n\nIn this paper, the authors leverage a recently proposed method for analyzing the generalization error of a model in terms of the radius and dimensionality of the error along each eigenvector of the input data (images viewed by both model and brain), to analyze the geometry of the generalization error of the regression model that maps a given DNN to neural data. \n\nThe authors use this method to analyze layer-wise differences in prediction error in a DNN, in terms of the error geometry. They find that the contributions of the error radius and error dimension change across layers. \n\nThey also analyze difference between trained and untrained DNNs at predicting neural responses. For V1 data, they find that the difference in predictivity and error geometry for trained vs. untrained networks is small. However, for V2 - IT data, they find interesting differences. In particular, they find that training reduces error radius, but increases error dimensionality, i.e.  spreads errors more evenly across error modes. They suggest that the improved predictivity of trained networks for V2 - IT data is primarily driven by changes in their intrinsic expressivity rather than significant changes in their representational alignment with neural data. \n\nIn addition, they examine differences between adversarially vs. standardly trained models. Replicating previous findings, they find that adversarial training overall improves model predictivity. They observe differences in how it affects error mode geometry across layers. In earlier layers, error radius is decreased with little change in error dimensionality. In later layers, error dimensionality is the main distinction between robust and standard models. \n\nLastly, they examine the dependence of the generalization error on the size of the training set of images. They find that for small datasets, the directions with small eigenvalues yield high error. For larger datasets, the additional samples permit learning along small eigenvalue directions. ", "questions": "I'd like to get a more intuitive handle on the significance of the error modes. I expect the eigenvectors of the gram matrix of the image data would be something like a Fourier basis. In this case, we'd expect large eigenvalues for low frequency bases and small eigenvalues for high frequency. Does this analysis essentially explain how error is distributed across frequency content in the images? If so, how does this give us insight into representational alignment?", "limitations": "The significance of the conclusions drawn from the experiments is not clear. The purpose of this tool is to provide insight into neural representations in DNNs and the brain. Greater effort should thus be made to make the meaning of the quantified values (error radius / dimension along eigenvalues of the data's Gram matrix) interpretable.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper applies a recently proposed methodology for analyzing the generalization error of a model to the problem of understanding DNN-neural data representational alignment. It is a solid idea, and the authors conduct a suite of well-posed experiments that analyze existing results from the literature in this frame.", "weaknesses": "The paper would benefit from several changes.\n\nFirst, the motivation of the methodology was unclear. Intuitively, why are we interested in understanding the radius and dimensionality of the error? What insight can this give us into understanding representational alignment? What's the motivation for projecting along the eigenvectors of the data's Gram matrix? I think strong answers to these questions can be formulated. They should appear front and center in the paper. \n\nSecond, the conclusions obtained by the experiments were difficult to tease out. These should be clearly and simply stated, and presented in a list as an overview of the results. The way the results are written and presented visually, the reader needs to do a lot of digging to parse the meaning and significance. Consider making more intuitive graphical figures to emphasize the main points / contributions / conclusions, and/or condensing the results figures into simpler plots that highlight the important trends.\n\nLastly, the standard approach for interpreting DNN-neural data alignment could be more clearly explained and illustrated, so the reader understands how your approach differs. Consider making a schematic plot that highlights this distinction. \n\nOverall, I felt the paper required considerable effort to understand. Given that this work draws from several disciplines and relies on a very recently proposed methodology, I believe the authors should do more work to help the reader understand the motivation and significance of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review of our work. Please see the responses below.\n\n## Weaknesses:\n\n1. **Motivation for methodology:**\n\n    We are thankful for the suggestions and made changes to improve the clarity of our work.\n    \n    \n    We found that the radius and dimensionality of the error modes are a convenient way to summarize spectral properties of the generalization error. Specifically the two measures jointly summarize whether the distribution of the error modes, $\\tilde W_i$ are relatively spread out (higher $D_{em}$, lower $R_{em}$), or tightly concentrated (lower $D_{em}$, higher $R_{em}$) at a fixed generalization error. Because the total error ($E_g(p)$) is the product of these two properties, we can easily visualize how the error mode geometry of many models differs. We include RFig 2 (which we will add to the main text) to give readers additional intuition for these quantities and how to interpret our plots.\n\n\n    The motivation for projecting along the eigenvectors of the Gram matrix is that it allows us to write the generalization error in terms of the alignment coefficients, $W_i$ and the spectrum, $\\{\\lambda_i\\}$. We have highlighted these interpretations in the revised figures (RFig 1C-D) and the revised section 2.2–see the global response for details.\n\n2. **Conclusions for experiments and clarity:**\n\n    Thank you for this suggestion. We have added a list of key contributions to the end of the introduction (see the global response). We have significantly revised Figure 1 (RFig 1), and have expanded Figure 1E (RFig 2) to help readers interpret the contour plots. We have also replaced Figure 4B-D with RFig 4, where we performed an explicit experiment analyzing the contribution of $\\lambda_i$ and $W_i$ to the generalization error to emphasize the main conclusions of this section (see global response for full details).\n    \n3. **Comparison to standard approaches:**\n\n    Our approach decomposes one commonly used measure of DNN-brain comparison (neural predictivity via regression) in terms of the spectral properties of the model and its alignment with the neural data. As such, our analyses primarily concern how and why  models achieve the predictivities that they do, rather than offering an alternative scalar metric for predictivity. Indeed, as one reviewer noted, \"Analyzing the eigenspectrum of a model's feature space [....] goes beyond the standard measures of representational similarity analysis or linear regression experiments that one usually comes across for this line of research...\" To make this more explicit we have added a schematic figure highlighting this aspect of our method (RFig 2).\n    \n4. **Overall:**\n\n    Thank you for this suggestion. We believe that the changes detailed in the global response have made the manuscript significantly easier to understand. \n    \n## Questions:\n\n1. **Intuitive picture for error modes and analogy to Fourier basis:**\n    \n    Please note that the Gram matrices that we refer to in the paper are defined on the artificial and biological network representations, rather than the images themselves. For example, the gram matrix $\\mathbf{G}_X$ is defined as $\\mathbf{G}_X = \\mathbf{X}\\mathbf{X}^T$, where $\\mathbf{X}$ is a $P\\times M$ matrix containing the activations of $M$ artificial network units responding to $P$ images. Given that the network responses have no direct connection to the frequency content of the images, neither do the Gram matrices.\n\n\n    Our work focuses on the alignment between the gram matrices of the artificial and biological network representations, denoted $\\mathbf{G}_X$ and $\\mathbf{G}_Y$, respectively. The alignment between these two matrices as expressed by the coefficients $W_i$ therefore quantifies the similarity between the artificial and biological network representations.\n\n\n    Nevertheless, we agree that it would be of interest to gain insight into how these error modes map onto the image space. One future direction we are interested in pursuing is visualizing error modes for a given network that are contributing the most and least to the error. It is possible that these could somewhat correspond to specific frequencies (especially when investigating early layers of the neural network). However, this type of analysis is beyond the scope of this paper.\n\n## Limitations:\n    \n1. **Main conclusions from experiments and their significance:**\n\n    Thank you for this suggestion. To make the interpretation of $R_{em}$ and $D_{em}$ clearer to the reader, we have significantly revised the figures in the paper with helpful schematics. Furthermore, we greatly simplified the presentation of the theory in section 2.2 (see global response), and we revised section 3.2 to more clearly explain our results involving $R_{em}$ and $D_{em}$ (see global response).\n"}, {"review_id": "FIe48ipXwd", "reviewer": "Reviewer_RZx6", "summary": "The authors studied linear regression-based DNN encoding models of macaque visual cortical areas by extending a recent generalization error theory. They provided a theoretical link between the predictivity and geometry of representations and showed that models with similar generalization errors may have quite different geometrical properties. They compared a randomly initialized ResNet50 with standard and adversarially trained ResNet50 models. Evaluations and analyses are a bit limited.", "questions": "- How do you explain a decrease in $R_{em}$ and an increase in $D_{em}$ that is caused by (standard) training?\n- Why did you choose a (wide) ResNet50 and not any other model? Is there a particular reason for that choice?\n- Why do you think \"[...] that trained networks do better than their random counterparts at predicting V1 data for large, but not small, training data set sizes\"? Is it possible that the size of the (training) data to fit the regression model is just too small to explain any variance at all or do you have another explanation that rules out that possibility? I think that $N = 135$ is too small to see any effects at all which is why I think there's no notable difference between pretrained and randomly initialized networks in your analyses. However, $N = 3600$ is large enough to observe statistically significant effects. So, I doubt that you can draw any conclusions at all about the smaller of the two datasets. Note that $N = 3600$ is still not a particularly large dataset.", "limitations": "Yes. The authors discussed limitations of their work in a dedicated section at the end of the main paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Analyzing the eigenspectrum of a model's feature space appears to be an interesting analysis that goes beyond the standard measures of representational similarity analysis or linear regression experiments that one usually comes across for this line of research at the intersection of cognitive neuroscience and machine learning. So, I appreciate that effort.", "weaknesses": "**Major** \n\n- **Tense**: There's a mix between present tense and past perfect in $\\S2$ which I think is not great. Choose one tense and use it consistently. I'd use the present tense to describe the problem settings and methods and either use present tense or past perfect for the results section. Also, use active rather than passive voice which I think is better readable (and more scientific).\n\n- **$\\S3.4$ + main results**: I find it very strange to denote the training set size with small $p$. As I mention below, $p$ is usually used to refer to the number of features in some representation space and not to the size of the (training) data. If I was just reading the title of the subsection, I'd think that the generalization error depends on the size of the feature space (although I would not know on which of the feature spaces). This is not implausible, but very different from what you mean/find. Actually, it's pretty trivial that the generalization error depends on the size of the training data for fitting the regression model. When would the generalization error not depend on the size of the training data? This section is in general a bit confusing. In lines 205-207 you write: \"[...] it may be that trained networks do better than their random counterparts at predicting V1 data for large, but not small, training data set sizes.\" Do you refer to the size of the (training) data with which the regression model is fit? If that's the case, I am a bit puzzled because from Bayesian statistics (and statistics in general) we know that the prior is most important for small data regimes. In the infinite data limit, the prior and, therefore, any regularization parameter stops mattering. A pretrained neural network has a much better initialization compared to a randomly initialized network. It has found a good local minimum over the course of pretraining which gives it a big head start. The pertaining is an implicit regularization for any downstream task. Here, we can treat the regression task as a downstream task. So, the prior should matter most when the training data to fit your regression model is small. You seem to find the opposite, or at least that's what you claim. The only possibility where I can see this being the case is when the training data size for the downstream task (e.g., for performing linear regression) is so small that the parameters of your (regression) model can basically explain no variance at all. But then the findings are pretty useless, aren't they? Actually, I am confident that $N=135$ is too small to find any statistically meaningful effects.\n\n- **Model choice**: It would have been interesting to see at least one, if not more, other architecture(s) being analyzed. I doubt that you can generalize any findings from a single ResNet50 to the whole space of DNNs. Even within the subclass of CNNs, there exist countless models. I find it more interesting to compare CNNs (pick two or three) to ViTs (pick two or three) but comparing a few CNNs (each with a different inductive bias --- e.g., AlexNet, VGG16, ResNet50, ResNext, ConvNeXt, EfficientNetV2), ideally trained with different recipes would have probably been sufficient.\n\n**Minor**\n\n\n- Lines 26-27: \"[...] many different architecture and training procedures lead to similar predictions of neural responses.\" While the former is true and has been demonstrated multiple times in various ways, it has recently been shown in a comprehensive study that both training data and objective function have a major impact on the degree of alignment with human behavior; much more than architecture (see [Muttenthaler, L., Dippel, J., Linhardt, L., Vandermeulen, R. A., & Kornblith, S. (2022)](https://openreview.net/pdf?id=ReDQ1OUQR0X)). A longer while ago it has also been shown that different data augmentation strategies lead to different biases and as such to different degrees of consistency with human errors. See [Hermann, K., Chen, T., & Kornblith, S. (2020)](https://proceedings.neurips.cc/paper_files/paper/2020/file/db5f9f42a7157abe65bb145000b5871a-Paper.pdf); [Hermann, K., & Lampinen, A. (2020)](https://proceedings.neurips.cc/paper/2020/file/71e9c6620d381d60196ebe694840aaaa-Paper.pdf); [Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2018)](https://arxiv.org/pdf/1811.12231.pdf); and [Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., & Brendel, W. (2021)](https://proceedings.neurips.cc/paper_files/paper/2021/file/c8877cff22082a16395a57e97232bb6f-Paper.pdf).\n\n\n- In line 58 is a small grammar mistake: \"[...] characterizing how fast the eigenvalues of data Gram matrix fall [...]\". There is an article missing before **the** data Gram matrix. \n\n\n- **Notation**: The notation in $\\S2.2$ is weird. Why is the matrix of neural activations $\\mathbf{R} \\in \\mathbb{R}^{P \\times N}$? First, $\\bf{R}$ can sometimes have a special meaning. So, I'd be careful with that. Second, although not bolded, you use $R$ for the generalization error terms. Third, generally $N$ is used to denote the number of stimuli/examples in your training data and not to refer to the number of features.  My suggestion is to denote the matrix of model features as $\\mathbf{X} \\in \\mathbb{R}^{N \\times D}$ and the matrix of neural activations $\\mathbf{Y} \\in \\mathbb{R}^{N \\times P}$. As such, the regression problem $\\mathbf{Y} = \\mathbf{X}\\mathbf{A}^{\\top} + b$ becomes more intuitive. I'd replace $\\mathbf{R}^{\\ast}(\\alpha_{reg}, p)$ with the full equation. $\\mathbf{R}^{\\ast}(\\alpha_{reg}, p)$ is not easy to parse. That being said, note that this is just my personal taste which is why it's a minor weakness. There may be people who don't care about that. I think that maths should be easy to follow, even if a reader skims the paper. I dislike decorative maths. If it's just decorative, it can as well be omitted.\n\n\n- In Equation 2 is an \"is equivalent to\" symbol. It is probably not clear to everyone why the two terms are equivalent but not equal. I'd like to see a (short) derivation of it. You can derive it in the Supplementary Material. Similarly, there are two \"$\\equiv$\" symbols in Equation 5. Why did you choose \"$\\equiv$\"? How are the terms \"equivalent\" but not \"equal\"? Do you perhaps mean \"$\\coloneqq$\" for an assignment of variables? That would make more sense to me. \"$\\equiv$\" seems to be an abuse of notation but maybe I am missing something here.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your careful, extensive and constructive assessment of our work which led us to improve the clarity of our paper! Please also see global response and Rebuttal Figures (RFig).\n\n**Tense**: Thank you for pointing this out. We edited our draft so that the tense is consistent in each section, and we now use active voice when possible. Please see global response updates to §2.2 and §3.2 for examples.\n\n**§3.4 + main results**:\n1. **Notation:** Thank you for pointing out this discrepancy in the literature. We use the convention where $p$ is the training set size and $P$ is the full dataset size to be consistent with previous theoretical work [1,2]. But if you feel that it is absolutely necessary, we are willing to change the notation.\n2. **Text and Title:** We fully agree that the title and text of this section are confusing. The main purpose of this section is that metrics for \"ranking\" models depend on dataset size and the ordering of models can change based on this decision. This will be clarified in the main text. We also changed the section title to “*Characterizing the Dependence of Generalization Error on Training Set Size*\", and updated the first sentence to “*Our spectral theory of neural predictivity characterizes the relationship between the training set size $p$, the generalization error, and the spectral properties of the model and brain responses.*”. \n\n    Note that this section mainly explains potential consequences from our theoretical results, and as such could be moved to the Discussion if you think it's appropriate.\n\n3. **Regarding model priors and dataset size:** We apologize for the confusion and thank you for bringing up this excellent example!\n\n    While we agree with your intuition, the theory is precisely useful for such examples. In our framework, having a good prior corresponds to the variance in neural responses $\\mathbf{Y}$ being fully characterized by the first few model eigenvectors $\\mathbf{v}_i$ with large eigenvalues $\\lambda_i$. Since the theory poses that the variance corresponding to the larger eigenvalues are learned in greater rates, the theory confirms your example when few samples are sufficient to describe the majority of the variance.\n\n    However, we observe that the distribution of the data variance among model eigenvectors on V1 data are quite similar between trained and untrained models, and widely spread. This means that in small data ($p$) regime, the explained variance remains roughly the same for both models. Then we concluded that the trained models *may* distinguish themselves in the large $p$ regime where the observed high dimensionality (hence expressivity) of the feature space ensures learning remaining portion of the variance in data.\n\n4. **Model Choice:** Thank you for expressing concerns regarding experiments, which we believe is a confusion due to lack of experimental details in the main text. The models investigated in Fig. 2-4 spanned many different CNNs and ViT architectures. This is now clear in main text lines 76-80 (see response #5 to PvxB). We also included RFig 3 of the error mode geometry for supervised neural networks grouped into different architecture types. For Fig. 5 we used publically available checkpoints for ResNet50 so that we had different training types with the same architecture. \n\n\n\n**Minor Concerns:**\n\n1. **Lines 26-27:** Thank you for highlighting this point and providing these references. We agree that the line of work focusing on human vs. model behavior is particularly interesting, especially given that many models have similar neural predictions. We tried to convey this in the original text Lines 27-33 (which already include some of the referenced papers), but agree that this could be further clarified. We modified the paragraph starting at line 27 and added in these citations. The final sentence of that paragraph now reads: \n\n    “*Given the increasing number of findings that demonstrate large variability among candidate computational models, it is an open question of why current neural prediction benchmarks are less sensitive to model modification, and how to design future experiments and stimulus sets to better test our models.*”\n    \n2. **Grammar Mistake in Line 58:** We updated this.\n\n3. **Notation in §2.2:** We have changed our notation for the neural data matrix from $\\mathbf{R}$ to $\\mathbf{Y}$ as suggested.\n\n4. **The $\\equiv$ symbol:**  Indeed, the “equivalent” symbols meant assignment of variables. We switched to the “:=” symbol as suggested.\n\n**Questions:**\n\n1. **$R_{em}$ and $D_{em}$ after training:** These points are now explicitly described in the revised §3.2 (see the global response), and the corresponding RFig 4.\n\n2. **Choice of ResNet50:** Please see the response above regarding other models. \n\n3. **Size of Training Set and V1:** We thank you for this question, which led to experiments presented in RFig 4 and RFig 6. A few things to note:\n    1. First, a similar finding has been reported before that the random networks do fairly well for early visual areas [3,4,5], and our observations that the differences are more pronounced in later regions is in line with these previous works. That said, the V1 data is still slightly better predicted by trained networks, which is now explicitly stated in RFig 4B. \n    2. Note that the V2 dataset has the same number of stimuli presented as the V1 dataset, and we see more difference between trained and random networks in this region, thus the difference cannot be only due to stimulus set size. \n    3. We have run an additional suite of experiments on V1 region data with larger data sizes ($P=1250$ stimuli) and find that our observations are consistent. Please see our response to reviewer t8VM and RFig 6 for more details.\n\n[1] Bordelon et. al., ICML, 2020\n\n[2] Canatar et. al., Nature communications, 2021\n\n[3] Schrimpf et. al., BioRxiv, 2018\n\n[4] Saxe et. al., Nature Reviews Neuroscience, 2021\n\n[5] Elmoznino et. al., bioRxiv, 2022"}, {"review_id": "t7vQhIRFZH", "reviewer": "Reviewer_3d9A", "summary": "Previous works have demonstrated that many different state-of-the-art deep neural networks (DNNs) perform similarly at neural responses prediction. But a complete understanding of which aspects of these DNNs lead to the similarity in predicting neural responses remains unknown. The authors proposes a spectral theoretical framework to explore how the geometrical properties of DNNs affect the performance of neural responses prediction. More specifically, the authors introduce a generalization error to serve as a measure of fit, which brings a geometrical interpretation of the model activations (representations) in DNNs. This geometrical interpretation could further give additional insights into how DNNs are achieving different performance of neural responses prediction. The authors also design several experiments to investigate the roles of layer depth, dataset size,  and different training approaches of DNNs in neural responses prediction.", "questions": "N/A", "limitations": "I have highlighted weaknesses above. I have nothing further to add here.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The authors establish a novel link betwwen the predictivity of neural responses and the geometry of DNN representations.\n* The authors provide a solid analysis and several comprehensive experiments to support their theoretical framework.", "weaknesses": "* The authors should improve the clarity of their conclusions from their results. Section 3 provides many details about their analysis and results, but it's hard for the readers to extract the essential arguments the authors want to convey. It would be better to summarize your conclusions at the beginning or end of each paragraph.\n*  The authors should improve their exposition of the spectral theoretical framework. I recommend the authors include more background information and provide some examples to show the role of each equation in the relationship between the generalization error and the geometry of model activations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review and suggestions. Please see our responses below.\n\n1. **Clarity issues and summary of results:**\n\nWe thank you for these suggestions and agree that the clarity should be improved. In response to this comment and others, we added a concise list outlining the principle contributions of paper to the end of the introduction. We have also internally updated our document to increase clarity of the results section focusing on the main conclusions for each section. Furthermore, we uploaded Rebuttal Figures (RFig) further explaining main concepts.\n\n2. **Improvement for spectral theory framework:**\n\nWe significantly simplified the exposition of our theory in response to this and other comments, please see the RFig 1 and RFig 2. Furthermore, please see the updated version of Sec. 2 below:\n\n> In response to a total of $P$ stimuli, we denote model activations with $M$ features (e.g. responses from one stage of a DNN) and neural responses with $N$ neurons (e.g. firing rates) by $\\mathbf{X}\\in\\mathbb{R}^{P\\times M}$ and $\\mathbf{Y} \\in \\mathbb{R}^{P\\times N}$, respectively. Sampling a training set $(\\mathbf{X_{1:p}}, \\mathbf{Y_{1:p}})$ of size $p < P$, the ridge regression solves: $$\\hat{\\beta}(p) = \\arg\\min_{\\beta \\in \\mathbf{R}^{M\\times N}} ||\\mathbf{X_{1:p}} \\beta - \\mathbf{Y_{1:p}}||_F^2 + {\\alpha^{reg}} ||\\beta||_F^2, \\quad \\hat{\\mathbf{Y}}(p) = \\mathbf{X} \\hat{\\beta}(p)$$\nWe analyze the neural predictivity based on the generalization error $E_g(p) = \\frac{||\\hat{\\mathbf{Y}}(p) - \\mathbf{Y}||_F^2}{||\\mathbf{Y}||_F^2}$ for which we utilize theoretical tools from learning theory, random matrix theory and statistical physics to extract geometrical properties of representations based on spectral properties of data. In particular, the theory introduced in [1,2] relies on the orthogonal mode decomposition (PCA) of the Gram matrix $\\mathbf{X}\\mathbf{X}^\\top$ of the model activations, and projection of the target neural responses onto its eigenvectors: \n>\n> $$\\mathbf{X}\\mathbf{X}^\\top = \\sum_{i=1}^P \\lambda_i \\mathbf{v}_i \\mathbf{v}_i^\\top,\\quad W_i :=  \\frac{||\\mathbf{Y}^T\\mathbf{v}_i||_2^2}{||\\mathbf{Y}||^2_F}, \\quad <\\mathbf{v}_i, \\mathbf{v}_j> = \\delta\\_{ij} .$$ \n>\n> Here, associated to each mode $i$, $W_i$ denotes the variance of neural responses $\\mathbf{Y}$ in the direction $\\mathbf{v}\\_i$, and $\\lambda_i$ denotes the $i^{th}$ eigenvalue. Then, the predicted generalization error is given by:\n$$E_g(p) = \\sum_{i=1}^P W_i E_i(p), \\quad E_i(p) := \\frac{\\kappa^2}{1-\\gamma} \\frac{1}{(p\\lambda_i + \\kappa)^2},$$\nwhere $\\kappa = \\alpha_{\\text{reg}} + \\kappa \\sum_{i=1}^P \\frac{\\lambda_i}{p\\lambda_i + \\kappa}$ must be solved self-consistently, and $\\gamma = \\sum_{i=1}^P\\frac{p \\lambda_i^2}{(p \\lambda_i + \\kappa)^2}$. Note that the theory depends not only on the model eigenvalues $\\lambda_i$, but also on the model eigenvectors $\\mathbf{v}_i$ along with the responses $\\mathbf{Y}$, which determine how the variance in neural responses distributed among its eigenmodes. \n>\n> Although the equations are complex, the interpretations of $W_i$ and $E_i(p)$ are simple: $W_i$ quantifies the projected variance in neural responses on model eigenvectors (alignment between neural data and model eigenvectors, i.e., *task-model alignment* [1]). Meanwhile, $E_i(p)$ determines the reduction in the error contributed from each $W_i$ when the training set size is $p$, and depends only on the eigenvalues, $\\lambda_i$ (i.e., *spectral bias* [1]).\n> \n> In this work, we combine both and introduce *error modes* $\\tilde W_i(p) := W_i E_i (p)$:\n$$\\tilde W_i(p) := \\frac{\\kappa^2}{1-\\gamma}\\frac{W_i}{(p\\lambda_i + \\kappa)^2}, \\quad E_g = \\sum_i \\tilde W_i(p)$$\nAs shown in RF 1C, it can be seen that $\\tilde W_i$ that are associated with larger eigenvalues $\\lambda_i$ will decay faster with increasing $p$ than those associated with small eigenvalues.\n>\n>While the error modes $\\tilde W_i$ completely characterize the generalization performance of a given model, it is difficult to use them for direct model comparison due to their high dimensionality. Our main goal here is to derive a geometric measure that analytically relates to the generalization error. This is in contrast to previous such measures, such as the effective dimensionality that only depends on model eigenvalues.\n>\n>Here, we define a set of geometric measures that characterize the distribution of a model's $\\tilde W_i$ via the ***error mode geometry*** (RFig 1D). Specifically, we rewrite the generalization error $E_g(p)$ as:\n> $$ R_{em}(p) := \\sqrt{\\sum_i \\tilde{W}_i(p)^2}, \\quad D\\_{em}(p) := \\frac{\\big(\\sum_i \\tilde W_i(p)\\big)^2}{\\sum_i \\tilde W_i(p)^2}, \\quad E_g(p) = R\\_{em}(p) \\sqrt{D\\_{em}(p)}.$$\n>\n> The error mode radius $R_{em}$ denotes the overall size of the error terms, while the error mode dimension $D_{em}$ represents how dispersed the total generalization error is across the different eigenvectors (RFig 1D). Note that, the generalization error $E_g(p)$ above has a degeneracy of error geometries; many different combinations of $R_{em}$ and $D_{em}$ may result in the same $E_g(p)$ (RFig 2).\"\n\n[1] Canatar et. al., Nature communications, 2021."}, {"review_id": "A4sYa9LXO1", "reviewer": "Reviewer_PvxB", "summary": "This study asks a crucial question: What is it about representations in ImageNet-trained neural networks that allow the successful regression of responses in mammalian visual cortex? The authors answer this question (in part) via an ingenious method combining learning theory and empirical analyses. \n\nEven after a decade of study (or more), it is unclear why the responses of neurons in the visual cortex can be so well predicted – better than any other approach – by linearly regressing them from the representations of pretrained DNNs. Here, the authors open the door to a potentially very fruitful method of answering why. Beginning from an analytical expression for the generalization error of ridge regression, the authors notice that it can be decomposed into a product of two terms describing the geometry of the error in the space of examples (the outer product a.k.a. Gram matrix space): one term which carries the interpretation of an average radius, and the other which carries the interpretation of the effective dimensionality of the error. A low generalization error can be achieved by decreasing this radius and dimension. Then, the authors empirically (and thoroughly) analyze how the generalization error decomposes into radius and dimension when predicting V1, V2, V4, and IT responses from a very wide range of pretrained networks, both supervised and unsupervised. The results are striking. For example, networks appear to be surprisingly constrained in their “error mode geometry”, apparently forced to trade between low radius and high dimension. There are many implications in this paper for how we go about understanding neural responses.\n", "questions": "Could a plot be created in which the error mode geometry is shown split by which objective function (supervised, Barlow Twins, etc) as well as each network? Even a null result (indistinguishable) would be interesting.\n\nDoes the theory require an alignment between the eigenmodes of $G_R$ and $G_X$? If so, how is this achieved in theory?", "limitations": "The limitations were thoroughly mentioned and addressed. I greatly appreciated the exploration (in the supplement) of the difference between the generalization error of ridge regression and the Pearson’s R2 measure for partial least squares regression commonly used elsewhere. The additional supplement plots for other small details (outliers, train/test split, $l_\\infty$ adversarial) were also appreciated. In general, I appreciate how the analyses and plots were chosen not to puff up results or push them into any particular narrative, but rather were an honest and complete presentation of the results. ", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "A pleasure to review. This paper tackles a question of high significance using a thorough and deep analysis, one that is both highly original and leverages recent advances. The empirical analyses are complete and leave nearly nothing wanting. ", "weaknesses": "The only weakness I might comment on, in the category of clarity, is the absolute density of results, many of which are not followed up or explored more deeply. To help the reader internalize these interesting results, it would be better to provide a writing structure with more summaries of the outcomes of interest both at the paragraph level and at the paper level. (I personally prefer the CCC style argued for by Mensh and Kording (2017)). At the moment the reader is left to determine for oneself which results to find interesting – meaning that to many quick readers some of these important lessons will be overlooked.\n\nI have a number of requests for expanding the exposition. Apologies, as not all of these will fit in the page limit! \n\nFirst, it would be nice to have a little more discussion and intuition of the meaning of the key error mode geometry terms, radius and dimension. \n\nThen, there were a number of interesting findings that were only given a sentence or not mentioned in the main text. \n - SI.5.4 is very interesting but not much mentioned in the main text. Consider moving to the main text; by way of contrast this might help distinguish your own definition of dimension.\n - Line 443 in the supplement, which mentions that the eigenspectra of the stimulus set in part controls generalization error, which means that it could be a tool for better selecting stimulus sets for neural predictions. What would this look like?\n - It would be good to mention in the main text (briefly) that you analyze both supervised and unsupervised networks, and of many architectures.\n - That “the improved predictivity of trained networks in regions V2, V4, and IT is primarily driven by changes in their intrinsic expressivity as summarized by their eigenspectrum decay, rather than significant changes in their alignment with neural data.” (160-162). This would have large implications for those believing the brain==ImageNet DNNs. Could this be unpacked?\n\nThese are just a few; it would be nice to have summaries of all main takeaways.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank you for reading our paper thoroughly, your positive thoughts and detailed comments/suggestions!\n\n\n## Weaknesses:\n\n1. **Clarity, density of results and writing:**\n\n    We thank the reviewer for the suggestion. To make it easier for the reader to get a high level summary of our results, we’ve added a list of our main contributions to the end of the introduction (see global response).\n    \n\n\n2. **Intuition for Error Geometry: Radius and Dimensionality:**\n\n    We appreciate this suggestion and have simplified and improved the presentation of the theory (§2.2; see global response). Furthermore, we substantially revised Fig. 1 (see RFig 1) and have added a helpful schematic (RFig 2) to make our results on the error mode geometry easier to understand. \n    \n3. **SI.5.4:**\n\n    Thank you for this suggestion, and we are glad that you enjoyed this section. We will consider moving some of this to the main text if space allows. We also added the following sentence to the end of Sec. 3.1: \n\n   “*These results highlight the need to study the spectral properties of both the model and the brain responses as summarized by the $\\tilde W_i$: Simply studying the properties of the eigenspectra of the model is insufficient to characterize neural predictivity (see Sec. SI.5.4 for more details).*”\n    \n4. **Line 443 in SI, tool for selecting stimulus sets:**\n\n    Thank you for this comment. In response to this, we expanded this line as: \n    \n    “*These differences are not due to the recorded neurons, as the model eigenspectra is only a function of the stimulus set, pointing out the importance of considering the stimuli used for the neural predictions. Future work can consider choosing stimulus sets to maximize the difference between model eigenspectra. This would likely lead to greater differences between models in their predictions of neural activity patterns.*”\n\n5. **Mentioning the analysis of various networks and training procedures in the main text:**\n\n    Thank you for the suggestion. We have included more details of the architecture and layer choices on lines 76-80 of the main text, and explicitly noted the use of different training objectives: \n\n    “*The neural responses and model activations were extracted for each stimulus (Fig. 1B), and each layer of each investigated neural network was treated as a separate encoding model. We examined 32 deep neural networks trained on the ImageNet classification task. Model architectures included convolutional neural networks, Vision Transformers, and “biologically inspired” architectures with recurrent connections, and model types spanned a variety of supervised and self-supervised training objectives (see SI.2 for full list of models). We extracted model activations from several stages of each model. For ViTs we extracted activations from all intermediate encoder layers, while in all other models we extracted activations from the ReLU non-linearity after each intermediate convolutional activation. This resulted in a total number of $516$ analyzed model stages. In each case we flattened the model activations for the layer with no subsampling.*”\n    \n6. **Lines (160-162) Implications for brain==ImageNet DNNs:**\n\n    Thank you very much for this suggestion! The comments around lines 160-162 were based primarily on visual inspection of differences in the respective $\\{\\lambda_i\\}$ and $\\{W_i\\}$. However, in response to this comment, we ran an explicit experiment to test this claim (see RFig 4 and Global Response). To our surprise, these results indicate that training primarily drives increased predictivity through changes in alignment with the neural data, rather than changes in the eigenspectra of the model. This highlights that even if model eigenspectra seem quite different (RFig 4D), the spectral differences may not be driving the underlying differences in predictivity. We hope this serves as the groundwork for identifying the contribution of these different properties. \n\n## Questions:\n\n7. **Plotting error mode geometry by objective functions and networks:**\n\n    We have added this as RFig 5 and will include in the SI for V1, V2, V4, and IT. On first pass it seems like there is little difference in the error mode geometry between the supervised and self-supervised networks, which is in line with other works that suggest these networks have similar representations [1-3]. We have also added a plot grouping supervised models based on architecture (RFig 3)\n    \n8. **Alignment between $G_R$ and $G_X$ eigenmodes:**\n\n    Our theory for the generalization error depends on the alignment between the entire matrix of biological neuron responses, which is now denoted $\\mathbf{G}_Y$, and the eigenvectors of the model Gram matrix $\\mathbf{G}_X$, denoted by $\\mathbf{v}_i$. This alignment is quantified by the weights $W_i = ||\\mathbf{Y}\\mathbf{v}_i||^2/||\\mathbf{Y}||^2_F$, which quantifies the fraction of the variance in $\\mathbf{Y}$ that the eigenvector $\\mathbf{v}_i$ accounts for. As such, the eigenmodes of $\\mathbf{G}_Y$ do not *explicitly* enter into the formula for the generalization error but rather $W_i = \\frac{\\mathbf{v}_i^\\top \\mathbf{G}_Y \\mathbf{v}_i}{\\text{Tr}\\, \\mathbf{G}_Y}$, i.e. its projection (see Eq. 2 in main text).\n\n[1] Geirhos et al. 2020 (NeurIPS SVRHM Workshop)\n\n[2] Zhuang et al. 2021 (PNAS)\n\n[3] Konkle & Alvarez 2022 (Nature Communications)"}, {"review_id": "ueYtZPdbB0", "reviewer": "Reviewer_t8VM", "summary": "The paper uses (but doesn't introduce) a theoretical framework that relates generalization error to spectral bias of network activations to geometrically/spectrally analyze what aspects of pretrained deep networks contribute to the predictive performance of neural activity in layers V1, V2, V4 and IT. The authors verify that the theory matches empirical values, and analyze different aspects of the generalization, such as the influence of the training data size and adversarial pretraining. ", "questions": "Q1-Q3: See above. \nQ4: I couldn't find how you extracted the 516 model activations (neither in the paper nor in the supp material). Please briefly describe this procedure in the main paper (i.e. did you use PCA? on how many activations/stimuli? did you only use one layer or many?). \n\nMinor:\n- l 97: \"self-consistently\" did you mean \"self-consistency\"?", "limitations": "See weakness above. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "It is inherently difficult to analyze what aspects of deep network representations contribute to the predictive performance on neural data. The authors make an important contribution that disentangle different factors. This will help to understand why and when deep representations predict neural data well. ", "weaknesses": "My main two criticisms are:\n- The clarity/accessibility could be improved at times. \n- There are a few more references that could be considered. \n- Some of the assumptions/limitations could be discussed more clearly. \n\nWhile I like the general approach, I think it could be clearer at some point. Since you, as the authors, have thought a long time about what the different $W_i$, $R$, $D$ and other values mean, the reader sees that for the first time. So I think it's important to give a good intuition about what they mean. Figure 1 didn't really help in that respect. Particularly, D and E need a bit more effort to be helpful. The reason why this is so crucial is that they mainly discuss the observations in terms of these values. So if the reader doesn't have a clear intuition what they mean, the observations/interpretations become meaningless. So I urge the authors to improve on the introduction part and on the observation part (--> Q1). \n\nI think the following papers would be worth considering to cite (no, I am not S. Cadena ;) ):\n* Santiago A. Cadena, Konstantin F. Willeke, Kelli Restivo, George Denfield, Fabian H. Sinz, Matthias Bethge, Andreas S. Tolias, Alexander S. Ecker Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks\n* Santiago A. Cadena, Fabian H. Sinz, Taliah Muhammad, Emmanouil Froudarakis, Erick Cobos, Edgar Y. Walker, Jake Reimer, Matthias Bethge, Andreas Tolias, Alexander S. Ecker How well do deep neural networks trained on object recognition characterize the mouse visual system?\n* S. A. Cadena, G. H. Denfield, E. Y. Walker, L. A. Gatys, A. S. Tolias, M. Bethge, and A. S. Ecker Deep convolutional models improve predictions of macaque V1 responses to natural images\n\nThe first, because it discusses different prediction performances depending on the pretrained representation. The second because it finds that random representations do almost equally well as pretrained (albeit for mouse). And the third, because it compared task driven and data driven prediction performance for monkey V1. As far as I know, the authors also provide the data. So you could even repeat some of the analysis with more data for V1 (135 texture stimuli are not a lot). \n\nRe Limitations/Assumptions: Some of the assumptions seem unrealistic to me. I would ask the authors to discuss that in more detail. Specifically \n- Q2: You write that model and brain responses need to be deterministic functions of the stimuli. For brain responses this is certainly not true since there is noise and also latent brain states. Can you discuss how the deviation from this assumption affects your results. \n- Q3: I am unsure about the  $M,P\\rightarrow \\infty$ and $M/P \\in O(1)$ assumption. This means that the model features need to go to infinity as the data grows. Doesn't that mean that the model has to be non-parametric, which is not true for deep network. I would ask the authors to discuss that in more detail. \nDiscussing the limitations in two very short paragraphs in section 5 is not enough. Especially, since they have two pages left. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough review. Please see our responses below.\n\n**Q1:** \nThank you for the suggestions regarding the clarity and presentation of the manuscript. We have made a number of changes that we hope address these concerns. These are detailed in the global response. We have also updated Fig. 1 (see RFig 1) and have expanded on Fig. 1E into its own Figure which we hope will guide interpretation of our results (RFig 2).\n\n\n**Re Cadena References:** Thank you for the references. All of these are very related to our work. We have added the following references: \n\n\n1. *Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks:* We reference this work while discussing how training typically benefits models in later visual regions more than earlier ones (see global response). \n2. *How well do deep neural networks trained on object recognition characterize the mouse visual system?:* We now reference this work in the revised section presenting our results regarding trained vs. random networks (see global response). \n\n\nFor the third reference, we analyzed the trained vs. random network error mode geometry using this V1 dataset (focusing on the $P=1250$ natural images, RFig 6), as suggested. This analysis reproduced the overall trends where trained networks do slightly better than random networks at predicting V1 neural data, but not by much. This finding was consistent when using 135 images (matched to the previous V1 dataset) or when using all 1250 images. Additionally, the same trends held when using neural recordings from Cadena et al. texture stimuli rather than the natural images (not shown for space reasons). Even though the overall trends replicated, we found that the generalization error for the Cadena et al. dataset was higher (corresponding to worse predictions) than observed in the Freeman et al. dataset. We are further investigating this difference, but it may stem from a lower number of trial presentations per stimulus in the Cadena et al. dataset (2 or 4 trials per stimulus, in comparison to 20 presentations for the Freeman et al. dataset) leading to larger noise levels. We will include RFig 6 in the SI of the paper.\n\n**Q2:** \nThank you for this comment. Previous works on the spectral theory of ridge regression have investigated the role that noise plays in this setting [1]. In the kernel regression setting, noise leads to an additional term in the generalization error formula which depends on the spectrum of the model. Explicitly modeling neuronal noise is an important direction for future work, and we expanded the following paragraph in the limitations section:\n\n“*...Moreover, this theory assumed that the brain responses are deterministic functions of the stimuli. As shown in [1], this assumption may be relaxed, and future work can examine the role of neuronal noise in these contexts.*” (Quote follows line 259 in the original manuscript.) \n\n**Q3:**\n\nThis limit is often employed to obtain asymptotic expressions whenever both the number of units and the number of data points is large–see for example [1-3]. Our framework does not require that the feature dimension $M$ grows with $P$. Here, we focus on this large $M$ and $P$ limit since the resulting theory accurately predicts the performance of kernel ridge regression when applied to real data. However, since our formula for $E_g$ is only exactly correct in the large $M,P$ limit, we explicitly confirmed that this theory predicted empirical generalization errors in Fig. 2A. \n\n**Q4:**\n    Thank you for highlighting this confusion. We do not apply any dimensionality reduction to the model activations and flatten activations from convolutional layers. As it was shared by other reviewers we have included more details of the architecture, layer choices, and extraction on lines 76-80 of the main text: \n“*The neural responses and model activations were extracted for each stimulus (Fig. 1B), and each layer of each investigated neural network was treated as a separate encoding model. We examined 32 deep neural networks trained on the ImageNet classification task. Model architectures included convolutional neural networks, Vision Transformers, and “biologically inspired” architectures with recurrent connections, and model types spanned a variety of supervised and self-supervised training objectives (see SI.2 for full list of models). We extracted model activations from several stages of each model. For ViTs we extracted activations from all intermediate encoder layers, while in all other models we extracted activations from the ReLU non-linearity after each intermediate convolutional activation. This resulted in a total number of $516$ analyzed model stages. In each case we flattened the model activations for the layer with no subsampling.*”\n    \n**Minor: l 97 \"self-consistently\" did you mean \"self-consistency\"?**:\n\nThis terminology refers to the fact that there is no closed-form expression for calculating $\\kappa$ as a function of the model spectrum [1-3]. Instead, given fixed $\\{\\lambda_i\\}$, we have to numerically solve for the value of $\\kappa$ that satisfies the equation\n$$\n\\kappa - \\alpha_{reg} - \\kappa \\sum_i \\frac{\\lambda_i}{p\\lambda_i + \\kappa}=0.\n$$\n\n[1] Canatar et. al., Nature communications, 2021.\n\n[2] Loureiro et. al., NeurIPS, 2021.\n\n[3] Seung et. al., Phys. Rev. A, 1992."}, {"review_id": "pldLlX9Tiy", "reviewer": "Reviewer_EnLw", "summary": "This submission uses spectral theory and simulation experiments to try to compare and  assess the representations of DNNs vs those of biological neural networks.", "questions": "There may be good ideas behind this paper and in time this direction of research may prove useful. But at this stage things look premature and very foggy. In future versions, it will be essential to improve the clarity and keep very clear distinctions between artificial and biological NNs. A term like \"neural predictivity\" is extremely ambiguous in your context as it could have several meanings.", "limitations": "In addition to the problems mentioned above, there are significant limitations posed by the size of the data sets. The authors are aware of this problem and mention it in Section 5 together with other limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper comes with a fairly extensive review of the literature.\nThe paper uses both theoretical ideas and simulations.", "weaknesses": "The paper is difficult to read and understand. The primary reason is that the problem being tackled is not stated in any clear way.  The boundary between artificial NNs and biological NNs is very fuzzy throughout the paper and it is often unclear if the authors are discussing biological NNs or artificial NNs. The results also are not very clear. This is already obvious from the abstract which does not contain a single quantitative result. Furthermore, there are too many concepts piled up in the paper in a messy way (spectral, geometry, alignment, adversarial vs standard training, error mode weights, etc.).\n\nThe authors cite 43 papers. If the goal is to provide a fairly complete overview of the relevant work, then it seems that some key papers are missing (e.g. Anderson and Zipser in the 1980s, more recent work by DiCarlo et al. )", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for raising these issues and your comments. Please see below for the responses.\n\n1. **Clarity of the results** \n\nWe have made several changes to the paper which we believe substantially address these issues. While we refer the reviewer to the global response for details, we note that we have substantially revised the figures of the paper to give clear schematics of the modeling problem that we consider and describe our results (Rebuttal Figures 1&2), streamlined the presentation of the theory in Sec. 2, and added a concise list of our main contributions to the first section.\n\n\n2. **Additional Citations** \n\nThank you for suggesting that we provide additional context for how our results relate to previous literature. Within this work, we are primarily discussing taking the activations from computational models (pre-trained neural network models or other hand-engineered stimulus computable models) and learning a regression mapping between the activations of the computational model and those measured in the brain for the same set of stimuli. This is directly related to the lines of work from DiCarlo et al. We already cite 6 papers from the group, but have also added the following citation to our section on Adversarial vs. Standard Training: \n\n* Guo, C., Lee, M., Leclerc, G., Dapello, J., Rao, Y., Madry, A., & Dicarlo, J. (2022, June). Adversarially trained neural representations are already as robust as biological neural representations. In International Conference on Machine Learning (pp. 8072-8081). PMLR.\n\n\n\n\nRegarding the Andersen and Zipser work, this is more along the lines of building non-linear models that predict neural responses. There are many recent works along these lines–see e.g., [1,2]. It is beyond the scope of this paper to apply our theory to this framework, but we have added the following to the discussion for future work along these lines: \n\n“Finally, we note that while we focused on DNN models trained on image data sets, a closely related line of work has investigated the properties of DNN models directly trained to predict neural responses  [1-3]. Future work can compare the error mode geometries from end-to-end networks to those obtained from networks trained on image data. Such an analysis could elucidate how these two vastly different training schemes are both able to predict neural data well. Furthermore, the spectral theory of kernel regression we used here has previously been applied to analyze kernels corresponding to different layers of DNNs at different stages of the training process [4]. Future work could take a similar approach to analyzing how representational geometry evolves throughout the training process of end-to-end DNN models by extracting layer wise kernels at different training checkpoints.”\n\n## Questions\n\n1. **Ambiguity in the term, \"neural predictivity\"** \n\nWe believe that our revisions make the manuscript substantially clearer. However, we decided to continue to use the term \"neural predictivity,\" as this is a standard term in this field. See for example [5,6] below.\n\n\n## Limitations \n\n1. **Data set size.** \n\nIn response to reviewer concerns regarding the data set size for V1, we reran these experiments on a significantly larger data set from Ref. [7] with $P=1250$ images (see response to Reviewer t8VM and Rebuttal Figure 6). Note that the V4 and IT data sets are significantly larger and are standard data sets for bench marking models of visual cortex [8]. Additionally, note that the V1/V2 data sets that we used were a small, publicly available, subset of the full data presented in [9]. We are in the process of obtaining the full data. \n\nReferences\n\n[1] Cadena et. al., PLoS Comp. Bio., 2019\n\n[2] Klindt et. al., NeurIPS, 2017\n\n[3] Zipser & Andersen, Nature, 1988\n\n[4] Canatar & Pehlevan, IEEE, 2022 \n\n[5] Zhuang et. al., PNAS, 2021 \n\n[6] Yamins et. al., PNAS, 2014 \n\n[7] Cadena et. al., PLoS Comp. Bio., 2019\n\n[8] Shrimpf et. al., BioRxiv, 2018\n\n[9] Freeman et. al., Nature Neuro, 2013"}]

Author rebuttal:
# Global Response

We thank the reviewers for their helpful comments and for highlighting the significance of our work. As noted by one reviewer, “Even after a decade of study...it is unclear why the responses of neurons in the visual cortex can be so well predicted...by linearly regressing them from the representations of pretrained DNNs.” Our work uses a recent theoretical framework that directly relates the generalization error from regression to the spectral bias of the model activations and the alignment of the neural responses onto the learnable subspace of the model. As noted by another reviewer, "Analyzing the eigenspectrum of a model’s feature space...goes beyond the standard measures of representational similarity analysis or linear regression experiments that one usually comes across for this line of research."

Here, we address a few concerns shared by multiple reviewers. We also upload a pdf with Rebuttal Figures (RFig).

### 1. We clarified key contributions 

We added the following to the end of the introduction: 

1.  We analytically decompose the generalization error of ridge regression from a model to a set of brain data in terms of the model eigenspectra, the alignment between the eigenvectors of the model and brain data, and the training set size.
2. We introduce two geometric measures that summarize these spectral properties and directly relate to the neural predictivity. We show that these measures distinguish between different models with similar predictivities using a wide variety of network architectures, learning rules, and firing rate datasets from visual cortex.
3. Using spectral theory, we demonstrate that for networks effective in predicting neural data, we can ascertain if their superior performance stems from the model's spectra or alignment with the neural data. Our findings indicate:

    (a) Trained neural networks predict neural data better than untrained ones due to better alignment with brain response.
 
    (b) The effect of adversarial training on these geometric measures interacts with both the cortical area and the neural network layer being analyzed.


### 2. We revised the presentation of the theory 

Section 2.2 has been rewritten to improve clarity and provide intuitions. We have also updated Figure 1 and added a helpful schematic–see RFig 1 & 2. We include part of the section here (see 3d9A for the full section): 

"...Although the equations are complex, the interpretations of $W_i$ and $E_i(p)$ are simple: $W_i$ quantifies the projected variance in neural responses on model eigenvectors (alignment between neural data and model eigenvectors, i.e., *task-model alignment* [1]). Meanwhile, $E_i(p)$ determines the reduction in the error contributed from each $W_i$ when the training set size is $p$, and depends only on the eigenvalues, $\lambda_i$ (i.e., *spectral bias* [1]).

In this work, we combine both and introduce *error modes* $\tilde W_i(p) := W_i E_i (p)$:
$$\tilde W_i(p) := \frac{\kappa^2}{1-\gamma} \frac{W_i}{(p\lambda_i + \kappa)^2}, \quad E_g = \sum_i \tilde W_i(p)
$$

(see Eq. SI.5 for details.) As shown in RFig 1C, $\tilde W_i$ associated with large eigenvalues $\lambda_i$ will decay faster with increasing $p$ than those associated with small eigenvalues." 

The generalization performance of a model is fully characterized by its error modes, $\tilde W_i$. However, due to its vector nature, $\tilde W_i$ is not ideally suited for model comparisons. To address this limitation, we condense the overall shape of $\tilde W_i$ into two geometric measures, while preserving their direct relationship to the generalization error.

### 3. New experiment and clarification of training vs. random model results

One common concern was the clarity of results. We rewrote our results sections to  more clearly connect the spectral properties of the model and its alignment to the brain data to the error mode geometry. Additionally, we ran a new experiment to directly test whether training neural networks improves predictivity through improved alignment with neural data as opposed to changes in model eigenspectra. To demonstrate these points, we give the revised section on trained vs. untrained networks below.

"*We analyzed how the error mode geometry for neural predictions differed between trained and randomly initialized DNNs (Fig. 4A). In line with previous results [2,3,4,5], we found that training yielded an improvement in neural predictivity as measured via smaller $E_g(p)$ (RFig 4B). This improvement was most notable in regions V2, V4, and IT, where there was also a characteristic change in the error mode geometry. In these regions, while $R_{em}$ decreased with training, $D_{em}$ surprisingly *increased**.

...*To gain insight into this we further investigated how the eigenspectra ($\lambda_i$) and the alignment coefficients ($W_i$) individually contributed to the observed error mode geometry. These two spectral properties can be varied independently. We performed an experiment on the trained and random ResNet50 model activations where we measured $\lambda_i$ from one model and paired it with the $W_i$ for IT data measured from the other model (RFig 4C). When using the eigenspectra of the random model, the $D_{em}$ was lower than when using the eigenspectra of the trained model. In contrast, when using the $W_i$ terms of the random model, the $R_{em}$ was much larger than that of the trained model. This $R_{em}$ decrease when using the $W_i$ terms from the trained model is the main driver of the improvement in $E_g(p)$ when we use the trained model to predict neural data...*

*...In other words, the eigenvectors of the trained model were overall better aligned with the neural data compared to the random model.*"

[1] Canatar et. al., Nat. Comm, 2021

[2] Schrimpf et. al., BioRxiv, 2018

[3] Saxe et. al., Nat. Reviews Neuro, 2021

[4] Cadena et. al., bioRxiv, 2022

[5] Cadena et. al., NeurIPS, 2019

Meta-review: The submission contributes theory to a paradigm dominated by correlational analysis, aiming to answer the question of what drives correlations between biological and neural network representations. The majority of reviewers praised the submission's rigor and noted the potential for broader impact beyond the neuroscience core at NeurIPS.
Decision: Accept (spotlight)