Title: A Spectral Theory of Neural Prediction and Alignment
Venue: NeurIPS 2023
Accepted: yes
Keywords: computational neuroscience

neural manifolds

neuro-AI

statistical physics

representational geometry
Abstract: The representations of neural networks are often compared to those of biological systems by performing regression between the neural network responses and those measured from biological systems. Many different state-of-the-art deep neural networks yield similar neural predictions, but it remains unclear how to differentiate among models that perform equally well at predicting neural responses. To gain insight into this, we use a recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target. We apply this theory to the case of regression between model activations and neural responses and decompose the neural prediction error in terms of the model eigenspectra, alignment of model eigenvectors and neural responses, and the training set size. Using this decomposition, we introduce geometrical measures to interpret the neural prediction error. We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression. The work demonstrates that carefully decomposing representational metrics can provide interpretability of how models are capturing neural activity and points the way towards improved models of neural activity.

A Spectral Theory of Neural Prediction and Alignment

Abstract

The representations of neural networks are often compared to those of biological systems by performing regression between the neural network responses and those measured from biological systems. Many different state-of-the-art deep neural networks yield similar neural predictions, but it remains unclear how to differentiate among models that perform equally well at predicting neural responses. To gain insight into this, we use a recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target. We apply this theory to the case of regression between model activations and neural responses and decompose the neural prediction error in terms of the model eigenspectra, alignment of model eigenvectors and neural responses, and the training set size. Using this decomposition, we introduce geometrical measures to interpret the neural prediction error. We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression. The work demonstrates that carefully decomposing representational metrics can provide interpretability of how models are capturing neural activity and points the way towards improved models of neural activity.

1 Introduction

Neuroscience has a rich history of using encoding models to understand sensory systems [1, 2, 3]. These models describe a mapping for arbitrary stimuli onto the neural responses measured from the brain and yield predictions for unseen stimuli. Initial encoding models were built from hand-designed parameters to model receptive fields in early sensory areas [4, 5], but it was difficult to extend these hand-engineered models to capture responses in higher-order sensory regions. As deep neural networks (DNNs) showed human-level accuracy on complex tasks, they also emerged as the best models for predicting brain responses, particularly in these later regions [6, 7, 8, 9, 10].

Although early work showed that better task performance resulted in better brain predictions [6], in modern DNNs, many different architectures and training procedures lead to similar predictions of neural responses [11, 12, 13]. This is often true even when model representations are notably different from each other using other metrics of comparison [14, 15]. For instance, previous lines of work have shown clear differences between models from their behavioral responses to corrupted or texturized images [16, 17, 18], from their alignment to human similarity judgments [19], from generating stimuli that are "controversial" between two models [20], and from stimuli that are metameric to models [15] or to human observers [21]. Other work has focused on directly comparing representations [22, 23] or behavioral patterns [24] of one neural network to another, finding that changing the training dataset or objective results in changes to the underlying representations. Given the increasing number of findings that demonstrate large variability among candidate computational models, it is an open

question of why current neural prediction benchmarks are less sensitive to model modification, and how to design future experiments and stimulus sets to better test our models.

Metrics of similarity used to evaluate encoding models of neural responses are often condensed into a single scalar number, such as the variance explained when predicting held-out images [6, 11]. When many models appear similar with such metrics, uncovering the underlying mechanisms driving high predictivity is a challenge [25]. To uncover the key factors behind successful DNN models of neural systems, researchers have proposed various approaches to explore the structural properties of internal representations in DNN models in relation to neural data. One promising approach focuses on analyzing the geometry of neural activities [25]. Several studies have examined the evaluation of geometries and dimensionalities of DNNs [26, 27, 28], highlighting the value of these geometric approaches. They provide mechanistic insights at the population level, which serves as an intermediate level between computational level (e.g., task encoding) and implementation level (e.g., the activities of single neurons or units). Furthermore, the measures derived from representation geometries can be utilized to compare the similarity between models and neural data [7, 25, 29]. Although these approaches have been successful, they do not directly relate the geometry of DNNs to the regression model used to obtain neural predictions, which uniquely captures the utility of a model for applications such as medical interventions [11] and experimental designs [30]. Recent empirical findings by Elmoznino et al. [31] have suggested a correlation between the high-dimensionality of model representations and neural predictivity in certain scenarios, but this work lacks explicit theoretical connections between the spectral properties and predictivity measures. Our work contributes to this line of investigation, by seeking a theoretical explanation and suggesting new geometric measures linked to a model's neural prediction error for future investigations.

To bridge the gap between DNN geometry and encoding models used for neural prediction, we turn to a recent line of work that derived a predictive theory of generalization error using methods from statistical physics [32, 33]. Specifically, two different spectral properties that affect the regression performance were identified: Spectral bias characterizing how fast the eigenvalues of the data Gram matrix decay, and task-model alignment characterizing how well the target aligns with the eigenvectors of this data Gram matrix [33]. While the former describes how large the learnable subspace is, the latter defines how much of the variance of the target is actually learnable. Note that in general, these two properties may vary independently across representations. Hence, the trends among regression scores across DNNs must be carefully studied by taking into account the joint trends of spectral bias and task-model alignment.

In this work, we explore how encoding models of neural responses are influenced by the geometrical properties of the DNN representations and neural activities. Our analyses primarily concern how and why models achieve the predictivities that they do, rather than offering an alternative scalar metric for predictivity. We find that the neural prediction error from ridge regression can be broken down into the radius and dimension of the error mode geometry , characterizing the error along each eigenvector. When investigating models along these error axes, we see a large spread in the error mode geometry, suggesting that the correlation values obtained from regression analyses obscure many geometrical properties of the representations.

Our primary contributions are:

We analytically decompose the neural prediction error of ridge regression from a model to a set of brain data in terms of the model eigenspectra, the alignment between the brain data and the eigenvectors of the model, and the training set size.

We introduce two geometric measures that summarize these spectral properties and directly relate to the neural prediction error. We show that these measures distinguish between different models with similar neural prediction errors using a wide variety of network architectures, learning rules, and firing rate datasets from visual cortex.

Using spectral theory, we demonstrate that for networks effective in predicting neural data, we can ascertain if their superior performance stems from the model's spectra or alignment with the neural data. Our findings indicate that (a) trained neural networks predict neural data better than untrained ones due to better alignment with brain response and (b) adversarial training leads to an improved alignment between the eigenvectors of networks and early visual cortex activity.

Example V1/V2 Stimuli

(FreemanZiemba2013)

Example V4/IT Stimuli

(MajajHong2015)

(P=135)

3200 Stimuli

(P=3200)

A

neuron k

V1

V2

V4

IT

Y:

Neural Response

N

neurons

P

stimuli

DNN Layers

X:

Model Activations

M

units

P

stimuli

Spectral Alignment Theory

Spectral Decomposition of Model Activations (PCA):

Eigenvalues:

Eigenvectors:

Error Modes:

Weight on Eigenvector

i

: model eigenvector index

10 -3

10 -2

0

50

100

Neural prediction

error:

Alignment Coefficients:

Measure of Task-Model Alignment via projection of Y

B

Neural prediction error as a

measure of fit:

p

: training set size

: regression weights

Response predictions:

Neural Prediction with

Regression

C

(d)gx = (d)

Wi

W;(p)

Wi

{

vi}

xxT =x

AivivT

2

W;(p)

2

[YTvi]l2

IYIl'

2.1 Encoding Models of Neural Responses

We investigated DNNs as encoding models of visual areas V1, V2, V4, and IT. The neural datasets were previously collected on 135 texture stimuli for V1 and V2 [34], and on 3200 object stimuli with natural backgrounds for V4 and IT [35] (Fig. 1A). Both datasets were publicly available and obtained from [11]. The neural responses and model activations were extracted for each stimulus (Fig. 1B), and each stage of each investigated neural network was treated as a separate encoding model. We examined 32 visual deep neural networks. Model architectures included convolutional neural networks, vision transformers (ViTs), and 'biologically inspired' architectures with recurrent connections, and model types spanned a variety of supervised and self-supervised training objectives (see Sec. SI.2 for full list of models). We extracted model activations from several stages of each model. For ViTs, we extracted activations from all intermediate encoder model stages, while in all other models we extracted activations from the ReLU non-linearity after each intermediate convolutional activation. This resulted in a total number of 516 analyzed model stages. In each case, we flattened the model activations for the model stage with no subsampling.

2.2 Spectral Theory of Prediction and Alignment

In response to a total of P stimuli, we denote model activations with M features (e.g. responses from one stage of a DNN) by X ∈ R P × M and neural responses with N neurons (e.g. firing rates) by Y ∈ R P × N . Sampling a training set ( X 1 : p , Y 1 : p ) of size p < P , ridge regression solves (Fig. 1B):

We analyze the normalized neural prediction error, E g ( p ) = || ˆ Y ( p ) -Y || 2 F || Y || 2 F , for which we utilize theoretical tools from learning theory [36, 37, 38], random matrix theory [39, 40, 41] and statistical physics [42, 32, 33] to extract geometrical properties of representations based on spectral properties of data. In particular, the theory introduced in [32, 33] relies on the orthogonal mode decomposition (PCA) of the Gram matrix XX ⊤ of the model activations, and projection of the target neural responses onto its eigenvectors:

Here, associated to each mode i , W i denotes the variance of neural responses Y in the direction v i , and λ i denotes the i th eigenvalue. Then, the neural prediction error is given by:

where κ = α reg + κ ∑ P i =1 λ i pλ i + κ must be solved self-consistently, and γ = ∑ P i =1 pλ 2 i ( pλ i + κ ) 2 (see Sec. SI.1 for details) [32, 33]. Note that the theory depends not only on the model eigenvalues λ i , but also on the model eigenvectors v i along with the responses Y , which determine how the variance in neural responses distributed among a model's eigenmodes.

Although the equations are complex, the interpretations of W i and E i ( p ) are simple. W i quantifies the projected variance in neural responses on model eigenvectors (alignment between neural data and model eigenvectors, i.e., task-model alignment [33]). Meanwhile, E i ( p ) , as a function of the training set size p , determines the reduction in the neural prediction error at mode i and depends only on the eigenvalues, λ i (i.e., spectral bias [33]).

In this work, we combine both and introduce error modes W i ( p ) := W i E i ( p ) :

As shown in (Fig. 1C), ˜ W i associated with large eigenvalues λ i will decay faster with increasing p than those associated with small eigenvalues.

The generalization performance of a model is fully characterized by its error modes, ˜ W i . However, due to its vector nature, ˜ W i is not ideally suited for model comparisons. To address this limitation, we condense the overall shape of ˜ W i into two geometric measures, while preserving their direct relationship to the neural prediction error. This is in contrast to previous such measures, such as the effective dimensionality, that only depend on model eigenvalues [31].

Here, we define a set of geometric measures that characterize the distribution of a model's ˜ W i via the error mode geometry (Fig. 2A). Specifically, we rewrite the neural prediction error E g ( p ) as:

The error mode radius R em denotes the overall size of the error terms, while the error mode dimension D em represents how dispersed the total neural prediction error is across the different eigenvectors (Fig. 2A). Note that, the neural prediction error E g ( p ) above has a degeneracy of error geometries; many different combinations of R em and D em may result in the same E g ( p ) (Fig. 2B). Given a fixed neural prediction error, a higher value of R em ( p ) indicates that only a few ˜ W i are driving the error, while a higher D em ( p ) indicates that many different ˜ W i are contributing to E g ( p ) .

3 Results and Observations

We first confirmed that Eq. 3 accurately predicted the neural prediction error for the encoding models considered here. We performed ridge regression from the model activations of each model stage of

0.24

0.28

0.32

0.36

0.40

0.44

v

1

v

2

v

3

Contours

denote

equal

E g

(p)

v

1

v

2

v

3

Model 1 and Model 2 have the same

neural prediction error (i.e. same E

g

(p)),

but differ in error mode geometry

Model 1:

Remaining

E g

(p)

distributed across

many model

eigenvectors

(

v

i

)

:

higher Dem

lower Rem

Model 2:

Remaining

E g

(p)

concentrated on

a small number

of model

eigenvectors

(

v

i

)

:

lower Dem

higher Rem

i

i

v

1

v

2

0.03

0.04

9

10

11

√Dem

Rem

W1

W3

̃

W1

̃

̃

W3

Example visualization

of error mode geometry

W1

W3

̃

W1

̃

̃

W3

1

2

Wi

̃ : error modes

Wi

̃ : error modes

W1

W3

̃

W1

̃

̃

W3

Note: error

modes in large

eigenvalue

directions

decay faster

v

3

A

B

each trained neural network to the neural recordings for each cortical region using a ridge parameter of α reg = 10 -14 , and also calculated the theoretical neural prediction error for each using Eq. 3. Given that Eq. 3 is only accurate in the large P limit in which both the number of training points and the number of test samples is large (see Sec. SI.1 and Fig. SI.4.2), we used a 60/40 train-test split ( p = 0 . 6 P ) in the experiments below. As shown in Fig. 3A, this split yielded near-perfect agreement between the theoretical and empirical neural prediction errors for areas V4 and IT ( R 2 =0.97 and 0.97, respectively), and very strong agreement for areas V1 and V2 ( R 2 =0.9 and 0.9, respectively). We present further data for this agreement in Sec. SI.4. Furthermore, we found that the ordering of models according to their neural prediction error is very similar to the ordering obtained via the partial least squares (PLS) regression method used in Brain-Score [11] (see Sec. SI.3). We maintained these choices for train-test split sizes and α reg for all subsequent experiments.

We visualized the spread of error mode geometries for each brain region across different trained models in Fig. 3B. Each point is colored by the empirical E g ( p ) , while the contour lines show the theoretical E g ( p ) value. Among trained models, there was a range of R em and D em values even when many models have similar E g ( p ) . This demonstrates that our geometrical interpretation of the error mode geometry can give additional insights into how models achieve low neural prediction error. In the following sections, we explore how these geometrical properties of the neural prediction error vary with model stage depth, training scheme, and adversarial robustness.

3.1 Error Mode Geometry Varies across Model Stages

We analyzed the neural prediction error obtained from activations at different model stages. In Fig. 4A, we plot the error mode geometry for all analyzed DNN model stages when predicting brain data from V1 and IT. Each point is color coded based on its relative model depth. Lower neural prediction errors were achieved (i.e. points lie on lower E g contours) for earlier model stages in early visual area V1, and for later model stages in downstream visual area IT. This observation is in agreement with previous findings [6, 11, 43] where artificial neural networks have a similar hierarchy compared to the visual cortex. Qualitatively, the trends across model stages were more prominent than differences observed from comparing model architectures for supervised models (Fig. SI.5.1), or from comparing supervised trained models to self-supervised models of the same architecture (Fig. SI.5.2).

To investigate the source of this geometric difference across model stages, we performed an experiment on each model where we took the eigenspectra λ i measured from the first (Early) or last (Late) model stage and paired this with the alignment coefficients W i measured from the Early or Late model stage (Fig. 4B). These two spectral properties can be varied independently, as W i depends on

0.05

0.10

Rem

8

9

10

11

0.30

0.45

0.60

0.75

0.90

1.05

0.025

0.050

0.075

8

9

10

11

0.30

0.45

0.60

0.75

0.90

0.005

0.010

0.015

40

45

50

0.2

0.3

0.4

0.5

0.6

0.7

0.005

0.010

0.015

40

45

50

0.15

0.30

0.45

0.60

0.75

0.90

0.2

0.4

0.6

0.8

1.0

Eg(p)

0.4

0.6

0.3

0.4

0.5

0.6

0.7

E g

(p): Theoretical

V1

0.4

0.6

0.3

0.4

0.5

0.6

V2

0.2

0.4

0.6

0.2

0.3

0.4

0.5

0.6

V4

0.2

0.4

0.6

0.8

0.2

0.4

0.6

0.8

IT

E g

(p): Empirical

E g

(p): Empirical

E g

(p): Empirical

E g

(p): Empirical

r

2

= 0.90

r

2

= 0.97

r

2

= 0.97

r

2

= 0.90

V1

V2

V4

IT

A

B

Rem

Rem

Rem

the model eigenvectors but not on the model eigenvalues. We then measured the neural prediction error that would be obtained from such a model. This experiment allows us to isolate whether the observed differences in neural prediction error are primarily due to the λ i or W i terms. In the V1 data, there was little difference in the error mode geometry between the location of the W i terms, however using early model stage λ i corresponds to lower neural prediction error (summarized in Fig. 4C). In contrast, for the IT data using the late stage W i terms corresponded lower E g ( p ) mainly through a reduction of the error mode R em (i.e. many error modes are better predicted). The spectral properties are shown in full for the early and late stages of ResNet50 as measured on the V1 and IT datasets (Fig. 4D). These results highlight the need to study the spectral properties of the model together with the brain responses as summarized by ˜ W i : Simply studying the properties of the eigenspectra of the model is insufficient to characterize neural prediction error (see SI.5.5 for more details).

3.2 Trained vs. Untrained

We analyzed how the error mode geometry for neural predictions differed between trained and randomly initialized DNNs (Fig. 5A). In line with previous results [11, 14, 44], we found that training yielded improved neural predictions as measured via smaller E g ( p ) (Fig. 5B). This improvement was most notable in regions V2, V4, and IT, where there was also a characteristic change in the error mode geometry. In these regions, while R em decreased with training, D em surprisingly increased. However, the decrease in R em was large enough to compensate for the higher D em , leading to an overall reduction in E g ( p ) . In V1, we observed only a small difference in neural prediction error between trained and untrained models, similar to previous results for early sensory regions [45, 14]. As shown in Fig. SI.5.7, we found qualitatively similar results in a different V1 dataset with a larger number of tested stimuli [46].

What differed between the trained and random model representations that led to differences in error mode geometry? To gain insight into this, we investigated how the eigenspectra ( λ i ) and the alignment coefficients ( W i ) individually contributed to the observed error mode geometry in visual area IT. We performed an experiment on the trained and random ResNet50 model activations where

0.03

0.04

0.05

9

10

11

0.24

0.30

0.36

0.42

0.48

0.54

0.005

0.010

40.0

42.5

45.0

47.5

50.0

0.18

0.24

0.30

0.36

0.42

0.48

0.54

Earliest

Latest

Stage Depth

(Normalized)

B

C

λ

i

(eigenspectra)

Wi

(alignment)

Early

Late

Early Late

V1

IT

A

R em

R em

0.04

0.06

0.08

8

9

10

11

V1: Early vs. Late

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.005

0.010

0.015

40

45

50

IT: Early vs. Late

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Early λ, E arly W

i

Early λ, L ate W

i

Late λ, E arly W

i

Late λ, L ate W

i

0.25

0.45

0.65

Eg(p)

V1

Early λ, E arly W

i

Early λ, L ate W

i

Late λ, E arly W

i

Late λ, L ate W

i

IT

R em

R em

0

50

100

0

1000 20003000

Early (First Stage)

Late (Last Stage)

V1

ResNet50

IT

ResNet50

Weight on Eigenvector

i

: model eigenvector index

10 -6

10 -4

10 -2

10 -6

10 -4

10 -2

D

√Dem

√Dem

Figure 4: Model stage depth and error mode geometry. (A) Each point represents the error mode geometry obtained from a specific model stage of a trained neural network, where the color of the value represents the depth of the model stage. The color is normalized for each model to the [0 , 1] range where the earlier and later model stages have lighter and darker colors, respectively. R em and √ D em values are obtained from theoretical values, and contour lines correspond to theoretical values of the neural prediction error as in Fig. 2. (B,C) Predicted E g ( p ) that would be obtained when using the eigenspectra ( λ i ) from the first (Early) or last (Late) stage of each of the 32 analyzed models, paired with the alignment coefficients ( W i ) terms from the Early or Late stage of each model. Full error mode geometry is given in (B) where each point corresponds to a single model with the chosen W i and λ i terms, and (C) shows box plots summarizing the E g ( p ) from each comparison. In V1, using the λ i from the early stage of the model resulted in better neural prediction error and a lower R em but higher √ D em , regardless of the W i terms used. In IT, using the alignment terms W i from the late model stage resulted in a smaller neural prediction error and lower R em , with little change in √ D em . (D) Full spectra for λ i , W i and ˜ W i from Early and Late model stages of ResNet50 on the V1 and IT datasets.

3.3 Adversarial vs. Standard Training

Recent work has suggested that adversarially trained networks have representations that are more like those of biological systems [47, 48, 15]. We analyzed the neural prediction error and error mode geometry of adversarially trained ("Robust") neural network models [49, 50], and their nonadversarially trained counterparts. The error mode geometry for an ℓ 2 ( ϵ = 3) ResNet50 is given in Fig. 6A, and similar results are shown for an ℓ ∞ ( ϵ = 4) ResNet50 in Fig. SI.5.8.

The effect of adversarial training on error mode geometry varied across model stages and cortical regions. In the V1 dataset, lower neural prediction error was observed for early stages of robust

0.04

0.06

R em

9

10

11

em

V1

0.24

0.32

0.40

0.48

0.56

0.64

0.04

0.06

9

10

11

V2

0.24

0.32

0.40

0.48

0.56

0.64

0.01

0.02

40

45

50

V4

0.15

0.30

0.45

0.60

0.75

0.90

0.01

0.02

40

45

50

IT

0.15

0.30

0.45

0.60

0.75

0.90

Trained

Random

Stage Depth

(Normalized)

Early

Late

A

R em

R em

R em

Trained

0.0

0.2

0.4

0.6

0.8

1.0

E g

(p)

V1

Random

V2

V4

IT

B

Trained

Random

Trained

Random

Trained

Random

IT: ResNet50

R em

√D em

λ

i

(eigenspectra)

Wi

(alignment)

trained

random

trained

random

Stage Depth

early

late

C

0.006

0.008

0.010

44

46

48

0.20

0.25

0.30

0.35

0.40

0.45

0.50

D

0

1000 2000 3000

Trained ResNet50

(layer4.2.relu)

0

1000 2000 3000

Weight on Eigenvector

10 -7

10 -5

10 -3

10 -1

λ

i

Wi

̃

Wi

Random ResNet50

(layer4.2.relu)

i

: model eigenvector index

models (Fig. 6B), in line with previous work [48]. This corresponded to a decrease in R em with little change in D em , suggesting that the decrease in neural prediction error was shared relatively equally across all error modes. In contrast, at late model stages, D em was smaller for the robust models compared to the standard models, but there was no difference in neural prediction error between the robust and standard networks. In regions V2, V4, and IT, we observed that there was little difference in neural prediction error between the standard and robust models, but that error mode geometry in robust models was associated with a lower D em and a higher R em . This suggests that the difference between robust and standard models in these regions was due to a small number of error modes having higher ˜ W i in the robust models. This did not lead to better neural prediction error for regions V2, V4, and IT, but nevertheless, the error mode geometry revealed that the data was predicted in different ways.

To better understand the differences between robust and standard networks in V1, we ran a series of experiments testing the effects of adversarial training on model eigenspectra and alignment with neural data. For each model stage of the ResNet50 architecture, we paired the eigenspectra ( λ i ) of the standard or robust network with the alignment coefficients ( W i ) of the standard or robust network (Fig. 6C). We then calculated the error mode geometry and neural prediction error for all four possible pairings. This approach allowed us to examine how adversarial training-related changes in the spectrum and alignment coefficients affected the error mode geometry for each model stage.

Wefound that the alignment coefficients of the robust models consistently yielded lower V1 prediction error (Fig. 6D). In early model stages, using the W i from the robust models led to a lower neural

0.03

0.04

9

10

11

V1

0.24

0.28

0.32

0.36

0.40

0.44

0.03

0.04

9

10

11

V2

0.24

0.28

0.32

0.36

0.40

0.44

0.005

0.010

42

44

46

48

50

V4

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.005

0.010

42

44

46

48

50

IT

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Standard

Robust

Stage Depth

(Normalized)

Early

Late

A

C

D

R em

R em

R em

R em

0.03

0.05

Late Stages

0.24

0.32

0.40

0.48

0.56

0.64

0.03

0.05

9

10

11

Early Stages

0.24

0.32

0.40

0.48

0.56

0.64

V1: Standard vs. Robust

Rem

Rem

λ

i

(eigenspectra)

Wi

(alignment)

Standard

Robust

Standard

Robust

early

late

√Dem

0.24

0.28

0.32

0.36

0.40

E g

(p)

Empirical

Theory

V1: E

g

(p)

Stage Depth

(Early -> Late)

B

Robust Model

Late Stages

Early Stages

E g

(p)

V1: Standard vs. Robust

0.3

0.4

0.5

Standard λ, Standard W

Standard λ, R obust W

Robust λ, Standard W

Robust λ, R obust W

Standard λ, Standard W

Standard λ, R obust W

Robust λ, Standard W

Robust λ, R obust W

prediction error via reduced R em , and the choice of λ i had a negligible effect on the prediction error for these stages. In the later model stages, the W i from the robust models also achieved a lower E g , but for these stages, the choice of λ i had a non-negligible impact on the resulting prediction error. Overall, these results suggest that adversarial training leads to an improved alignment between the eigenvectors of the model and early visual cortex activity.

4 Discussion

Many metrics have been proposed to compare model representations to those observed in biological systems [51, 52, 22, 23, 11]. However, it is typically unclear which aspects of the representational geometry lead to a particular similarity score. In this work, we applied the theory of spectral bias and task-model alignment for kernel regression [33] to the generalization error for encoding models of neural activity. This provides a theoretical link between a model's ability to predict neural responses and the eigenspectra and eigenvectors of model representations. We introduced geometric measures R em and D em of the error modes to summarize how the spectral properties relate to neural prediction error, and showed that models with similar errors may have quite different geometrical properties. We applied this theory to investigate the roles of stage depth, learned representations, and adversarial training in neural prediction error throughout the ventral visual stream.

Dimensionality of neural and model representations

Recent work has investigated spectral properties of neural activity in biological [53, 54] and artificial [31, 28] systems. One empirical study suggested that high-dimensional model representations have

lower neural prediction error, where effective dimensionality was defined as the participation ratio of the eigenvalues alone [31]. However, we empirically found that this effective dimensionality of representations was not well correlated with prediction error for our datasets (Sec. SI.5.5). The lack of a consistent relationship between effective dimensionality and neural prediction error was expected based on our theory, as the neural prediction error depends both on the eigenvalues and eigenvectors of a model. These quantities co-vary from model to model, and thus model comparisons should include information from both. In this work, we demonstrated how our theory can be used to examine whether differences in observed prediction error are primarily due to the eigenvalues or alignment coefficients (which are dependent only on the eigenvectors), and found that in many instances, the alignment coefficients were the main driver of lower prediction error.

Dependence of Neural Prediction Error on Training Set Size

Our spectral theory of neural prediction characterizes the relationship between the training set size, the generalization error, and the spectral properties of the model and brain responses. This highlights that model ranking based on neural prediction error may be sensitive to the train/test split ratio. In Eq. 3, we see that, for small p , the error modes corresponding to small λ i generally remain close to their initial value ( ˜ W i ( p ) ≈ ˜ W i (0) ), and hence their contribution to neural prediction error does not decay (Eq. 4). On the other hand, at large sample size p , these small λ i error modes change from their initial values and therefore may change the ranking of the models. These observations regarding the role of model eigenspectra and dataset size may be used in the future to design more comprehensive benchmarks for evaluating models.

Limitations

Although our empirical analysis follows what we expect from theory and shows a diversity of error mode geometries, the datasets that we used (particularly for V1 and V2) have a limited number of samples. In some cases, such as the large p limit, theoretical results for the neural prediction error may not match empirical results. Additionally, we only investigated one dataset for V2, V4, and IT, and only two datasets for V1. Future work may reveal different trends when using different datasets. Moreover, this theory assumes that the brain responses are deterministic functions of the stimuli. As shown in [33], this assumption may be relaxed, and future work can examine the role of neuronal noise in these contexts.

Future work

The results in this paper demonstrate that our theory can characterize the complex interplay between dataset size, representational geometry, and neural prediction error. It provides the basis for numerous future directions relating spectral properties of the model and data to measures of brain-model similarity. Insights from our theory may suggest ways that we can build improved computational models of neural activity.

While we focused on DNN models pre-trained on image datasets, a closely related line of work has investigated the properties of DNN models directly trained to predict neural responses [46, 55, 56]. Future work could compare the error mode geometries from end-to-end networks to those obtained from networks pre-trained on image data. Such an analysis could elucidate how these two vastly different training schemes are both able to predict neural data well. Additionally, our experiments focused on the case of the same small regularization coefficient for all regression mappings, but future work could explore the case of optimal regularization. The spectral theory can also be applied to other types of brain-model comparisons, such as Partial Least Squares or Canonical Correlation Analysis.

Although we were able to differentiate many error mode geometries based on R em and √ D em , we initially expected to see many more values of ( R em , √ D em ) for each fixed E g ( p ) . The investigated vision models seem to be constrained to a particular region in the ( R em , √ D em ) -space (Fig. 3). Future work may shed light on whether this is due to implicit biases in model architectures or due to a fundamental coupling of ( R em and √ D em ) due to constraints on the E i ( p ) term.

There are many architectural differences between the tested DNNs and biological systems. Future work may investigate how biologically plausible constraints [9] (e.g., stochasticity [48, 57], divisive normalization [58], power-law spectra [59]) and diverse training objectives [44, 13]) lead to improved neural prediction error through changes in error mode geometry.

Acknowledgments and Disclosure of Funding

We would like to thank Nga Yu Lo, Eero Simoncelli, N Apurva Ratan Murty, Josh McDermott and Greta Tuckute for helpful discussions, and Jim DiCarlo for comments on an early version of this work. We would also like to thank the reviewers for their helpful comments, which strengthened the paper. This work was funded by the Center for Computational Neuroscience at the Flatiron Institute of the Simons Foundation and the Klingenstein-Simons Award to S.C. All experiments were performed on the Flatiron Institute high-performance computing cluster.

References

Matteo Carandini, Jonathan B Demb, Valerio Mante, David J Tolhurst, Yang Dan, Bruno A Olshausen, Jack L Gallant, and Nicole C Rust. Do we know what the early visual system does? Journal of Neuroscience , 25(46):10577-10597, 2005.

Marcel AJ van Gerven. A primer on encoding models in sensory neuroscience. Journal of Mathematical Psychology , 76:172-183, 2017.

Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage , 56(2):400-410, 2011.

Christina Enroth-Cugell and John G Robson. The contrast sensitivity of retinal ganglion cells of the cat. The Journal of physiology , 187(3):517-552, 1966.

Edward H Adelson and James R Bergen. Spatiotemporal energy models for the perception of motion. Josa a , 2(2):284-299, 1985.

Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences , 111(23):8619-8624, 2014.

Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology , 10(11):e1003915, 2014.

Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haignere, and Josh H McDermott. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron , 98(3):630-644, 2018.

Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience , 22(11):1761-1770, 2019.

Grace W Lindsay. Convolutional neural networks as a model of the visual system: Past, present, and future. Journal of cognitive neuroscience , 33(10):2017-2031, 2021.

Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv , page 407007, 2018.

Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit hierarchical region correspondence. bioRxiv , pages 2022-09, 2022.

Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRxiv , pages 2022-03, 2022.

Andrew Saxe, Stephanie Nelli, and Christopher Summerfield. If deep learning is the answer, what is the question? Nature Reviews Neuroscience , 22(1):55-67, January 2021.

Jenelle Feather, Guillaume Leclerc, Aleksander Madry, and Josh H McDermott. Model metamers reveal divergent invariances between biological and artificial neural networks. Nature Neuroscience , pages 1-18, 2023.

Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between

human and machine vision. Advances in Neural Information Processing Systems , 34:2388523899, 2021.

Katherine Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems , 33:19000-19015, 2020.

Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems , 31, 2018.

Lukas Muttenthaler, Lorenz Linhardt, Jonas Dippel, Robert A Vandermeulen, Katherine Hermann, Andrew K Lampinen, and Simon Kornblith. Improving neural network representations using human similarity judgments. arXiv preprint arXiv:2306.04507 , 2023.

Tal Golan, Prashant C Raju, and Nikolaus Kriegeskorte. Controversial stimuli: Pitting neural networks against each other as models of human cognition. Proceedings of the National Academy of Sciences , 117(47):29330-29337, 2020.

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems , 30, 2017.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning , pages 3519-3529. PMLR, 2019.

Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems , 33:9995-10006, 2020.

SueYeon Chung and LF Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology , 70:137-144, 2021.

Uri Cohen, SueYeon Chung, Daniel D. Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature Communications , 11(1):746, Feb 2020.

Mehrdad Jazayeri and Srdjan Ostojic. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Current opinion in neurobiology , 70:113-120, 2021.

Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences , 119(43):e2200800119, 2022.

Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science , 364(6439):eaav9436, 2019.

Eric Elmoznino and Michael F Bonner. High-performing neural network models of visual cortex benefit from high latent dimensionality. bioRxiv , pages 2022-07, 2022.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning , pages 1024-1034. PMLR, 2020.

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications , 12(1):2914, 2021.

Jeremy Freeman, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature Neuroscience , 16(7):974-981, Jul 2013.

Najib J Majaj, Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience , 35(39):13402-13418, 2015.

Bernhard Schölkopf, Alexander J Smola, Francis Bach, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press, 2002.

Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning , volume 2:3. MIT press Cambridge, MA, 2006.

Thomas Hofmann, Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine learning. The Annals of Statistics , 36(3):1171, 2008.

Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices , volume 20. Springer, 2010.

Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Implicit regularization of random feature models. In International Conference on Machine Learning , pages 4631-4640. PMLR, 2020.

Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems , 34:18137-18151, 2021.

Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A , 45(8):6056, 1992.

Soma Nonaka, Kei Majima, Shuntaro C Aoki, and Yukiyasu Kamitani. Brain hierarchy score: Which deep neural networks are hierarchically brain-like? IScience , 24(9):103013, 2021.

Santiago A Cadena, Konstantin F Willeke, Kelli Restivo, George Denfield, Fabian H Sinz, Matthias Bethge, Andreas S Tolias, and Alexander S Ecker. Diverse task-driven modeling of macaque v4 reveals functional specialization towards semantic tasks. bioRxiv , pages 2022-05, 2022.

Santiago A Cadena, Fabian H Sinz, Taliah Muhammad, Emmanouil Froudarakis, Erick Cobos, Edgar Y Walker, Jake Reimer, Matthias Bethge, Andreas Tolias, and Alexander S Ecker. How well do deep neural networks trained on object recognition characterize the mouse visual system? In Real Neurons {\ & } Hidden Units: Future directions at the intersection of neuroscience and artificial intelligence@ NeurIPS 2019 , 2019.

Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS computational biology , 15(4):e1006897, 2019.

Chong Guo, Michael Lee, Guillaume Leclerc, Joel Dapello, Yug Rao, Aleksander Madry, and James Dicarlo. Adversarially trained neural representations are already as robust as biological neural representations. In International Conference on Machine Learning , pages 8072-8081. PMLR, 2022.

Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. Advances in Neural Information Processing Systems , 33:13073-13087, 2020.

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In Proceedings of the 34th International Conference on Neural Information Processing Systems , pages 3533-3545, 2020.

Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019.

Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems , 14, 2001.

Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience , page 4, 2008.

Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometry of population responses in visual cortex. Nature , 571(7765):361365, 2019.

Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality, dynamics and measurement. BioRxiv , page 214262, 2017.

David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating 'what' and 'where'. Advances in neural information processing systems , 30, 2017.

David Zipser and Richard A Andersen. A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons. Nature , 331(6158):679-684, 1988.

Joel Dapello, Jenelle Feather, Hang Le, Tiago Marques, David Cox, Josh McDermott, James J DiCarlo, and SueYeon Chung. Neural population geometry reveals the role of stochasticity in robust perception. Advances in Neural Information Processing Systems , 34:15595-15607, 2021.

Michelle Miller, SueYeon Chung, and Kenneth D Miller. Divisive feature normalization improves image recognition performance in alexnet. In International Conference on Learning Representations , 2022.

Josue Nassar, Piotr Sokol, SueYeon Chung, Kenneth D Harris, and Il Memming Park. On 1/n neural representation and robustness. Advances in Neural Information Processing Systems , 33:6211-6222, 2020.

Harold Widom. Lectures on integral equations . Courier Dover Publications, 2016.

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Out-of-distribution generalization in kernel regression. Advances in Neural Information Processing Systems , 34:12600-12612, 2021.

James B. Simon, Madeline Dickens, Dhruva Karkada, and Michael R. DeWeese. The eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks, 2022.

Noureddine El Karoui. The spectrum of kernel random matrices. The Annals of Statistics , 38(1):1 - 50, 2010.

Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity with statistical testing. arXiv preprint arXiv:2108.01661 , 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09 , 2009.

Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997 , 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11976-11986, 2022.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel LK Yamins, and James J DiCarlo. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv , page 408385, 2018.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Selfsupervised learning via redundancy reduction. In International Conference on Machine Learning , pages 12310-12320. PMLR, 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Computer vision with a single (robust) classifier. In ArXiv preprint arXiv:1906.09453 , 2019.

Christopher Williams and Carl Rasmussen. Gaussian processes for regression. Advances in neural information processing systems , 8, 1995.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'Ridgeless' regression can generalize. The Annals of Statistics , 48(3):1329 - 1347, 2020.

Dmitry Kobak, Jonathan Lomond, and Benoit Sanchez. The optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization. The Journal of Machine Learning Research , 21(1):6863-6878, 2020.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020.

Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. On the surprising similarities between supervised and selfsupervised models. In NeurIPS 2020 Workshop SVRHM , 2020.

Talia Konkle and George A Alvarez. A self-supervised domain-general learning framework for human ventral stream representation. Nature communications , 13(1):491, 2022.

Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences , 118(3):e2014196118, 2021.

Broader Impacts

Our work provides a novel approach towards comparing the fidelity of neural encoding model predictions, and may provide insight that leads to improved models of neural activity. Such models are applicable for neuroengineering purposes, such as retinal prosthetics or cochlear implants. However, training the deep learning models tested in the paper consumes a large amount of energy, contributing to carbon emissions. Although we used pre-trained neural networks here, future work may need to train networks with biases specifically for low neural prediction error.

SI.1 Spectral Theory of Generalization Error in Regression

The spectral theory of generalization error for regression, as developed in [32, 33], relates the spectral properties of the data and target to the generalization error in an analytical way. Here, we summarize the key results from these works and describe how we use the results in our context.

For a generic regression task, we begin by considering a model x ( s ) : R D → R M , which maps D -dimensional stimuli s to a set of M -dimensional features x . Additionally, we assume that the neural responses are generated by a ground truth function y ( s ) : R D → R N , mapping the stimuli to the activations of a set of N biological features y (e.g., firing rates). Finally, we assume that the stimuli are drawn from a generic distribution s ∼ ν ( s ) .

In regression, the goal is to estimate the ground truth y ( s ) from a training set of size p and test it on all possible stimuli as drawn from ν ( s ) . Given a training dataset with p -samples D tr = { s µ , y µ } p µ =1 where s µ ∼ ν ( s ) and y µ = y ( s µ ) , ridge regression returns an estimator ˆ y ( s ) as:

where α reg is the ridge parameter introduced to regularize the solution. Here, the regression weights and estimator depend on the particular choice of the training set D tr of size p . We define the normalized generalization error as:

which also depends on the training set D tr. Explicit solution of the objective in Eq. SI.1 gives:

where X 1 : p ∈ R p × M is the matrix of the model responses to the stimuli in D tr , Y 1 : p ∈ R p × N is the matrix of the neural responses in D tr, and I denotes the p × p identity matrix. Defining the kernel function K ( s , s ′ ) = x ( s ) ⊤ x ( s ′ ) , the predictor can be written as:

Note that every time a new training set is drawn from the stimuli distribution ν ( s ) , the generalization error may change. We are interested in the dataset averaged generalization error which only depends on the size of the training set p rather than the particular instances of each D tr. This is defined as:

where K is p × p matrix with elements K µν ≡ K ( s µ , s ν ) and k ( s ) is a p -dimensional vector with elements k ( s ) µ ≡ K ( s , s µ ) . Hence, the generalization error depends on the kernel K ( s , s ′ ) , a particular draw of a training set D tr of size p , and the ridge parameter α reg .

where the average is taken over all possible choices for D tr of size p . Now, this quantity only depends on the training set size p , input distribution ν ( s ) , model kernel K ( s , s ′ ) and the ground truth function y ( s ) . This quantity can also be thought of as the cross-validated generalization error, where the generalization errors for each regressor corresponding to a particular instance of D tr are averaged.

The theory of generalization error in [32, 33] finds that this dataset averaged E g ( p ) depends on the spectral properties of the kernel. Mercer's theorem [60] suggests that the kernel can be decomposed into orthogonal modes as K ( s , s ′ ) = ∑ i λ i v i ( s ) v i ( s ′ ) where the eigenvalues λ i and orthonormal eigenfunctions v i ( s ) are obtained by solving the equation

Here we emphasize that the resulting eigenvalues and eigenfunctions strongly depend on the distribution of the stimuli ν ( s ) . For example, for the same model kernel, the spectrum might look completely different depending on the stimuli being textures or natural scenes.

The dependence of the neural response function y ( s ) to the generalization error comes from the mode decomposition of y ( s ) on to the eigenfunction v i ( s ) :

where w i is the projection of the neural response function y ( s ) onto the i th eigenfunction v i ( s ) .

As described in [33, 32, 61, 62], using techniques from statistical mechanics, the generalization error is given by the self-consistent equations

where κ must be solved self-consistently, and we defined the alignment coefficients W i

∥ ∥ ∥ ∥ which give the ratio of the variance of neural responses along mode i to its total variance. In most cases, it is not possible to find an analytical solution to κ , but it is straightforward to solve it numerically.

In this paper, since our data are finite, we take the distribution ν ( s ) to be the empirical distribution on the full dataset D full = { s µ , y µ } P µ =1 of P data points:

which is an approximation to the true stimuli distribution where P →∞ .

In this case, the functions x ( s ) and y ( s ) are fully described by the data matrices, X and Y , and the generalization error in Eq. SI.1.2 becomes the neural prediction error,

Since the full distribution of stimuli are characterized by the empirical distribution ν ( s ) , training sets are generated by randomly sampling p samples from D full. Furthermore, the eigenvalue problem in Eq. SI.1.6 reduces to a PCA problem for the empirical kernel K = XX ⊤

and the projections of neural responses Y on the eigenvectors v i reduces to

We note that this method may misestimate the neural prediction error, since the true eigenvalues and eigenvectors of the kernel K ( s , s ′ ) are approximated by the finite empirical kernel matrix K for P samples [63].

With the spectral data { λ i } and { w i } , we obtain the dataset averaged neural prediction error E g ( p ) (Eq. 3) as:

˜ To understand the behavior of the functions E i ( p ) , we look at their limits. When p = 0 , E i (0) = 1 for all modes and no variance is explained. As p → ∞ , we either learn the variance W i entirely ( E i ( p →∞ ) → 0 ) if the corresponding λ i > 0 , or we do not learn it at all if λ i = 0 ( E i ( p →∞ ) → 1 ). It can be further shown that, for all p > 0 , E i ( p ) < E j ( p ) if λ i > λ j [33], which implies that the percent reduction in ˜ W i ( p ) is greater for modes with larger eigenvalues. Several remarks are in order:

Here, we normalized the weights so that ∑ i W i = 1 and E g ( p = 0) = 1 . The functions E i ( p ) depend only on the eigenvalues and training set size p . Since each W i is the normalized variance of Y along the eigenvector v i , W i E i ( p ) gives the contribution to the generalization error along mode i when the training set has size p . In our work, we defined these as error modes W i ( p ) = W i E i ( p ) .

The ordered eigenvalues { λ i } characterize the spectral bias of the model: for a fixed training set size p , the reduction in E i ( p ) is larger for modes associated with large eigenvalues λ i than for modes associated with small λ i . Therefore, a sharply decaying eigenvalue spectrum implies that there is a bias toward certain modes being learned at higher fidelity than others, while a flat eigenvalue spectrum implies that learning will be equally distributed across all modes.

W i characterizes how much of the variance of the neural responses Y lie in the i th mode. These alignment coefficients { W i } characterize task-model alignment ; because of spectral bias , if most of the neural response variance is associated with eigenvectors with high λ i terms, the neural prediction error will decrease faster compared to when most neural response variance is on eigenvectors with low λ i . For example, if W i are largest for the modes for which λ i are smallest, we would expect a large generalization error when the training set size is small.

We consider both { λ i } and { W i } as spectral properties of the model for the measured data (e.g. see Fig. 1C), and generalization error is determined by the interplay between these two spectral properties, together with the training set size p . For a fixed set of neural responses D = { s µ , y µ } P µ =1 , different DNN models have different { λ i } and { W i } since for each model both eigenvalues (spectral bias) and the eigenvectors (task-model alignment) of the model activations change in a complicated way. Since we must specify both of these aspects of the model, comparison of model activations to neural responses should require at least two measures to be complete. Drawing from this argument, we may also conclude that reducing model/data comparisons to commonly used single scalar measures might be misleading as they may miss some of the factors contributing to the predictivity the others capture [64].

In this work, we considered two geometrized measures R em and √ D em that directly relate to the neural prediction error. The two measures jointly summarize whether the distribution of the error modes, ˜ W i , are relatively spread out (higher D em , lower R em ) or tightly concentrated (lower D em , higher R em ) at a fixed generalization error. We found that these terms were a convenient way to summarize spectral properties of the generalization error, but in principle, other measures could be defined.

SI.2 Experimental Details

Code for analyzing the empirical and theoretical neural prediction error can be found at https://github.com/chung-neuroai-lab/SNAP.

We conducted our experiments with various vision models trained with different learning paradigms and training schemes. In total, we used 32 models and picked several model stage activations from each. Except for vision transformers (ViT), chosen model stages were intermediate convolutional model stage activations after passed through a ReLU non-linearity. For ViT, we choose intermediate encoder model stages. For untrained models, models were randomly initialized for each iteration

for each experiment. All experiments were conducted on single Nvidia H100 or A100 GPUs with 80GB RAM using PyTorch 2 [65] and JAX 0.4 [66] libraries. Additional details on model stage selection and model architectures can be found in the associated code. In the remainder of this section, we report the models used in this study and explain step-by-step how we obtain the model stage activations to perform regression.

SI.2.1 Supervised Models

We investigated supervised vision models trained on the ImageNet 1000-way categorization task [67] with publicly available architectures and checkpoints obtained from the PyTorch library [65]. We included Alexnet [68], several ResNets [69], WideResNets [70], DenseNets [71], ConvNeXts [72], EfficientNets [73], MobileNet [74] and various vision transforms (ViT) [75].

We also investigated ANN models that were "biologically inspired" with a CorNet architecture [76]. For these models, the pre-trained weights are pulled using the library https://github.com/ dicarlolab/CORnet .

In Table SI.2.1, we list all standard supervised models used in this study, together with their Top-1 ImageNet accuracies.

Table 1: List of supervised models studied in this work together with their Top-1 ImageNet accuracies on the validation set.

SI.2.2 Self-Supervised and Adversarially Trained Models

We also considered models trained either with self-supervised training (SSL) [77, 78, 79] or adversarial training (referred to as "robust" models) [80, 81]. All analyzed models were ResNet50 or ResNet101 architectures.

For SSL models, we considered Barlowtwins [77], SimCLR [78] and MoCo [79] models whose pre-trained weights were pulled from https://github.com/facebookresearch/ barlowtwins , https://github.com/facebookresearch/vissl and https://github.com/ facebookresearch/moco-v3 , respectively.

For the robust models, we obtained the pre-trained weights from https://github.com/MadryLab/ robustness [50].

In Table SI.2.2, we list all SSL and robust models used in this study, together with their Top-1 ImageNet accuracies.

SI.2.3 Obtaining Model Stage Activations

We obtained activations from each model stage on the stimuli from neural datasets. We employ the image preprocessing steps that were used during model training. For all models, this preprocessing is resizing image stimuli to (224 , 224) pixels and normalizing them with the mean and variance of the ImageNet dataset, which are µ = (0 . 485 , 0 . 456 , 0 . 406) and σ = (0 . 229 , 0 . 224 , 0 . 225) , respectively. All activations were flattened before regression.

SI.2.4 Ridge Regression Experiment

After obtaining the model stage activations, we perform ridge regression by computing the predictions as:

where I p denotes the p × p identity matrix and α reg denotes the ridge parameter. In this notation, Y ∗ ( p ) is the P × N matrix of the predictions for the entire neural response set that is inferred using only a subset of p samples as the training set. Then, X 1 : p denotes the p × M matrix of the training set that is the model activations for randomly sampled p stimuli and Y 1 : p denotes the p × N matrix of the corresponding neural activations.

Note that the least-squares regression is ill-defined when the number of data points are more than the number of regression variables (i.e. p > M ) since the matrix X 1: p X ⊤ 1: p + α reg I is singular for α reg = 0 . In order to account for this issue, one can use the push-through identity to obtain:

We repeat the same computation over 5 random choices of a training set and report neural prediction error Eq. SI.1.11 averaged over these trials. Furthermore, in Sec. SI.4, we show the agreement between empirically calculated and theoretically predicted errors.

where now I M is the M × M identity matrix. The form in Eq. SI.2.2 is the conventional form of ridge regression, while the equation Eq. SI.2.1 is the form obtained via the so-called kernel trick [82, 36]. However, we only use the form in Eq. SI.2.1, since the model activations often are high-dimensional ( ∼ 1 M ) and hence M ≫ P , where P ∼ 10 3 or less.

In our experiments, we choose α reg = 10 -14 for numerical stability, which roughly corresponds to double-precision. Note that this choice of ridge parameter is to get close to least-squares regression, in which case α reg is identically zero. We considered a very small ridge parameter, since even for ridgeless regression, strong spectral biases of overparameterized models avoid overfitting and effectively behave as if there is a non-zero ridge parameter [83, 32, 40, 84, 33, 85].

Here we chose a single regularization parameter for all experiments, however the choice of regularization parameters and the relation to the error modes is critical for fully characterizing the responses of encoding models. Future work could address this by choosing ridge parameters separately for different models and model stages, or allowing flexibility for different regularizations for individual unit responses (rather than one regularization for each neural region).

SI.2.5 Computing Theoretical Error Mode Geometry

After obtaining the model stage activations, we compute the eigenvalues and eigenvectors of each activation using the formula in Eq. SI.1.6. In terms of empirical Gram matrices, this corresponds to

performing Principal Component Analysis (PCA) on the Gram matrix. We project the neural response data onto the obtained eigenvectors and calculate W i . Then, theoretical values of √ D em and R em are calculated using the formula given in Eq. 5. For each p , the value of κ is solved self-consistently using the JAX auto-differentiation library [66] and SciPy optimizers [86].

SI.3 Relation to Brain-Score Neural Prediction Metrics

In this work, we considered ridge regression to compare model activations to neural responses. While there are many other choices for model comparisons such as canonical correlation analysis (CCA), partial least squares regression (PLS) and kernel-target alignment (KTA) [22, 23, 51], we chose to focus on ridge regression as it is theoretically well studied in terms of spectral properties [32, 33].

To demonstrate that the neural prediction error we obtained from ridge regression is comparable to other widely-used metrics, we directly compared our predicted values to those obtained from PLS using the Brain-Score [11] framework. Brain-Score uses PLS regression and reports the noisecorrected Pearson r 2 score on test data to rank models. Note that PLS regression finds a new basis that maximizes the correlations between two sets. Then the regression is done on new variables by performing dimensionality reduction on both sets using the common basis. If one uses all the components in this new basis, then PLS is equivalent to least squares regression.

To calculate the PLS Pearson r 2 value, we first extracted model activations as described in SI.2.3. We then fit PLS regression using the standard 25 latent components [11] and calculated the Pearson correlations between PLS-based predictions and actual neural responses on held-out data. We took the median correlation across all neurons and divided it by the median noise ceiling, which was calculated using split-half reliability across trials featuring identical stimuli. Train and test data points were randomly assigned, and the final Pearson r 2 values were calculated by averaging over 10 random splits. This procedure is described in more detail in [11] and corresponds to the default analysis for public Brain-Score neural prediction benchmarks in the following regions and datasets:

V1: movshon.FreemanZiemba2013public.V1-pls

V2: movshon.FreemanZiemba2013public.V2-pls

V4: dicarlo.MajajHong2015public.V4-pls

IT: dicarlo.MajajHong2015public.IT-pls

0.0

0.1

0.2

0.3

PLS Scores BrainScore

Noise corrected Pearson r 2

0.4

0.6

Eg(p = 0.6P)

V1

0.499073

- 0.781028

0.0

0.2

0.4

PLS Scores BrainScore

Noise corrected Pearson r 2

0.3

0.4

0.5

0.6

Eg(p = 0.6P)

V2

0.515764

- 0.668697

0.2

0.4

0.6

PLS Scores BrainScore

Noise corrected Pearson r 2

0.2

0.3

0.4

0.5

0.6

Eg(p = 0.6P)

V4

0.748517

- 0.896917

0.2

0.4

0.6

PLS Scores BrainScore

Noise corrected Pearson r 2

0.2

0.4

0.6

0.8

Eg(p = 0.6P)

IT

0.699778

-0.847348

r

2 =

ρ=

r

2 =

ρ=

r

2 =

ρ=

r

2 =

ρ=

In Fig. SI.3.1, we compare the Brain-Score Pearson r 2 from each model stage to the empirical ridge-regression E g ( p ) . The Brain-Score metric and neural prediction error from ridge regression are moderately correlated, despite the fact that we are (1) comparing values between ridge regression and PLS regression and (2) comparing a Pearson r 2 metric to E g ( p ) values, which are not directly comparable (Pearson r 2 measures the angle between predictions and true labels, while neural prediction error measures the distance between the two).

To directly address the difference in metrics, in Fig. SI.3.2, we compare the Brain-Score PLS regression Pearson r 2 on the test data to ridge regression Pearson r 2 on the test data (using the same regression weights as were used to obtain the E g ( p ) values above. When using the same metrics on the two different regression methods, there is a strong correlation for all regions. Combined, these results show that by using the ridge regression formulation we generally preserve the ordering of the commonly used Brain-Score metric.

0.0

0.1

0.2

0.3

PLS Scores BrainScore

Noise corrected Pearson r 2

0.05

0.10

0.15

0.20

2

V1

r

2 = 0.938693

ρ=0.925348

0.0

0.2

0.4

PLS Scores BrainScore

Noise corrected Pearson r 2

0.00

0.05

0.10

0.15

0.20

V2

0.911480

0.853052

0.2

0.4

0.6

PLS Scores BrainScore

Noise corrected Pearson r 2

0.1

0.2

0.3

0.4

V4

0.884471

0.917968

0.2

0.4

0.6

PLS Scores BrainScore

Noise corrected Pearson r 2

0.1

0.2

0.3

IT

0.855466

0.900416

2

2

2

r

2 =

ρ=

r

2 =

ρ=

r

2 =

ρ=

SI.4 Empirical vs. Theoretical Neural Prediction Error

Complimentary to the discussion around Fig. 3, we confirmed that the theoretical predictions (see Sec. SI.1) obtained from the spectral properties of data match with the empirical regression discussed in Sec. SI.2 for various values of p . In Fig. SI.4.1, we plot the learning curves for selected model stages of a pre-trained ResNet18 architecture for all brain regions. We see that the theory matches the empirical values for all regions.

Note that particular choices of train/test splits may alter the ranking of which encoding models have the lowest neural prediction error (for instance, see layer4.1.relu changing from the lowest neural prediction error at small p to the highest error at large p in Fig SI.4.1 IT). Additionally, as we mentioned in the main text, increasing the train/test split yielded worse agreement between empirical and theoretical neural prediction errors. The main reason for this disagreement is that the theoretical values depend highly on the estimation of the true eigenvalue distribution of the empirical Gram matrix of data. However, as discussed in Sec. SI.1, the tail of the true eigenvalues are often underestimated by the empirical eigenvalues and that creates a discrepancy between observed E g ( p ) and theoretically predicted one [63]. This discrepancy can be clearly seen in Fig. SI.4.2, where we plot the empirical neural prediction errors against the theoretically predicted ones. As the amount of training data increases (corresponding to a decrease in amount of test data), the correlation between the two quantities decreases monotonically.

10 1

10 2

10 3

p

0.00

0.25

0.50

0.75

1.00

Eg(p)

IT - resnet18

layer1.1.relu

layer2.1.relu

layer3.1.relu

layer4.1.relu

10 1

10 2

p

0.00

0.25

0.50

0.75

1.00

Eg(p)

V1 - resnet18

layer1.1.relu

layer2.1.relu

layer3.1.relu

layer4.1.relu

10 1

10 2

p

0.00

0.25

0.50

0.75

1.00

Eg(p)

V2 - resnet18

layer1.1.relu

layer2.1.relu

layer3.1.relu

layer4.1.relu

10 1

10 2

10 3

p

0.00

0.25

0.50

0.75

1.00

Eg(p)

V4 - resnet18

layer1.1.relu

layer2.1.relu

layer3.1.relu

layer4.1.relu

0.4

0.6

Eg(p), em pirical

0.3

0.4

0.5

0.6

0.7

Eg(p), theoretical

Tr/Test Split: 0.6

V1 (R 2 = 0.899)

0.4

0.6

Eg(p), em pirical

0.3

0.4

0.5

0.6

Eg(p), theoretical

Tr/Test Split: 0.6

V2 (R 2 = 0.897)

0.2

0.4

0.6

Eg(p), em pirical

0.2

0.3

0.4

0.5

0.6

Eg(p), theoretical

Tr/Test Split: 0.6

V4 (R 2 = 0.971)

0.2

0.4

0.6

0.8

Eg(p), em pirical

0.2

0.4

0.6

0.8

Eg(p), theoretical

Tr/Test Split: 0.6

IT (R 2 = 0.973)

0.2

0.4

0.6

Eg(p), em pirical

0.2

0.3

0.4

0.5

0.6

Eg(p), theoretical

Tr/Test Split: 0.7

V1 (R 2 = 0.895)

0.2

0.3

0.4

0.5

Eg(p), em pirical

0.2

0.3

0.4

0.5

Eg(p), theoretical

Tr/Test Split: 0.7

V2 (R 2 = 0.886)

0.2

0.4

Eg(p), em pirical

0.2

0.3

0.4

0.5

Eg(p), theoretical

Tr/Test Split: 0.7

V4 (R 2 = 0.954)

0.2

0.4

0.6

Eg(p), em pirical

0.2

0.3

0.4

0.5

0.6

0.7

Eg(p), theoretical

Tr/Test Split: 0.7

IT (R 2 = 0.959)

0.1

0.2

0.3

0.4

Eg(p), em pirical

0.1

0.2

0.3

0.4

Eg(p), theoretical

Tr/Test Split: 0.8

V1 (R 2 = 0.875)

0.1

0.2

0.3

Eg(p), em pirical

0.10

0.15

0.20

0.25

0.30

0.35

Eg(p), theoretical

Tr/Test Split: 0.8

V2 (R 2 = 0.838)

0.1

0.2

0.3

0.4

Eg(p), em pirical

0.1

0.2

0.3

0.4

Eg(p), theoretical

Tr/Test Split: 0.8

V4 (R 2 = 0.921)

0.2

0.4

Eg(p), em pirical

0.1

0.2

0.3

0.4

0.5

Eg(p), theoretical

Tr/Test Split: 0.8

IT (R 2 = 0.933)

0.1

0.2

Eg(p), em pirical

0.05

0.10

0.15

0.20

Eg(p), theoretical

Tr/Test Split: 0.9

V1 (R 2 = 0.812)

0.05

0.10

0.15

Eg(p), em pirical

0.050

0.075

0.100

0.125

0.150

0.175

Eg(p), theoretical

Tr/Test Split: 0.9

V2 (R 2 = 0.763)

0.1

0.2

Eg(p), em pirical

0.05

0.10

0.15

0.20

0.25

Eg(p), theoretical

Tr/Test Split: 0.9

V4 (R 2 = 0.859)

0.1

0.2

0.3

Eg(p), em pirical

0.05

0.10

0.15

0.20

0.25

0.30

Eg(p), theoretical

Tr/Test Split: 0.9

IT (R 2 = 0.901)

Increasing Train/Test Split

SI.5 Additional Experimental Results

SI.5.1 Geometry grouped by architecture and training types

For visualization purposes, we show the error mode geometries grouped by the architecture (Fig. SI.5.1) and for a subset of models all with ResNet50 backbones but with different training types (Fig. SI.5.2).

0.03

0.04

0.05

9

10

11

V1

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.03

0.04

0.05

9

10

11

V2

0.24

0.30

0.36

0.42

0.48

0.54

0.004

0.008

0.012

40

45

50

V4

0.18

0.24

0.30

0.36

0.42

0.48

0.54

0.004

0.008

0.012

40

45

50

IT

0.18

0.24

0.30

0.36

0.42

0.48

0.54

ResNets

WideResNets

ConvNexts

DenseNets

EfficientNets

MobileNets

CORNets

ViTs

em

R em

R em

R em

R em

0.03

0.04

9

10

11

V1

0.28

0.32

0.36

0.40

0.44

0.48

0.025

0.030

10

11

V2

0.24

0.26

0.28

0.30

0.32

0.34

0.36

0.004

0.006

46

48

50

V4

0.200

0.225

0.250

0.275

0.300

0.325

0.350

0.004

0.006

0.008

46

48

IT

0.20

0.24

0.28

0.32

0.36

0.40

Training Type

(ResNet50)

Supervised

BarlowTwins

MoCo

SimCLR

Stage Depth

(Normalized)

Early

Late

R em

em

R em

R em

R em

SI.5.2 Geometry across model stages

In the main text, we focused on data from regions V1 and IT when analyzing the error mode geometry's dependence on model stage depth. Analysis of regions V2 and V4 is included in Fig. SI.5.3. Additionally, to show the error mode geometries from different model stages in Fig. 4A and Fig. SI.5.3A, we zoomed in on the parts of the space with the most data points and maintained the same axes limits for regions that shared the same stimulus set. In Fig. SI.5.4, we present all model stages, including outliers, colored by model stage depth.

0.03

0.04

0.05

9

10

11

0.24

0.30

0.36

0.42

0.48

0.54

0.005

0.010

40.0

42.5

45.0

47.5

50.0

0.18

0.24

0.30

0.36

0.42

0.48

0.54

Earliest

Latest

Stage Depth

(Normalized)

B

C

λ

i

(eigenspectra)

Wi

(alignment)

Early

Late

Early

Late

V2

V4

A

R em

R em

V2: Early vs. Late

V4: Early vs. Late

0.25

0.45

0.65

Eg(p)

V2

V4

R em

R em

0

50

100

0

1000 20003000

Early (First Stage)

Late (Last Stage)

V2

ResNet50

V4

ResNet50

Weight on Eigenvector

i

: model eigenvector index

10 -6

10 -4

10 -2

10 -6

10 -4

10 -2

D

√Dem

√Dem

Early λ, E arly W

i

Early λ, L ate W

i

Late λ, E arly W

i

Late λ, L ate W

i

Early λ, E arly W

i

Early λ, L ate W

i

Late λ, E arly W

i

Late λ, L ate W

i

0.025

0.050

0.075

0.100

8

9

10

11

0.30

0.45

0.60

0.75

0.90

1.05

0.005

0.010

0.015

40

45

0.16

0.24

0.32

0.40

0.48

0.56

0.64

0.72

0.05

0.10

Rem

8

9

10

11

V1

0.30

0.45

0.60

0.75

0.90

1.05

0.025

0.050

0.075

Rem

8

9

10

11

V2

0.30

0.45

0.60

0.75

0.90

0.005

0.010

0.015

Rem

40

45

50

V4

0.2

0.3

0.4

0.5

0.6

0.7

0.005

0.010

0.015

Rem

40

45

50

IT

0.15

0.30

0.45

0.60

0.75

0.90

0.0

0.2

0.4

0.6

0.8

Stage Depth

(Normalized)

We further elaborate on the spectral properties of different model stages and the corresponding error mode geometry by using the ℓ ∞ trained ResNet50 [50] model as an example. We plot the eigenvalues of the model activations from each model stage for all tested visual regions(Fig. SI.5.5). For V1 and V2, we saw that earlier model stages had better predictions, while for V4 and IT, later model stages performed better.

0

1000

2000

3000

i

10

-

5

10

-

4

10

-

3

10

-

2

λ

i

IT - robust resnet50 linf 8

layer3.5.relu 0.273

layer1.0.relu 0.419

0

1000

2000

3000

i

1

2

3

4

5

6

˜

W

i

×

10

-

4

0

50

100

i

10

-

3

10

-

2

10

-

1

λ

i

V1 - robust resnet50 linf 8

layer2.3.relu 0.267

layer4.2.relu 0.397

0

50

100

i

0

.

5

1

.

0

1

.

5

˜

W

i

×

10

-

2

0

50

100

i

10

-

3

10

-

2

10

-

1

λ

i

V2 - robust resnet50 linf 8

layer3.5.relu 0.277

layer4.2.relu 0.348

0

50

100

i

0

.

5

1

.

0

1

.

5

˜

W

i

×

10

-

2

0

1000

2000

3000

i

10

-

5

10

-

4

10

-

3

10

-

2

λ

i

V4 - robust resnet50 linf 8

layer2.3.relu 0.239

layer4.2.relu 0.360

0

1000

2000

3000

i

1

2

3

4

5

6

˜

W

i

×

10

-

4

The spectral properties of each model stage depend highly on the input distribution. We notice that, under the input distribution of V1/V2 data, the eigenvalues of earlier model stages decayed more slowly than later model stages, while under the input distribution of V4/IT data this pattern is reversed where later model stages decayed more slowly. These differences are not due to the recorded neurons, as the eigenspectra is only a function of the stimulus set, pointing out the importance of considering the stimuli used for the neural predictions. Future work could consider choosing stimulus sets to maximize the difference between model eigenspectra. This would likely lead to greater differences between models in their predictions of neural activity patterns.

SI.5.3 Geometry for Trained vs. Untrained Models

In the main text analysis of trained vs. untrained models, we zoomed in on the parts of the space with the most data points and maintained the same axes limits for regions that shared the same stimulus set (Fig. 5)A. In Fig. SI.5.6, we show all the points of the trained and untrained networks, including the outliers. We also include an analysis of the error mode geometry for trained vs. untrained networks when predicting an additional V1 dataset [46] Fig. SI.5.7.

0.05

0.10

Rem

8

9

10

11

V1

0.30

0.45

0.60

0.75

0.90

1.05

0.025

0.050

0.075

Rem

8

9

10

11

V2

0.30

0.45

0.60

0.75

0.01

0.02

Rem

35

40

45

50

V4

0.2

0.4

0.6

0.8

1.0

1.2

0.025

0.050

Rem

35

40

45

50

IT

0.5

1.0

1.5

2.0

2.5

3.0

Trained

Random

Stage Depth

(Normalized)

Early

Late

.

0.05

0.10

8

9

10

11

0.30

0.45

0.60

0.75

0.90

1.05

0.050

0.075

0.100

8.0

8.5

9.0

9.5

10.0

10.5

11.0

V1 (Cadena et al.): 135 Stim

0.30

0.45

0.60

0.75

0.90

1.05

0.02

0.04

24

26

28

30

32

0.30

0.45

0.60

0.75

0.90

1.05

1.20

Trained

Random

Stage Depth

(Normalized)

Early

Late

R em

em

V1 (Cadena et al.): 1250 Stim

R em

R em

A

B

C

em

em

Random

Trained

0.0

0.2

0.4

0.6

0.8

1.0

Random

Trained

0.0

0.2

0.4

0.6

0.8

1.0

Random

Trained

0.0

0.2

0.4

0.6

0.8

1.0

Eg(p)

Eg(p)

Eg(p)

SI.5.4 Geometry for Robust vs. Standard Models

In this section, we consider the model stage-wise changes in error mode geometry for an ℓ ∞ adversarially trained network, as opposed to the ℓ 2 trained network presented in Fig. 6A. In Fig. SI.5.8, we find that the error mode geometry is very similar for this network, suggesting a consistent effect of adversarial training on the error mode geometry.

0.03

0.04

Rem

9

10

11

√Dem

V1

0.25

0.30

0.35

0.40

0.45

0.03

0.04

Rem

9

10

11

V2

0.25

0.30

0.35

0.40

0.45

0.004 0.006 0.008 0.010

Rem

42

44

46

48

50

V4

0.20

0.25

0.30

0.35

0.40

0.45

0.004 0.006 0.008 0.010

Rem

42

44

46

48

50

IT

0.20

0.25

0.30

0.35

0.40

0.45

Standard

Linf

Robust

Stage Depth

(Normalized)

Early

Late

For visualization purposes, in Fig. SI.5.9, we plot the eigenspectra and error mode weights for a standard and adversarially trained model.

SI.5.5 Effective Dimensionality and Neural Prediction Error

Previous work has suggested that the effective dimensionality of a model's responses to visual stimuli correlates strongly with its neural predictivity [31]. Here, we did not find a relationship between neural prediction error and the dimensionality of the model activations. We measured the effective dimensionality ( ED ) of the model stage responses for the stimuli in each neural experiment, extracted as described in SI.2.3, with the participation ratio of the eigenvalues of G X :

We tested the correlation between ED and both Brain-Score performance and neural prediction error. In Fig. SI.5.10, we show that neural prediction error does not correlate well with model dimensionality across any of the datasets considered here. This is to be expected based on the theory presented - although the spectral properties of the model activations to the stimulus set is one of the factors

̃

̃

̃

̃

0

25

50

75

100

125

i

10 -4

10 -3

10 -2

10 -1

10 0

λi

robust_resnet50_l2_3

resnet50

0

25

50

75

100

125

i

10 -7

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

Wi

V2

0

1000

2000

3000

i

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

λi

robust_resnet50_l2_3

resnet50

0

1000

2000

3000

i

10 -8

10 -7

10 -6

10 -5

10 -4

10 -3

Wi

IT

0

25

50

75

100

125

i

10 -4

10 -3

10 -2

10 -1

10 0

λi

robust_resnet50_l2_3

resnet50

0

25

50

75

100

125

i

10 -7

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

Wi

V1

0

1000

2000

3000

i

10 -6

10 -5

10 -4

10 -3

10 -2

10 -1

λi

robust_resnet50_l2_3

resnet50

0

1000

2000

3000

i

10 -8

10 -7

10 -6

10 -5

10 -4

10 -3

Wi

V4

leading to the observed values of ˜ W i , the direction of the model eigenvectors is also important for determining the error mode geometry. These properties are captured by our proposed geometric quantities R em and D em , which are directly related to the neural prediction error. Additionally, in the main text, we investigated the neural prediction error in cases where we modified the eigenvalues or alignment coefficients (which are dependent on the model eigenvectors).

0

50

100

Effective Dimensionality

0.25

0.30

0.35

0.40

0.45

V1

Eg(p=0.6P)

r

2

: 0 .0 2 3 4 5 5

ρ

: -0 .3 4 9 5 7 5

0

50

100

Effective Dimensionality

0.25

0.30

0.35

0.40

0.45

V2

r

2

: 0 .2 1 3 0 3 4

ρ

: 0.209479

0

500

1000

Effective Dimensionality

0.3

0.4

0.5

IT

r

2

: 0 .0 0 0 0 9 6

ρ

: -0 .1 6 4 1 2 8

0

500

1000

Effective Dimensionality

0.20

0.25

0.30

0.35

0.40

V4

r

2

: 0 .0 4 4 3 8 1

ρ

: -0 .4 1 7 1 6 1

We note that the authors of [31] applied either an average-pooling or principle-components-based dimensionality reduction method to model activations before carrying out their regression analyses. To ensure that we were able to reproduce the results of [31], we performed similar pooling on our model activations, which is complete average-pooling on activations from each convolutional filter. When average-pooling to our model activations, we found that setting α reg = 10 -14 as above yielded several outliers with unusually high E g values. We therefore set α reg to the value that yielded the lowest possible E g for each set of model activations. When pooling model activations, we found a relatively strong relationship between prediction error and model dimensionality in IT similar to that reported in [31] (Fig. SI.5.11). We further replicated the results in [31] by directly comparing the effective dimensionalities obtained from the spatially averaged activations to the Brain-Score noise-corrected PLS regression values (Fig. SI.5.12). Taken together, these results indicate that pooling discards important variability in the model responses, and that when we use the full model activations the model dimensionality does not correlate well with neural prediction error.

5

10

15

0.4

0.6

0.8

Eg(p = 0.6P)

V1

r 2 : 0.211417

ρ: -0.578106

5

10

15

0.3

0.4

0.5

0.6

0.7

0.8

V2

r 2 : 0.169005

ρ: -0.535735

0

25

50

75

0.5

0.6

0.7

0.8

0.9

V4

r 2 : 0.506195

ρ: -0.841676

0

25

50

75

0.4

0.6

0.8

IT

r 2 : 0.482205

ρ: -0.860328

Effective Dimensionality

(Averaged Unit Responses)

Effective Dimensionality

(Averaged Unit Responses)

Effective Dimensionality

(Averaged Unit Responses)

Effective Dimensionality

(Averaged Unit Responses)

5

10

15

Effective Dimensionality

(Averaged Unit Responses)

0.0

0.1

0.2

0.3

PLS Scores BrainScore

Pearson r 2  (Noise corrected)

V1

r 2 : 0.013537

ρ: -0.084701

5

10

15

0.0

0.1

0.2

0.3

0.4

V2

r 2 : 0.133616

ρ: 0.477885

0

50

0.3

0.4

0.5

0.6

V4

r 2 : 0.300794

ρ: -0.470344

0

50

0.2

0.3

0.4

0.5

0.6

IT

r 2 : 0.318168

ρ: 0.794994

Effective Dimensionality

(Averaged Unit Responses)

Effective Dimensionality

(Averaged Unit Responses)

Effective Dimensionality

(Averaged Unit Responses)

Peer reviews:
comments: SUMMARY:
This submission uses spectral theory and simulation experiments to try to compare and  assess the representations of DNNs vs those of biological neural networks.

STRENGTHS:
The paper comes with a fairly extensive review of the literature.
The paper uses both theoretical ideas and simulations.

WEAKNESSES:
The paper is difficult to read and understand. The primary reason is that the problem being tackled is not stated in any clear way.  The boundary between artificial NNs and biological NNs is very fuzzy throughout the paper and it is often unclear if the authors are discussing biological NNs or artificial NNs. The results also are not very clear. This is already obvious from the abstract which does not contain a single quantitative result. Furthermore, there are too many concepts piled up in the paper in a messy way (spectral, geometry, alignment, adversarial vs standard training, error mode weights, etc.).

The authors cite 43 papers. If the goal is to provide a fairly complete overview of the relevant work, then it seems that some key papers are missing (e.g. Anderson and Zipser in the 1980s, more recent work by DiCarlo et al. )

QUESTIONS:
There may be good ideas behind this paper and in time this direction of research may prove useful. But at this stage things look premature and very foggy. In future versions, it will be essential to improve the clarity and keep very clear distinctions between artificial and biological NNs. A term like "neural predictivity" is extremely ambiguous in your context as it could have several meanings.

LIMITATIONS:
In addition to the problems mentioned above, there are significant limitations posed by the size of the data sets. The authors are aware of this problem and mention it in Section 5 together with other limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper uses (but doesn't introduce) a theoretical framework that relates generalization error to spectral bias of network activations to geometrically/spectrally analyze what aspects of pretrained deep networks contribute to the predictive performance of neural activity in layers V1, V2, V4 and IT. The authors verify that the theory matches empirical values, and analyze different aspects of the generalization, such as the influence of the training data size and adversarial pretraining.\n\nSTRENGTHS:\nIt is inherently difficult to analyze what aspects of deep network representations contribute to the predictive performance on neural data. The authors make an important contribution that disentangle different factors. This will help to understand why and when deep representations predict neural data well.\n\nWEAKNESSES:\nMy main two criticisms are:\n- The clarity/accessibility could be improved at times. \n- There are a few more references that could be considered. \n- Some of the assumptions/limitations could be discussed more clearly. \n\nWhile I like the general approach, I think it could be clearer at some point. Since you, as the authors, have thought a long time about what the different $W_i$, $R$, $D$ and other values mean, the reader sees that for the first time. So I think it's important to give a good intuition about what they mean. Figure 1 didn't really help in that respect. Particularly, D and E need a bit more effort to be helpful. The reason why this is so crucial is that they mainly discuss the observations in terms of these values. So if the reader doesn't have a clear intuition what they mean, the observations/interpretations become meaningless. So I urge the authors to improve on the introduction part and on the observation part (--> Q1). \n\nI think the following papers would be worth considering to cite (no, I am not S. Cadena ;) ):\n* Santiago A. Cadena, Konstantin F. Willeke, Kelli Restivo, George Denfield, Fabian H. Sinz, Matthias Bethge, Andreas S. Tolias, Alexander S. Ecker Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks\n* Santiago A. Cadena, Fabian H. Sinz, Taliah Muhammad, Emmanouil Froudarakis, Erick Cobos, Edgar Y. Walker, Jake Reimer, Matthias Bethge, Andreas Tolias, Alexander S. Ecker How well do deep neural networks trained on object recognition characterize the mouse visual system?\n* S. A. Cadena, G. H. Denfield, E. Y. Walker, L. A. Gatys, A. S. Tolias, M. Bethge, and A. S. Ecker Deep convolutional models improve predictions of macaque V1 responses to natural images\n\nThe first, because it discusses different prediction performances depending on the pretrained representation. The second because it finds that random representations do almost equally well as pretrained (albeit for mouse). And the third, because it compared task driven and data driven prediction performance for monkey V1. As far as I know, the authors also provide the data. So you could even repeat some of the analysis with more data for V1 (135 texture stimuli are not a lot). \n\nRe Limitations/Assumptions: Some of the assumptions seem unrealistic to me. I would ask the authors to discuss that in more detail. Specifically \n- Q2: You write that model and brain responses need to be deterministic functions of the stimuli. For brain responses this is certainly not true since there is noise and also latent brain states. Can you discuss how the deviation from this assumption affects your results. \n- Q3: I am unsure about the  $M,P\\rightarrow \\infty$ and $M/P \\in O(1)$ assumption. This means that the model features need to go to infinity as the data grows. Doesn't that mean that the model has to be non-parametric, which is not true for deep network. I would ask the authors to discuss that in more detail. \nDiscussing the limitations in two very short paragraphs in section 5 is not enough. Especially, since they have two pages left.\n\nQUESTIONS:\nQ1-Q3: See above. \nQ4: I couldn't find how you extracted the 516 model activations (neither in the paper nor in the supp material). Please briefly describe this procedure in the main paper (i.e. did you use PCA? on how many activations/stimuli? did you only use one layer or many?). \n\nMinor:\n- l 97: \"self-consistently\" did you mean \"self-consistency\"?\n\nLIMITATIONS:\nSee weakness above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study asks a crucial question: What is it about representations in ImageNet-trained neural networks that allow the successful regression of responses in mammalian visual cortex? The authors answer this question (in part) via an ingenious method combining learning theory and empirical analyses. \n\nEven after a decade of study (or more), it is unclear why the responses of neurons in the visual cortex can be so well predicted – better than any other approach – by linearly regressing them from the representations of pretrained DNNs. Here, the authors open the door to a potentially very fruitful method of answering why. Beginning from an analytical expression for the generalization error of ridge regression, the authors notice that it can be decomposed into a product of two terms describing the geometry of the error in the space of examples (the outer product a.k.a. Gram matrix space): one term which carries the interpretation of an average radius, and the other which carries the interpretation of the effective dimensionality of the error. A low generalization error can be achieved by decreasing this radius and dimension. Then, the authors empirically (and thoroughly) analyze how the generalization error decomposes into radius and dimension when predicting V1, V2, V4, and IT responses from a very wide range of pretrained networks, both supervised and unsupervised. The results are striking. For example, networks appear to be surprisingly constrained in their “error mode geometry”, apparently forced to trade between low radius and high dimension. There are many implications in this paper for how we go about understanding neural responses.\n\nSTRENGTHS:\nA pleasure to review. This paper tackles a question of high significance using a thorough and deep analysis, one that is both highly original and leverages recent advances. The empirical analyses are complete and leave nearly nothing wanting.\n\nWEAKNESSES:\nThe only weakness I might comment on, in the category of clarity, is the absolute density of results, many of which are not followed up or explored more deeply. To help the reader internalize these interesting results, it would be better to provide a writing structure with more summaries of the outcomes of interest both at the paragraph level and at the paper level. (I personally prefer the CCC style argued for by Mensh and Kording (2017)). At the moment the reader is left to determine for oneself which results to find interesting – meaning that to many quick readers some of these important lessons will be overlooked.\n\nI have a number of requests for expanding the exposition. Apologies, as not all of these will fit in the page limit! \n\nFirst, it would be nice to have a little more discussion and intuition of the meaning of the key error mode geometry terms, radius and dimension. \n\nThen, there were a number of interesting findings that were only given a sentence or not mentioned in the main text. \n - SI.5.4 is very interesting but not much mentioned in the main text. Consider moving to the main text; by way of contrast this might help distinguish your own definition of dimension.\n - Line 443 in the supplement, which mentions that the eigenspectra of the stimulus set in part controls generalization error, which means that it could be a tool for better selecting stimulus sets for neural predictions. What would this look like?\n - It would be good to mention in the main text (briefly) that you analyze both supervised and unsupervised networks, and of many architectures.\n - That “the improved predictivity of trained networks in regions V2, V4, and IT is primarily driven by changes in their intrinsic expressivity as summarized by their eigenspectrum decay, rather than significant changes in their alignment with neural data.” (160-162). This would have large implications for those believing the brain==ImageNet DNNs. Could this be unpacked?\n\nThese are just a few; it would be nice to have summaries of all main takeaways.\n\nQUESTIONS:\nCould a plot be created in which the error mode geometry is shown split by which objective function (supervised, Barlow Twins, etc) as well as each network? Even a null result (indistinguishable) would be interesting.\n\nDoes the theory require an alignment between the eigenmodes of $G_R$ and $G_X$? If so, how is this achieved in theory?\n\nLIMITATIONS:\nThe limitations were thoroughly mentioned and addressed. I greatly appreciated the exploration (in the supplement) of the difference between the generalization error of ridge regression and the Pearson’s R2 measure for partial least squares regression commonly used elsewhere. The additional supplement plots for other small details (outliers, train/test split, $l_\\infty$ adversarial) were also appreciated. In general, I appreciate how the analyses and plots were chosen not to puff up results or push them into any particular narrative, but rather were an honest and complete presentation of the results.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious works have demonstrated that many different state-of-the-art deep neural networks (DNNs) perform similarly at neural responses prediction. But a complete understanding of which aspects of these DNNs lead to the similarity in predicting neural responses remains unknown. The authors proposes a spectral theoretical framework to explore how the geometrical properties of DNNs affect the performance of neural responses prediction. More specifically, the authors introduce a generalization error to serve as a measure of fit, which brings a geometrical interpretation of the model activations (representations) in DNNs. This geometrical interpretation could further give additional insights into how DNNs are achieving different performance of neural responses prediction. The authors also design several experiments to investigate the roles of layer depth, dataset size,  and different training approaches of DNNs in neural responses prediction.\n\nSTRENGTHS:\n* The authors establish a novel link betwwen the predictivity of neural responses and the geometry of DNN representations.\n* The authors provide a solid analysis and several comprehensive experiments to support their theoretical framework.\n\nWEAKNESSES:\n* The authors should improve the clarity of their conclusions from their results. Section 3 provides many details about their analysis and results, but it's hard for the readers to extract the essential arguments the authors want to convey. It would be better to summarize your conclusions at the beginning or end of each paragraph.\n*  The authors should improve their exposition of the spectral theoretical framework. I recommend the authors include more background information and provide some examples to show the role of each equation in the relationship between the generalization error and the geometry of model activations.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nI have highlighted weaknesses above. I have nothing further to add here.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors studied linear regression-based DNN encoding models of macaque visual cortical areas by extending a recent generalization error theory. They provided a theoretical link between the predictivity and geometry of representations and showed that models with similar generalization errors may have quite different geometrical properties. They compared a randomly initialized ResNet50 with standard and adversarially trained ResNet50 models. Evaluations and analyses are a bit limited.\n\nSTRENGTHS:\n- Analyzing the eigenspectrum of a model's feature space appears to be an interesting analysis that goes beyond the standard measures of representational similarity analysis or linear regression experiments that one usually comes across for this line of research at the intersection of cognitive neuroscience and machine learning. So, I appreciate that effort.\n\nWEAKNESSES:\n**Major** \n\n- **Tense**: There's a mix between present tense and past perfect in $\\S2$ which I think is not great. Choose one tense and use it consistently. I'd use the present tense to describe the problem settings and methods and either use present tense or past perfect for the results section. Also, use active rather than passive voice which I think is better readable (and more scientific).\n\n- **$\\S3.4$ + main results**: I find it very strange to denote the training set size with small $p$. As I mention below, $p$ is usually used to refer to the number of features in some representation space and not to the size of the (training) data. If I was just reading the title of the subsection, I'd think that the generalization error depends on the size of the feature space (although I would not know on which of the feature spaces). This is not implausible, but very different from what you mean/find. Actually, it's pretty trivial that the generalization error depends on the size of the training data for fitting the regression model. When would the generalization error not depend on the size of the training data? This section is in general a bit confusing. In lines 205-207 you write: \"[...] it may be that trained networks do better than their random counterparts at predicting V1 data for large, but not small, training data set sizes.\" Do you refer to the size of the (training) data with which the regression model is fit? If that's the case, I am a bit puzzled because from Bayesian statistics (and statistics in general) we know that the prior is most important for small data regimes. In the infinite data limit, the prior and, therefore, any regularization parameter stops mattering. A pretrained neural network has a much better initialization compared to a randomly initialized network. It has found a good local minimum over the course of pretraining which gives it a big head start. The pertaining is an implicit regularization for any downstream task. Here, we can treat the regression task as a downstream task. So, the prior should matter most when the training data to fit your regression model is small. You seem to find the opposite, or at least that's what you claim. The only possibility where I can see this being the case is when the training data size for the downstream task (e.g., for performing linear regression) is so small that the parameters of your (regression) model can basically explain no variance at all. But then the findings are pretty useless, aren't they? Actually, I am confident that $N=135$ is too small to find any statistically meaningful effects.\n\n- **Model choice**: It would have been interesting to see at least one, if not more, other architecture(s) being analyzed. I doubt that you can generalize any findings from a single ResNet50 to the whole space of DNNs. Even within the subclass of CNNs, there exist countless models. I find it more interesting to compare CNNs (pick two or three) to ViTs (pick two or three) but comparing a few CNNs (each with a different inductive bias --- e.g., AlexNet, VGG16, ResNet50, ResNext, ConvNeXt, EfficientNetV2), ideally trained with different recipes would have probably been sufficient.\n\n**Minor**\n\n\n- Lines 26-27: \"[...] many different architecture and training procedures lead to similar predictions of neural responses.\" While the former is true and has been demonstrated multiple times in various ways, it has recently been shown in a comprehensive study that both training data and objective function have a major impact on the degree of alignment with human behavior; much more than architecture (see [Muttenthaler, L., Dippel, J., Linhardt, L., Vandermeulen, R. A., & Kornblith, S. (2022)](https://openreview.net/pdf?id=ReDQ1OUQR0X)). A longer while ago it has also been shown that different data augmentation strategies lead to different biases and as such to different degrees of consistency with human errors. See [Hermann, K., Chen, T., & Kornblith, S. (2020)](https://proceedings.neurips.cc/paper_files/paper/2020/file/db5f9f42a7157abe65bb145000b5871a-Paper.pdf); [Hermann, K., & Lampinen, A. (2020)](https://proceedings.neurips.cc/paper/2020/file/71e9c6620d381d60196ebe694840aaaa-Paper.pdf); [Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2018)](https://arxiv.org/pdf/1811.12231.pdf); and [Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., & Brendel, W. (2021)](https://proceedings.neurips.cc/paper_files/paper/2021/file/c8877cff22082a16395a57e97232bb6f-Paper.pdf).\n\n\n- In line 58 is a small grammar mistake: \"[...] characterizing how fast the eigenvalues of data Gram matrix fall [...]\". There is an article missing before **the** data Gram matrix. \n\n\n- **Notation**: The notation in $\\S2.2$ is weird. Why is the matrix of neural activations $\\mathbf{R} \\in \\mathbb{R}^{P \\times N}$? First, $\\bf{R}$ can sometimes have a special meaning. So, I'd be careful with that. Second, although not bolded, you use $R$ for the generalization error terms. Third, generally $N$ is used to denote the number of stimuli/examples in your training data and not to refer to the number of features.  My suggestion is to denote the matrix of model features as $\\mathbf{X} \\in \\mathbb{R}^{N \\times D}$ and the matrix of neural activations $\\mathbf{Y} \\in \\mathbb{R}^{N \\times P}$. As such, the regression problem $\\mathbf{Y} = \\mathbf{X}\\mathbf{A}^{\\top} + b$ becomes more intuitive. I'd replace $\\mathbf{R}^{\\ast}(\\alpha_{reg}, p)$ with the full equation. $\\mathbf{R}^{\\ast}(\\alpha_{reg}, p)$ is not easy to parse. That being said, note that this is just my personal taste which is why it's a minor weakness. There may be people who don't care about that. I think that maths should be easy to follow, even if a reader skims the paper. I dislike decorative maths. If it's just decorative, it can as well be omitted.\n\n\n- In Equation 2 is an \"is equivalent to\" symbol. It is probably not clear to everyone why the two terms are equivalent but not equal. I'd like to see a (short) derivation of it. You can derive it in the Supplementary Material. Similarly, there are two \"$\\equiv$\" symbols in Equation 5. Why did you choose \"$\\equiv$\"? How are the terms \"equivalent\" but not \"equal\"? Do you perhaps mean \"$\\coloneqq$\" for an assignment of variables? That would make more sense to me. \"$\\equiv$\" seems to be an abuse of notation but maybe I am missing something here.\n\nQUESTIONS:\n- How do you explain a decrease in $R_{em}$ and an increase in $D_{em}$ that is caused by (standard) training?\n- Why did you choose a (wide) ResNet50 and not any other model? Is there a particular reason for that choice?\n- Why do you think \"[...] that trained networks do better than their random counterparts at predicting V1 data for large, but not small, training data set sizes\"? Is it possible that the size of the (training) data to fit the regression model is just too small to explain any variance at all or do you have another explanation that rules out that possibility? I think that $N = 135$ is too small to see any effects at all which is why I think there's no notable difference between pretrained and randomly initialized networks in your analyses. However, $N = 3600$ is large enough to observe statistically significant effects. So, I doubt that you can draw any conclusions at all about the smaller of the two datasets. Note that $N = 3600$ is still not a particularly large dataset.\n\nLIMITATIONS:\nYes. The authors discussed limitations of their work in a dedicated section at the end of the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn recent years, DNNs trained on image recognition tasks have emerged as strong predictive models of neural activity in the visual cortex. Typically, a regression model is trained to map DNN responses onto biological neural activity in response to the same inputs. The standard method for evaluating the predictive capacity of given DNN is to quantify the variance explained by the model on held-out images. This compresses the structure of each model into a single scalar value. While this can be used to rank models, it provides little insight into the structure of the model's representations and how they align with the structure of the biological neural representations. Another approach analyzes the similarities in representational geometry between models and neural data. However, the authors, state, these approaches do not relate the geometry of the DNN to the regression model used for the neural predictions.\n\nIn this paper, the authors leverage a recently proposed method for analyzing the generalization error of a model in terms of the radius and dimensionality of the error along each eigenvector of the input data (images viewed by both model and brain), to analyze the geometry of the generalization error of the regression model that maps a given DNN to neural data. \n\nThe authors use this method to analyze layer-wise differences in prediction error in a DNN, in terms of the error geometry. They find that the contributions of the error radius and error dimension change across layers. \n\nThey also analyze difference between trained and untrained DNNs at predicting neural responses. For V1 data, they find that the difference in predictivity and error geometry for trained vs. untrained networks is small. However, for V2 - IT data, they find interesting differences. In particular, they find that training reduces error radius, but increases error dimensionality, i.e.  spreads errors more evenly across error modes. They suggest that the improved predictivity of trained networks for V2 - IT data is primarily driven by changes in their intrinsic expressivity rather than significant changes in their representational alignment with neural data. \n\nIn addition, they examine differences between adversarially vs. standardly trained models. Replicating previous findings, they find that adversarial training overall improves model predictivity. They observe differences in how it affects error mode geometry across layers. In earlier layers, error radius is decreased with little change in error dimensionality. In later layers, error dimensionality is the main distinction between robust and standard models. \n\nLastly, they examine the dependence of the generalization error on the size of the training set of images. They find that for small datasets, the directions with small eigenvalues yield high error. For larger datasets, the additional samples permit learning along small eigenvalue directions.\n\nSTRENGTHS:\nThe paper applies a recently proposed methodology for analyzing the generalization error of a model to the problem of understanding DNN-neural data representational alignment. It is a solid idea, and the authors conduct a suite of well-posed experiments that analyze existing results from the literature in this frame.\n\nWEAKNESSES:\nThe paper would benefit from several changes.\n\nFirst, the motivation of the methodology was unclear. Intuitively, why are we interested in understanding the radius and dimensionality of the error? What insight can this give us into understanding representational alignment? What's the motivation for projecting along the eigenvectors of the data's Gram matrix? I think strong answers to these questions can be formulated. They should appear front and center in the paper. \n\nSecond, the conclusions obtained by the experiments were difficult to tease out. These should be clearly and simply stated, and presented in a list as an overview of the results. The way the results are written and presented visually, the reader needs to do a lot of digging to parse the meaning and significance. Consider making more intuitive graphical figures to emphasize the main points / contributions / conclusions, and/or condensing the results figures into simpler plots that highlight the important trends.\n\nLastly, the standard approach for interpreting DNN-neural data alignment could be more clearly explained and illustrated, so the reader understands how your approach differs. Consider making a schematic plot that highlights this distinction. \n\nOverall, I felt the paper required considerable effort to understand. Given that this work draws from several disciplines and relies on a very recently proposed methodology, I believe the authors should do more work to help the reader understand the motivation and significance of this work.\n\nQUESTIONS:\nI'd like to get a more intuitive handle on the significance of the error modes. I expect the eigenvectors of the gram matrix of the image data would be something like a Fourier basis. In this case, we'd expect large eigenvalues for low frequency bases and small eigenvalues for high frequency. Does this analysis essentially explain how error is distributed across frequency content in the images? If so, how does this give us insight into representational alignment?\n\nLIMITATIONS:\nThe significance of the conclusions drawn from the experiments is not clear. The purpose of this tool is to provide insight into neural representations in DNNs and the brain. Greater effort should thus be made to make the meaning of the quantified values (error radius / dimension along eigenvalues of the data's Gram matrix) interpretable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}