SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Abdulkadir Canatar $^{1,2*}$  Jenelle Feather $^{1,2*}$  Albert J. Wakhloo $^{1,3}$  SueYeon Chung $^{1,2}$
<sup>1</sup>Center for Computational Neuroscience, Flatiron Institute
$^{2}$ Center for Neural Science, New York University
$^{3}$ Zuckerman Institute, Columbia University
{acanatar,jfeather,awakhloo,schung}@flatironinstitute.org
The representations of neural networks are often compared to those of biological systems by performing regression between the neural network responses and those measured from biological systems. Many different state-of-the-art deep neural networks yield similar neural predictions, but it remains unclear how to differentiate among models that perform equally well at predicting neural responses. To gain insight into this, we use a recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target. We apply this theory to the case of regression between model activations and neural responses and decompose the neural prediction error in terms of the model eigenspectra, alignment of model eigenvectors and neural responses, and the training set size. Using this decomposition, we introduce geometrical measures to interpret the neural prediction error. We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression. The work demonstrates that carefully decomposing representational metrics can provide interpretability of how models are capturing neural activity and points the way towards improved models of neural activity.
Neuroscience has a rich history of using encoding models to understand sensory systems [1, 2, 3]. These models describe a mapping for arbitrary stimuli onto the neural responses measured from the brain and yield predictions for unseen stimuli. Initial encoding models were built from hand-designed parameters to model receptive fields in early sensory areas [4, 5], but it was difficult to extend these hand-engineered models to capture responses in higher-order sensory regions. As deep neural networks (DNNs) showed human-level accuracy on complex tasks, they also emerged as the best models for predicting brain responses, particularly in these later regions [6, 7, 8, 9, 10].
Although early work showed that better task performance resulted in better brain predictions [6], in modern DNNs, many different architectures and training procedures lead to similar predictions of neural responses [11, 12, 13]. This is often true even when model representations are notably different from each other using other metrics of comparison [14, 15]. For instance, previous lines of work have shown clear differences between models from their behavioral responses to corrupted or texturized images [16, 17, 18], from their alignment to human similarity judgments [19], from generating stimuli that are "controversial" between two models [20], and from stimuli that are metameric to models [15] or to human observers [21]. Other work has focused on directly comparing representations [22, 23] or behavioral patterns [24] of one neural network to another, finding that changing the training dataset or objective results in changes to the underlying representations. Given the increasing number of findings that demonstrate large variability among candidate computational models, it is an open
question of why current neural prediction benchmarks are less sensitive to model modification, and how to design future experiments and stimulus sets to better test our models.
Metrics of similarity used to evaluate encoding models of neural responses are often condensed into a single scalar number, such as the variance explained when predicting held-out images [6, 11]. When many models appear similar with such metrics, uncovering the underlying mechanisms driving high predictivity is a challenge [25]. To uncover the key factors behind successful DNN models of neural systems, researchers have proposed various approaches to explore the structural properties of internal representations in DNN models in relation to neural data. One promising approach focuses on analyzing the geometry of neural activities [25]. Several studies have examined the evaluation of geometries and dimensionalities of DNNs [26, 27, 28], highlighting the value of these geometric approaches. They provide mechanistic insights at the population level, which serves as an intermediate level between computational level (e.g., task encoding) and implementation level (e.g., the activities of single neurons or units). Furthermore, the measures derived from representation geometries can be utilized to compare the similarity between models and neural data [7, 25, 29]. Although these approaches have been successful, they do not directly relate the geometry of DNNs to the regression model used to obtain neural predictions, which uniquely captures the utility of a model for applications such as medical interventions [11] and experimental designs [30]. Recent empirical findings by Elmoznino et al. [31] have suggested a correlation between the high-dimensionality of model representations and neural predictivity in certain scenarios, but this work lacks explicit theoretical connections between the spectral properties and predictivity measures. Our work contributes to this line of investigation, by seeking a theoretical explanation and suggesting new geometric measures linked to a model's neural prediction error for future investigations.
To bridge the gap between DNN geometry and encoding models used for neural prediction, we turn to a recent line of work that derived a predictive theory of generalization error using methods from statistical physics [32, 33]. Specifically, two different spectral properties that affect the regression performance were identified: Spectral bias characterizing how fast the eigenvalues of the data Gram matrix decay, and task-model alignment characterizing how well the target aligns with the eigenvectors of this data Gram matrix [33]. While the former describes how large the learnable subspace is, the latter defines how much of the variance of the target is actually learnable. Note that in general, these two properties may vary independently across representations. Hence, the trends among regression scores across DNNs must be carefully studied by taking into account the joint trends of spectral bias and task-model alignment.
In this work, we explore how encoding models of neural responses are influenced by the geometrical properties of the DNN representations and neural activities. Our analyses primarily concern how and why models achieve the predictivities that they do, rather than offering an alternative scalar metric for predictivity. We find that the neural prediction error from ridge regression can be broken down into the radius and dimension of the error mode geometry, characterizing the error along each eigenvector. When investigating models along these error axes, we see a large spread in the error mode geometry, suggesting that the correlation values obtained from regression analyses obscure many geometrical properties of the representations.
Our primary contributions are:
- We analytically decompose the neural prediction error of ridge regression from a model to a set of brain data in terms of the model eigenspectra, the alignment between the brain data and the eigenvectors of the model, and the training set size.
- We introduce two geometric measures that summarize these spectral properties and directly relate to the neural prediction error. We show that these measures distinguish between different models with similar neural prediction errors using a wide variety of network architectures, learning rules, and firing rate datasets from visual cortex.
- Using spectral theory, we demonstrate that for networks effective in predicting neural data, we can ascertain if their superior performance stems from the model's spectra or alignment with the neural data. Our findings indicate that (a) trained neural networks predict neural data better than untrained ones due to better alignment with brain response and (b) adversarial training leads to an improved alignment between the eigenvectors of networks and early visual cortex activity.
Figure 1: Experimental and theoretical framework. (A) Example stimuli from visual experiments used for neural prediction error measures of V1, V2, V4, and IT. (B) Framework for predicting neural responses from model activations. Responses are measured from a brain region  $\mathbf{Y}$  and from a model stage of a DNN  $\mathbf{X}$ . Weights for linear regression are learned to map  $\mathbf{X}$  to  $\mathbf{Y}$ , and the empirical neural prediction error  $E_{g}(p)$  is measured from the response predictions. (C) Spectral alignment theory. Neural prediction error can be described as a function of the eigenvalues and eigenvectors of the model Gram matrix. The neural response matrix  $\mathbf{Y}$  is projected onto the eigenvectors  $\mathbf{v}_{i}$  with alignment coefficients  $W_{i}$ . For the number of training samples  $p$ , the value of each error mode is given as  $W_{i}E_{i}(p) = \widetilde{W}_{i}(p)$ . The total neural prediction error can be expressed as a sum of  $\widetilde{W_{i}}(p)$  terms, visualized as the area under the  $\widetilde{W_{i}}(p)$  curve.
2.1 Encoding Models of Neural Responses
We investigated DNNs as encoding models of visual areas V1, V2, V4, and IT. The neural datasets were previously collected on 135 texture stimuli for V1 and V2 [34], and on 3200 object stimuli with natural backgrounds for V4 and IT [35] (Fig. 1A). Both datasets were publicly available and obtained from [11]. The neural responses and model activations were extracted for each stimulus (Fig. 1B), and each stage of each investigated neural network was treated as a separate encoding model. We examined 32 visual deep neural networks. Model architectures included convolutional neural networks, vision transformers (ViTs), and "biologically inspired" architectures with recurrent connections, and model types spanned a variety of supervised and self-supervised training objectives (see Sec. SI.2 for full list of models). We extracted model activations from several stages of each model. For ViTs, we extracted activations from all intermediate encoder model stages, while in all other models we extracted activations from the ReLU non-linearity after each intermediate convolutional activation. This resulted in a total number of 516 analyzed model stages. In each case, we flattened the model activations for the model stage with no subsampling.
2.2 Spectral Theory of Prediction and Alignment
In response to a total of  $P$  stimuli, we denote model activations with  $M$  features (e.g. responses from one stage of a DNN) by  $\mathbf{X} \in \mathbb{R}^{P \times M}$  and neural responses with  $N$  neurons (e.g. firing rates) by  $\mathbf{Y} \in \mathbb{R}^{P \times N}$ . Sampling a training set  $(\mathbf{X}_{1:\mathbf{p}}, \mathbf{Y}_{1:\mathbf{p}})$  of size  $p < P$ , ridge regression solves (Fig. 1B):
$$
\hat {\beta} (p) = \underset {\beta \in \mathbf {R} ^ {M \times N}} {\arg \min } \| \mathbf {X} _ {\mathbf {1}: \mathbf {p}} \beta - \mathbf {Y} _ {\mathbf {1}: \mathbf {p}} \| _ {F} ^ {2} + \alpha_ {\mathrm {r e g}} \| \beta \| _ {F} ^ {2}, \quad \hat {\mathbf {Y}} (p) = \mathbf {X} \hat {\beta} (p) \tag {1}
$$
We analyze the normalized neural prediction error,  $E_{g}(p) = \frac{||\hat{\mathbf{Y}}(p) - \mathbf{Y}||_{F}^{2}}{||\mathbf{Y}||_{F}^{2}}$ , for which we utilize theoretical tools from learning theory [36, 37, 38], random matrix theory [39, 40, 41] and statistical physics [42, 32, 33] to extract geometrical properties of representations based on spectral properties of data. In particular, the theory introduced in [32, 33] relies on the orthogonal mode decomposition (PCA) of the Gram matrix  $\mathbf{X}\mathbf{X}^{\top}$  of the model activations, and projection of the target neural responses onto its eigenvectors:
$$
\mathbf {X} \mathbf {X} ^ {\top} = \sum_ {i = 1} ^ {P} \lambda_ {i} \mathbf {v} _ {i} \mathbf {v} _ {i} ^ {\top}, \quad W _ {i} := \frac {\left\| \mathbf {Y} ^ {T} \mathbf {v} _ {i} \right\| _ {2} ^ {2}}{\left\| \mathbf {Y} \right\| _ {F} ^ {2}}, \quad \langle \mathbf {v} _ {i}, \mathbf {v} _ {j} \rangle = \delta_ {i j}. \tag {2}
$$
Here, associated to each mode  $i$ ,  $W_{i}$  denotes the variance of neural responses  $\mathbf{Y}$  in the direction  $\mathbf{v}_{i}$ , and  $\lambda_{i}$  denotes the  $i^{th}$  eigenvalue. Then, the neural prediction error is given by:
$$
E _ {g} (p) = \sum_ {i = 1} ^ {P} W _ {i} E _ {i} (p), \quad E _ {i} (p) := \frac {\kappa^ {2}}{1 - \gamma} \frac {1}{\left(p \lambda_ {i} + \kappa\right) ^ {2}}, \tag {3}
$$
where  $\kappa = \alpha_{\mathrm{reg}} + \kappa \sum_{i=1}^{P} \frac{\lambda_i}{p\lambda_i + \kappa}$  must be solved self-consistently, and  $\gamma = \sum_{i=1}^{P} \frac{p\lambda_i^2}{(p\lambda_i + \kappa)^2}$  (see Sec. SI.1 for details) [32, 33]. Note that the theory depends not only on the model eigenvalues  $\lambda_i$ , but also on the model eigenvectors  $\mathbf{v}_i$  along with the responses  $\mathbf{Y}$ , which determine how the variance in neural responses distributed among a model's eigenmodes.
Although the equations are complex, the interpretations of  $W_{i}$  and  $E_{i}(p)$  are simple.  $W_{i}$  quantifies the projected variance in neural responses on model eigenvectors (alignment between neural data and model eigenvectors, i.e., task-model alignment [33]). Meanwhile,  $E_{i}(p)$ , as a function of the training set size  $p$ , determines the reduction in the neural prediction error at mode  $i$  and depends only on the eigenvalues,  $\lambda_{i}$  (i.e., spectral bias [33]).
In this work, we combine both and introduce error modes  $\widetilde{W}_i(p) \coloneqq W_iE_i(p)$ :
$$
\widetilde {W} _ {i} (p) := \frac {\kappa^ {2}}{1 - \gamma} \frac {W _ {i}}{\left(p \lambda_ {i} + \kappa\right) ^ {2}}, \quad E _ {g} (p) = \sum_ {i} \widetilde {W} _ {i} (p) \tag {4}
$$
As shown in (Fig. 1C),  $\widetilde{W}_i$  associated with large eigenvalues  $\lambda_i$  will decay faster with increasing  $p$  than those associated with small eigenvalues.
The generalization performance of a model is fully characterized by its error modes,  $\widetilde{W}_i$ . However, due to its vector nature,  $\widetilde{W}_i$  is not ideally suited for model comparisons. To address this limitation, we condense the overall shape of  $\widetilde{W}_i$  into two geometric measures, while preserving their direct relationship to the neural prediction error. This is in contrast to previous such measures, such as the effective dimensionality, that only depend on model eigenvalues [31].
Here, we define a set of geometric measures that characterize the distribution of a model's  $\widetilde{W}_i$  via the error mode geometry (Fig. 2A). Specifically, we rewrite the neural prediction error  $E_g(p)$  as:
$$
R _ {e m} (p) := \sqrt {\sum_ {i} \widetilde {W} _ {i} (p) ^ {2}}, \quad D _ {e m} (p) := \frac {\left(\sum_ {i} \widetilde {W} _ {i} (p)\right) ^ {2}}{\sum_ {i} \widetilde {W} _ {i} (p) ^ {2}}, \quad E _ {g} (p) = R _ {e m} (p) \sqrt {D _ {e m} (p)}. \tag {5}
$$
The error mode radius  $R_{em}$  denotes the overall size of the error terms, while the error mode dimension  $D_{em}$  represents how dispersed the total neural prediction error is across the different eigenvectors (Fig. 2A). Note that, the neural prediction error  $E_{g}(p)$  above has a degeneracy of error geometries; many different combinations of  $R_{em}$  and  $D_{em}$  may result in the same  $E_{g}(p)$  (Fig. 2B). Given a fixed neural prediction error, a higher value of  $R_{em}(p)$  indicates that only a few  $\widetilde{W}_{i}$  are driving the error, while a higher  $D_{em}(p)$  indicates that many different  $\widetilde{W}_{i}$  are contributing to  $E_{g}(p)$ .
3 Results and Observations
We first confirmed that Eq. 3 accurately predicted the neural prediction error for the encoding models considered here. We performed ridge regression from the model activations of each model stage of
$$
E _ {g} (p) = R _ {e m} (p) \sqrt {D _ {e m} (p)}
$$
$$
R _ {e m} (p) = \sqrt {\sum_ {i} \widetilde {W} _ {i} (p) ^ {2}}
$$
$$
D _ {e m} (p) = \frac {\left(\sum_ {i} \widetilde {W} _ {i} (p)\right) ^ {2}}{\sum_ {i} \widetilde {W} _ {i} (p) ^ {2}}
$$
Figure 2: Summary measures for error mode geometry. (A) Error modes  $(\widetilde{W}_i)$  are related to the model eigenspectra  $(\lambda_i)$  and alignment coefficients  $(W_i)$ . Error mode geometry summarized by  $R_{em}$  and  $D_{em}$  characterizes the distribution of error modes and directly relates to the neural prediction error  $E_g(p)$  (B) Interpreting contour plots. Error mode geometry distinguishes between models with the same prediction error  $(E_g(p))$ . Neural prediction error is the product of  $R_{em}$  and  $\sqrt{D_{em}}$  and is thus easily visualized on a contour plot where each contour represents equal  $E_g(p)$ .
each trained neural network to the neural recordings for each cortical region using a ridge parameter of  $\alpha_{\mathrm{reg}} = 10^{-14}$ , and also calculated the theoretical neural prediction error for each using Eq. 3. Given that Eq. 3 is only accurate in the large  $P$  limit in which both the number of training points and the number of test samples is large (see Sec. SI.1 and Fig. SI.4.2), we used a 60/40 train-test split  $(p = 0.6P)$  in the experiments below. As shown in Fig. 3A, this split yielded near-perfect agreement between the theoretical and empirical neural prediction errors for areas V4 and IT  $(R^2 = 0.97$  and 0.97, respectively), and very strong agreement for areas V1 and V2  $(R^2 = 0.9$  and 0.9, respectively). We present further data for this agreement in Sec. SI.4. Furthermore, we found that the ordering of models according to their neural prediction error is very similar to the ordering obtained via the partial least squares (PLS) regression method used in Brain-Score [11] (see Sec. SI.3). We maintained these choices for train-test split sizes and  $\alpha_{\mathrm{reg}}$  for all subsequent experiments.
We visualized the spread of error mode geometries for each brain region across different trained models in Fig. 3B. Each point is colored by the empirical  $E_{g}(p)$ , while the contour lines show the theoretical  $E_{g}(p)$  value. Among trained models, there was a range of  $R_{em}$  and  $D_{em}$  values even when many models have similar  $E_{g}(p)$ . This demonstrates that our geometrical interpretation of the error mode geometry can give additional insights into how models achieve low neural prediction error. In the following sections, we explore how these geometrical properties of the neural prediction error vary with model stage depth, training scheme, and adversarial robustness.
3.1 Error Mode Geometry Varies across Model Stages
We analyzed the neural prediction error obtained from activations at different model stages. In Fig. 4A, we plot the error mode geometry for all analyzed DNN model stages when predicting brain data from V1 and IT. Each point is color coded based on its relative model depth. Lower neural prediction errors were achieved (i.e. points lie on lower  $E_{g}$  contours) for earlier model stages in early visual area V1, and for later model stages in downstream visual area IT. This observation is in agreement with previous findings [6, 11, 43] where artificial neural networks have a similar hierarchy compared to the visual cortex. Qualitatively, the trends across model stages were more prominent than differences observed from comparing model architectures for supervised models (Fig. SI.5.1), or from comparing supervised trained models to self-supervised models of the same architecture (Fig. SI.5.2).
To investigate the source of this geometric difference across model stages, we performed an experiment on each model where we took the eigenspectra  $\lambda_{i}$  measured from the first (Early) or last (Late) model stage and paired this with the alignment coefficients  $W_{i}$  measured from the Early or Late model stage (Fig. 4B). These two spectral properties can be varied independently, as  $W_{i}$  depends on
Figure 3: Error mode geometry of model predictions for each brain region. (A) Theoretical values of neural prediction error match empirical values from ridge regression. Each point corresponds to predictions obtained from one of the 516 analyzed model stages. (B) Error mode dimension and radius for each brain region. Each point corresponds to the error mode  $\sqrt{D_{em}}$  and  $R_{em}$  for the region from a specific model stage of trained neural network models. Points are colored by the empirical error from performing ridge regression between the model activations and neural responses.  $R_{em}$  and  $\sqrt{D_{em}}$  values are obtained from theoretical values, and contour lines correspond to theoretical values of the neural prediction error, such that points along the contours have the same error.
the model eigenvectors but not on the model eigenvalues. We then measured the neural prediction error that would be obtained from such a model. This experiment allows us to isolate whether the observed differences in neural prediction error are primarily due to the  $\lambda_{i}$  or  $W_{i}$  terms. In the V1 data, there was little difference in the error mode geometry between the location of the  $W_{i}$  terms, however using early model stage  $\lambda_{i}$  corresponds to lower neural prediction error (summarized in Fig. 4C). In contrast, for the IT data using the late stage  $W_{i}$  terms corresponded lower  $E_{g}(p)$  mainly through a reduction of the error mode  $R_{em}$  (i.e. many error modes are better predicted). The spectral properties are shown in full for the early and late stages of ResNet50 as measured on the V1 and IT datasets (Fig. 4D). These results highlight the need to study the spectral properties of the model together with the brain responses as summarized by  $\widetilde{W}_{i}$ : Simply studying the properties of the eigenspectra of the model is insufficient to characterize neural prediction error (see SI.5.5 for more details).
3.2 Trained vs. Untrained
We analyzed how the error mode geometry for neural predictions differed between trained and randomly initialized DNNs (Fig. 5A). In line with previous results [11, 14, 44], we found that training yielded improved neural predictions as measured via smaller  $E_{g}(p)$  (Fig. 5B). This improvement was most notable in regions V2, V4, and IT, where there was also a characteristic change in the error mode geometry. In these regions, while  $R_{em}$  decreased with training,  $D_{em}$  surprisingly increased. However, the decrease in  $R_{em}$  was large enough to compensate for the higher  $D_{em}$ , leading to an overall reduction in  $E_{g}(p)$ . In V1, we observed only a small difference in neural prediction error between trained and untrained models, similar to previous results for early sensory regions [45, 14]. As shown in Fig. SI.5.7, we found qualitatively similar results in a different V1 dataset with a larger number of tested stimuli [46].
What differed between the trained and random model representations that led to differences in error mode geometry? To gain insight into this, we investigated how the eigenspectra  $(\lambda_{i})$  and the alignment coefficients  $(W_{i})$  individually contributed to the observed error mode geometry in visual area IT. We performed an experiment on the trained and random ResNet50 model activations where