SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Among the five modules evaluated in this analysis, cortical magnification emerged as the most influential factor in enhancing V1 alignment. This mechanism substantially improved the ResNet's ability to explain the variance in stimulus responses, and the artificial neurons exhibited tuning properties that were more closely aligned with those of biological neurons, particularly in terms of
ImageNet Acc	V1 Overall	V1 Predictivity	V1 Property
Center-surround antagonism	.610 ± .001	.545 ± .002	.304 ± .016	.786 ± .018
Local receptive fields	.609 ± .001	.550 ± .006	.300 ± .002	.799 ± .012
Divisive normalization	.606 ± .001	.543 ± .003	.271 ± .014	.815 ± .011
Tuned normalization	.608 ± .002	.547 ± .004	.274 ± .004	.820 ± .009
Cortical magnification	.548 ± .008	.587 ± .014	.370 ± .008	.805 ± .021
ResNet50 (Baseline)	.613 ± .002	.550 ± .004	.295 ± .003	.805 ± .011
Table 1: ImageNet object recognition classification performance  $(64\times 64$  images) and primary visual cortex (V1) alignment scores of ResNet50 augmented with each architectural component. Mean and standard deviations are reported across three runs (random initialization, training, and evaluating) of each architecture. Scores higher than baseline are presented in green and those lower are presented in red (the more saturated the color is, the greater the difference from baseline).
Orientation	Spatial frequency	Response selectivity	RF size	Surround modulation	Texture modulation	Response magnitude
Center-surround	.876 ± .027	.831 ± .030	.632 ± .012	.853 ± .046	.773 ± .027	.757 ± .025	.783 ± .024
Local receptive fields	.904 ± .021	.817 ± .016	.648 ± .008	.852 ± .054	.847 ± .083	.743 ± .036	.780 ± .022
Divisive normalization	.908 ± .017	.840 ± .014	.689 ± .007	.858 ± .046	.860 ± .070	.746 ± .030	.846 ± .019
Tuned normalization	.907 ± .035	.841 ± .013	.689 ± .023	.865 ± .031	.852 ± .020	.742 ± .029	.844 ± .015
Cortical magnification	.907 ± .037	.848 ± .039	.708 ± .011	.803 ± .044	.664 ± .063	.789 ± .058	.917 ± .071
ResNet50 (Baseline)	.893 ± .023	.826 ± .048	.684 ± .059	.832 ± .080	.820 ± .009	.786 ± .058	.790 ± .042
Table 2: Model alignment across the seven primary visual cortex (V1) tuning properties that constitute the V1 Property score. Mean and standard deviation of scores observed across three trials of model training and evaluation are reported.
orientation tuning, spatial frequency tuning, response selectivity, and most of all, stimulus response magnitude. However, the artificial neuronal responses of the cortical magnification network showed lower resemblance to those observed in primate V1 with regard to surround modulation, as compared to the baseline network.
Simulating neural normalization within the ResNet resulted in artificial neurons that displayed improved alignment with primate V1 in terms of response properties. Noteworthy enhancements were observed in the spatial frequency, receptive field size, surround modulation, and response magnitude properties of neurons within the modified network, leading to improvements in the V1 Property score. These results applied to both tuned and untuned forms of normalization.
In contrast, the introduction of center-surround convolutions yielded minimal improvements in neural predictivity and slight reductions in overall neuron property similarity. Surprisingly, the surround modulation properties of the artificial neurons decreased compared to the baseline model, contrary to our expectations.
Finally, replacing the final  $3 \times 3$  convolution preceding the artificial V1 readout with a locally connected layer resulted in modest changes in V1 alignment. This was one of the two mechanisms that led to improvements in the surround modulation response property score (tuned normalization being the other).
These findings collectively provide valuable insights into the individual contributions of each specialized mechanism. Although mechanisms simulating center-surround antagonism (i.e., DoG convolution) and local connectivity provide little benefit to overall predictivity and property scores in isolation, we observed that they reduce the property dissimilarity gap among tuning properties that are nonetheless important and complement alignment scores where divisive normalization and cortical magnification do not.
4.2 Complementary Components Explain V1 Activity
Constraining a general-purpose deep learning model with a single architectural component is likely insufficient to explain primate V1 activity given our knowledge that a composition of known circuits play pivotal roles in visual processing. This was empirically observed in Section 4.1, wherein cortical
Center-Surround	Local RF	Tuned Nor-malization	Cortical Mag-nification	ImageNet Acc	V1 Overall	V1 Predictivity	V1 Property
✓	✓	✓	✓	.551	.605	.357	.852
✓	✓	✓	.543	.605	.353	.857
✓	✓	✓	.541	.599	.340	.858
✓	✓	✓	.552	.592	.364	.820
✓	✓	✓	.603	.555	.276	.834
✓	✓	.541	.598	.351	.845
✓	✓	.555	.593	.384	.803
✓	✓	.606	.561	.287	.835
ResNet50 (Baseline)	.613	.550	.295	.805
Table 3: Backward component elimination results. Checkmarks denote whether or not the architectural component was included in the model. Adversarial training was performed on the two models that tied for the top V1 Overall Score.
magnification was the only architectural component found to improve the overall V1 alignment score. Taking inspiration from this design principle, we supplemented a ResNet50 with each architectural component and discern the necessary components to achieve optimal V1 alignment in an ablation study. We omitted the architectural component implementing divisive normalization, however, as it cannot be integrated simultaneously with tuned normalization, which was observed to yield slightly higher V1 Predictivity and Property scores in isolated component evaluation. Starting with a ResNet which features all of these architectural components, we employed a greedy approach reminiscent of backward elimination feature selection to deduce critical components without having to evaluate every permutation of component subsets. In each round of this iterative approach, we selectively removed the architectural component that reduced overall V1 alignment the most until only one feature remained. This analysis allowed us to identify the subset of components that collectively yielded the most significant improvements in V1 alignment, and unraveled the intricate relationship between these specialized features and their combined explanation of V1.
The results of the ablation study are presented in Table 3. With the exception of center-surround antagonism, removing any neural mechanisms from the modified residual network reduced overall V1 alignment, suggesting that (1) each architectural component contributed to V1 alignment (the utility of center-surround antagonism is detailed in Section 4.5) and (2) nontrivial interactions between these mechanisms explain V1 more than what is possible with any single mechanism. Seven of the eight models evaluated in this ablation study substantially outperformed all existing models on the Brain-Score platform in modeling V1 tuning property distributions. Furthermore, four models were observed to achieve state-of-the-art V1 Overall scores, explaining both V1 stimulus response activity and neural response properties with high fidelity.
Whether or not feed-forward, ImageNet-trained ANNs can fully approximate activity in primate V1 has stood as an open question. Previous studies have argued that no current model is capable of explaining all behavioral properties using neurons from a single readout layer [17]. The top performing models of the current evaluation stand out as the first examples of CNNs with neural representations that accurately approximate all evaluated V1 tuning properties (Appendix C), offering positive evidence for the efficacy of explaining primate V1 with neuro-inspired deep learning architectures.
4.3 Component Contributions to V1 Alignment
Supplementing the quantitative studies of Sections 4.1 and 4.2, studying learned component parameters and visual stimuli that maximally activated artificial V1 neurons provided further insights into the role of these architectural components in explaining biological V1 activity.
The V1 Property scores found in our ablation studies (Table 5) suggest the contribution of center-surround antagonism in aligning spatial frequency properties between artificial and biological V1 representations. Optimized DoG kernels implementing center-surround antagonism were commonly observed to have center Gaussian distributions with low variance (i.e., small  $\sigma_{center}^2$ ), suggesting preferences for patterns and textures with high spatial frequencies (Appendix Fig. F.2).
Center-Surround	Local RF	Tuned Normalization	Cortical Magnification	Adversarial Training	ImageNet Acc	V1 Overall	V1 Predictivity	V1 Property
✓	✓	✓	✓	✓	.448	.629	.430	.829
✓	✓	✓	✓	.448	.625	.430	.819
Adversarily trained ResNet50 (Baseline)	.555	.581	.352	.809
Table 4: Adversarial training was performed on the two models that tied for the top V1 Overall Score. Checkmarks denote whether or not the architectural component was included in the model.
Regarding local receptive fields, we hypothesized that removing weight sharing would enable a greater diversity of response selectivity patterns to be learned. While multi-component ablation studies revealed a reduction in response selectivity property scores when local filtering was omitted, the same finding was surprisingly not observed in the single-component analyses of Section 4.1.
Given the inter-neuron competition enforced by tuned normalization, one would expect networks with this component to learn more diverse artificial V1 representations. Analyses of the visual stimuli that maximally activated artificial neurons of these networks (i.e., optimized visual inputs that maximally excite neurons of each channel of the artificial V1 layer, computed via stochastic gradient ascent) provide evidence for this. Quantitatively, we found that the mean Learned Perceptual Image Patch Similarity (LPIPS) [57] between pairs of maximally activating stimuli of the tuned normalization network was less than that of the baseline ResNet50 network ( $p < 0.01$ , one-way ANOVA [58]). We suggest that this learned feature diversity contributed to the improvements in spatial frequency, response selectivity, receptive field size, surround modulation, and response magnitude tuning property scores when tuned normalization was present.
Finally, given the retinotopic organization of V1, we hypothesized that cortical magnification would give rise to better-aligned response selectivity and receptive field size tuning distributions, meanwhile improving V1 neuron predictivity. In each trial of our ablation studies for which cortical magnification was removed, these respective scores dropped, supporting this hypothesis.
4.4 Object Recognition Robustness to Corrupted Images
In contrast with the human visual system, typical CNNs generalize poorly to out-of-distribution data. Small perturbations to an image can cause a model to output drastically different predictions than it would on the in-tact image. Recent studies have demonstrated a positive correlation between model-brain similarity and robustness to image corruptions [24-26, 42, 59] After fine-tuning each model's classification head on Tiny-ImageNet (see Section 3.2), we evaluated the object recognition accuracy of each model from Section 4.1 and the top two overall models from Section 4.2 on the Tiny-ImageNet-C dataset. The results of these evaluations for each category of corruption and corruption strength are provided in Appendix H.
Among the evaluated components, only tuned normalization was observed to yield improved corrupt image classification accuracy over the entire test set, albeit slight, beating the baseline accuracy (0.278) by 0.005 (i.e., an absolute improvement of  $.5\%$ ). More substantial improvements were observed on 'brightness', 'defocus_blur', 'elastic_transform', and 'pixelate' corruptions (improvements over the baseline of .00986, .00989, .0105, and .0133, respectively).
4.5 Adversarily Training Neuro-Constrained ResNets
Adversarial training has previously been shown to enhance the brain-similarity of artificial neural representations without any modification to the underlying network [25, 60]. Curious as to whether adversarial training would further align the neuro-constrained ResNet50s with V1 activity, we selectively trained the two networks most aligned with V1 (one model with all architectural components and the other with all components except center-surround convolution) from Section 4.2 using "Free" adversarial training [61] (Appendix G). The results are shown in Table 4. Despite the drop in object recognition accuracy, the artificial neural representations that emerged in each network were drastically better predictors of stimulus response variance representations. Tuning property alignment dropped in the process, but remained above previous state-of-the-art regardless. Interestingly, we found that the main difference in V1 scores between these two models can be traced to surround modulation tuning alignment. Center-surround convolutions indeed contributed to improved surround
modulation tuning learned while training with on corrupted images, contrasting its apparent lack of contribution to the overall network suggested in the ablation study.
In sum, both networks achieved Rank-1 V1 Overall, Predictivity, and Property scores by large margins, setting a new standard in this breed of brain-aligned CNNs. At the time of writing, the previous Rank-1 V1 Overall, Predictivity, and Property scores were .594, .409, and .816, respectively, and all achieved by separate models.
Throughout this work we presented a systematic evaluation of four architectural components derived from neuroscience principles and their influence on model-V1 similarity. Specifically, we studied task-driven CNNs enhanced with ANN layers that simulate principle processing mechanisms of the primate visual system including center-surround antagonism, local receptive fields, tuned normalization, and cortical magnification. Through an ablation study and isolated component analyses, we found that each component contributed to the production of latent ANN representations that better resemble those of primate V1, as compared to a traditional baseline CNN. When these four components were assembled together within a neuro-constrained ResNet50, V1 tuning properties were explained better than any previous deep learning model that we are aware of. Furthermore, this neuro-constrained model exhibited state-of-the-art explanation of V1 neural activity and is the first of its kind to do so, by a large margin nonetheless, highlighting a promising direction in biologically constrained ANNs. Training this model with "free" adversarial training greatly improved its ability to predict primate neural response to image stimuli at a minor sacrifice to tuning property similarity, establishing an even larger gap between previous state of the art.
Among all architectural components examined in this work, cortical magnification was the most influential to improving V1 alignment. This mechanism on its own could not explain the neural activity as comprehensively as the top models of this study, however. Our implementation of tuned normalization provided substantial improvement to V1 tuning property alignment, and was the only component that contributed to model robustness. The importance of center-surround antagonism seemed to be training data-dependent. In our ablation study, for which all models were trained on ImageNet, center-surround convolutional layers did not contribute to overall V1 scores. This did not surprise us, as deep CNNs have the capacity to learn similar representations without these specialized layers. When training on adversarially perturbed data, however, the center-surround antagonism provided by this layer appeared to improve surround modulation tuning properties of artificial V1 neurons. While previous attempts at improving model-brain similarity have been highly dataset dependent, our results highlight the importance of artificial network design.
A notable limitation to our work is the reduction in ImageNet classification performance that was observed upon the introduction of cortical magnification. While maintaining baseline model accuracy was not a motivation of this work, we can imagine situations in which object recognition performance needs to be preserved alongside these improvements in brain-model alignment. The implementation of cortical magnification in this work assumed that the model's gaze was focused at the center of each image. Consequently, visual stimuli at greater eccentricities from the image center were under-sampled during the polar transformation (Fig. F.1), making images for which the object of interest was not located at the image center (common in ImageNet) more challenging to classify. One scope of future work involves implementing saliency- or attention-driven polar transformations that dynamically focus the center of the polar map on, or near, an object of interest as opposed to being fixed at the image center. We demonstrate the efficacy of this strategy with a naive proof of concept in which classification is performed by averaging predictions from the five static crops of each image [62]. This simple strategy improved the validation accuracy of the network with all components from  $55.1\%$  to  $59.9\%$  without affecting V1 scores. A more sophisticated, dynamic strategy could further reduce this accuracy drop. We additionally plan to extend this work to model architectures other than ResNet to validate the widespread application of each of these neuro-constrained components.
This work highlights an important advancement in the field of NeuroAI, as we systematically establish a set of neuro-constrained architectural components that contribute to state-of-the-art V1 alignment. We argue that our architecture-driven approach can be further generalized to additional areas of the brain as well. The neuroscience insights that could be gleaned from increasingly accurate in-silico models of the brain have the potential to transform the fields of both neuroscience and AI.
[57] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[58] Amanda Ross and Victor L. Willson. One-Way Anova, pages 21-24. SensePublishers, Rotterdam, 2017.
[59] Shahd Safarani, Arne Nix, Konstantin Willeke, Santiago Cadena, Kelli Restivo, George Denfield, Andreas Tolias, and Fabian Sinz. Towards robust vision by multi-task learning on monkey visual cortex. In Advances in Neural Information Processing Systems, volume 34, pages 739-751. Curran Associates, Inc., 2021.
[60] Alexander Riedel. Bag of Tricks for Training Brain-Like Deep Neural Networks. March 2022.
[61] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[62] Manish Reddy Vuyyyuru, Andrzej Banburski, Nishka Pant, and Tomaso Poggio. Biologically inspired mechanisms for adversarial robustness. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2135-2146. Curran Associates, Inc., 2020.
A Supplemental Model Diagrams
Fig. A.1 depicts the modifications made to ResNet50 residual layer 1 in the isolated component analyses of section 4.1. All multi-component (composite) models analyzed in Section 4.2 relied on combinations of these modifications (as exemplified in Fig. 2).
(A) ResNet50 baseline
(B) Center-surround antagonism
(C) Local receptive fields
(D) Divisive normalization
(E) Tuned normalization
(F) Cortical magnification
Figure A.1: ResNet50 residual layer 1, supplemented with individual neuro-constrained architectural components, as in section 4.1. (A) No modification (baseline ResNet50 layer 1), (B) with center-surround antagonism, (C) with local receptive field (RF), (D) with divisive normalization, (E) with tuned normalization, (F) with cortical magnification.
B V1 Scores of Alternate Layers of Baseline Network
When evaluating a model on Brain-Score, users are permitted to commit a mapping between model layers and areas of the ventral stream. Model-brain alignment is computed for each mapped pair in the Brain-Score evaluation. To promote a fair evaluation, we sought to find the layer that yielded optimal V1 alignment from the baseline ResNet50 model and fix this layer as the artificial V1 readout layer in all of our tested models. It is worth noting that after supplementing the base ResNet50 with neuro-constrained components, this layer may no longer offer optimal V1 alignment in the augmented network. In spite of this, we maintain this layer as our artificial V1 readout layer for fair evaluation.
To find the ResNet50 layer with the best V1 Overall, Predictivity, and Property scores, we compared a total of 20 different hidden layers (Fig. B.1). 16 of these layers corresponded to the post-activation hidden states of the network. The remaining 4 were downsampling layers of the first bottleneck block of each residual layer in the network, as these have previously demonstrated good V1 alignment [25]. Aside from these downsampling layers, hidden layers that did not follow a ReLU activation were omitted from this evaluation as the activities of these states can take on negative values and are therefore less interpretable as neural activities. Among all evaluated layers, the final output of ResNet50 residual layer 1 (i.e., the output of the third residual block of ResNet50) offered the highest V1 Overall score, and was therefore selected as the artificial V1 readout layer in all of our experiments.
Figure B.1: V1 alignment Brain-Scores for 20 different hidden layers of ResNet50. In the plot above, readout location 'X.Y' denotes that artificial V1 activity was evaluated from residual block 'Y' of residual layer 'X'. Readout location suffixed with '.d' correspond to downsampling layers of the associated residual bottleneck. Highest V1 overall score came from block 3 of residual layer 1.
C Expanded Model Tuning Properties
Primary visual cortex (V1) tuning property alignments for each composite model evaluated in Section 4.2 are presented in Table 5. Tuning property similarities are computed asCeiled Kolmogorov-Smirnov distance between artificial neural response distributions from the model and empirical distributions recorded in primates [16, 17].
Center-Surround	Local RF	Tuned Norm.	Cortical Mag.	Adv. Training	Orientation	Spatial Frequency	Response Selectivity	RF Size	Surround Modulation	Texture Modulation	Response Mag.
✓	✓	✓	✓	.891	.925	.756	.840	.779	.844	.930
✓	✓	✓	.858	.919	.780	.834	.808	.871	.930
✓	✓	✓	.894	.932	.750	.851	.775	.858	.946
✓	✓	✓	.878	.873	.739	.816	.719	.802	.910
✓	✓	✓	.875	.873	.702	.808	.890	.815	.870
✓	✓	.873	.886	.735	.840	.794	.825	.959
✓	✓	.902	.866	.715	.801	.625	.841	.869
✓	✓	.915	.817	.691	.811	.898	.802	.911
✓	✓	✓	✓	✓	.924	.863	.773	.797	.733	.815	.899
✓	✓	✓	✓	.944	.834	.768	.806	.673	.811	.900
D V1 Brain-Scores of Untrained Models
Table 5: Ablation study model alignment across the seven primary visual cortex (V1) tuning properties that constitute the V1 Property score ('Marques2020') of Brain-Score. Checkmarks denote whether or not the architectural component was included in the model.
V1 Overall	V1 Predictivity	V1 Property
Center-surround antagonism	.298	.245	.551
Local receptive fields	.477	.210	.743
Divisive normalization	.499	.207	.792
Tuned normalization	.471	.218	.724
Cortical magnification	.497	.276	.718
All Components	.483	.225	.741
ResNet50	.466	.223	.710
E V2, V4, and IT Brain-Scores of Top Model