Neural Regression, Representational Similarity, Model Zoology & Neural Taskonomy at Scale in Rodent Visual Cortex
Authors: Colin Conwell
David Mayo
Andrei Barbu
Michael A Buice
George A. Alvarez
Boris Katz
Venue: NeurIPS 2021 Poster (2021)
Keywords: neuro_ai
deep neural networks
rodent visual cortex
mouse brains
optical physiology

Abstract: How well do deep neural networks fare as models of mouse visual cortex? A majority of research to date suggests results far more mixed than those produced in the modeling of primate visual cortex. Here, we perform a large-scale benchmarking of dozens of deep neural network models in mouse visual cortex with both representational similarity analysis and neural regression. Using the Allen Brain Observatory's 2-photon calcium-imaging dataset of activity in over 6,000 reliable rodent visual cortical neurons recorded in response to natural scenes, we replicate previous findings and resolve previous discrepancies, ultimately demonstrating that modern neural networks can in fact be used to explain activity in the mouse visual cortex to a more reasonable degree than previously suggested. Using our benchmark as an atlas, we offer preliminary answers to overarching questions about levels of analysis (e.g. do models that better predict the representations of individual neurons also predict representational similarity across neural populations?); questions about the properties of models that best predict the visual system overall (e.g. is convolution or category-supervision necessary to better predict neural activity?); and questions about the mapping between biological and artificial representations (e.g. does the information processing hierarchy in deep nets match the anatomical hierarchy of mouse visual cortex?). Along the way, we catalogue a number of models (including vision transformers, MLP-Mixers, normalization free networks, Taskonomy encoders and self-supervised models) outside the traditional circuit of convolutional object recognition. Taken together, our results provide a reference point for future ventures in the deep neural network modeling of mouse visual cortex, hinting at novel combinations of mapping method, architecture, and task to more fully characterize the computational motifs of visual representation in a species so central to neuroscience, but with a perceptual physiology and ecology markedly different from the ones we study in primates.

Neural Regression, Representational Similarity, Model Zoology & Neural Taskonomy at Scale in Rodent Visual Cortex
Colin Conwell" Department Psychology
David Mayo CSAIL CBMM MIT
Harvard University
Michael A. Buice
Boris Katz CSAIL & CBMM MIT
Modeling, Analysis & Theory Allen Institute MindScope Program
George A. Alvarez
Andrei Barbu CSAIL CBMM
Department Psychology
Harvard University
MIT
Abstract
How well do deep neural networks fare models of mouse visual cortex? A majority of research to date suggests results far more mixed than those produced in the modeling of primate visual cortex. Here, we perform large-scale bench- marking of dozens of deep neural network models in mouse visual cortex with both representational similarity analysis and neural regression. Using the Allen Brain Observatory's ' 2-photon malcium-imaginn dataset of activity in over 6,000 reliable rodent visual cortical neurons recorded in response natural scenes, we replicate previous findings and resolve previous discrepancies, ultimately demonstrating that modern neural networks can in fact used to explain activity in the mouse visual cortex to more reasonable degree than previously suggested. Using our benchmark as an atlas, we offer preliminary answers to overarching questions about levels of analysis (e.g. do models that better predict the representations of individual neurons also predict representational similarity across neural popula- tions?); questions about the properties of models that best predict the visual system overall (e.g. is convolution or category- suprrvviiiiiiiiioo necessary to better predict neural activity?); and questions about the mapping between biological and artificial representations (e.g. does the information processing hierarchy in deep nets match the anatomical hierarchy of mouse visual cortex?). Along the way, we catalogue a number of models (including vision transformers, MLP-Mixers, normalization free networks, Taskonomy encoders and self-supervised models) outside the traditional circuit of convolutional object recognition. Taken together, our results provide a reference point for future ventures in the deep neural network modeling of mouse visual cortex, hinting at novel combinations of mapping method, architecture, and task to more fully characterize the computational motifs of visual representation species central to neuroscience, but with a perceptual physiology and ecology markedly different from the ones we study in primates.
*Correspondence: @onwell@g.harvar... Project Website: wuth.h..ww///ciiicommo/////etete
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Introduction
To date, the most successful models of biological visual cortex are object-recognizing deep neural networks applied to the prediction of neural activity in primate visual cortex [1-5]. Corresponding to the biology not only at the level of individual layers, but across the feature hierarchy, these models are so powerful they can now effectively be used as neural controllers, synthesizing stimuli that drive neural activity far beyond the range evoked by any handmade experimental stimulus [6]. The correspondence of these same models to mouse visual cortex, on the other hand, has proven a bit more tenuous [7, 8], with recent finding even suggesting that randomly initialized networks are as predictive of rodent visual cortical activity as trained ones [9].
Often implicit in interpretations of these results is the notion that the visual milieu and machinery of mice simply different something characterized more, perhaps, by brute force predator avoidance and the 'flexible random associations' thought to define senses like olfaction [10] than by the sophisticated active sampling and representational compositionality enabled by primate central vision. And yet, mice do recognize objects [11, 12] and do engage other sophisticated visual behaviors [13] that suggest they must have visual solutions that at least functionally approximate the kinds of solutions learned by modern computer vision algorithms. If these models perform well in monkeys, but not in mice, are we overfitting to an artifact? Are the object recognition capabilities of mice simply the byproduct of a representational competence learned through other (even more behaviorally relevant) tasks? Have mice perhaps converged on solutions to visual problems that fundamentally differ from the solutions that undergird the emergent similarity between monkeys and machines? To even begin to answer these questions, we need substantially more comprehensive modeling statistics than we currently have. Our main goal in this work was to provide exactly that re-examine at large scale the state of neural network modeling in the visual cortices of mice, using many thousands of neurons, over 1 10 distinct neural network models, and two methods of mapping models to brain.
We summarize the statistics from our benchmarking survey in five main results:
1. Training matters. The randomly initialized variants of some convolutional architectures fare well when predicting individual neural responses, but representational similarity is always better captured by features learned in service of some task. (Segmentation seems best.)
2. Features of intermediate complexity dominate the prediction of all cortical sites, but both our mapping methods do demonstrate an upwards gradient in complexity from primary visual cortex onwards that roughly matches the information processing hierarchy proposed elsewhere in the rodent neurophysiology literature.
3. Taskonomic tools that have previously been shown to approximate functional organization in primates fail to strongly differentiate anatomical regions in mice, with the same kinds of tasks dominant across multiple, distinct neural sites.
4. When aggregated in similar ways, representational similarity and neural regression methods capture similar trends in the kinds of feature spaces that best predict the biology.
5. While still far from the overall noise ceiling for this highly reliable neural data, variety of the artificial deep net models in our survey make predictions only slightly less accurate than "biological conspecific models' composed of the neurons from other mice.
2 Methods
2.1
Neural Dataset
For neural data, we use the Allen Brain Observatory Visual Coding' dataset [14] collected with two- photon calcium-imagin from the visual cortex of 256 awake adult transgenic mice and consisting of approximately 59,610 unique, individual neurons. Calcium iieeiuunimmaing fluorescence patterns are preprocessed and deconvolved by the Allen Institute2. The neurons sampled include neurons from 6 visual cortical areas at 4 cortical depths across 12 genetic cre lines. The visual experiments recorded activity for both artificial images (e.g., diffraction gratings) and 118 natural scenes. We analyze only
Available with non-commercial license under the Allen Institute terms of use:
http://ww.aalleee www.alennstiiute...mmm allleminssiuueeooggggga//eessss
2 MMore details are available in the whitepapers released with the observatory data: http://obserratorrybbr brain wwwainmaaannooooooeee map.org/ www.oo/wwssuoooooonnaaggniiiiiii

---

the latter to ensure comparable inputs to what is typically used in the training of deep nets. Each natural scene is displayed 50 times over the course of an assay.
To ensure an optimal signal to noise ratio, we perform a significant amount of subsetting on the full neural population, beginning by subsetring only excitatory neurons. Recent analyses suggest neural activity throughout mouse visual cortex is often impacted by extraneous, external body movements [15]. For this reason, we subsequently filter out any neurons whose peak responses the presentation of natural scene images are significantly modulated by the mouse's 's running speed, using an ANOVA metric provided by the Allen Institute. We further subselect neurons by assessing their split-half reliability across trials (with each split-half constituting 25 of 50 presentations for each image), keeping only those neurons exhibiting 0.8 reliability and above. This thresholding still leaves 6619 neurons for analysis, is in line with prior work on primates, and supports, for example, the construction of cortical representational dissimilarity matrices (RDMs) with split-half reliabilities as high as 0.93. (More details on the relationship between our metrics and neural reliabiliit, including visualizations of some of our results across many degrees of thresholding. can be found A.4 of the Appendix.)
2.2 Model Zoology
To explore the influence of model architecture on predictive performance, we use 26 model ar- chitectures from the Torchvision (PyTorch) model zoo [16] and 65 model architectures from the Timm [17] model zoo [18-52]. These models include convolutional networks, vision transformers, cormalization-free networks and MLP-Mixer models. For each of these models, extract the features from one trained and one randomly initialized variant (using whatever initialization scheme the model authors deemed best) as to better disentangle what training object recognition affords us in terms of predictive power.
2.3 Neural Taskonomy
Model zoology provides decent perspective on the computations related to object recognition, but the responsibilities of the visual cortex (no matter the species) extend far beyond identifying the category of an object. To probe a wider range of tasks, we turn to Taskonomy: a single architecture trained on 24 different common computer vision tasks [53], ranging from autoencoding to edge detection. The model weights we use are from updated PyTorch implementations of the original Tensorflow models design in which only
[54]. Key to the engineering of Taskonomy is the use of an encoder-decode the construction of the decoder varies across tasks. While recent analyses using a similar approach in human visual cortex with fMRI data [55] have tended focus only on the latent space of each task's encoder, we extract representations across all layers, better situating Taskonomy within the same empirical paradigm that has so far defined the modeling of object recognition in the primate brain. For further clarity, we cluster the 24 tasks according to their "Taskonomic' category - a total of 5 clusters (2D, 3D, semantic, geometric or other) that we further collapse into 4 clusters (lumping the only member of the 'other' category - denoising autoencoder - in with its closest cousin a vanilla autoencoder in the '2D' category). These purely data-driven clusters are derived from estimates of how effectively a set of features learned for one task transfer to (or boost the performance in) another task [53]. Use of the Taskonomy models provides a unique opportunity to test variance in training regimes without the confound of simultancous changes in architecture.
2.4 Sell-Supervised Models
Full category supervision, while robust in its ability to build representations that transfer well to variety of tasks, suffers in its neuroscientific relevance as an ethologically plausible mode of learning. Recently, self-supervised models have begun to provide viable alternatives the representations learned by dategory-superviine
models in both computer vision [56, 57] and neural mapping [58, 59]. Here. assess 22 supervupervi models from the VISSL model zoo [60], ranging from earlier iterations (e.g. DeepCluster [61]) to modern contrastive learning algorithms (e.g. Barlow and Dino (62-65]). We use these models assess whether category-supervision in predicting neural activity. might eventually be supplanted by these more realistic alternatives. 14 of these models have as their base architecture a standard ResNet50; 8 are built atop vision transformers.
however powerful it is
2.5 Comparing Representations across Biological & Artificial Networks
Two methods predominate in the comparison of neural recordings deep neural networks: at the most abstract level, one of these compares representational geometries computed across the activations
3

---

of many individual neurons [66, 67]: the other attempts to predict the activity of individual neurons directly [67, 68]. Both of these techniques are grounded in the use of image-computable models and a shared stimulus set, but differ in the types of transformation applied to the neural activity generated by those stimuli. Given the difference both target (neural populations versus individual neurons) and transforms (correlation matrices versus dimensionality reduction) we attempt variant of each type of analysis here, comparing the two directly on the exact same neural data, with the same models and the same stimulus set, and in granular, layer-by-layer fashion. (A more comprehensive review of neural mapping methods is provided in Section A.2 of the Appendix.)
2.5.1 Representational Similarity Analysis
To compare the representational geometries of given model to the representational geometries of the brain, we begin by computing classic representational dissimilarity matrices (RDMs) [69]. We compute these RDMS by calculating the pairwise correlation coefficients between the neural response vectors for each image (one for each of the 6 cortical areas surveyed). We then repeat this procedure for the artificial networks, aggregating the responses of the artificial neurons in a given layer, before aggregating them once more into a correlation matrix. We then measure the relationship between the RDMs computed from the biological and artificial networks with a second-order Pearson correlation between the flattened upper triangles of each. The resultant coefficient constitutes the score for how well given model layer predicts the representational similarity of given cortical area.
2.5.2 Neural Regression (Encoding Models)
To more directly compare the biological and artificial neural activations in our data, we use a style of regression made popular in the modeling of primate visual cortex, epitomized by BrainScore [4]. Variants of this approach abound, but most consist of extracting model activations, performing dimensionality reduction, and then some form of cross-validated penalized or principal components regression. The dimensionality-redum reduccd feature spaces of the model are used as the regressors of the activation patterns in a given neuron. After testing number of these variants, we settled on sparse random projection for dimensionality reduction (which proved far more computationally efficient than standard PCA, without sacrifice in terms of regression scores), followed by ridge regression (in place of the more frequently used partial least squares regression).
The details of our method (programmed with [70]) are as follows: Given a network, we first extract a predetermined number of sparse random projections (4096, in this case) from the features of each layer- boyer-iie in line with the Baanson-Lindenstrann to the mice) in our data set 3. After extracting these projections, we regress them on the activity of each individual neuron using ridge regression (with default lambda penalty of 1.0). The use of a penalized regression in this case allows us to monopolize generalized cross- gross-validation (a linear algebraic form of leave-one-out cross- cross-validates validation), yielding a set of predictions for the activity of each neuron for each image". We then compute the Pearson correlation between the predicted and actual activity for each neuron to obtain score per neuron per model layer, which then aggregate by taking the mean of scores per neuron across cortical area.
lemma for the number of observations (images shown
We verify the efficacy of this method on the publicly available benchmarks of primate BrainScore. where (relative to BrainScore's in-house regression method) we demonstrate provisional gains not only in terms of predictive score (sometimes up to r = 34%), but also in terms of speed and computational efficiency. (Details may be found in Section A.1 of the Appendix.)
2.6 Model Rankings
To rank the models according to how well they predict the variance in a given cortical area, we take the max across layers. In effect, this requires that a model 'commit' only one layer to the prediction of each area. In the case of our neural regression metric we call these scores the Ridge Max'; in the case of our representational similarity metric call these scores the 'RSA Max'. A final mean taken over the SRP-Ridge Max and RSA Max scores per model per cortical area yields our overall model rankings, which serve the basis for the bulk of our analyses.
NNte that in cases where the dimensionality of features is less than the number of projections suggested by the lemma, sparse random projections will actually upsample the feature space, rather than downsample it. 'The use of generalized cross- validation is particularly beneficial in datasets with fewer probe images, where follld cross-validalee validation means losing significant degree of information in each fit.

---

2.7 Non-Neural Network Baselines
Prior the ascendancy of neural network models, significant amount of time and craft was invested in the hand-engineering of features to simultaneously facilitate image recognition and capture meaningful subsets of neural variance. In this work, test how well small subset of those features are able to explain the variance in rodent visual cortex, using both our neural encoding and representational similarity metries. Our non-neural network baselines consist of random fourier features [71] (computed specifically to match the dimensionality of our neural network predictors), handcrafted gabor filters and GIST (spatial envelope) descriptors [72].
3 Results
3.1 How do trained models compare to randomly initialized models?
Previous work in the deep neural network modeling of mouse visual cortex found that randomly initialized VGG16 predicted neural responses as well as, not slightly better than, a VGG16 trained on ImageNet [9], suggesting that the neural predictivity of the features produced by trained object recognition model are perhaps no better than the features produced by a randomly initialized one. Our results, on the other hand, suggest that the neural predictivity of trained versus randomly initialized models more generally depends on both the particular model being tested and the particular method used produce the mappings between model and brain.
At the level of individual neurons (neural regression), 17 of the 91 model architectures we tested had randomly initialized variants that either matched outperformed their ImageNNet-rrained counterparts. Replicating previous findings, we found these 17 architectures to include VGGI6, as well as all 3 other VGG variants (11, 13 & 19), AlexNet, the DenseNet architectures (121, 169, 201), and architectures. Despite this, a paired t-test of the difference in
almost all of the normalization-Fii scores across all models demonstrates that ImageNet-trained architectures performant than their randomly initialized counterparts (Student's t = 7.74, 1.37e 11, Hedge's " 0.81). At the level of emergent representational similarity (RSA), ImageNet-trained models categorically outperform their randomly initialized counterparts, and by a large margin (Student's t 22.66, p 5.81e 39, Hedge's 2.36)..
still overall more
Taken together, these results strongly affirm that training matters, and that randomly initialized features can only go far in the prediction of meaningful neural variance. Differences between ImageNet-trained and randomly initialized models are shown in Figure 1.
SAP-Ridge Max
RSA Max
0.21
Inceppion Resnet-V2)
Incestion Resnat
ResAve50
WIT 224
lincapionn Resnet-VV
0.4
P26
ECA- WAT.
Score [Pearson's (
0.10
ResNerSe
VOC16
Incection Reanet-V2
ResNes50
Nett
0.2
L
ECANDNeTL
Ne
IGSST
0.12
VI-L3P16-24
0.1
Imagenat
Rendom
Imagenat
Random
Model Weights
Model Weights
Figure 1: Each pair of connected points in these plots is the score of a trained versus randomly- initialized version of the same model architecture. Here, we've labeled 5 of these pairs for demonstra- tion's sake. The key takeaway of these plots is that ImageNet-trained models significantly outperform their randomly initialized counterparts majoritively in the case of the SRP-Ridge Max metric, and categorically in the case of the RSA Max metric.
5

---

3.2 What kinds of architectures best predict rodent visual cortex?
The overall best architecture for predicting mouse visual cortex across both individual neurons (SRP-Ridge) and population-Ievel representation (RSA) was an Inception-ResNNt hybrid (Inception- ResNet-V2), There is small, positive correlation between the depth of model (the number of distinct layers) for both the RSA-Max metric and SRP-Ridge metric (Spearman's 's r = 0.22, p = 0.001 and r = 0.192, p 0.007, respectively), and a small, negative correlation for the total number of trainable parameters in the RSA Max metric (Spearman's r -0.18, p = 0.007). The latter of these most likely driven by the relatively poor performance of parameter-dens architectures like VGG.
Markedly, trends previously noted in macaques [73] fail to materialize here. In particular, models with higher top-1 accuracies on ImageNet do not perform significantly better than models with lower top-1 accuracies. This relative parity is driven in large part it seems by newer models like EfficientNets, which across the board have dominant scores ImageNet, but sometimes middling poor scores in the predictions of rodent visual cortex we've ve tabulated here.
Compared to all other architectures, transformers on average fare slightly worse in the RSA Max metric (Student's & = -3.96, p = 0.004, Hedge's s 9 = -0.113), but moderately better in the SRP-Ridge Max metric (Student's i f = 2.45, 0.023, Hedge's 9 0.633).
Strikingly, transformers and MLP-Mixers boast the largest differences between ImageNet- trained and randomly initialized variants in the SRP-Ridge Max metric, with all pairwise t-tests significant at alpha 0.05 after Bonferroni correction for multiple comparisons. This strongly suggests that the advantage of those randomly initialized variants that matched or outperformed their ImageNet-trained counterparts is an advantage conferred by properties of convolutional architectures (e.g., translation invariance), and not necessarily an advantage shared across random feature spaces writ large. (The rankings of these and other architectures may be found in Figure 8 in the Appendix).
3.3
What kinds of tasks best predict rodent visual cortex?
The overall best Taskonomy encoder across both the RSA and SRP-Ridge Max 2D segmentation (ranking second and first respectively; neurons (SRP-Ridge), 2D tasks (keypoints, autoencoding, inpainting) dominate. At the level of representational similarity (RSA), all 2D tasks but 2D segmentation fall to the bottom of the rankings, and Semantic tasks (object recognition and semantic segmentation) rise to 2nd and 3rd place.
Figure 9 in the Appendix). At the level of individual
This reshifting in rank presents a curious case for interpretation, suggesting most likely that while the representations of individual neurons may be coordinated more by the lower level, less abstract features necessary for performing well on most 2D tasks, the overall neural population codes are coordinated more by the parsing of the visual input into ethologically and spatially relevant units via the segmentation and classification tasks. Notably, the original research from which these PyTorch models were adopted offers an auxiliary data point that may anchor this interpretation more concretely. The top 3 models in our RSA Max metric (2D segmentation. object classification, semantic segmentation) are likewise in the top 5 of a ranking the original researchers produced by pitting the Taskonomy encoders against one another pretrained 'perceptual systems' for reinforcement learning agents learning to navigate a virtual environment (see [54], figure 13 in the appendix). This raises the possibility that the reason these models are optimally predicting the visual neural population code for mice is simply because that code is coordinated in service of navigation.
3.4 How do category-superrised models compare to self-supervised models?
Whether with ResNet50 as their base, or a vision transformer, self-supervised models seem to be verging closer and closer to the predictive power of their category-supervised counterparts. Our most
its predictive self-supervised ResNet50, for example, (a MocoV2 model) effectively matches counterpart in the SRP-Ridge Max metric (with scores of ,182 and ,184, respectively), while slightly outperforming its category-superiieed counterpart in the RSA Max metric (with scores of .422 and ,415, respectively). While this single comparison by no means denotes statistically significant superiority of self-supervised models (which would require training multiple iterations of each). it does begin to provide preliminary evidence for parity.

---

3.5
How well do non-neural network baselines predict rodent visual cortex?
Non-neural network baselines somewhat uniformly fail predict neural activity as accurately as deep net features (though see Section A. of the Appendix for a counterexample). We tested three baselines: 1) a bank of Gabor filters of applied to 8x8 grids of each image; 2) the PCs of the resultant feature matrices (i.e. the Gist descriptors [72]); 3) and the max across 600 iterations of 4096 random Fourier features dimensionality matching that of our SRPs). Ridge regressed with generalized cross-validation, these feature models yield average scores of 0.07, 0.06 and -0.014, respectively. Compared via representational similarity, they yield average scores of 0.20, 0.25 and 0.011.
3.6 How 'deep' are the layers that best predict rodent visual cortex?
Echoing previous results [7, 8], we find across all ImageNet-traid trained architectures, regardless of metric. that the features most predictive of rodent visual cortex are found about a third of the way into the model (though see Section A.5 of the Appendix for some caveats). These early to intermediate visual features go beyond basic edge detection but are far from the highly abstracted representations adjacent to final fully connected layers. Across Taskonomy encoders, 2D & Geometrie tasks yield their best features in earlier layers; 3D & Semantic tasks yield their best features in more intermediate and later layers. Note that these aggregate motifs do not preclude subtler differences across cortical area, which we discuss in the section below.
3.7
Are there differences in model predictions across cortical area?
In this work, address this question from two perspectives: that of hierarchy and that of function.
In primate visual cortex, it common consensus that there exists distinct information processing hierarchy along the ventral visual stream [74-76], with posterior sites like VI and V3 defined by features like oriented edge detectors, and more anterior sites like V4 and IT defined by more complex morphologies. While there continues to be some debate as to whether a similar hierarchy exists in rodent visual cortex, a large body of anatomical, functional and physiological work [77-83] has coalesced around a meaningful hierarchy that consists first of ventral dorsal split after primary visual cortex (VISp), with VISp leading to VISI the ventral stream and VISp leading to VISrl -VSSal VISpm VISam in the dorsal stream. Strikingly, our modeling does seem to provide corresponding evidence for this circuit in the form of a data-driven hierarchy produced purely by taking the median depths of the model layers that best predict the neural activity in each of these cortical areas, and assessing for difference across them. A nonparametric ANOVA shows an overall difference in depth across cortical area to be significant for both our SRP- metric (Friedman's 's
x2 = 34.08. p = 2.29c 06, Kendall's s W = 0.04) and our RSA metric (Friedman's 's x² 37.05, 5.86e - 076, Kendall's W = 0.06). Subsequent pairwise comparisons show many of the differences that underlie this group-level effect to be differences between earlier and later layers of the information processing hierarchy established in the literature. (For further details, see Figure 2.)
Other differences across cortical area that we might expect are differences driven by function. Research into primate visual cortex over the last two decades has unveiled a significant degree of functional organization over and above purely anatomical organization [84-86], with distinct subregions defined large part by their differential activity in response to different kinds of stimuli. To try and replicate this in mouse visual cortex we search for Taskonomic organization, proxy of functional organization wherein distinct neural sites are better or worse predicted by the features from different taskonomy encoders. Curiously, and in contrast to previous findings in human IMRI [55], it seems to be the case that the scores of different Taskonomic clusters are relatively consistent across cortical area; see Figure 3.) This suggests that mouse visual cortex may be more functionally (or Taskonomically) homogenous than primate visual cortex, with anatomical descriptors providing little to cue of functional difference though this seems unlikely given other analyses we' ve performed showing greater similarities of neurons within cortical site than between cortical site (see Section A. for details). Another (more likely) alternative is that the tasks of computer vision are just not so neatly mapped onto the tasks of biological vision in mice.
3.8 How do the predictions compare across RSA and neural regression?
While prior work has addressed this question theoretically [87], it's rarely the case that representational similarity and neural regression are compared directly and empirically. Here, we compare our RSA and SRP-Ridge metric both at the level of overall rankings (taking the max across layers) and at
7

---

SRP--RO0e
1.00
1.00
Model Layer Depth
0.75
0.50
0.50
0.28
00.00
+.299
0.26
0.28
0.25
+ 00.35
0.50
0.38
+.0.86
0.00
0.00
VI8I
ViSp
VISIT
ViGal
V18pm
ViSam
VI8I
ViSp
VIBI
ViSal
Vi8pm
ViSia
Ventral
Dorsal Cortical Areas
Ventral
Dorsal Cortical Areas
Figure In these plots, we demonstrate purely data- driven approach for recapitulating the proposed information processing hierarchy in mouse visual cortex. On the x axis in these plots are the cortical areas our optical physiology sample, arranged roughly according to the schematic proposed by [79] (Figure 3C). On the axis is the average depth from O (the first layer) to 1 (the last layer) of the layer in each model that best predicts a given cortical area. Each point is an individual model. The horizontal brackets are drawn only in the case of significant difference in a Games-Howel (Holm-correeted, pairwise comparison test. While noisy, these plots demonstrate that the deeper into the information processing hierarchy given cortical area is, the deeper the average depth of the maximally correspondent model layers (and more complex the features). Even those cases where
violation of the order suggested by the physiology (as in the case of the difference between in median depths between VISp and VISrl in the SRP-Ridge metric), these differences are insignificant. All significant differences point in the direction proposed by the physiology.
the level of individual layers, the latter of which provides a much more detailed assessment of how different feature spaces map to cortical representation.
In terms of overall rankings, the Spearman rank order correlation between the methods is either 0.56 (p 8.36 10 10-19) or 0.59 (p = 3.17 x 10 10-12), depending on whether you include or exclude the randomly initialized architectures. In terms of layer by layer comparisons, Spearman correlation across distinct combinations of model and cortical area. The average coefficient between the two methods, along with bootstrapped 95% confidence intervals is 0.468 (0.447,0.48) or 0.626 (0.613.0.63) again depending on the inclusion or exclusion of the random models. This suggests significant degree of overlap between the kinds of features that optimally predict the representations of both individual neurons and neural populations, Of course, the averages here do obscure some meaningful subtrends and idiosyncrasies. For details, see Figure
decompose the
3.9
How well are doing overall in predicting mouse visual cortex?
The overall best model any cortical area across either our metries is unsupervised 2D segmenta- tion in anterolateral visual area (VISal), with an RSA Max score of 0.538. The (Spearman-Broww splithalf reliability of the RDM for this area (an effective proxy of its explainable variance) 0.89. This means our most predictive model in any cortical area across any metric is little more than halfway to the noise ceiling.
Of course, it's possible this noise ceiling a bit too strict. Instead of requiring the model to predict the neural data as well as the neural data predicts itself, another possible target which we might recalibrate is the relative performance we would expect if (instead of an artificial neural network) we used the responses of another biological network the model predict neural activity. Inspired by recent work [88], and to better contextualize the scores of our SRP- ERP-Ridge metric, we attempted version of this here. To compute this reference, we proceeded again neuron by neuron using the exact same neural regression method (dimesnonality reduction, and hyperparameters) described in Section 2.5.2, instead of using responses of a deep net layer as the predictors in our ridge regression, we used the responses of the neurons from the same cortical area in all other mice (conspecifics) across the donor sample. Conceptually, this 'intermouse score' represents how well we might do model of given mouse brain were other mouse brains.

---

RSA
VISI
VISp
Vier
ViSal
ViSpm
ViSam
0.4
Score (Pearson's R
0.2
0.0-
SAP-Ricge
ViSpm
0.4
Soore
ImageNat-
Category- Superviiee
Taskonomy
Taskonomy
Geometric
ImagoNat- Selt Sell-Superwfeed
Teskanomy
Figure 3: A mosaic scores across distinct kinds of models in different cortical areas. Each bar the average score across all models of certain kind of training regimen task affinity cluster (denoted by color). Error bars are 95% bootstrapped confidence intervals across models. The dashed lines above each bar are the scores of the best performing models in that category. Within each facet, the averages are arranged in ascending order, with the best performing models on the right and the worst performing the left. For the SRP-Ridge metric, we' included reference point call the 'intermouse score', which leverages the neural activity of other mice to predict the activity in the cells target mouse. The band corresponding to the intermouse score the 95% confidence interval across the scores for all cells in given cortical area. Intermouse scores (as sort of noise ceiling) tend be much more attainable target than the average splithalf' reliability, which ranges from 0.886 to 0.906 across cortical area. There are a few takeaways from this plot: first, we see that there do not seem be major 'taskonomic' dissociations across cortical area. 2D tasks (driven by unsupervised segmentation) dominate in the SRP-Ridge max metric, and despite similarly strong showing from unsupervised segmentation, Semantic tasks are the evident superlative in the RSA Max metric. Second, we see that our models are in some cases on the verge of providing estimates of neural activity as well suited to the prediction of a given biological neuron as the reweighted activity of dozens of other biological neurons sampled from the same species. Finally, we see that (overall) there tend to be very few divergences or punctuated gaps in the average scores across model underscoring that even sometimes substantive differences in the tools of computer vision do not always translate immediately to differences in neural predictivity.
Averaging across both cortical area and model, the average distance (with 95% bootstrapped confi- dence intervals) between the best performing deep net feature spaces and the mean the intermouse scores (expressed in the same units of Pearson's we' ve used heretofore) is 0.0985 [0.0940, 0.103]. Compare this to the same distance computed relative to the splithalf reliability: 728 [0.726, 0.731]. On average, then, while variance relative to the splithalf noise ceiling, they're re verging increasingly close to the predictive threshold suggested the reweighting of biological neurons from the same species. The performance of models relative to the intermouse score may be seen in the lower half of Figure 3.
artificial models capturing only fraction of the total explainable
Discussion
Our intent with this work was to provide preliminary atlas for future ventures into the deep neural network modeling of rodent visual cortex. To this end, we have deliberatcly invested in introspective analyses of the tools used, as well as the curation of deep neural networks we hope will provide informative waypoints. Obviously, the atlas is far from complete. Other model classes like recurrent models [89, 90], equivariant models [91]. and robotic models (e.g. for visual odometry [92]) are promising candidates for inclusion in future benchmarks, and our neural encoding & representational similarity metrics are just two of many variants.
Nevertheless, the results we have presented here indicate that neural recordings from visual brains of mice can (with care and caution) be compared to deep neural networks using many of the same tools we' 've used better characterize the visual brains of monkeys. Having as reference two animal models that occupy very different ecological niches and are separated by tens of million

---

09
2222
0.10
3.10-
(Paanon
0.00
0.2
0.1
0.0
3.25
0.50
0.75
120
0.25
2.55
0.75
1.00
0.00
2.25
2.52
6.75
1230
0.25
0.50
0.75
1.50
Model Loyer Depth (Binned)
Model Layer Depth
Architesture
Conwilttonal
Transtrm
Task Cluster
Samant
Figure 4: Aggregate layer layer comparisons across ImageNet-trained models (left) and Taskonomy models (right). The horizontal axis shows relative model layer depth, from the first to the last layer. Because the models on the left vary significantly in size, we discretize the relative depths into bins of size 0.01 (100 bins). The jagged, semitransparent lines are the mean scores at given depth. (These are particularly jagged for transformers due to the heterogeneous nature of the computational blocks, which engenders large troughs in predictive power). The opaque, smoothed trend lines are the outputs of locally weighted linear regressions (lowess) computed at each point with a span of 2/3 the width of the axis. Plots show that convolutional networks and transformers have similar predictive power at similar depths despite their significant computational differences. (right) Features most predictive of cortical responses across the Taskonomy models vary in their relative depth. The reversal in the ranks of 2D models between RSA Max and SRP Ridge Max seen as vertical shift in the intercept of the curve relating depth to score though note that the slope and shape of the curve remain similar across metrics. 2D models are most predictive in the early layers, but are nevertheless superseded by other tasks in the later layers at the level of representational similarity.
years of evolution makes it far more likely that insights into vision gleaned across both are actually fundamental to perceptual meaning-making and not just some idiosyncratic quirk specific to any one evolutionary trajectory. Primate and rodent vision do differ rather drastically, even in fairly basic ways: mice lack a fovea, have a retina dominated by rods for vision under low light, and spatial acuity less than 20/1000 [93], making their primary visual system more akin to the primate peripheral system - and making it all the more curious that the same models explain decent amounts of variance in both. The differences between the species, it seems, may not be irreconcilable at the level of modeling, but only with future work more carefully controlling for distinct aspects of each organism's 's unique physiology (see Section A.5) can more concrete conclusions of this kind be made.
Beyond considerations distinctive physiology is the indispensable point that perceptual systems should always be considered in service of behavior. It's possible that mice mostly rely vision as sort of broad bandpass filter for lower-frequency. dynamic stimuli the animal can then flee, fight, or further investigate with its whiskers perhaps its most sophisticated sensory organ [94]. Another possibility is that mice use vision to facilitate navigation. The dominance in our Taskonomy results of 2D segmentation, object recognition and semantic segmentation (all tasks that have elsewhere been shown to provide effective, transferable features for the simulation of robotic navigation) provide some evidence for this. Of course, the behavioral roles of rodent vision may very well be manifold. Understanding this plurality in a readily available model species could in the end be key for bridging the gaps that remain between biological and computer vision [95]. The unparalleled access, resolution, and control afforded by rodent neuroimaging have already revolutionized our understanding of the relationship between perceptual representation and behavioral output. Combined with novel methods like the embedding of neural networks in virtual agents [96] in ecologically realistic environments, this kind of data may well provide a testbed for better situating the tasks of computer vision in the broader behavioral context of agentic scene understanding.
In summary, only novel combinations of architecture, task and mapping will help to explain the highly reliable neural variance we' ve yet to explain in our current survey. Already this recombination under way: Shi et al. [97] have created custom CNN designed specifically match (processing stage by processing stage) the anatomy of rodent visual cortex, while Nayebi et al. [88] have combined the power of self-supervised learning with smaller, shallower architectures to more fully account for the ethological realities of rodent behavior and the differences in computational bandwidth that shape and constrain their visual systems. More work of this variety will be necessary to more fully model the rich diversity and fiendish complexity of biological brains at scale even the very smallest ones.
10

---

4.1
Acknowledgements
We thank Martin Schrimpf, Tiago Marques, Jim DiCarlo, well many others the BrainScore team for helpful discussion, feedback, and inspiration. We would also like to thank the Allen Institute founder, Paul G. Allen, for his vision, encouragement, and support.
4.2
Code Availability
More results and code for the replication of our analysis may be found at this GitHub repository: github.com/
(License GPL v2)
4.3
Compute Required
We used single machine with 8 Nvidia RTX 3090 GPUs, 755gb of RAM, and 96 CPUs. GPUs were used only for extracting model activations, and could (without major slowdown) be removed from the analytic pipeline. Dimensionality reduction and regression computations were CPU and RAM intensive. Replicating all of our results would take approximately two weeks on similar machine.
4.4 Ethics Statement
Lest our science forget the life that powers it, we must note that behind the phenomenal dataset provided by the Allen Institute are 256 laboratory mice, each of which was subjected to multiple surgeries, a highly invasive neuroimaging technique and genetic engineering. The moral parameters of this particular praxis of neuroscience are contentious, and not without reason. While we believe centralized, comprehensive and (most importantly) public datasets like those provided by the Allen Institute may actually decrease the total number of laboratory animals required for similar kinds of empirical projects, we acknowledge with solemnity the cost to life required.
4.5 Funding Statement
This work was supported by the Center for Brains, Minds and Machines, NSF STC award 1231216. the MIT CSAIL Systems that Learn Initiative, the CBMM-Siemens Graduate Fellowship, the MIT- IBM Watson AI Lab, the DARPA Artificial Social Intelligence for Successful Teams (ASIST) program, the United States Air Force Research Laboratory and United States Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-19-2-1000 and the Office of Naval Research under Award Number N00014-20-1-2552 and Award Number N00014- 20-1-2643, The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed implied, of the U.S. Govermment. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

---

References
[1] Daniel I.K Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James DiCarlo. Performance-opiimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):811988244 2014.
[2] Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models understand sensory cortex. Nature neuroscience, 19(3):356, 2016.
[3] Santiago Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque responses to natural images. PLoS computational biology, (5(4))e100689) 2019.
[4] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj. Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-score Which artificial neural network for object recognition is most brain-like? bioRxiv preprint, 2018.
[5] Rishi Rajalingham, E.B. Issa, P. Bashivan, K. Kar, K. Schmidt, and J.J. DiCarlo. Large-scale high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of offt-oofthhe-ar deep artificial neural networks. The Journal of Neuroscience, 38(33): 7255-7269. doi: JNEUROSCI. 0388- 18. 2018.
118888-18.200 2018. URL https: htopp:://doo /doi. org 10/0 1523/
[6] Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 364(6439), 2019.
[7] Jianghong Shi, Eric Shea-Brown, and Michael Buice. Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex. In Advances in Neural Information Processing Systems, pages 5765-5775, 2019.
[8] Kasper Vinken and Hans Op de Beeck. Deep neural networks point to mid-level complexity rodent object vision. bioRxiv, 2020.
[9] S. A. Cadena, F. H. Sinz, T. Muhammad, E. Froudarakis, E. Cobos, E. Y. Walker, J. Reimer, M. Bethge, A. Tolias, and A. S. Ecker. How well do deep neural networks trained on object recognition characterize the mouse visual system? NeurIPS Neuro Al Workshop, 2019.
[10] Leslie M Kay. Olfactory coding: random scents make sense. R928-R R929, 2011.
Current Biology, 21 21(2)):
[11] Davide Zoccolan, Nadja Ocrtelt, James J DiCarlo, and David D Cox. rodent model for the study of invariant visual object recognition. Proceedings of the National Academy of Sciences, 106(21):8748 8753, 2009.
[12] Matthew B Broschard, Jangjin Kim, Bradley C Love, and John H Freeman. Category learning rodents using touchscreen-basee tasks. Genes, Brain and Behavior, 20(1):e 1265, 2021.
[13] Jennifer L Hoy, Iryna Yavorska, Michael Wehr, and Cristopher M Niell. Vision drives accurate approach behavior during prey capture in laboratory mice. Current Biology, 26622)3046-3005, 2016.
[14] Saskia EJ de Vries, Jerome A Lecoq, Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse visual cortex. Nature Neuroscience, 23(1):138-155 2020.
[15] Simon Musall, Matthew T Kaufman, Ashley L Juavinett, Steven Gluf, and Anne K Churchland. Single-trial neural dynamics are dominated by richly varied movements. Nature neuroscience, 12(10):1677-1666 2019.
12

---

[16] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. 7achary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. 2019.
[17] Ross Wightman. Pytorch image models. pytorch- mooeed, 2019.
[18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, pages 1097-1105, 2012.
[19] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Conference on Computer Vision and Pattern Recognition, pages 4700-4708, 2017.
[20] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.
[21] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019.
[22] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[24] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Conference on Computer Vision and Putlern Recognition, pages 1492-1500, 2017.
[25] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Conference on Computer Vision and Pattern Recognition, pages
2018
[26] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. 2014. https://arxxv.or//abb// 1409.
[27] Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
2016.
https://aniivarr//b///66..1.1110
[28] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. 2016.
602.0000.
[29] Alexey Bochkovskiy, Chien-Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. CVPR, 2020. https//arrivv.rr/abb///oa 10934.
[30] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alex Alemi. Inception- v4, inception-resnet and the impact of residual connections on learning. AAAI, 2017. http::/aarrii.orraaa/11222......
[31] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019. https//arxii.org/ab///999 1946.
[32] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. CVPR, 2018. https///arivoor/aas/777...44444
13

---

[33] Qilong Wang. Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca-net: Efficient channel attention for deep convolutional neural networks. CVPR, 2020.
https///ari..orraas//1100001111
[34] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search, 2019. https://arxxv.orr/abb//// 18.00343.....4..
[35] Mingxing Tan and Quoc V. Le. Mixconv: Mixed depthwise convolutional kernels. BMVC,
2019. https//arxivvorr/abbs. 19011/909700999
[36] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. CVPR, 2020. https:/axxivoor//bbb000011111
[37] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. Selective kernel networks. CVPR, 2019. https://arxii.or//bs///0000 003.0///6669000666666
[38] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. ECCV, 2018. https:/aaxxiv.or//bb///1111 100.055
[39] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze- networks. CVPR, 2019.
http://aaxii.orr/aa////0000000
[40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Bain- ing Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
https://arsiv.orr/ab//111... 14030.
[41] François Chollet. Xception: Deep learning with depthwise separable convolutions, 2017. https//arxivoorg/as//6600033333
[42] Niv Nayman, Yonathan Aflalo, Asaf Noy, and Lihi Zelnik-Manor. Hardcore-nas: Hard constrained differentiable neural architecture search, 2021. hutps//arxixoor/ab/222222116
[43] Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases, 2021. http:://ariivorr//ab//111....
[44] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. ICCV, 2021. http://aaxivoorgbb22200000099999
[45] Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. CVPR, 2020. https:/arrii.orr//bb//// 19977
[46] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer convnet's clothing for faster inference, 2021. https:/aaxxv.orr//b///110...1
[47] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lu- cic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision, 2021.
https:/arrki.orr//a///155.0...1
[48] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. ICCV, 2019. https:/aarii..rr//////1955.00000
[49] Andrew Brock, Soham De, Samuel L. Smith, and Karen Simonyan. High-performance large-scale image recognition without normalization, 2021. https//aaxii.orrgaas/2111222
[50] Andrew Brock, Soham De, and Samuel L. Smith. Characterizing signal propagation to close the performance gap in unnormalized resnets. ICLR, 2021. http::/aaxiiorraab/2000..0//
14

---

[51] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. CVPR, 2021.
https:/anxiiorrghhs21100000000
[52] Hugo Touvron, Piotr Bojanowski. Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training, 2021.
https//aaxivoorraah2221110.03000
[53] Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3712-3722, 2018.
[54] Alexander Sax, Jeffrey O Zhang. Bradley Emi, Amir Zamir, Silvio Savarese, Leonidas Guibas, and Jitendra Malik. Learning navigate using mid-level visual priors. CoRL, 2019. https//arxxivorr/bbb/11112
[55] Aria Wang, Michael Tarr, and Leila Wehbe. Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. In Advances in Neural Information P Processing Systems, pages 15475-15485, 2019.
[56] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parane non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018.
[57] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607, PMLR, 2020.
[58] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C. Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3), 2021.
[59] Talia Konkle and George A Alvarez. Beyond cutegory-supervvsson:: instance-level contrastive learning models predict human visual system responses to objects. bioRxiv, 2021.
[60] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra. Vissl.
2021.
[61] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning visual features. In Pruceedings of the European Conference on Computer Vision (ECCV), pages 132-149, 2018
[62] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedingy of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[63] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020. https//arrii.org//bb2220......
09882.
[64] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[65] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. ICMI., 2021. hutps:/arriv.or/aa/2211.....
[66] Nikolaus Kriegeskorte, Marieke Mur, Douglas A Ruff, Roozbeh Kiani, Jerzy Bodurka, Hossein Esteky, Keiji Tanaka, and Peter Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 12666)61666111111 121-1141-11 1141, 2008.
15

---

[67] Umut Güglü and Marcel AJ van Gerven. Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. Neurolmage. 145:329-336, 2017.
[68] Matteo Carandini, Jonathan B Demb, Valerio Mante, David Tolhurst, Yang Dan, Bruno A Olshausen, Jack L. Gallant, and Nicole C Rust. Do we know what the early visual system does? Journal of Neuroscience, 25(46):10577-1059) 2005.
[69] N. Kriegeskorte, M. Mur. D.A. Ruff, R. Kiani, J. Bodurka, H. Esteky, and P.A. Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60(6): 121-11261111 141. doi: 10. www.:jeeeroo200...1000 1016/j 2008 10. 043.
URL https::t///eet org/10.
[70] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[71] Ali Rahimi, Benjamin Recht, et al. Random features for large-scale kernel machines. In NIPS, volume 3, page 5. Citeseer, 2007.
[72] Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International journal of computer vision, 42(3):145-175, 2001.
[73] Kamila Maria Jozwik, Martin Schrimpf, Nancy Kanwisher, and James DiCarlo. To find better neural network models of human vision, find better neural network models of primate vision. BioRxiv, page 688390, 2019.
[74] D.J. Felleman and D.C. Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1(1):1-47.
[75] J.J. DiCarlo, D. Zoccolan, and N.C. Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-434, doi: 10. m0016fj.ccroo..21220..00 URL htttp:://oii org/10. 1016/j neuron. 2012. 01.010.
[76] Nicole C Rust and James J DiCarlo. Selectivity and tolerance ("invariance") both increase visual information propagates from cortical area v4 it. Journal of Neuroscience, 30(39): 11978-12992) 2010.
[77] Quanxin Wang, Enquan Gao, and Andreas Burkhalter. Gateways of ventral and dorsal streams in mouse visual cortex. Journal of Neuroscience, (1(5):195-199)) 2011.
[78] Quanxin Wang, Olaf Sporns, and Andreas Burkhalter. Network analysis of corticocortical connections reveals ventral and dorsal processing streams in mouse visual cortex. Journal of Neuroscience, 32( 31:3333443633399 2012.
[79] Rinaldo D D'Souza, Quanxin Wang, Weiqing Ji. Andrew M Meier, Henry Kennedy, Kenneth Knoblauch, and Andreas Burkhalter. Canonical and noncanonical features of the mouse visual cortical hierarchy. bioRxiv, 2020.
[80] Joshua H Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K Ramirez, Hannah Choi, Jennifer A Luviano, et al. Survey of spiking in the mouse visual system reveals functional hierarchy. Nature, 592(7852):86-99) 2021.
[81] Mark L Andermann, Aaron M Kerlin, Demetris K Roumis, Lindsey L Glickfeld, and R Clay Reid. Functional specialization of mouse higher visual cortical areas. Neuron, 72(6): 1025- 1039, 2011.
[82] Joshua H Siegle, Peter Ledochowitsch, Xiaoxuan Jia, Daniel Millman, Gabriel K Ocker, Shiella Caldejon, Linzy Casal, Andy Cho, Daniel J Denman, Séverine Durand, et al. Reconcil- ing functional differences populations of neurons recorded with two-photon imaging and electrophysiology. Elife, 10:c69068, 2021.
16

---

[83] Julie A Harris, Stefan Mihalas, Karla E Hirokawa, Jennifer D Whitesell, Hannah Choi, Amy Bernard, Phillip Bohn, Shiella Caldejon, Linzy Casal, Andrew Cho, et al. Hierarchical organization of cortical and thalamic connectivity. Nature, 575(7781):195-202 2019.
[84] Nancy Kanwisher. Functional specificity in the human brain: window into the functional architecture of the mind. Proceedings of the National Academy of Sciences, 107(25): 1633 1163-
11 11170, 2010.
[85] Kalanit Grill-Spector and Kevin Weiner. The functional architecture of the ventral temporal cortex and its role in categorization. Nature Reviews Neuroscience, 15t 15(8):536-548 2014.
[86] Dwight Kravitz, Kadharbatcha S Saleem, Chris I Baker, and Mortimer Mishkin. A new neural framework for visuospatial processing. Nature Reviews Neuroscience, 12(4):217-230,
2011.
[87] Jörn Diedrichsen and Nikolaus Kriegeskorte. Representational models: A common framework for understanding encoding, pattern-componet, and representational-simiinn analysis. PLoS computational biology, 13(4)ee 2017.
[88] Aran Nayebi, Nathan CL Kong, Chengxu Zhuang, Justin L. Gardner, Anthony M Norcia, and Daniel LK Yamins. Unsupervised models of mouse visual cortex. bioRxiv, 2021.
[89] K. Kar, J. Kubilius, K. Schmidt, E.B. Issa, and J.J. DiCarlo. Evidence that recurrent eir- cuits are critical to the ventral stream's execution of core object recognition behavior. Na- ture Neuroscience, doi: 10. 10.0088/4415922--1--9922-- URL pttttp:://oi... org/10.10 1038/ 941593-019-0399-95
[90] Jonas Kubilius, Martin Schrimpf, Ha Hong, Najib J. Majaj. Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Like Object Recognition with High- Performing Shallow Recurrent ANNs. In Neural Information Processing Systems (NeurIPS), Curran Associates, Inc., 2019.
pages
12796.
[91] Ivan Ustyuzhaninov, Santiago A Cadena, Emmanouil Froudarakis, Paul G Fahey, Edgar Walker, Erick Cobos, Jacob Reimer, Fabian H Sinz, Andreas S Tolias, Matthias Bethge, et al. contatio-invaaiin clustering of neuronal responses in primary visual cortex. In International Conference on Learning Representations, 2019.
[92] Sen Wang, Ronald Clark, Hongkai Wen, and Niki Trigoni. Deepvo: Towards end-to-ent visual odometry with deep recurrent convolutional neural networks. 2017 IEEE International Conference on Roboties and Automation (ICRA), May 2017. doi: 10.1109/cc22221777989226.
URL http: //dx doi. www...org//.... htrg/10.1199/ICCO 1109/ICRAA 2011 7989236.
[93] GT Prusky and RM Douglas. Characterization of mouse cortical spatial vision. Vision research, 1-134134 2004.
[94] Chengxu Zhuang, Jonas Kubilius, Mitra JZ Hartmann, and Daniel L Yamins. Toward goal- driven neural network models for the rodent whisker-trigeminal system. In Advances in Neural Information Processing Systems, pages 2555-2565, 2017.
[95] Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, 10(1):1-7, 2019.
[96] Josh Merel, Diego Aldarondo, Jesse Marshall, Yuval Tassa, Greg Wayne, and Bence Ölveczky. Deep neuroethology virtual rodent. 2019. https//arriv.orr/aas//9/ 1.0451.
[97] Jianghong Shi, Bryan Tripp, Eric Shea-Brown, Stefan Mihalas, and Michael Buice. Con mousenet: A biologically constrained convolutional neural network model for mouse visual cortex. bioRxiv, 2021.
[98] Meyed-Mahdi Khaligh-Razay and Nikolaus Kriegeskorte. Deep supervised, but not unsuper- vised, models may explain it cortical representation. PLoS computational biology, 10(11), 2014.
17

---

[99] David A Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. ral system identification for large populations separating" what" and" where". 2017.
Neu-
https://arriv.orr/aas/// http:/..rrrgaa///1112225355 1711.02633.
[100] Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems, pages 5727-5736, 2018.
[101] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. ICML, 2019, https://arxiv.orrgbbb////// hatps//99000000011
[102] Natalia Y Bilenko and Jack L Gallant. Pyreca: regularized kernel canonical correlation analysis python and its applications to neuroimaging. Frontiers in neuroinformatics, 10:49, 2016.
[103] Ryan M Rifkin and Ross A Lippert. Notes on regularized least squares. 2007.
[104] Leyla Tarhan and Talia Konkle. Reliability-based voxel selection. Neurolmage, 207:116350, 2020.
[105] S.A. Cadena, G.H. Denfield, E.Y. Walker, L.A. Gatys, A.S. Tolias, M. Bethge, and A.S. Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images author summary. PLoS Computational Biology, 1-28. doi: 10. 1/255//-nodeeeee https://doi.org org. 10l..11211/ggoooo 2e31e3.
URL
[106] Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22(12):20602000 2065, 2019.
[107] Tiago Marques, Martin Schrimpf, and James DiCarlo. Multi-scale hierarchical neural network models that bridge from single neurons in the primate primary visual cortex object recognition behavior. bioRxiv, 2021.
[108] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D Cox, and James DiCarlo. Simulating primary visual cortex at the front of cnns improves robustness image perturbations. BioRxiv, 2020.
18


Reviews:
[{"review_id": "xSPFij5RmsQ", "reviewer": "Reviewer_omuc", "summary": "Extracts intermediate features (across all layers) for models pretrained on ImageNet classification (and other computer vision tasks) and tests its modeling performance of mice two-photon neural responses (from 6 cortical areas) using regression and RSA. Features at intermediate depths are better, there is a slight hierarchy between neural network depth and cortical area, and there seems to be no relation between the pretraining task and the cortical area the trained network models best.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\nThough techniques used are standard, the extent of the experiments and the animal model (mice) is novel and significant.\n\nQuality\nExperiments are sound and enough information is provided to allow replicability.\n\nThe main drawback of the paper in its current form is the figures, though they contain information to draw the conclusions drawn in the paper, it is not easy to see. Fig 1, for instance shows performance metrics of pretrained vs randomly initialized models but the models have been sampled (from the total of 110), the red bar goes below the blue bar if the value is lower and the error bars overlap so it is hard to see where they start or end. Perhaps a two-sided violin plot per model (instead of the bars) could show the distribution for blue and red better. Or maybe a scatter plot (x axis trained, y axis random) where each point is a model could show all models and more easily show which version is better (in comparison to the diagonal), error bars could also be included per point or perhaps omitted for clarity, and important points (i.e., models) can be labelled (with colors or marker styles or text above the point). All results from Sec 3.3 (other than the best network being Inception-ResnetV2) are close to impossible to see in Fig. 1. In Fig. 2, perhaps adding a table with the averages per task group will help show the trends and differences between the two metrics (as done for figure 3). Fig. 3 could have also been a table (or perhaps two), it will be easier to read the numbers and gauge the difference that way (it is also currently unclear what is your assumed cortical hierarchy and task hierarchy). Given the amount of work put into obtaining these good results, spending time finding a better way to convey them might be worthy and will surely make the paper more attractive/citable.\n\nI do not think the method comparison with BrainScore regression (Sec 3.1) adds to this paper (and it may take away from it). I understand this was asked of you before but it does not actually show that results from this paper (in mice) will hold if using the BrainScore regression method, it only shows that both methods give similar regression results on the same data (which is expected given that the changes are minor). That the method transfers to primate data (with the very different visual system and recording methods) might be interesting but I presume so will many other methods (including BrainScore’s) and regression performance or transferability across animal models is not the focus of this paper so it feels disconnected. It may be worth considering bumping it to the end of the result section (as an extra transfer result) or even to the appendix.\n\nClarity\nWell written, though sometimes wordy; for instance, the main results of the paper at the end of the introduction could be a lot bolder/more direct even if obviating some details (for instance, 1 → “Training matters: features from trained models predict/model neural responses better than features from randomly initialized models.” or phrases like “features learned in service of some task” → “learned features” or “there may be a slight upwards gradient in complexity” → “complexity increases”).  Of course, the work and results still hold but I think ML readers will appreciate a less hedged writing style.\n\nFor reproducibility, can you add a sentence about how images were modified to work as ImageNet images, how do you deal with the aspect ratio difference (if any), sizing and color. \n\nPerhaps it is worth providing the proportion of cells (out of the 6.6K) from each cortical area (and the name of the areas). A single sentence should do. “Dataset areas are divided as V1 (50%), AM (10%),  PM(5%), ….” (could be in supplementary).\n\nFor the Taskonomy models, you say “we choose to extract representations across all layers”(l.90), to confirm, this means through all encoder+decoder layers (i.e., different tasks may have different depths)?\n\nI am confused about your ridge regression, do you cross-validate or not? because it says you use “a lambda penalty of 1.0” (l.132) but then in the same line you say you “monopolize generalized cross-validation” (?), I presume you do use LOOCV (if so, I would drop the sentence about using a lambda of 1 and perhaps provide the lambda grid you searched over).\n\nAlso, to confirm, once you cross-validate the penalty, you use all the data to fit the final linear regressor (with the chosen lambda), right? if so, the regression correlation may be optimistic (i.e., it reports cross-validation results). This is ok as long as what’s reported is clear.\n\nThe interpretation in Sec 3.4 seems premature given that it is only truly one task (Semantic segmentation) that moves up from SRP to RSA (and at the same time, scene classification moves down from SRP to RSA). It is “only” a hypothesis so it is up to the authors to decide whether they are ok with it.\n\nChanges in performance in Sec. 3.6 are so small I do not think one can even say they point in any direction (l.251).\n\nResults from Sec 3.8 are neat, I thought the lack of functional specialization across different cortical areas in mice was well known; maybe you can find some references for it and frame your results as further confirmation. I would also reference some of the discussions about cortical area hierarchies. \n\nIn l. 296, pointing regression scores are lower than RSA scores seems unimportant. They are both correlation scores but in very different spaces.\n\nGreat discussion.\n\nOther minor remarks\n\n- the word “later” in l.33 is odd.\n- you end up using 6619 cells (not 59K as claimed in the abstract)\n- in l. 132, it will be better to name scikit-learn [52] rather than just the citation or “implemented with a popular machine learning toolkit [57]” if you don’t want to name it.\n- in l. 152, drop “In this work,”\n- i would reference the figures before the written results (i.e., closer to the top of each 3.x subsection rather than at the very end), for instance, the line in 195-196 could go up to l.184\n- l.219 ‘writ large”?\n- Reference to fig 4 (l.257) appears before reference to fig 3 (l.281), fig 3 should probably be fig 4 and viceversa.\n- For 3.x sections, perhaps consider putting result directly in the titles, for instance, section 3.5: \"how well do non-neural network baselines predict rodent visual cortex\" -> \"non-neural network baselines fail to predict rodent visual cortex”\n- Showing the neural RDMs for different areas and perhaps some for a selected layer in a trained model could be interesting (perhaps in appendix)\n\nSignificance\nIt fills a research gap in visual neuroscience and will be a good reference for future studies.\n\nOverall\nThis is a good paper that is on the verge of acceptance. With some improvements I would have no problem accepting it.\n\n\nUpdate:\nI have bumped my score from 5 to 6 after the authors addressed some of my concerns and those of other reviewers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Biological datasets are always limited, though not directly pointed out, it is understood. Discussion about societal impact of a better understanding of the visual system is not directly provided, though I would think it may be unnecessary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "ptd7w7LyD3N", "reviewer": "Reviewer_5VYT", "summary": "Several other reviewers have expressed that they find the current scientific contributions of the paper significant enough, so I will increase my score to a 6, even though the authors did not quite address my major concern about the novelty and utility of the proposed neural regression (i.e. what is the empirical benefit of neural regression over just ridge regression? if that is not understood, the authors should disclose that and not put so much emphasis on the importance of the regression method (in the title, in the abstract, and in the main text)).  \n\nI believe that the paper can benefit from reducing the amount of time spent discussing the neural regression method and the RSA vs encoding model comparisons, and more time focusing on the actually novel and significant part which is the comparison of the findings in mice to the previously established findings in monkeys and humans. And just to clarify, I think that only discussing what the findings are in mice is not enough on its own, because all the analyses that the authors carry out have been previously done in other organisms. So to actually have a significant contribution, I think it's important to emphasize the comparisons of the results in mice to the previously established results in other organisms.\n\n=============== post-rebuttal =================\n\nThe manuscript presents an exploratory empirical study of the relationships between the mouse visual system and representations extracted from computer vision neural network models. The authors investigate representations from a large number of computer vision models, as well as representations from the same models but when the models are initialized randomly. They also explore two ways of relating the mouse recordings to the representations from the computer vision models: encoding models and representational similarity analysis. The authors additionally propose what they claim is a new encoding model methodology, though I remain unconvinced of its utility.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript was easy to read and provides a broad exploration of an interesting question. I do however have several concerns about this work. I've summarized these below.\n\n1. Methodological concerns. The authors claim to propose a \"novel, highly optimized neural regression method\" for encoding models. However I'm unclear about what parts of the proposed method (sparse random projection + ridge regression via generalized cross validation) the authors are claiming to be novel, and I'm also unclear about which of those parts are actually leading to a boost in prediction performance. The authors claim that ridge regression is a less popular encoding model method, though it is in fact the default method in encoding models in the language literature (Wehbe et al 2014, Huth et al. 2016, Jain and Huth 2018, Toneva and Wehbe 2019,...). The authors use leave one out cross validation which is indeed an uncommon choice, but it's also the default setting on the RidgeCV method in sklearn, so it's unclear what is \"highly optimized\" about this method. Also, it would be helpful if the authors could provide some clarification (both in the rebuttal and in the paper) about exactly what samples are used to train the encoding models (both in the mouse and the monkey experiments). Were separate encoding models built for different individuals? If separate encoding models were built, it's not clear how the predictions or predictions performance is aggregated across individuals. How were the repetitions of different image stimuli handled in the leave one out CV setting? I imagine that leaving out the data corresponding to one image while keeping data that corresponds to other repetitions in the training set can lead to some unfair underestimate of the generalization error. It would be interesting to compare the zero-shot performance of the leave-one-out proposed method to the zero-shot performance of the default Brain Score encoding model (i.e. zero-shot meaning that all repetitions of a test image are removed from the training set).\n\n2. Questions about some of the interpretations. \n- L214-220: The authors interpret the result that transformer-based architectures show the largest differences in prediction performance between the trained and randomly initialized computer vision models as evidence that some of the better performing randomly initialized models may be due to an underlying convolutional architecture. Another possibility is that the randomly initialized models predict the neural recordings well across architectures, but the trained transformer-based architectures are much better at predicting the neural recordings. This possibility is consistent with a large difference in prediction performance between the trained and randomly initialized transformer-based models. To disentangle these, it would be helpful if the authors compared the prediction performances across different randomly initialized architectures with a statistical test.\n- The authors compare the results from RSA with those of encoding models and state that \"with more probe images, we expect that the parametric linear mapping would be more competitive with the nonparameteric distances of representational similarity\". It's not clear to me that the correlations from encoding models being lower than those from RSA means that the encoding model results are less \"competitive\". The results from an encoding model are from generalizing to a previously unseen datapoint, and the RSA results are not. It would be helpful if the authors provide an estimate of the chance performance in both the encoding model and the RSA setting to show that one is not inherently easier than the other.\n\n3. Some lack of clarity in the methodology and experimental settings. The manuscript has a clear exposition for the most part but there are some methodologically details that need to be better explained. Some examples:\n- The questions that I brought up in the previous point \n- L92-98: what is being clustered and how? Are these clusters from the original Taskonomy paper or did the authors actually obtain these \"purely data-driven clusters\". \n- L111-120: for a reader who is unfamiliar with RSA, the description in these lines may not be easy to parse. For example \"We compute RDMs by correlating the activations of all neurons in a given neural site to the 118 images in the stimulus set\" sounds like the authors are correlating neural activity to images, instead of computing pairwise correlations across the neural activations. Here it is again not clear how separate subjects are handled.\n- L185-196: are these results using the mouse or the monkey data?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "B9O79CO6kjw", "reviewer": "Reviewer_ezgN", "summary": "This paper proposes a benchmark for the comparison of neural network models of vision to rodent visual cortex. The benchmark compares models with various architectures (including various CNNs, transformers, and other architectures) and one model trained different tasks to a large database of neural recordings from mouse visual cortex. It relies on newly proposed metrics that are validated on a previous benchmark. These metrics evaluate both the fit to individual neurons and the entire population of neurons. Previous results are replicated and explained in the context of a large scale study. The paper provides analysis of the results as well as predictions on the function and structure of the rodent visual cortex areas.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a benchmark, new evaluation metrics, a large scale evaluation of vision models trained on various tasks as models rodent visual cortex, many analyses of the results and interesting interpretations of these results. \n\nstrengths:\n- The benchmark is extensive, it explores a large number of models and tasks when compared to previous work in the field.\n- They propose new metrics for neural regression which are faster and computationally inexpensive. Furthermore, the two metrics evaluate different aspects of the neural code (fitting individual neuron activity and fitting neural populations).\n- Previous results on randomly initialized models are replicated and extended to other architectures. Which offers a new perspective on the inductive bias contributes to the scores.\n- The layer-by-layer and area-by-area analyses offer many interesting hypotheses and predictions on rodent visual cortex.\n- The paper is clear, thorough, well detailed and well explained. The code is provided for reproducing the results.\n- The authors discuss the paper's limitations, provide detailed descriptions of the computational resources they used and acknowledge the ethical concerns behind their work. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors addressed some of the papers limitations. However there are other issues to address, I have split them into major and minor weaknesses:\n\nMajor\n- It's important for analysis papers to have a clear objective beyond fitting neural data. The paper lacks hypotheses and predictions on brain function that constrain the space of models and tasks, allow for more in-depth analysis which, in turn, gives solid conclusions.\n- In brainscore's benchmark, data was recorded from monkeys tasked with object recognition, so evaluating only models trained on classification makes sense. To my understanding, the mice that are recorded in the dataset were only passively viewing images. Thus, it might be more reasonable to, for example, shift the analysis towards unsupervised/self-supervised objective functions and test models with different hypotheses about perception in that respect. \n- Inline with this remark, not all tasks included in the benchmark are behaviorally relevant for rodents. This might be why no clear conclusions can be deduced from the analysis of the task clusters. Another possible reason could be that some visual areas are responsible for detecting motion, which is why static stimuli do not allow for their differentiation from other areas. \n- In the absence of a guiding hypothesis and as mentioned by the authors in the discussion, the benchmark should include a wider range of architecture types and objective functions. Especially biologically plausible models of vision, such as recurrent models. Objective functions include, but are not restricted, self-supervised objectives and generative models of vision. \n\nMinor\n- Calling a neural regression method state of the art is inaccurate and might cause confusion. A neural network architecture is state of the art if it performs better than previous architectures whereas the higher pearson's R on brainscore only highlights the new neural regression method's ability to extract more correlation from the data.\n- The paper explores differences of neural regression on two axis; task and architecture. However, scores might change for different task-architecture combinations. It is definitely a formidable challenge to perform such an analysis at a large scale. But it is possible for a subset of architectures that do not share the same inductive bias (like convolutions), and using only behaviorally relevant tasks. \n- Results in 3.6 which claim the slight importance of architecture over task are far from conclusive because many biologically relevant tasks are not included and the space of architectures is not fully covered as mentioned previously.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "AY_X9H9FvyB", "reviewer": "Reviewer_HWeU", "summary": "The authors present a large-scale empirical study comparing artificial neural networks against mouse brain recordings using correlation-based metrics. They utilize a variety of publicly available neural networks as well as a large brain data corpus provided by the Allen Brain Institute. Their metrics are sampled from the field and rely on linear correlation based comparisons. They do also provide a variant metric that is more performant than the recently proposed BrainScore. They demonstrate that an artificial network’s training task and architecture individually influence metric performance. They also replicate some analyses previously done with primate data to emphasize similarities and differences.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths:*\nThe study is timely and comprehensive. It provides a sufficient level of rigor to establish itself as a viable benchmark for relating mouse visual neuron recordings to artificial network activations. Overall it is well written with easy-to-parse plots. The appendix also provides a concise and convincing defense for the methods chosen in the main manuscript.\n\n*Weaknesses:*\nThe claimed contribution of the work is to better understand what does/doesn't matter when comparing deep neural network activations against mouse brain activations. The abstract offers some argument for why we would want to do such a comparison in the first place, but in general the paper is pretty well fixated on the opening question of how well the networks fare as models of mouse cortex instead of what does comparing network responses to mouse cortex tell us. I realize that this work is building on an ongoing research thread from a couple of labs, who all provide their own motivations. Nonetheless, I still find myself wondering what new information the results gives us regarding biological & artificial networks. Maybe their emphasis on equivalences between primate and mouse is key? My recommendation is for them to try to make this point of *why the comparison matters in the first place* more salient in the intro or discussion.\n\nThe references need to be reviewed carefully. There are many [10, 26-28, 30-34, 36-42, 44-48] that do not include a publication venue. Others [24, 51, 74] are listed as arxiv preprints, but have since been published at peer-reviewed venues. Finally, [53] and [56] appear to be the same citation.\n\n*Additional minor comments:*\n\n1) Assuming you have this sort of resolution in your dataset, how well do the non-neural network baselines & neural networks perform on layer 4 primary cortex data? It has been suggested that this region has a dearth of “complex cells” as compared to primate data {1}. So I would imagine that the Gabor bank in particular will perform comparatively well in this area. I am asking because the Gabor filter bank model is still widely accepted as the standard among “interpretable” methods {2}, and given your discussion at the end about the meager overall performance it would be interesting to see if this focused area has a higher ceiling. Maybe this question is answered with the cre line split, but I’m not familiar with the different lines.\n\n2) While I realize this is far from the authors’ problem to solve, I take issue with the phrasing of correlational scores as a “geometric” analysis. For example, one of a few quotes from the paper: “The resultant coefficient constitutes the score for how well a given model layer predicts the representational geometry of a given cortical area.” I found no mention of geometry in [56]. A quick internet search got me to {3}, which says “The dissimilarities can be interpreted as distances in the multivariate response space. The RDM thus describes the geometry of the arrangement of patterns in this space.” Maybe that’s where they got the term from? I find that it is a stretch to relate correlation values to distances and then further to a geometric description. My recommendation to the authors: use the term if you want, but I would suggest that you at least specify exactly what you mean by it. You should probably also differentiate your definition from others, e.g. that used by {4}, {5}, and {6}.\n\n3) It appears to me that the overall differentiation among the models is quite small. This could lead to a possible alternative conclusion that none of these differentiations (task, architecture, depth, etc) really matter. Or, alternatively, that the metrics themselves are not useful for this type of differentiation. The noted winners/losers among architectures & tasks are certainly interesting observations, but I think it would help balance the paper if the authors commented on the general lack of variation in scores reported from figures 1 and 2. This would go well next to their already helpful discussion on the lack of total performance at the end of the paper.\n\n\n{1} https://www.jneurosci.org/content/jneuro/28/30/7520.full.pdf\n\n{2} https://www.annualreviews.org/doi/abs/10.1146/annurev-vision-091718-014731\n\n{3} https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003553\n\n{4} https://www.nature.com/articles/s41586-019-1346-5\n\n{5} https://elifesciences.org/articles/44526\n\n{6} https://www.sciencedirect.com/science/article/pii/S0042698915003600", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, although see my comment #3 above.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a large-scale / systematic study fitting many modern deep nets to mouse calcium data from the Allen institute. This paper received two accepts, one borderline accept and one borderline reject and was discussed on the forum. There was a bit of a discussion regarding the actual contributions of this paper. There was some agreement between the reviewers that the authors oversold the claim that they introduced a novel regression methods and the authors agreed to remove or significantly downplay this claim.There was also some discussion about the lack of clear hypotheses being tested or significant results contributing to our understanding of the visual cortex. The study is an impressive feat from the engineering perspective but at the end of the day from the dozens of models and results presented the fit are relatively low (correlation coefficients r<0.2 which means r^2<0.04). This is to be expected given how noisy calcium data are and given the small number of stimuli available (~100). However, given these scores it is possible that we are essentially fitting noise and indeed  the models are all performing very similarly so there is no really salient results. With that said, the positive reviewers also emphasized that the present study seem to be at least partially addressing a controversy in the field (because a previous study had found that random models did about as well as trained ones). Here at least the authors report the accuracy of trained models that significantly outperform untrained ones. The code is also made freely available and positive reviewers see that as a "launching point" for additional work on mouse (as opposed to primate cortex). All in all, this really is a borderline submission. Given the issues highlighted above, this paper could be accepted if space permits.
Decision: Accept (Poster)