Neural Regression, Representational Similarity, Model Zoology & Neural Taskonomy at Scale in Rodent Visual Cortex

How well do deep neural networks fare as models of mouse visual cortex ? A majority of research to date suggests results far more mixed than those produced in the modeling of primate visual cortex . Here , we perform a large-scale benchmarking of dozens of deep neural network models in mouse visual cortex with both representational similarity analysis and neural regression . Using the Allen Brain Observatory ’ s 2-photon calcium-imaging dataset of activity in over 6,000 reliable rodent visual cortical neurons recorded in response to natural scenes , we replicate previous findings and resolve previous discrepancies , ultimately demonstrating that modern neural networks can in fact be used to explain activity in the mouse visual cortex to a more reasonable degree than previously suggested . Using our benchmark as an atlas , we offer preliminary answers to overarching questions about levels of analysis ( e.g . do models that better predict the representations of individual neurons also predict representational similarity across neural populations ? ) ; questions about the properties of models that best predict the visual system overall ( e.g . is convolution or category-supervision necessary to better predict neural activity ? ) ; and questions about the mapping between biological and artificial representations ( e.g . does the information processing hierarchy in deep nets match the anatomical hierarchy of mouse visual cortex ? ) . Along the way , we catalogue a number of models ( including vision transformers , MLP-Mixers , normalization free networks , Taskonomy encoders and self-supervised models ) outside the traditional circuit of convolutional object recognition . Taken together , our results provide a reference point for future ventures in the deep neural network modeling of mouse visual cortex , hinting at novel combinations of mapping method , architecture , and task to more fully characterize the computational motifs of visual representation in a species so central to neuroscience , but with a perceptual physiology and ecology markedly different from the ones we study in primates . * Correspondence : conwell @ g.harvard.edu ; Project Website : github.com/ColinConwell/DeepMouseTrap 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . To date , the most successful models of biological visual cortex are object-recognizing deep neural networks applied to the prediction of neural activity in primate visual cortex [ 1–5 ] . Corresponding to the biology not only at the level of individual layers , but across the feature hierarchy , these models are so powerful they can now effectively be used as neural controllers , synthesizing stimuli that drive neural activity far beyond the range evoked by any handmade experimental stimulus [ 6 ] . The correspondence of these same models to mouse visual cortex , on the other hand , has proven a bit more tenuous [ 7 , 8 ] , with a recent finding even suggesting that randomly initialized networks are as predictive of rodent visual cortical activity as trained ones [ 9 ] . Often implicit in interpretations of these results is the notion that the visual milieu and machinery of mice is simply different – something characterized more , perhaps , by brute force predator avoidance and the ‘ flexible random associations ’ thought to define senses like olfaction [ 10 ] than by the sophisticated active sampling and representational compositionality enabled by primate central vision . And yet , mice do recognize objects [ 11 , 12 ] – and do engage in other sophisticated visual behaviors [ 13 ] that suggest they must have visual solutions that at least functionally approximate the kinds of solutions learned by modern computer vision algorithms . If these models perform well in monkeys , but not in mice , are we overfitting to an artifact ? Are the object recognition capabilities of mice simply the byproduct of a representational competence learned through other ( even more behaviorally relevant ) tasks ? Have mice perhaps converged on solutions to visual problems that fundamentally differ from the solutions that undergird the emergent similarity between monkeys and machines ? To even begin to answer these questions , we need substantially more comprehensive modeling statistics than we currently have . Our main goal in this work was to provide exactly that – to re-examine at large scale the state of neural network modeling in the visual cortices of mice , using many thousands of neurons , over 110 distinct neural network models , and two methods of mapping models to brain . We summarize the statistics from our benchmarking survey in five main results : 1 . Training matters . The randomly initialized variants of some convolutional architectures fare well when predicting individual neural responses , but representational similarity is always better captured by features learned in service of some task . ( Segmentation seems best . ) 2 . Features of intermediate complexity dominate in the prediction of all cortical sites , but both our mapping methods do demonstrate an upwards gradient in complexity from primary visual cortex onwards that roughly matches the information processing hierarchy proposed elsewhere in the rodent neurophysiology literature . 3 . Taskonomic tools that have previously been shown to approximate functional organization in primates fail to strongly differentiate anatomical regions in mice , with the same kinds of tasks dominant across multiple , distinct neural sites . 4 . When aggregated in similar ways , representational similarity and neural regression methods capture similar trends in the kinds of feature spaces that best predict the biology . 5 . While still far from the overall noise ceiling for this highly reliable neural data , a variety of the artificial deep net models in our survey make predictions only slightly less accurate than ‘ biological conspecific models ’ composed of the neurons from other mice . 2 Methods . 2.1 Neural Dataset . For neural data , we use the Allen Brain Observatory Visual Coding1 dataset [ 14 ] collected with twophoton calcium-imaging from the visual cortex of 256 awake adult transgenic mice and consisting of approximately 59,610 unique , individual neurons . Calcium-imaging fluorescence patterns are preprocessed and deconvolved by the Allen Institute2 . The neurons sampled include neurons from 6 visual cortical areas at 4 cortical depths across 12 genetic cre lines . The visual experiments recorded activity for both artificial images ( e.g. , diffraction gratings ) and 118 natural scenes . We analyze only 1Available with a non-commercial license under the Allen Institute terms of use : http : //www.alleninstitute.org/legal/terms-use/ 2More details are available in the whitepapers released with the observatory data : http : //observatory.brain-map.org/visualcoding/transgenic the latter to ensure comparable inputs to what is typically used in the training of deep nets . Each natural scene is displayed 50 times over the course of an assay . To ensure an optimal signal to noise ratio , we perform a significant amount of subsetting on the full neural population , beginning by subsetting only excitatory neurons . Recent analyses suggest neural activity throughout mouse visual cortex is often impacted by extraneous , external body movements [ 15 ] . For this reason , we subsequently filter out any neurons whose peak responses to the presentation of natural scene images are significantly modulated by the mouse ’ s running speed , using an ANOVA metric provided by the Allen Institute . We further subselect neurons by assessing their split-half reliability across trials ( with each split-half constituting 25 of 50 presentations for each image ) , keeping only those neurons exhibiting 0.8 reliability and above . This thresholding still leaves 6619 neurons for analysis , is in line with prior work on primates , and supports , for example , the construction of cortical representational dissimilarity matrices ( RDMs ) with split-half reliabilities as high as 0.93 . ( More details on the relationship between our metrics and neural reliability , including visualizations of some of our results across many degrees of thresholding , can be found in A.4 of the Appendix . ) 2.2 Model Zoology . To explore the influence of model architecture on predictive performance , we use 26 model architectures from the Torchvision ( PyTorch ) model zoo [ 16 ] and 65 model architectures from the Timm [ 17 ] model zoo [ 18–52 ] . These models include convolutional networks , vision transformers , normalization-free networks and MLP-Mixer models . For each of these models , we extract the features from one trained and one randomly initialized variant ( using whatever initialization scheme the model authors deemed best ) so as to better disentangle what training on object recognition affords us in terms of predictive power . 2.3 Neural Taskonomy . Model zoology provides decent perspective on the computations related to object recognition , but the responsibilities of the visual cortex ( no matter the species ) extend far beyond identifying the category of an object . To probe a wider range of tasks , we turn to Taskonomy : a single architecture trained on 24 different common computer vision tasks [ 53 ] , ranging from autoencoding to edge detection . The model weights we use are from updated PyTorch implementations of the original Tensorflow models [ 54 ] . Key to the engineering of Taskonomy is the use of an encoder-decoder design in which only the construction of the decoder varies across tasks . While recent analyses using a similar approach in human visual cortex with fMRI data [ 55 ] have tended to focus only on the latent space of each task ’ s encoder , we extract representations across all layers , better situating Taskonomy within the same empirical paradigm that has so far defined the modeling of object recognition in the primate brain . For further clarity , we cluster the 24 tasks according to their ‘ Taskonomic ’ category — a total of 5 clusters ( 2D , 3D , semantic , geometric or other ) that we further collapse into 4 clusters ( lumping the only member of the ‘ other ’ category — a denoising autoencoder — in with its closest cousin — a vanilla autoencoder in the ‘ 2D ’ category ) . These purely data-driven clusters are derived from estimates of how effectively a set of features learned for one task transfer to ( or boost the performance in ) another task [ 53 ] . Use of the Taskonomy models provides a unique opportunity to test variance in training regimes without the confound of simultaneous changes in architecture . 2.4 Self-Supervised Models . Full category supervision , while robust in its ability to build representations that transfer well to a variety of tasks , suffers in its neuroscientific relevance as an ethologically plausible mode of learning . Recently , self-supervised models have begun to provide viable alternatives to the representations learned by category-supervised models in both computer vision [ 56 , 57 ] and neural mapping [ 58 , 59 ] . Here , we assess 22 self-supervision models from the VISSL model zoo [ 60 ] , ranging from earlier iterations ( e.g . DeepCluster [ 61 ] ) to modern contrastive learning algorithms ( e.g . BarlowTwins and Dino [ 62–65 ] ) . We use these models to assess whether category-supervision , however powerful it is in predicting neural activity , might eventually be supplanted by these more realistic alternatives . 14 of these models have as their base architecture a standard ResNet50 ; 8 are built atop vision transformers . 2.5 Comparing Representations across Biological & Artificial Networks . Two methods predominate in the comparison of neural recordings to deep neural networks : at the most abstract level , one of these compares representational geometries computed across the activations of many individual neurons [ 66 , 67 ] ; the other attempts to predict the activity of individual neurons directly [ 67 , 68 ] . Both of these techniques are grounded in the use of image-computable models and a shared stimulus set , but differ in the types of transformation applied to the neural activity generated by those stimuli . Given the difference in both target ( neural populations versus individual neurons ) and transforms ( correlation matrices versus dimensionality reduction ) we attempt a variant of each type of analysis here , comparing the two directly on the exact same neural data , with the same models and the same stimulus set , and in a granular , layer-by-layer fashion . ( A more comprehensive review of neural mapping methods is provided in Section A.2 of the Appendix . )

Summary: Several other reviewers have expressed that they find the current scientific contributions of the paper significant enough, so I will increase my score to a 6, even though the authors did not quite address my major concern about the novelty and utility of the proposed neural regression (i.e. what is the empirical benefit of neural regression over just ridge regression? if that is not understood, the authors should disclose that and not put so much emphasis on the importance of the regression method (in the title, in the abstract, and in the main text)).    I believe that the paper can benefit from reducing the amount of time spent discussing the neural regression method and the RSA vs encoding model comparisons, and more time focusing on the actually novel and significant part which is the comparison of the findings in mice to the previously established findings in monkeys and humans. And just to clarify, I think that only discussing what the findings are in mice is not enough on its own, because all the analyses that the authors carry out have been previously done in other organisms. So to actually have a significant contribution, I think it's important to emphasize the comparisons of the results in mice to the previously established results in other organisms.  =============== post-rebuttal =================  The manuscript presents an exploratory empirical study of the relationships between the mouse visual system and representations extracted from computer vision neural network models. The authors investigate representations from a large number of computer vision models, as well as representations from the same models but when the models are initialized randomly. They also explore two ways of relating the mouse recordings to the representations from the computer vision models: encoding models and representational similarity analysis. The authors additionally propose what they claim is a new encoding model methodology, though I remain unconvinced of its utility.