Wiring Up Vision: Minimizing Supervised Synaptic Updates Needed to Produce a Primate Ventral Stream
Authors: Franziska Geiger
Martin Schrimpf
Tiago Marques
James J. DiCarlo
Venue: ICLR 2022 Spotlight (2022)
Keywords: computational neuroscience
primate visual ventral stream
convolutional neural networks
biologically plausible learning

Abstract: After training on large datasets, certain deep neural networks are surprisingly good models of the neural mechanisms of adult primate visual object recognition. Nevertheless, these models are considered poor models of the development of the visual system because they posit millions of sequential, precisely coordinated synaptic updates, each based on a labeled image.  While ongoing research is pursuing the use of unsupervised proxies for labels, we here explore a complementary strategy of reducing the required number of supervised synaptic updates to produce an adult-like ventral visual stream (as judged by the match to V1, V2, V4, IT, and behavior). Such models might require less precise machinery and energy expenditure to coordinate these updates and would thus move us closer to viable neuroscientific hypotheses about how the visual system wires itself up. Relative to standard model training on labeled images in ImageNet, we here demonstrate that the total number of supervised weight updates can be substantially reduced using three complementary strategies: First, we find that only 2% of supervised updates (epochs and images) are needed to achieve 80% of the match to adult ventral stream. Specifically, training benefits predictions of higher visual cortex the most whereas early visual cortex predictions only improve marginally over the course of training. Second, by improving the random distribution of synaptic connectivity, we find that 54% of the brain match can already be achieved “at birth" (i.e. no training at all). Third, we find that, by training only 5% of model synapses, we can still achieve nearly 80% of the match to the ventral stream. This approach further improves on ImageNet performance over previous attempts in computer vision of minimizing trained components without substantially increasing the relative number of trained parameters. These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's updates based on visual experience).

Published as a conference paper at ICLR 2022
WIRING UP VISION:
MINIMIZING SUPERVISED SYNAPTIC UPDATES NEEDED TO PRODUCE A PRIMATE VENTRAL STREAM
Franziska Geiger" 1,2,44 Martin Schrimpf" 1,156, Tiago Marques M55 1566, and James J. DiCarlo 1.666
McGovern Institute for Brain Research, MIT
Universsity of Augsburg
LLdwig Maximilian University "Technical University of Munich Department of Brain and Cognitive Sciences, MIT
CCenter for Brains, Minds and Machines, MIT joint first authors
ABSTRACT
After training on large datasets, certain deep neural networks are surprisingly good models of the neural mechanisms of adult primate visual object recognition. Nevertheless, these models are considered poor models of the development of the visual system because they posit millions of sequential, precisely coordinated synaptic updates, each based a labeled image. While ongoing research is pur- suing the use of unsupervised proxies for labels, we here explore complementary strategy of reducing the required number of supervised synaptic updates to pro- duce an adult-like ventral visual stream (as judged by the match to VI, V2, V4, IT, and behavior). Such models might require less precise machinery and energy expenditure to coordinate these updates and would thus move closer to viable neuroscientific hypotheses about how the visual system wires itself up. Relative standard model training on labeled images in ImageNet, we here demonstrate that the total number of supervised weight updates can be substantially reduced using three complementary strategies: First, we find that only 2% of supervised updates (epochs and images) are needed to achieve ~80 a fully trained model's match to adult ventral stream. Specifically, training benefits predictions of higher visual cortex the most whereas predictions of earlier areas improve only marginally over the course of training. Second. by improving the random distribution of synap- tic connectivity, we find that 54% of the brain match can already be achieved "at birth" (i.e. no training all). Third, find that, by training only ~5% of model synapses, we can still achieve nearly 80% of the match to the ventral stream. This approach further improves on ImageNet performance over previous attempts in computer vision of minimizing trained components without substantially increas- ing the number of trained parameters. These results reflect first steps modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's updates based on visual experience).
INTRODUCTION
Particular artificial neural networks (ANNs) are the leading mechanistic models of visual process- ing in the primate visual ventral stream (Schrimpf et al.) 2020; Kubilius et al., 2019 Dapello et al.) 2020). After training on large-scale datasets such as ImageNet (Deng et al. 2009) by updating weights based on labeled images, internal representations of these ANNs partly match neural repre- sentations in the primate visual system from early visual cortex V1 through V2 and V4 to high-level
IT (Yamins et al.) 2014; Sealigh-Razav & Kriegeskorte| 2014 Cadena et al. 2017: Tang et al.) 2018 Schrimpl et al. 2018 Kubilius et al. 2219),, and model object recognition behavior can partly account for primate object recognition behavior (Rajalingham et al. 2018 Schrimmp et al.) 2018).

---

Published as a conference paper at ICLR 2022
Recently, such models have been criticized due to how their learning departs from brain develop- ment because they require many more labeled examples than is reasonable for biological systems' limited waking (visual) experience (Scibert 2018: Zador 2019). For example, all the current top models of the primate ventral stream rely on trillions of supervised synaptic updates, i.e. the training of millions of parameters with millions of labeled examples over dozens of epochs. In biological systems, on the other hand, the at-birth synaptic wiring as encoded by the genome already pro- vides structure that is sufficient for macaques to exhibit adult-like visual representations after a few months (Movshon & Kiorpes, 1988) Kiorpes & Movshon 2004, (Seibert 2018), which restricts the amount of experience dependent learning. Furthermore, different neuronal populations in cortical circuits undergo different plasticity mechanisms: neurons in supragranular and infragranular layers adapt more rapidly than those in layer which receives inputs from lower areas (Diamond et al.) 1994 [Schoups et al. (00T), while current artificial synapses, on the other hand, all change under the same plasticity mechanism. While current models provide a basic understanding of the neural mechanisms of adult ventral stream inference, can we start to build models that provide an understanding of how the ventral stream "wires itself up" models of the initial state at birth and how it develops during postnatal life?
Related Work. Several papers have addressed related questions in machine learning: Distilled student networks can be trained on the outputs of a teacher network (Hinton et al. 2013 Cho & Hariharan 2019 Tian al.] 2019), and, in pruning studies, networks with knocked out synapses perform reasonably well (Cheney et al. [017]] Morcos et al. (al](2018) demonstrating that models with
many trained parameters can be compressed (Wu et al. 2018) which is further supported by the convergence of training gradients onto a small subspace (Gur-Ari et al. 2018)- [Tian et al.] (2020) show that a pre-trained encoder's fixed features can be used to train thin decoder with performance close to full fine-tuning and recent theoretically-diee
work has found that training only BatchNorm
layers (Frankle al.] 202T) or determining the right parameters from a large pool of weights (Frankle et al. 2019: Ramanujan et al. 2019) can already achieve high classification accuracy. Unsupervised approaches are also starting to develop useful representations without requiring many labels by inferring internal labels such as clusters or representational similarity (Caron et al. 2018: Wu et al.)
2018 Zhuang et al., 2019; Hénaff et al. 2019 Konkle & Alvarez, 2020; /Zhuang et al. 2020).
Many attempts are also being made to make the learning algorithms themselves more biologically plausible (c.g. [illicrap et al. 2016 Scellier& & Bengio 2017 Pozzi et al. (022)) Nevertheless, all of these approaches require many synaptic updates in the form of labeled samples or precise machinery to determine the right set of weights. In this work, we take first steps of relating findings in machine learning to neuroscience and using such models to explore hypotheses about the product of evolution (a model's "birth state") while simultaneously reducing the number of supervised synaptic updates (a model's visual experience dependent development) without sacrificing high brain predictivity.
2cellSeelllerr
Our contributions follow from framework in which evolution endows the visual system with well-chosen, yet still largely random "birth" pattern of synaptic connectivity (architecture + initial- ization), and developmental learning corresponds training a fraction of the synaptic weights using very few supervised labels. We do not view the proposed changes as fully biological models of post-natal development, only that they more concretely correspond to biology than current models. Solving the entire problem of development all at once is too much for one study, but even partial improvements in this direction will likely be informative to further work. Specifically,
1. we build models with a fraction of supervised updates (training epochs and labeled images) that retain high similarity to the primate ventral visual stream (quantified by brain predic- tivity score from benchmarks on Brain-Score (Schrimpf et al., 2018 2020b) and find that layers corresponding to higher visual regions such as IT are most dependent training.
2. we improve the "at-birth" synaptic connectivity to show that even low- oowccppcctyy evolution- arily encoded information might lead reasonable initial representations with no training at all,
3. we propose a thin, "critical training" technique which reduces the number of trained synapses while maintaining high brain predictivity and improves over previous computer vision attempts to minimize trained components,
4. we combine these three techniques to build models with two orders of magnitude fewer supervised synaptic updates but high brain predictivity relative to fully trained model Code and pre- trained models are available via GitHub:
https: / / wtcbbb.ccom// franz www.geeeieerrreeeeeeeeeeee inirrriiinnnnnnnnn wec.iinggreduuiiooo
2

---

Published as a conference paper at ICLR 2022
2 MODELING PRIMATE VISION
We evaluate all models on a suite of ventral stream benchmarks in Brain-Score (Schrimpf et al.j 2018 2020), and we base the new models presented here on the CORnet-S architecture, one of the most accurate models of adult primate visual processing (Kubilius et al. 2019).
Brain-Score benchmarks. To obtain quantified scores for brain-likeness, we use a thorough set of benchmarks from Brain-Score (Schrimpf et al. 2018). To keep scores comparable, only included those neural benchmarks from Brain-Score with the same predictivity metric. All benchmarks feed the same images candidate model that were used for primate experiments while "recording" activations or measuring behavioral outputs. Specifically, the VI and V2 benchmarks present 315 images of naturalistic textures and compare model representations primate single-unit recordings from Freeman et al. (2013) (102 VI and 103 V2 neurons); the V4 and IT benchmarks present 2,560 naturalistic images and compare models to primate Utah array recordings from Majaj al. (2015) (88 V4 and 168 IT electrodes). A linear regression is fit from model to primate representations in response to 90% of the images and its prediction score on the held-out 10% of images is evaluated with Pearson correlation, cross-validated 10 times. The behavioral benchmark presents 240 images and compares model to primate behavioral responses from Rajalingham et al.] (2018). logistic classifier is fit on models' penultimate representations on 2,160 separate labeled images. The clas- sifier is then used to estimate probabilities for 240 held-out images. Per-image confusion patterns between model and primate compared with Pearson correlation. All benchmark scores are normalized by the respective ceiling. We primarily report the average brain predictivity score as the mean of VI, V2, V4, IT, and behavioral scores.
We note that the Brain-Score benchmarks in this study are based on limited data and thus present a possible limitation. Nonetheless, they are the most extensive set of primate ventral stream neuronal and behavioral benchmarks that is currently available and the scores generalize to new experiments Kubilius al. 2019).
Brain-Score provides separate sets of data as public benchmarks which we use to determine the type of distribution in Section 4 and the layer-to-region commitments of reference models.
CORnet-S, One of the current best model architectures on the Brain-Score benchmarks is CORnet-S (Kubilius et al. 2019), a shallow recurrent model which anatomically commits to ventral stream regions. CORnet-S has four computational areas, analogous to the ventral visual areas V1, V2, V4, and IT, and a linear decoder that maps from neurons in the model's last visual area to its behavioral choices. The recurrent circuitry (Figure 33) uses up- and down-sampling convolutions to process features and is identical in each of the models visual areas (except for Vlçon). but varies by the total number of neurons in each area. We base all models developed here on the CORnet-S chitecture and use the same syper-parameter as proposed in (Kubilius et al. 2019). Representations are read out the end of anatomically corresponding areas.
HIGH SCORES IN BRAIN PREDICTIVITY CAN BE ACHIEVED WITH FEW SUPERVISED UPDATES
We evaluated the brain predictivity scores of CORnet-S variants that were trained with a combination of fewer epochs and images. Models were trained with an initial learning rate of 0.1, divided by 10 when loss did not improve over 3 epochs, and stopping after three decrements.
Figure shows model scores on neural and behavioral Brain-Score measures, relative model trained for 43 epochs on all 1 1.28M labeled ImageNet images. In Panel A, we compare the average score over the five brain measures of various models to the number of supervised updates that each model was trained with, defined as the number of labeled images times the number of epochs. While a fully trained model reaches an average score of .42 after 55,040,000 supervised updates (43 epochs 1.28M images), a model with only 100,000 updates already achieves 50% of that score, and 1,000,000 updates increase brain predictivity scores to 76%. Models are close to convergence score after 10,000,000 supervised updates with performance nearly equal to full training (97%). Scores grow logarithmically with an approximate 5% score increase for every order magnitude more supervised updates.
3

---

Published as a conference paper at ICLR 2022
Standarrd traniig
10205
20% spdatts 97% score
100%
85%
updaree 7e0ii
80%
Scare [% sarrrrr vunnee
800
02%
updates sccre
Score uorddard
00%
Siate Prod
00%
Mose Biyis Pred
Pradictivity
40%
40%
60%
25%
Behaxor
20%
Benwer
0%5
0%
10a
32*
30
Supervised updatts (Traiing Epocto broges)
Training Epochs
Labeled Images
Figure 1: High scores in brain predictivity be achieved with few supervised updates (log x- axes). A Average brain predictivity of models trained with a range of supervised updates (different combinations of epochs x images). Each dot is different hypothesis of how the ventral visual stream might have developed and shows the adult Fairly brain-like representations are already realized with few supervised updates, relative to fully trained model (black dot; see also Figure 6). Standardly trained CORnet-S is set to be 100% score in brain predictivity on the benchmarks. A "pixels" baseline achieves 7% of this brain predictivity score. B Individual brain predictivity scores over epochs. Models start to approximate the primate ventral stream with few training epochs. Lower visual areas (VI, V2) are approximated earlier in training. C Like B, but number of training images instead of epochs. Few images are sufficient to approximate especially early visual areas.
score that achieved by that model.
Figures IB and show individual neural and behavioral scores of models trained with fewer training epochs or labeled images independently. Early to mid visual representations (V1, V2, and V4 scores) are especially closely met with only few supervised updates, reaching 50% of the final trained model in fractions of the first epoch (Figure DB). After only one full iteration over the training set, V1, V2, and V4 scores are close to their final score (all >80% while IT requires two epochs to reach comparable level. Behavioral scores take slightly longer to converge (> 80% after 7 epochs).
Similarly, when training until convergence with fractions of the 1 1.28M total images, 50,000 images are sufficient to obtain high neural scores (80% of full training in V1, V2, V4, IT). Behavioral scores again require more training: half the standard number of labeled images is needed to surpass 80%.
Concretely relating supervised updates to primate ventral stream development, SSibbrr (2018) estab- lishes that no more than 44 months - or 10 million seconds- - of waking visual experience is needed to reach adult-level primate IT cortex (as assessed by its capability to support adult level object recognition). From this estimate, we can compute how many supervised updates per second differ- models in Figure [TA would require (assuming those updates are evenly distributed million seconds). For instance, the fully trained model's 55 million supervised updates translate to 5.5 updates every second, whereas the model with I million updates and 76% relative brain predic- tivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans Yaarbus 1967 Gibaldi & Sabatini 2020
the 10
"AT-BIRTH" SYNAPTIC CONNECTIVITY YIELDS REASONABLE BRAIN PREDICTIVITY WITH NO TRAINING AT ALL
If few supervised updates can get model representations fairly close to fully trained model (Fig- ure [ how close are the initial representations without any training? In relation to biology and following the introduced framework of treating all consecutive training as developmental learning, these "at-birth" synaptic connections would result from information encoded in the genome as a product of evolution.
Due to the genome's ' capacity bottleneck, it is infeasible to precisely encode every synapse. Pri- mary visual cortex alone contains 1..E8 neurons per hemisphere (Leuba & Kraftsik 1994), 1EB synapses per neuron, each requiring ~37 bits per synapse (Zador) 2019). Thus, without any clever rules, specifying the connections in only one hemisphere of V1 could require up ~5.2E12 bits
4

---

Published as a conference paper at ICLR 2022
A
Birth Initialization (per layer)
B
V1.covl
Brain Predictivity [% of standard training]
100%
trained
weights
80%
V1. conv2
Compress
60%
Weight Compression
v2.com2
Weight Compression
Kaiming Normal
p,c. O.
40%
Kaiming
V4 conv2
Normal
Sample
Sample
IT conov2
Paoks
weights
weights
0%
tririig
Full trairing
Figure 2: "At-birth" synaptic connectivity yields reasonable scores in brain predictivity. A Specifying the initial weight distribution: Kaiming Normal (KN, (H et al. (015) samples from a generic Gaussian. Weight Compression (WC) compresses trained weights into low-parameter clustered distributions that weights can be initialized from. B Visualization of WC compressed parameters: Gabor filters for first layer and cluster centers for following layers with kernel size >> 3. The cluster centers capture intuitive variety of kernel types. C "At-birth" representations with WC achieve 54% score in brain predictivity of a fully trained model, with no training at all. Scores after training remain virtually unchanged (shaded bars).
orders of magnitude more than the entire genome's 1GB 8E9 bits (Zador 2019). Sampling synaptic weights from reasonably compressed distributions on the other hand places little memory requirements genetic encoding while potentially yielding useful initial weights. Current machine learning techniques for initializing weights, such as Kaiming Normal (He et al. 2015 sample from Gaussian distribution centered around zero.
To test the hypothesis that the genome might already encode more powerful initial representations with synaptic wiring sampled from distributions specified by only few bits, we explored multi- dimensional distributions as more expressive alternative. These distributions only require a small number of parameters, but unlike current generic initializers, we explicitly specify them for each layer. To determine the right parameterization, we compress trained model's weights into clusters which we then sample from ("Weighs Compression, WC").
More specifically, for all convolutional layers except the first, we cluster the kernel weights and later sample from the clusters. We determine the number of clusters with the elbow method (Thorndike) 1953): 11 for V1, 13 for V2, 16 for V4, and 15 for IT. To capture the relative importance of clusters we fit a normal distribution to the cluster frequency over kernels. In batch normalization layers, fit normal distribution each to the weights and biases. For the first convolutional layer only, we employ a Gabor prior on the weights following studies in VI (Hubel & Wiesel, 1962 Jones & Palmer 1987) (Appendix B. This results in 33 KB (4,166 parameters) specify network initialization, compared to 423 MB for a trained model's weights (assuming 8 byte per parameter).
Model interpretability studies (Zeiler & Fergus 2013 Olah et al., 2020 Cammarata al. 2020)
classify model weights, comparable to WC's representation. Visualizing the weight compressions from trained CORnet-S weights (Figure 2B), we find that the first layer's Gabor filters qualitatively align with an analysis by Cammarata et al. (2020). Cluster centers seem to represent an intuitive division of channel types with opposite types every layer.
Applying WC to CORnet-S, we first obtain a compressed and clustered set of parameters, from which we sample entirely new weights to yield new model CORnet-Swc. This model is not trained at all and we only evaluate the goodness of its initial wiring on the suite of Brain-Score benchmarks. Strikingly, we find that even without any training, CORnet-Swe achieves 54 + 1.5%of the brain predictivity score relative to fully-trained model (Figure 2). representing 12 percent point improvement (n = 10 seeds; permutation test p < 1E-5) over the Kaiming Normal initialized model with score of 43 + 1.7%. Early ventral stream regions I and V2 are predicted especially well with no loss in score but we note that these two benchmarks are less well predicted by the trained model to begin with. V4 scores also approximate those of a trained model relatively well
5

---

Published as a conference paper at ICLR 2022
A
Standard Training
V1
V2
V4
Brain Predictivity (% of standlad training
100%
Standard Training
Cntical Training
80%
Dgeneveel Copp
Downstream Training
60%
Downstream Training
40%
20%
Critical Training
0%
Dounsimple Corv
Number of trained parameters [Million]
Imagenent performance (%% stardard traiinn]
100%
Standard Training
80%
Critical Training
Frozen
Trained
60%
Upsample
Downsample Conv
Down CarEo
40%
Conv
Input
20%
Downstream Training
Sklp
Gate
Add
Number of trained parameters [Milion]
Batchnarm
Non-Linaarit
Figure 3: Training only critical layers reduces the number of updated synapses while maintain- ing high brain predictivity. We could naively reduce the parameters of fully-trained model by freezing layers from the bottom up, training only the top layers (" ("Downstream Training DT"; gray box). We instead propose Critical Training (CT) which only trains thin "critical" layers (e.g. down- sampling layers, blue box). B CORnet-S circuitry. CT only trains critical layers, leaving the rest frozen. C Naively reducing parameters from standard training (black dot, top right) quickly deteri- orates brain predictivity scores (DT, gray line) whereas Critical Training reduces parameters while retaining high scores (blue line, CT). D Like C, but measuring ImageNet score. CT retains nearly half the score with fraction of parameters.
(75%). The major drop occurs in the IT and especially behavioral scores where CORnet-Swc only reaches 39% and 6% of the trained model's 's score respectively. Similarly, a trained linear decoder on CORnet-Swe's 's IT representations only reaches 5% of trained model's ImageNet top-1 accuracy. While intuitively sampling weights from a compression trained weights should somewhat recover reasonable brain predictivity scores, note that not every implementation satisfies both high brain predictivity and sufficient compression for the genome bottleneck (Appendix B.3).
Weight Compression explores the hypothesis that evolution may have discovered initialization strategy with improved at-birth representations (relative to current initializations). WC is most likely not how evolution found the at-birth synaptic connections, but shows that with nearly identical pacity, an alternative initialization distribution leads to networks that are more brain-like in their adult state revealing new space of possibilities (hypotheses) that should be considered (see Ap- pendix [or more details on biological plausibility). These findings further suggests that matching representations in higher visual regions might be especially dependent on visual experience whereas early visual regions might already be reasonably well specified without experience.
5 TRAINING THIN DOWN-SAMPLIN LAYERS REDUCES THE NUMBER OF UPDATED SYNAPSES WHILE MAINTAINING HIGH BRAIN PREDICTIVITY
While improved "at-birth" connectivity can reach 54% of a fully-trained model's score, additional experience dependent updates appear necessary reach higher predictivities. With standard train-

---

Published as a conference paper at ICLR 2022
ing, each iteration simultaneously updates all of the millions of synaptic weights in the neural net- work, which may be difficult to implement biologically. Alternatively, learning could take place preferentially in specific components. Cortical circuits are heterogeneous and different neuronal populations undergo distinct plasticity mechanisms. For example, neurons in supra- and infra- granular layers adapt more rapidly than those in layer 4, where inputs from lower areas arrive, as observed in rat somatosensory cortex (Diamond et al. 1994) and primate VI (Schoups et al. 2001).
As proo--of-rrncciple that training a reduced set of layers can retain high performance, we propose a novel thin training technique, which we term Critical Training (CT; Figure AA). CT updates only the weights in critical layers, instead of updating every single model synapse. In CORnet-S, each of the blocks has one down- hown-sampling layer to produce an area's final representation (Figure B3. We explore successive variants of applying CT up to block in the architecture and then training the following blocks, e.g. freezing VI, V2, V4 with critical training of the respective down-sampling layers and additional IT training. The final CT' ventral stream model is almost completely frozen and only the synapses generating each cortical area's 's output are trained.
We compared Critical Training against two alternative approaches: 1) reducing the trained parame- ters by freezing entire model blocks, for instance keeping V1 and V2 blocks fixed while training V4 and IT blocks. We term this block-wing freezing and training approach Downstream Training (DT).
And 2) an approach proposed by Frankle et al. (02T where only the BatchNorm parameters in network are trained while all other parameters are kept at their initial values ("BatchNorm, BN").
Compared to standard pacpppropagttion training all the weights, all three approaches (CT, DT, BN) reduce the number of trained parameters (Figure 3C). However, while the average score with DT (gray) already drops below 65% with over a quarter of trained parameters remaining and BN drops to 62% with very few parameters, CT (blue) maintains over 75% with only 1.4 out of 52.8 million parameters trained. The choice of which critical layers to train also matters: training the connecting layers between regions- i.c. the last (down- dlmwpisampling, default) or first (up-sampling) layer - retains most of the performance whereas training layers such as BatchNorm performs worse.
By reducing the number of trained parameters, Critical Training also yields engineering benefits with more than 40% of the ImageNet score maintained at < 3% of parameters trained- significant improvement over 22% accuracy with BatchNorm (Frankle et al.) (02T) while adding only a small number of additional weights. CT further reduces training time by 30% per epoch.
HIGH BRAIN PREDICTIVITY CAN BE ACHIEVED WITH A RELATIVELY SMALL NUMBER OF SUPERVISED SYNAPTIC UPDATES
All three training reduction methods independently minimize the number of supervised synaptic updates required to reach reasonably high brain predictivity score. Reducing the number of super- vised updates minimizes the number of epochs and images (Section 3): Weight Compression (WC) improves the at-birth synaptic connectivity for high initial scores with no training all (Section 4) and Critical Training (CT) reduces the number of synapses that are updated during training (Sec- tion 5). Testing synergies between these strategies, we combined all three methods build novel models that only require a small number of supervised synaptic updates to reasonably capture the mechanisms of adult ventral visual stream processing and object recognition behavior.
Figure A shows the average brain predictivity of a range of models with varying numbers of super- vised synaptic updates relative to a standard trained CORnet-S (black dot, 3,000 trillion supervised synaptic updates). With a reduced number of supervised updates (training epochs and labeled im- ages) but standard initialization and training all weights (light blue dots), models require 5.2 trillion updates to achieve >50% of the score of a fully trained model and about 100 trillion updates to reach 80% brain score. Adding WC+CT (dark blue dots), the corresponding model already reaches 53% at birth with 0 supervised synaptic updates. At 0.5% the updates of fully trained model (14 trillion vs. 3,000 trillion), models then reach 79% of the score (17 model with modeling choices marked in Figures to 3.. Continuing standard training from this 79% model, we can achieve 100% of the score with I5 additional epochs of 1,028 trillion supervised synaptic updates (one third of the fully trained model's 3,000 trillion updates). Reference models (gray dots) MobileNet (Howard et al. 2017) and ResNet (He et al. 2016) obtain high scores, but also require many supervised synaptic updates. HMAX (Riesenhuber & Poggio 1999) is fully specified with updates but lacks in score.
7

---

Published as a conference paper at ICLR 2022
A
B
Standard Training...
Mobilenet
Brain Predictivity [percent points]
100%
Brain Predictivity standard training)
Resnet
WCC better
100
.5% updates
79% score
80%
1000
11
27
19
15
8
updates
Labeled Images
54% score
10000
11
26
9
8
-2
-1
-3
60%
50000
-2
40%
100000
11
-14
-11
-19
WC+CT worse
HMAX
500000
11
-3
-8
-11
-17
-17
20%
plaels
All
-11
-17
supervized updates +C
20 Convergence
10
10t2
1033
10-4
10t3
Supervised synaptic updates
Training Epochs
(training epochs labeled images trained synapses)
Figure 4: Benefits of combining training reductions for high brain predictivity with few super- vised synaptic updates and limitations of techniques in bigh-update regimes. A (log x-axis) By reducing updates with a combination of fewer supervised updates (Figure [T]. improved initialization WC (Figure 2). and training only down-sampling layers CT (Figure 3). the resulting models (dark blue dots; fewer supervised updates alone in light blue) maintain high brain predictivity scores while requiring only a fraction of supervised synaptic updates compared to standard CORnet-S (black dot, top right). Comparison between WC-initialized models trained with CT versus standardly initial- ized models training all weights, when varying training epochs and labeled images. Colors represent their percent point difference in brain predictivity scores. WC+ CT improve performance in regimes with few epochs and images, but fall behind in regimes with many labeled images and epochs.
We next examined interactions between methods by comparing models initialized with WC and trained with CT to models with standard initialization and training all weights, when both are trained with fewer epochs and images. F shows the percent point difference between the two model families. WC+C7 yield strong benefits (green numbers) in a regime with few supervised updates, improving by up to 27 percent points when training for only epoch on 1,000 images. With many updates on the other hand, WC+C7's is less advantageous than standard training (red numbers): with all 43 epochs and 1.28M images, the score reduces by 17 percent points. WC+CT therefore most positively interact with small budget of supervised updates (which is the focus of this work).
7 DISSECTING TRAINING REDUCTIONS
We asked whether the developed techniques would generalize to architectures other than the CORnet-S architecture they were based on. Beyond establishing the methods as more general, this can be seen as novel way to construct model taxonomies. We therefore applied Weight Compres- sion (WC) and Critical Training (CT) to ResNet-50 (He et al. 2016) and MobileNet-VI (Howard) et al. 2017) architectures, both high- nigh-performing models on Brain-ccore. We used WC distributions determined on CORnet-S, i.e. we tested transfer without re- re-fitting. WC+CT maintain 91 of the score in ResNet despite an almost 80% reduction in parameters. When applied to MobileNet, the average score drops by 22% and parameters are reduced less strongly (43%). This difference in performance could be due to MobileNet already being very compressed, or having less similar architecture.
With most analyses far comparing an average score, we dissected the relative contributions of WC and CT to individual benchmarks (Figure (B). We compared KN to WC initialization, as well resulting models after critical training (KN+CT and WC+CT). WC initialization improves most over KN early visual regions VI and V2, while additional training with CT is most beneficial in mid- high-level visual cortex V4 and IT, well the behavioral benchmark.
8

---

Published as a conference paper at ICLR 2022
Figure 5: Transfer other networks and individual scores comparison. A Transfer other networks. We sample from WC initializations determined on CORnet-S, followed by Critical Training of only down-sampling layers. B Absolute scores on individual benchmarks of combi- nations of initialization (KN/WC, Figure 2). and with critical training (CT, Figure techniques.
Drain Preductivtty (otoo.n))
MobileNet
Resnet50
Brain Predictiiny (atcam]]
21% paramsters 91%
57% parameters 578% soore
Behavior
Number trained parameters [Milion]
DISCUSSION
We developed a range of models with neural and behavioral scores approaching those the current leading model of the adult ventral visual stream as quantified in Brain-Score, while requiring only a fraction of supervised synaptic updates. These models were built by complementarily 1) reducing the number of supervised updates, i.e. training epochs and labeled images; 2) improving the "at birth" distribution of synaptic connectivity; and 3) training only critical synapses at the end of each model area. The techniques and resulting models proposed here first steps to more closely modeling not just adult primate visual processing, but also exploring the underlying mechanisms of evolution and developmental learning.
These proof-of-principl demonstrations are far from accounting for the rich information encoded in the genome or the developmental learning that together result in adult mechanisms of visual processing, and require further experimental validation. We here started from CORnet-S, one of the leading models on Train-Sccre, but does not fully predict all brain measurements (0.42 absolute score). We verified favorable transfer to models with similar architectures such as ResNet, but generalization to an already compressed MobileNet was limited (Figure BJA).
Relating to genomic mechanisms, the proposed techniques should generalize to other domains such as auditory processing. With the capacity bottleneck the genome, mechanisms for wiring up would likely be shared between similar systems. The fact that early visual areas converge earlier during training (Figure I) and are better predicted than higher areas by WC initialization is consistent with developmental studies of the primate ventral stream. In humans, behaviors that rely on low- level spatial and temporal processing of visual inputs reach adult-like performance considerably earlier than complex visual behaviors that rely on higher cortical regions, such as face perception (Ellemberg et al. 1999 Grill-Spector et al. 2008).
A critical component in more closely modeling primate development is reduce the dependence on labels altogether. Recent unsupervised approaches are starting to rival the classification perfor- mance of supervised models (Caron et al. 2018 Hénnfff al. 2019 Zhuang al.) (2020) and com- bining them with the advances presented here could further reduce the number of synaptic updates. More precise biological measurements are required quantify the number of (parallel) experience- dependent updates. Current unsupervised techniques however still require back tack-propagation which routinely criticized as non-biological, among others due to the propagation of gradients (Gross- berg 1987, [Wittington & Bogac7 2019 Hunsberger] 2017). Local learning rules (Löwe et al.) 2019 Xiong al. 2020) might alleviate these concerns and with critical training (Figure) 3). could be sufficient to learn in only a subset of layers.
The changes to model initialization and training presented here serve as proof-of-principle that models can be changed to more closely align with primate development by reducing training steps with labeled images and improving initialization. It is also possible to achieve high brain predictivity when training only a fraction of weights. but all these models still far from the actual biolog- ical mechanisms. We expect future work in this direction to further close the gap with improved evolutionarily encoded wiring mechanisms and developmental learning rules.

---

Published as a conference paper at ICLR 2022
REFERENCES
Martin Schrimpf, Idan A Blank, Greta Tuckute, Carina Kauf, Eghhal A Hosseini, NANCY G KANWISHER, Joshua B. Tenenhaum, and Evelina Fedorenko. Artificial Neural Networks Accurately Predict Language Processing in the Brain. bioRxiv preprint, 2020.
Jonas Kubilius, Martin Schrimpf, Ha Hong. Najib Majaj. Rishi Rajalingham, Elias Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs. In Neural Information Processing Systems (NeurIPS), pp. 12785- 12796. 2019.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D. Cox, and James J. DiCarlo. Simu- lating Primary Visual Cortex the Front CNNs Improves Robustness to Image Perturbations. Neural Information Processing Systems (NeurlPS). 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. ImageNet: large-scale hicrarchical image databasc. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-28888 245. 2009.
Daniel LK Yamins. Ha Hong. Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James DiCarlo. Performance-optmninn opttnnize hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 11 111(23):8619 8624, 2014.
Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised. models may explain it cortical representation. PLoS computational biology, 10(1 ((11))e1000 1003915, 2014.
Santiago A Cadena, George H Denfield, Edgar Y Walker, Icon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Fikker Deep convolutional models improve predictions of macaque v1 responses natural images. bioRxiv. pp. 201764, 2017.
Hanlin Tang, Martin Schrimpf. William Lotter, Charlotte Moerman, Ana Paredes, J.O. Josue Ortega Caro, Walter Hardesty, David Cox, and Gabriel Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences (PNAS), 1 11533))8835-8880, 2018.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott- Roy, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Score: Which artificial neural network for object recognition most brain-like? bioRxiv, 2018
Rishi Rajalingham, Elias Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James DiCarlo. Large- scale, regouursoottion comparison of the core visual object recognition behavior humans, monkeys, and statc-of-the-att deep artificial ncural networks. Journal of Neuroscience, pp. 7255-7269, 2018
Darren Seibert. High-level visual object representation in juvenile and adult primates. PhD thesis, Mas- sachusetts Institute of Technology, 2018.
Anthony Zador. A Critique of Pure Learning: What Artificial Neural Networks can Learn from Animal Brains. bioRxiv preprint, 2019.
J. Anthony Movshon and Lynne Kiorpes. Analysis of the development of spatial contrast sensitivity in monkey and human infants. Journal of the Optical Society of America A (JOSA A), ((1))2266 1988.
Lynne Kiorpes and J. Anthony Movshon. Development of sensitivity to visual motion in macaque monkeys. Visual Neuroscience, 21(6):8118558 2004.
Mathew E. Diamond, Wei Huang, and Ford F. Ebner. Laminar comparison of somatosensory cortical plasticity. Science,
1994.
Aniek Schoups, Rufin Vogels, Ning Qian, and Guy Orban. Practising orientation identification improves ori- cntation coding in V1 neurons. Nature, 44246656) 2001.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv preprint, 2015.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In International Conference on Computer Vision (ICCV), volume 2019-Octob pp. 4793-4801 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Representation Distillation, arXiv preprint, 2019. Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the Robustness of Convolutional Neural Net- works to Interal Architecture and Weight Perturbations, arXiv preprint, 2017.
Ari S. Morcos. David G. T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference Learning Representations (ICLR), 2018.
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-Means: Re-Traine Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions. In International Conference on Machine Learning (ICML), 2018.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. preprint, 2018
Gradient Descent Happens in a Tiny Subspace. arXiv
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenhaum, and Phillip Isola. Rethinking Fhw-Shot Image Classification: a Good Embedding Is All You Need? arXiv preprint, 2020.
10

---

Published as a conference paper at ICLR 2022
Jonathan Frankle, David J Schwab, and Ari S Morcos. Training BatchNorm and Only BatchNorm: On the Expressive Power Random Features CNNs. In International Conference on Learning Representations (ICLR), 2021.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. The Lottery Ticket Hy- pothesis at Scalc. arXiv preprint, 2019.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's Hidden in Randomly Weighted Neural Network? arXiv preprint, 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep Clustering for Unsupervised Learning of Visual Features. In European Conference Computer Vision (ECCV), pp. 132 149, 2018. Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised Feature Learning via Non- parameeeer Instance Diserimination. In Computer Vision and Pattern Recognition (CVPR), pp. 3733-3742, 2018.
Chengxu Zhuang, Alex Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual em- beddings. In International Conference on Computer Vision (ICCV), pp. 6001-6011 2019.
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-Efficient Image Recognition with Contrastive Predictive Coding. In Computer Vision and Pattern Recognition (CVPR), 2019.
Talia Konkle and George A. Alvarez. Instance-level contrastive learning yiclds human brain-like representation without -uurgooossuprriision bioRxiy preprint, 2020.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C. Frank, James J. DiCarlo, and Daniel K. Yamins. Unsupervised Neural Network Models of the Ventral Visual Stream. bioRxiv preprint, 2020. Timothy P. Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1):13276 2016.
Benjamin Scellier and Yoshua Bengio. Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation. Frontiers in Computational Neuroscience, 1 11:24, 2017.
Isabella Pozzi, Sander M Bohté, and Pieter R Roelfsema. Attention- Gated Brain Propagation: How the brain can implement reward-basee error backpropagation. In Neural Information Processing Systems (NeurIPS), volume 33, 2020.
Jeremy Freeman, Corey M Ziemba, David Heeger, Fero P Simoncelli, and J Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature Neuroscience, 16(7):974-981, 2013. Najib Majaj, Ha Hong, Ethan A Solomon, and James DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 35(39): (000)):1322-1411111 13418, 2015.
A. L. Yarbus. Eye movements and vision. Plenum Press, 1967.
Agostino Gibaldi and Silvio P. Sabatini. The saccade main sequence revised: fast and repeatable tool for oculomotor analysis. Behavior Research Methods, pp. 1 1-21, 2020.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human- ImageNet Classification. In Proceedings of the IEEE international conference on
Level Performance computer vision, pp. 1026- 1036-103,222 2015.
Leuba and R Kraftsik. Changes in volume, surface estimate, three-dimensional shape and total number of neurons of the human primary visual cortex from midgestation until old age. Anatomy and Embryology. 190 (4):351-366 1994.
Robert L. Thomdike, Who belongs in the family? Psychometrika, 18(4):2672776 1953,
David H Hubel and Torsten N Wiesel, Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1):106-144, 1962.
J. P. Jones and L. A. Palmer. The two-dimensional spatial structure of simple receptive fields in cat striate cortex. Journal of Neurophysiology, 58(6): 187-1211, 1987.
Matthew D Zeiler and Rob Fergus. Visualizing and Understanding Convolutional Networks. arXiv preprint, 2013.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 5(3), 2020.
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petroy, and Ludwig Schubert. Thread: Circuits. Distill, 2020.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016. Maximilian Riesenhuher and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 1019) 1999.
11

---

Published as a conference paper at ICLR 2022
Dave Ellemberg. Terri L. Lewis. Chang Hong Liu, and Daphne Maurer. Development of spatial and temporal vision during childhood. Vision Research, 39(14):3325-333 1999.
Kalanit Grill-Spector, Golijeh Golarai, and John Gabrieli. Developmental neuroimaging of the human ventral visual cortex. Trends in Cognitive Sciences, 12(4):152-162, 2008.
Stephen Grossherg. Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11(1):23-33, 1987.
James C.R. Whittington and Rafal Bogacz. Theories of Error Back-Propagation the Brain. Trends Cogni- tive Sciences, 23(3)233 250, 2019.
Eric Hunsberger. Spiking Deep Neural Networks: Engineered and Biological Approaches Object Recogni-
tion. PhD thesis, University of Waterloo, 2017.
Sindy Löwe, Peter O' O'Connor, and Bastiaan S. Vceling. Putting An to End-to-End:
Learning of Representations. In Neural Information Processing Systems (NeurIPS), volume 32. 2019.
Yuwen Xiong. Mengye Ren, and Raquel Urtasun. LoCo: Local Contrastive Representation Learning. arXiv, 2020.
Evelyn Fix and J. L.. Hodges. Discriminatory analysis, nonparametric discrimination. Technical report, United States Air Force, 1951.
Gideon Schwarz. Estimating the Dimension of a Model. Annals of Statistics, 6(2)4611464, 1978.
Andrew D. Huberman, Marla B. Feller, and Barbara Chapman. Mechanisms Underlying Development of Visual
Maps and Receptive Fields. Annual Review of Neuroxcience, 31(1):479-509, 2008.
12

---

Published as a conference paper at ICLR 2022
Figure Findings generalize across hyperparameters. Notations like in Figure[] To test whether fast learning could be achieved by purely changing hyperparameters, we evaluated dif- ferent regularizations (weight decay 0.001 and 0.00001 vs. 0,0001), learn- ing rates (initial 1.0 and 0.01 vs. 0.1 the main manuscript) and learn- ing rate schedules ("Scheduler alter- native" decreases learning rate more agressively). Trends are qualitatively similar across all choices, and quanti- tatively nearly optimal with respect fast convergence to high brain predic- tivity for the hyperparameters used in the manuscript. We suspect that more careful tuning of e.g. the learning rate could lead to improved trade-off between supervised updates and brain predictivity.
100%
Brain Predictivity 1% of standard training)
Weight decay 0.001
Weight docay le-5
80%
Learning rate 1.0
60%
Scandard training
Learning rate 0.1
40%
Scheduler alternative
20%
0%
10
10
10
10
A BENCHMARK DETAILS
We use the benchmarks implemented in www. www..gg.tnb..coobbbaa--eeeo brain- score at commit 96b0711, and convert www wwwhhbtthh.......g.. wwon:/baainscorreee wwwcorremoooe-eeee
base models to brain models with at commit 21778c6. Images were presented at degrees without aperture for the V1 and V2 benchmarks and at 8 degrees for the V4, IT, and behavior benchmarks. Models committed to input size of degrees visual angle.
B WEIGHT COMPRESSION DETAILS
For all convolutional layers except the first, we cluster kernel weights in a layer using the k-means algorithm (Fix & Hodges 195T). The number of clusters determined using elbow (Thorndike) 1953) (see Table To capture the relative importance of clusters we fit normal distribution Ni for each cluster with μs as the cluster frequency over kernels and øy as the frequency standard deviation. To sample weights for a kernel, we first sample cluster distribution i i~ N2 per kernel and then obtain channel weights by sampling from a Gaussian with μi as the cluster center and the standard deviation o; of clustered weights. In batch normalization layers, we fit one normal distribution each to the weights and biases.
B. COMPRESSING THE FIRST LAYER WITH GABOR PRIOR
The weight compression approach we use in Section 4 is based on different initialization techniques, applied to different layers. For the very first layer of size 7 x we found Gabor filter most effective following studies in V1 (Hubel & Wiesel 1962: Jones & Palmer 1987) kernels trained channel weights a Gabor function
To generate the Gabor
exp
-0.5( /roe/ cos (2T + 6) C
(I)
where
n2
Frot x cos(0) (0)
0x
(2)
(3)
Brot = -x - sin )) + (0)
F2y
0y f
13

---

Published as a conference paper at ICLR 2022
LAYER
NUM CLUSTERS
VI.covv
V1.con2
7
V2.conv
V2.conv1
2
V2.conv2
V2.conv3
2
V2.skip
2
V4.conv _nnput
V4.conv1
2
V4.conv2
V4.conv3
2
V2.skip
ITCc
2
IT.con22
IT.conv3
2
IT.skip
Table I: Number of clusters used for each layer by Weight Compression, determined by the elbow method. Note that for V1.convl, I, there are four components of a multivariate Gaussian as described in Appendix B.I
Trot and Urot are the orthogonal and parallel orientations relative to the grating, 0 is the angle of the grating orientation, f is the spatial frequency of the grating. is the phase of the grating relative to the Gaussian envelope, a* and By the standard deviations of the Gaussian envelope orthogonal and parallel to the grating, which can be defined as multiples (ng and nyy of the inverse the grat- ing firqueeccy and C' is scaling factor.
The function is fit per channel, which leads to set of Gabor parameter for each of the 3 RGB chan- nels. We then fit multidimensional mixture of Gaussians to the combination of all filter parameter per kernel, resulting in a kernel parameter set. For the three RGB input channels in the first layer and the Gabor parameters of components (number of distinct Gaussian distributions) based on the Bayesian Information Crite- rion (Schwarz 1978) and use 4 components for the first layer of CORnet-S. To generate new kernels we sample kernel parameter set from this mixture distribution and apply them to the described Gabor function that spans the weight values.
therefore fit to 3
27 parameters. We evaluate the best number
B.2 COMPRESSING BATCHNORM LAYERS
In addition to convolutional layers, models consist of several Batchnorm layers, which contain learnable bias and weight term. To initialize these terms, we fit normal distribution per weight and bias vector of the trained values and sample from this distribution. Note that BatchNorm layers contain running average means and standard deviations for normalization purposes, which are ap- plied at validation time. Those terms are set to zero when no training has happened, but cause score changes once the model has processed the dataset. During training the mean and standard deviation of the current batch are used instead.
B.3 ALTERRNATVVE APPROACHES
We have explored a variety of weight compression methods applied to different layers and evaluate their performance "at birth" without training and when trained with critical training.
Figure shows brain predictivities of several alternative compression methods implemented fol- lows:
WC Weight compression approach with clustering described in Section 4, using a Gabor prior approach for the first layer, noisy cluster sampling for convolutional layers and fitted normal distributions for Batchnorm layers (4, 166 parameters for CORnet-S).
14

---

Published as a conference paper at ICLR 2022
Figure 7: Alternative weight compression methods Compar- ison of different initializations that compress weights, "at birth" i.e. without any training (gray) and after training critical layers (shades of blue) for 6 epochs. Our best clustering-basct ap- proach WC achieved similar re- sults the Mixture Gaussian approach (~33 percent points mean difference) but leads to more diverse clusters. Perfor- mance drops when solely sam- pling weights from kernel based normal distributions (Kernel nor- mal) and additionally disabling the Gabor prior (No Gabor prior)
100%
Brain Predictivity 1% of standard training]
80%
Criiial Erain ng
Criics training
train mng
60%
No training
No training
40%
20%
Cluster
Mixture gaussian
Kernel normal
No gabor prior
Mixture Gaussian Instead of sampling weights from cluster centers, this approach uses multidimensional distributions for convolutional layers with kernel size > 1. We fit mix- ture Gaussian distribution per layer to the weights of a channel over all kernels. To sample a new kernel, sample individual channels from this distribution. For convolutional layers with a kernel size of we draw weights from a normal distribution adjusted per kernel as described in the next item (428, 11 14 parameters for CORnet-S).
Kernel normal All weights are sampled based on normal distributions. We mean and standard deviation to the weights of one trained kernel and resample new kernel from this distribution. We do this separately for every kernel to generate a whole layer. This approach is similar to the BatchNorm sampling method where we compress BatchNorm weight and bias terms instead of kernels (433,735 parameters for CORnet-S).
No Gabor prior To evaluate the importance of the Gabor prior we use the Kernel normal model and apply the same normal distribution approach layer one instead of Gabor sam- pling. Performance drops by 13 percent points without training, and by 9 percent points after critical training. (20,026 parameters for CORnet-S)
We speculate that increasing the number of clusters would follow logarithmic growth with respect to the performance of the model: i.e. small number of clusters can already yield useful wiring (as explored here), while adding more clusters will have positive effect but with decreasing effect size.
B.4 BIOLOGICAL PLAUSIBILITY
Weight Compression (WC)'s cluster-based initialization generates an "at-birth" network that already captures some useful aspects of the visual inputs in a compressed manner and allows the system to learn faster. The way that these clusters determined (from previously trained network) is biologically plausible and does not correspond to any evolutionary mechanism we are aware of. However, independent of how these clusters are determined, WC shows that is possible to encode certain priors about system's 's wiring diagram in very compressed manner. Evolution also acquired aspects of the visual inputs (with a different strategy) and encoded them in the genome lower information regime.
For example, we know that there are multiple neuro-devclopmental mechanisms, such as sponta- neous retinal waves and axon guidance cues, that significantly shape the architecture and function of the visual system requiring no (or very little) visual experience ((Huberman et al., 2008)). These mechanisms depend on relatively small number of proteins encoded in genome and give rise to a highly complex pattern of at-birth synaptic connectivity representing a very large compression of information.
15

---

Published as a conference paper at ICLR 2022
Standad trilking spach /eeege/
2005
Drse Prodocaiity (1/ of tolle
80%
1000
12
37
Lakded
13
Sandhiind spriptic upd Libeed
152
Tradking Epecit
Tridking Eaecs
Figure 8: Detailed analysis of WC+CT. A When reducing the number of supervised synaptic up- dates, adding critical training (dark grey) and adding weight compression initialization (dark blue) both improve scores in brain predictivity at the same number of supervised synaptic updates, in comparison to a model with standard initialization and training all weights (bright blue). B Brain predictivities for the WC+CT model when trained with a range of epochs and labeled images. C Same as B, but for a standardly initialized (KN) model training all weights.
C WC INITIALIZED AND CT TRAINED MODEL ANALYSIS
Our best model WC+CT benefits from a combination of improved initialization through weight compression, and critical training. Figure shows models with standard initialization and training all weights, but with fewer supervised updates (cf. Figure 1), models that only train down- down-sampling layers (CT), and models that combine critical training with weight compression (WC+CT). A model initialized with weight compression achieves (only WC) 54% brain predictivity score with super- vised synaptic updates. Figure 88 and C show detailed brain predicitivity scores, relative to fully trained model, for models initialized and trained with WC+CT (B) and models initialized with stan- dard Kaiming Normal and training all weights (C) when trained with range of epochs and labeled images. The specific benchmark scores when either training with all labeled images for a vary- ing number of epochs (Figure A or when training with fewer labeled images until convergence (Figure 3) show the benchmarks of early visual achieve the best results, relative to a fully trained model. The VI score identical over all training states, since we do not train the V1 area.
Notably, ImageNet performance of these networks seems to not be predictive of their brain predic- tivity, since even untrained networks with at-chance ImageNet performance correspond reasonably well to c.g. VI (Figures [ and 2). New normative tasks might be required to explain these results, such as model robustness to image corruptions (Dapello al. 2020).
D DISSECTING TRAINING REDUCTIONS - DETAILS
D.I TRANSFER TO RESNET AND MOBILENET
To show the generalization of our approach we applied the weight compression methods to ResNet- 50 (He et al. 2016) and a MobileNet (Howard et al. 2017) (version 1, multiplier 1.0, image size 224) architecture. We do regenerate sampling distributions or clusters based the new chitectures trained weights, but used the CORnet-S based distributions to sample new weights for the different architectures. Since CORnet-S is inspired by ResNet modules, we applied our criti- cal training approach by training all conv3 layers (equivalent down sampling layers) of ResNet50. For MobileNet we explored various layer mappings. When training only the very few layers that result in reduced feature size, which are implemented as depthwise separable convolutional layers and appear three times overall, performance performance dropped close to random. Those layers however are mapped to CORnet-S' conv2 layers due to their 3 x 3 kernels whereas critical training in CORnet-S trains conv3 down-sampling layers with a kernel size of 1 x 1. To transfer our criti- cal training approach, therefore additionally train the 1 1 MobileNet layers corresponding to conv3. This training version allows for more training but still reduces the amount of trained parame- ters by 43% while maintaining 78% of the original score. For both transfer methods we initialize the
16

---

Published as a conference paper at ICLR 2022
100%
2005
Score of itardard training
Scure 1%% of standard traitine)
695
Mean
Moan
42%
Behowoo
Doharior
0%
Training
Images
Figure 9: Individual brain benchmark scores for WC+CT model A Individual brain predictiv- itiy scores of WC+CT models trained with range of epochs on all images. These models score especially high VI, V2 and V4 already after one epoch in comparison initialization training all weights. IT and Behavior benchmarks continuously improve over later epochs well but fall short of a fully trained model. B Like A, but with models trained until con- vergence on different numbers of labeled images, up to the full dataset of 1.28M images (rightmost points). As in IT and behavioral scores, more images are required.
model with standard
80% V1, V2 and V4 scores with only 100,000 images. For comparable
first layer using the Gabor method based on CORnet-S's mixture-off of-Gaussian distribution. Since the Gabor function is scalable we can produce Gabor kernels varying size. Furthermore we disable BatchNorm biases and weights in all transfer models by freezing them to default values. We found that transferring those distributions on new architectures harms brain predictivity scores. Neverthe- less, the BatchNorm layers still normalize activations by applying the running average and standard deviation.
D.2 COMPARISON OF TECHNIQUES TO REDUCE SUPERVISED SYNAPTIC UPDATES (FIG. 5B)
To analyse the relative contributions of Weight Compression and Critical Training we compare brain predictivitiy scores of different models Figure 5B:
KN A model initialized by standard Kaiming Normal initialization without training.
WC A model initialized by our Weight Compression initialization, described in Section 4, without training.
KN+CT The KN-initialized model trained with Critical Training until convergence, i.c. three downstream layers and the decoder are trained and all other layers remain unchanged.
WC+CT The WC-initialized model with Critical Training. V1 scores do not change be- cause weights in the V1 model area are all frozen.
E TRAINING DETAILS
We used PyTorch 0.4.1 and trained the model using the ImageNet 2012 training set Deng et al. (009). We used a batch size of 256 images and trained on QuadroRTX6000 GPU until conver- gence. We start with a learning rate of 0.1 and decrease it four times by a factor of ten when training loss does not decrease over period of three epochs. For optimization, we use Stochastic Gradient Descent with a weight decay 0.0001, momentum 0.9, and cross-entropy loss between image labels and model logits. We trained all models with these settings except the standard Mobilenet, where we used the pretrained tensorflow model. Since the number of epochs for this model are not clearly stated, use the published value of 100 training epochs Howard al. (2017). The training time of a full CORnet-S with standard Imagenet dataset for 43 epochs is 22. days. All variations with less weights/images/epochs trained in shorter time. Reference models trained for 4 days at most under
17

---

Published as a conference paper at ICLR 2022
the described settings. If not further specified, we show results of one training run. When showing error bars we used seeds 0, 42 and 94.
Code to reproduce analyses from scratch, including the framework for weight compression and critical training, as well as pre-trained models, will be made available through GitHub.
18


Reviews:
[{"review_id": "VGl7TQOdxrG", "reviewer": "Reviewer_FEh9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\nThe paper addresses an important problem in theoretical neuroscience: while trained deep networks appear to be good models of the visual stream, the amount of training they need to get there is incompatible with biology. \n\nTwo of the proposed approaches to reduce the amount of training are somewhat straightforward: training for fewer epochs and freezing the amount of trained layers. However, I think the provided results are important as they show good fit to brain data can be achieved without closely following the training procedures of deep learning.\n\nThe last approach, weight clustering, is perhaps more interesting to the theoretical neuro community, as it works in the same way as genomically encoding the network (the “at birth” state). \n\nImportantly, the proposed ways to reduce the amount of training are tested not only using ImageNet performance, but also ventral stream data, which is arguably more important for neuroscience. \n\nOverall, the paper presents a novel set of results for an important problem, and supports its claims with a significant number of experiments.\n \n## Weaknesses\nAs mentioned before, I think the weight compression part is the most valuable one, and therefore it should receive more attention in the paper.\n1. Clustering should be better explained in the main text. Especially how the number of clusters was chosen.\n2. In the beginning of Sec. 4, you discuss how many bits it would take to encode different networks. I think the same should be done for your experiments. E.g. what is the size of CORnet-S vs. the size of a compressed network?\n3. I think similar weight compression techniques have been used before, see http://proceedings.mlr.press/v80/wu18h.html for example\n4. In general, it’d be interesting to see how at-birth performance changes with the amount of “genetically encoded” information, i.e. the number of clusters passed through compression. E.g., would doubling the amount of clusters you use drastically improve performance? I’m not expecting this experiment to appear in the paper, but perhaps it’s worth discussing.\n\nThe majority of results is limited to one architecture, CORnet-S, which is not a very standard deep learning model. However, I think it is OK as repeating the experiments on something like ResNet50 would be very resource-consuming, and CORnet-S appears to be a better model of the visual stream anyway.\n\n## Recommendation\nAs the paper addresses an important problem with a novel approach and extensive experiments, I recommend to accept the paper.\n\n## Minor points\nIn Fig.1, were learning rates tuned individually for each length of training, or were short runs just snapshots of the longer ones? If it’s the latter, I’d imagine you can get better results by cranking up the learning rate and later decreasing it before you hit a plateau (e.g., by spacing out the same three LR decreases within a shorter run). I guess doing it for each point in Fig.1A is too much training, but repeating all points in Fig.1B-C should be possible?\n\n> For instance, the fully trained model’s 55 million supervised updates translate to 5.5 updates every second, whereas the model with 1 million updates and 76% relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans\n\nI would say that a more correct conclusion is that supervised learning is a bad concept for development, but I also think your results are applicable beyond supervised learning. Basically, you show that reducing the amount of updates in backprop retains good Brain-Score performance. But that probably tells more about SGD dynamics in deep networks rather than a specific loss function, and might easily translate to, say, more realistic self-supervised learning. (I guess you briefly mention this point in the discussion.)\n\n> Abstract\n\n> First, we find that only 2% of supervised updates (epochs and images) are needed to achieve ∼80% of the match to adult ventral stream.\n\n80% of the best performing network -- currently it reads as 80% of absolute performance, which would presumably be 1.0 Brain-Score?\n\nThe caption of Fig.3C-D should better explain what exactly is trained in each setting\n\nIn Sec.6, you should specify that you used MobileNet V1 and ResNet-50.", "summary_of_the_paper": "The paper is concerned with closing the gap between the amount of training in deep networks and in developing brains, as the current deep learning models use an unrealistically large number of synaptic updates. The authors address that with three strategies: less training, weight clustering and training in a subset of layers. All methods are tested individually and in combination with each other on primate ventral stream data.  ", "summary_of_the_review": "The paper provides novel results and important evaluations. There are some methodological issues, but they are not critical to the main point. I therefore favor acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RzRdO_6tQT", "reviewer": "Reviewer_pctr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors present thorough analyses of standard feedforward networks that quantitatively demonstrate that such networks can model the visual stream well while requiring far fewer coordinated parameter changes than occur with traditional training methods.  In particular, they show that very little supervision is required for strong matches to neural responses, that plausible distributions of initial random weights can match neural responses very well, and that weights can be updated very sparsely while still achieving quite high neural response matches. \n\nSuch findings are highly relevant to allowing for such networks to be more plausibly accepted by the neuroscience community as valid models of cortical networks.  The striking finding that well-chosen random weights (Weight Compression) can predict neural responses with more than 50% accuracy is  especially interesting, as it suggests that indeed there may be less training required in mammalian brains compared to randomly initialized neuronal networks. This possibility opens up further avenues of study relating to the ability of real brains to learn with far fewer examples compared to ANNs.  \n\nFigure 4b further demonstrates this possibility, showing that, under the ethologically relevant condition of little supervision, WC + Critical Training, where only a small subset of weights in so-called critical layers are trained, greatly outperforms traditional training.  We note that, in the ideal case, a biologically plausible method might be found that performs well both under little supervision, as WC+CT does here, and under a large amount of supervision, as traditional backprop does, the demonstration of principle for the former case already represents appreciable progress.\n\nHowever, little in the way of tying model predictivity to accuracy on another task was explored. While tying brain predictivity to network performance is of course a very active area in its own right, we feel that addressing this directly within the context of the present work would facilitate readers' abilities to contextualize the present study with respect to ANN performance.\n\nOverall, we find the authors' presented work to be of great interest to those interested in applying ML models to theoretical questions in neuroscience and to have a thorough set of analyses that both illustrates progress in this area while suggesting future avenues of work.\n\nMinor:\n- opening quotes are backwards throughout paper ", "summary_of_the_paper": "While deep networks are known to be biologically implausible, the strong similarities between the feature maps in such networks and the response characteristics (tuning curves) in different areas of the mammalian visual cortex (in particular the ventral stream) has prompted much work into trying to reconcile the known differences. The authors present work that contributes to such inquiries, demonstrating that at least some of the strong impediments in using such networks as plausible visual stream models may be more easily surmountable than they have thus far appeared to be.", "summary_of_the_review": "The authors provide a compelling case that much more biologically plausible sets of initial conditions and sparse, selective supervision and synaptic updates might help substantially close the gap in viewing deep networks as plausible visuocortical models.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8XgiSituROK", "reviewer": "Reviewer_Rdb1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- The paper is clear and well written\n- The analysis is thorough and supports the authors' arguments\n- The method of initializing weights via clusters of trained kernels is interesting  \n- The \"critical training\" method of training only selected layers is also interesting \n\nWeaknesses: \n- \"Recently, such models have been criticized due to how their learning departs from brain development (Marcus, 2004; Grossberg, 2020; Zador, 2019).\" This seems to be the motivation for the work, but those authors don't raise the issues that are addressed in this paper. The Marcus book predates deep learning. Grossberg focuses on catastrophic forgetting. Zador focuses on the sparsity of labelled data in life. I am not sure whether the number of weight updates needed to train deep networks presents a substantial independent concern about their biological plausibility, particularly given the possibility of constant unsupervised learning. If the issue has been raised in the literature, more specific references would help here. If not, I think the authors should argue more specifically that it is a problem, before going on to solve it. \n- The cluster-based initialization method seems to be presented as a way of bringing some learning results through a genetic bottleneck, but the biological plausibility isn't obvious. Further discussion of how this relates to and departs from biology would be helpful. \n- It's interesting that critical training worked so well, but in the context of the paper's goals, the biological relevance of this method should also be laid out more clearly. \n\nMinor: \n- I don't think it was shown how classification performance is affected by the cluster-based initialization method, but it would be nice to see that. \n- \"37 bits per synapse (Zador, 2019)\" This refers to the binary connectivity pattern (an issue of architecture) rather than the weights (which are addressed in this paper). \n", "summary_of_the_paper": "The paper addresses the question of how many weight updates are needed to train a deep network before it takes on biologically realistic representations. The paper uses CORnet-S (a network that has been proposed to resemble primate ventral stream), and BrainScore (a benchmark of how closely related deep network responses are to visual responses in primate ventral stream). Three ways of reducing the numbers of weight updates are explored, each of which is found to vastly reduce updates while moderately reducing BrainScore. First, the network is simply trained for fewer epochs. Second, weights are initialized with clusters of weights found after training. Third, only a subset of layers is updated. A combination of methods leads to 80% of full brain predictivity with 0.5% of the standard number of weight updates. ", "summary_of_the_review": "I think the approach and the results are new and interesting, but I have reservations about the motivation for the work and its connection to biology. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6QcCfqCHr1p", "reviewer": "Reviewer_Ekkd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength:\n\nFirst of all, the question addressed in the paper is essential for both the machine learning and neuroscience community and the author aim at addressing in a well-written piece of work that also clearly contextualizes and motivates the issue at hand. This work uses a state-of-the-art neural architecture, instead of a reduced version of models as is often the case in this type of work. The authors also propose a good analysis and clearly lay out their results, although their interpretation is one of the weaknesses of the paper. \n\n### Weakness:\n\nAlthough it is true that this work has many strengths it is hard to overcome the following weaknesses, which question the motivation of the approach undertaken. First, there exists a set of more principled approaches that have been developed over the last 4-5 years for training ANNs in a biologically plausible way.  Second, the results obtained are very subjective to the authors as the result of the improvement of the score is minor. \n\nAlthough it is true that we don’t expect the authors to solve how the visual system develops and that this work corresponds to a proof of principle, it is still hard to navigate the plausibility of the assumptions made among all possibilities. On the other hand, there is a large set of work that has been proposed to build biologically plausible deep neural networks starting from clear objective function minimization and enforcing locality of learning rules. Compared to this set of works this one seems more adhoc than principled. This represents an important weakness when addressing the more neuroscience-friendly community. \n\nIndeed the approach proposed is more of a combination of existing approaches rather than a novel idea: the use of training of a subset of layers, different weights initialization based on the distribution of trained weights. It is not clear how this provides new insight to the statistical learning community. \n\n", "summary_of_the_paper": "## Updated the score\n\nThis paper proposes to address an important research question for connecting biological (BNN) and artificial neural networks (ANN). Although after training, ANN replicates various salient features of BNN, the way they are often trained is biologically implausible and thus, it is hard to argue that ANNs are suitable for modeling BNNs convincingly. In particular, this work focuses on the already existing CORnet who has shown a high Brain-score. The idea of the authors is to show that they can largely reduce the number of updates when using their methods while still retaining a high Brain-score, thus proposing a potential training mechanism for BNNs.", "summary_of_the_review": "This paper is well-written and addresses an important question possibly helping bridge neuroscience and machine learning. However, this work falls short on both the relevance of the experimental results and the ad-hoc nature of the proposed approach. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper experiments with what is required for a deep neural network to be similar to the visual activity in the ventral stream (as judged by the brainscore benchmark). The authors have several interesting contributions, such as showing that a small number of supervised updates are required to predict most of the variance in the brain activity, or that models with randomized synaptic weights can also predict a significant portion of the variance. These different points serve to better connect deep learning to important questions in neuroscience and the presence of the paper at ICLR would create good questions. The discussion between authors and reviewers resulted in a unanimous vote for acceptance, and the authors already made clarifications to the paper. I recommend acceptance.
Decision: Accept (Spotlight)