Wiring Up Vision: Minimizing Supervised Synaptic Updates Needed to Produce a Primate Ventral Stream
Authors: Franziska Geiger
Martin Schrimpf
Tiago Marques
James J. DiCarlo
Venue:  (2021)
Keywords: computational neuroscience
primate ventral stream
convolutional neural networks
biologically plausible learning

Abstract: After training on large datasets, certain deep neural networks are surprisingly good models of the neural mechanisms of adult primate visual object recognition. Nevertheless, these models are poor models of the development of the visual system because they posit millions of sequential, precisely coordinated synaptic updates, each based on a labeled image. While ongoing research is pursuing the use of unsupervised proxies for labels, we here explore a complementary strategy of reducing the required number of supervised synaptic updates to produce an adult-like ventral visual stream (as judged by the match to V1, V2, V4, IT, and behavior). Such models might require less precise machinery and energy expenditure to coordinate these updates and would thus move us closer to viable neuroscientific hypotheses about how the visual system wires itself up. Relative to the current leading model of the adult ventral stream, we here demonstrate that the total number of supervised weight updates can be substantially reduced using three complementary strategies: First, we find that only 2% of supervised updates (epochs and images) are needed to achieve ~80% of the match to adult ventral stream. Second, by improving the random distribution of synaptic connectivity, we find that 54% of the brain match can already be achieved “at birth" (i.e. no training at all). Third, we find that, by training only ~5% of model synapses, we can still achieve nearly 80% of the match to the ventral stream. When these three strategies are applied in combination, we find that these new models achieve ~80% of a fully trained model's match to the brain, while using two orders of magnitude fewer supervised synaptic updates. These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's updates based on visual experience).

Under review as a conference paper at ICLR 2021
WIRING UP VISION: SYNAPTIC UPDATES NEEDED TO PRODUCE A PRIMATE VENTRAL STREAM
MINIMIZING SUPERVISED
Anonymous authors Paper under double-blind review
ABSTRACT
After training on large datasets, certain deep neural networks are surprisingly good models of the neural mechanisms of adult primate visual object recognition. Nevertheless, these models are poor models of the development of the visual system because they posit millions of sequential, precisely coordinated synaptic updates, each based on labeled image. While ongoing research is pursuing the use of unsupervised proxies for labels, we here explore a complementary strategy of reducing the required number of supervised synaptic updates produce an adult-like ventral visual stream (as judged by the match to VI, V2, V4, IT, and behavior). Such models might require less precise machinery and energy expenditure to coordinate these updates and would thus move closer to viable neuroscientific hypotheses about how the visual system wires itself up. Relative to the current leading model of the adult ventral stream, we here demonstrate that the total number supervised weight updates can be substantially reduced using three complementary strategies: First, we find that only 2% of supervised updates (epochs and images) are needed to achieve ~880 the match to adult ventral stream. Second, by improving the random distribution of synaptic connectivity, we find that 54% of the brain match can already be achieved "at birth" (i.e. no training all). Third, we find that, by training only ~5% of model synapses, we can still achieve nearly 80% of the match to the ventral stream. When these three strategies are applied combination, we find that these new models achieve 80% of a fully trained model's match to the brain, while using two orders of magnitude fewer supervised synaptic updates. These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's 's updates based on visual experience).
INTRODUCTION
Particular artificial neural networks (ANNs) are the leading mechanistic models of visual processing in the primate visual ventral stream (Schrimpf al., 2018; Kubilius et al., 2019; Dapello et al., 2020). After training on large-scale datasets such as ImageNet (Deng et al., 2009) by updating weights based on labeled images, internal representations of these ANNs partly match neural representations in the primate visual system from early visual cortex V1 through V2 and V4 to high-level IT (Yamins et al., 2014: Khaligh-Razzi
& Kriegeskorte, 2014: Cadena et al., 2017: Tang et al., 2018; Schrimpf et al., 2018; Kubilius et al., 2019), and model object recognition behavior can partly account for primate object recognition behavior (Rajalingham et al., 2018: Schrimpf et al., 2018)
Recently, such models have been criticized due to how their learning departs from brain development (Marcus, 2004; Grossberg, 2020; Zador, 2019). For example, all the current top models of the primate ventral stream rely on trillions of supervised synaptic updates, i.e. the training of millions of parameters with millions of labeled examples over dozens of epochs. In biological systems, on the other hand, the at-birth synaptic wiring as encoded by the genome already provides structure that is sufficient for macaques to exhibit adult-like visual representations after few months (Movshon & Kiorpes, 1988; Kiorpes & Movshon, 2004; Seibert, 2018), which restricts the amount of experience

---

Under review as a conference paper at ICLR 2021
dependent learning. Furthermore, different neuronal populations in cortical circuits undergo different plasticity mechanisms: neurons in supragranular and infragranular layers adapt more rapidly than those in layer 4 which receives inputs from lower areas (Diamond et al., 1994; Schoups et al., 2001), while current artificial synapses, on the other hand, all change under the same plasticity mechanism. While current models provide basic understanding of the neural mechanisms of adult ventral stream inference, can we start to build models that provide an understanding of how the ventral stream "wires itself up" models of the initial state at birth and how develops during postnatal life?
Related Work. Several papers have addressed related questions in machine learning: Distilled student networks can be trained on the outputs of teacher network (Hinton et al., 2015; Cho & Hariharan, 2019; Tian et al., 2019), and, in pruning studies, networks with knocked out synapses perform reasonably well (Cheney al., 2017: Morcos et al., 2018), demonstrating that models with many trained parameters can be compressed which is further supported by the convergence of training gradients onto a small subspace (Gur-Ari et al., 2018). Tian et al. (2020) show that pre-trained encoder's ' fixed features can be used to train thin decoder with performance close to full fine- tuning and recent theoretically-drivee work has found that training only BatchNorm layers (Frankle et al., 2020) or determining the right parameters from large pool of weights (Frankle et al., 2019; Ramanujan et al., 2019) can already achieve high classification accuracy. Unsupervised approaches are also starting to develop useful representations without requiring many labels by inferring internal labels such as clusters or representational similarity (Caron et al., 2018; Wu et al., 2018; Zhuang et al., 2019; Hénaff et al., 2019; Konkle & Alvarez, 2020; Zhuang al., 2020). Many attempts are also being made to make the learning algorithms themselves more biologically plausible (e.g. Lillicrap al., 2016; Scellier & Bengio. 2017; Pozzi al., 2020) Nevertheless, all of these approaches require many synaptic updates in the form of labeled samples or precise machinery to determine the right set of weights. In this work, we take first steps of relating findings in machine learning to neuroscience and using such models to explore hypotheses about the product of evolution (a model's "birth state") while simultaneously reducing the number of supervised synaptic updates (a model's visual experience dependent development) without sacrificing high brain predictivity.
Our contributions follow from a framework in which evolution endows the visual system with a well-chosen, yet still largely random "birth" pattern of synaptic connectivity (architecture ini- tialization), and developmental learning corresponds to training fraction of the synaptic weights using very few supervised labels. We do not view the proposed changes as fully biological models of post-natal development, only that they more concretely correspond to biology than current models. Solving the entire problem of development all once is too much for one study, but even partial improvements in this direction will likely be informative to further work. Specifically,
1. we build models with fraction of supervised updates (training epochs and labeled im- ages) that retain high similarity to the primate ventral visual stream (quantified by a brain predictivity score from benchmarks on Brain-Score (Schrimpf et al., 2018)),
2. we improve the "at-birth" synaptic connectivity to achieve reasonable brain predictivity with no training at all,
3. we propose a thin, "critical training" technique which reduces the number of trained synapses while maintaining high brain predictivity.
4. we combine these three techniques to build models with two orders of magnitude fewer supervised synaptic updates but high brain predictivity relative to fully trained model
Code and pre-trained models will be available through GitHub.
2 MODELING PRIMATE VISION
We evaluate all models on suite of ventral stream benchmarks in Brain-Score (Schrimpf al., 2018), 8), and we base the new models presented here on the CORnet-S architecture as this is currently the most accurate model of adult primate visual processing (Kubilius et al., 2019).
Brain-Score benchmarks. To obtain quantified scores for brain-likeness, we use thorough set of benchmarks from Brain-Score (Schrimpf et al., 2018). To keep scores comparable, we only included those neural benchmarks from Brain-Score (Schrimpf al., 2018) with the same predictivity
2

---

Under review as a conference paper at ICLR 2021
metric. All benchmarks feed the same images to candidate model that were used for primate experiments while "recording" activations or measuring behavioral outputs. Specifically, the VI and V2 benchmarks present 315 images of naturalistic textures and compare model representations to primate single-unit recordings from Freeman et al. (2013) (102 V1 and 103 V2 neurons); the V4 and IT benchmarks present 2,560 naturalistic images and compare models to primate Utah array recordings from Majaj et al. (2015) (88 V4 and 168 IT electrodes). A linear regression is fit from model to primate representations in response 90% of the images and its prediction score on the held-out 10% of images is evaluated with Pearson correlation, cross-valiee validated 10 times. The behavioral benchmark presents 240 images and compares model to primate behavioral responses from Rajalingham et al. (2018). A logistic classifier is fit on models' penultimate representations on 2, 10 separate labeled images. The classifier then used to estimate probabilities for 240 held-out images. Per-image confusion patterns between model and primate are compared with Pearson correlation. All benchmark scores are normalized by the respective ceiling. We primarily report the average brain predictivity score as the mean of VI, V2, V4, IT, and behavioral scores.
We note that the Brain-Score benchmarks in this study are based on a limited number of data and thus present a possible limitation. Nonetheless, they are the most extensive set of primate ventral stream benchmarks that is currently available and the scores seem to generalize to new experiments (Kubilius al., 2019).
Brain-Score provides separate sets of data as public benchmarks which we use to determine the type of distribution in Section 4, and the layer-to-region commitments of reference models.
CORnet-S. The current best model on the Brain-Score benchmarks is CORnet-S (Kubilius et al., 2019), shallow recurrent model which anatomically commits to ventral stream regions. CORnet-S has four computational areas, analogous to the ventral visual areas V1, V2, V4, and IT, and a linear decoder that maps from neurons in the model's last visual area its behavioral choices. The recurrent circuitry (Figure 3B) uses up- and down-sampling convolutions to process features and identical in each of the models visual areas (except for VlCoR), but varies by the total number of neurons in each area.
We base all models developed here on the CORnet-S architecture and the same hyper- as proposed in (Kubilius et al., 2019). Representations are read out at the end of anatomically corresponding areas.
3. HIGH SCORES IN BRAIN PREDICTIVITY CAN BE ACHIEVED WITH FEW SUPERVISED UPDATES
We evaluated the brain predictivity scores of CORnet-S variants that were trained with fewer epochs and images. Models trained with an initial learning rate of 0. divided by 10 when loss did not improve over 3 epochs, and stopping after three decrements.
Figure 1 shows model scores neural and behavioral Brain-Score measures, relative to model trained for 43 epochs on all 1.28M labeled ImageNet images. In Panel A, we compare the average score over the five brain measures of various models the number of supervised updates that each model was trained with, defined as the number of labeled images times the number of epochs. While fully trained model reaches average score of .42 after 55,040,000 supervised updates (43 epochs x 1. 1.28M images), a model with only 100,000 updates already achieves 50% of that score, and 1,000,000 updates increase brain predictivity scores to 76%. Models are close to convergence score after 10,000,000 supervised updates with performance nearly equal to full training (97%). Scores grow logarithmically with an approximate 5% score increase for every order of magnitude more supervised updates.
Figures IB and show individual neural and behavioral scores of models trained with fewer training epochs or labeled images independently. Early mid visual representations (V1, V2, and V4 scores) are especially closely met with only few supervised updates, reaching 50% of the final trained model in fractions of the first epoch (Figure IB). After only one full iteration over the training set, V1, V2, and V4 scores are close to their final score (all >80% while IT requires two epochs to reach a comparable level. Behavioral scores take slightly longer to converge (>80% after 7 epochs).
3

---

Under review as a conference paper at ICLR 2021
Standarrd toaniig
10235
20% spdates 97% score
100%
85%
updaree 7e0ii
80%
Scare [% sarrrrr vuinee
800
02%
updates sccre
Score of uorddard
00%
Siate Prod
00%
Mose Biyis Pred
Prudictivity
40%
40%
60%
25%
Behaxor
20%
Benwer
0%5
0%
107
30*
20
Supervised updates (Traiing Epocto Labelod broges)
Training Epochs
Labeled Images
Figure 1: High scores in brain predictivity can be achieved with few supervised updates (log x-axes). Average brain predictivity of models trained with a range of supervised updates (epochs x images). Each dot is different hypothesis of how the ventral visual stream might have developed and shows the adult brain-likeness score that is achieved by that model. Fairly brain-like representations are already realized with few supervised updates, relative
fully trained model (black dot; see also Figure 6). Standardly trained CORnet-S is set to be 100% score in brain predictivity on the benchmarks (0.42 absolute). "pixels" baseline achieves 7% brain predictivity score (0.03 absolute). B Individual brain predictivity scores over epochs. Models start to approximate the primate ventral stream with few training epochs. Lower visual areas (VI, V2) are approximated earlier in training. C Like B, but number of training images instead of epochs. Few images are sufficient to approximate especially early visual areas.
Similarly, when training until convergence with fractions of the 1.28M total images, 50,000 images are sufficient to obtain high neural scores (80% full training in VI, V2, V4, IT). Behavioral scores again require more training: half the standard number of labeled images is needed to surpass 80%.
Concretely relating supervised updates to primate ventral stream development, Seibert (2018) estab- lishes that more than 44 months to reach adult-level primate IT cortex (as assessed by its capability to support adult level object recognition). From this estimate, we can compute how many supervised updates per second different models in Figure 1A would require (assuming those updates are evenly distributed over the 10 million seconds). For instance, the fully trained model's 55 million supervised updates translate to 5.5 updates every second, whereas the model with 1 million updates and 76% relative brain predictivity translates to one labeled image update every seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans (Yarbus, 1967; Gibaldi & Sabatini, 2020)
10 million seconds of waking visual experience is needed
4 "AT-BIRTH" SYNAPTIC CONNECTIVITY YIELDS REASONABLE BRAIN PREDICTIVITY WITH NO TRAINING AT ALL
If few supervised updates can get model representations fairly close to fully trained model (Figure 1). how close are the initial representations without any training? In relation to biology and following the introduced framework of treating all consecutive training as developmental learning, these "at-birth" synaptic connections would result from information encoded in the genome as a product of evolution.
Due to the genome's ' capacity bottleneck, it is infeasible to precisely encode every synapse. Primary visual cortex alone contains ~1.488 neurons per hemisphere (Leuba Kraftsik, 1994), 1EB synapses per neuron, each requiring ~37 bits per synapse (Zador, 2019). Thus, without any clever rules, specifying the connections in only one hemisphere of V1 could require up to ~55.E112 bits orders of magnitude more than the entire genome's 's 1GB 8E9 bits (Zador, 2019). Sampling synaptic weights from reasonably compressed distributions on the other hand places only little memory requirements genetic encoding while potentially yielding reasonably useful initial weights. Current machine learning techniques for initializing weights, such as Kaiming Normal (He et al., 2015), sample from a Gaussian distribution with μ = and o = V2/N where N is the number of incoming connections per layer.
4

---

Under review as a conference paper at ICLR 2021
A
Birth Initialization (per layer)
B
V1.covl
Brain Predictivity [% of standard training]
100%
trained
weights
80%
V1. conv2
Compress
60%
Weight Compression
v2.com2
Weight Compression
Kaiming Normal
p,c. O.
40%
Kaiming
V4 conv2
Normal
Sample
Sample
IT conv22
Paoks
weights
weights
0%
tmiring
Full trairing
Figure "At-birth" synaptic connectivity yields reasonable scores in brain predictivity, Specifying the initial weight distribution: Kaiming Normal (KN, He et al., 2015) samples from a generic Gaussian. Weight Compression (WC) compresses trained weights into low-parameter clustered distributions that weights can be initialized from. B Visualization of WC compressed parameters: Gabor filters for first layer and cluster centers for following layers with kernel size > 3. The cluster centers capture intuitive variety of kernel types. C "At-birthh" representations with WC achieve 54% score brain predictivity of a fully trained model, with no training at all. Scores after training remain virtually unchanged.
To improve on Kaiming Normal initialization, we explored multi-dimensional distributions as more expressive alternative. These distributions only require small number of parameters, but unlike current generic initializers, we explicitly specify them for each layer. To determine the right parameterization, we compress trained model's weights into clusters which we then sample from (Weighl Compression, WC).
More specifically, for all convolutional layers except the first, we cluster the kernel weights and later sample from the clusters. To capture the relative importance of clusters we fit a normal distribution to the cluster frequency over kernels. In batch normalization layers, we one normal distribution each to the weights and biases. For the first convolutional layer only, we employ Gabor prior on the weights following studies in VI (Hubel & Wiesel, 1962; Jones & Palmer, 1987) (supplement).
Model interpretability studies (Zeiler & Fergus, 2013; Olah et al., 2020; Cammarata et al., 2020) classify model weights, comparable to WC's representation. Visualizing the weight compressions from trained CORnet-S weights (Figure 2B), find that the first layer's Gabor filters qualitatively align with an analysis by Cammarata et al. (2020). Cluster centers seem to represent an intuitive division of channel types with opposite types every layer.
While intuitively sampling weights from a compression of trained weights should somewhat recover reasonable brain predictivity scores, not every implementation satisfies both high brain predictivity and sufficient compression for the genome bottleneck (Appendix B.3).
Applying WC to CORnet-S, we first obtain a compressed and clustered set of parameters, from which we sample entirely new weights to yield a new model CORnet-Swe- This model is not trained at all and we only evaluate the goodness of its initial wiring on the suite of Brain-Score benchmarks. Strikingly, we find that even without any training. CORnet-Swc achieves 54 + 1.5%of the brain predictivity score relative to mully-trained model (Figure 2). representing 12 percent point improvement (n 10 seeds; permutation test p < E 5) over the Kaiming Normal initialized model with a score of 43 1.7% Early ventral stream regions V1 and V2 are predicted especially well with no loss in score but we note that these two benchmarks are less well predicted by the trained model to begin with. V4 scores also approximate those of a trained model relatively well (75%). The major drop occurs in the IT and especially behavioral scores where CORnet-Swe only reaches 39% and 6% of the trained model's 's score respectively. Similarly, a trained linear decoder on CORnet-Swe's 's IT representations only reaches 5% of a trained model's ImageNet top-1 accuracy.
Weight Compression explores the hypothesis that evolution may have discovered an initialization strategy with improved at-birth representations (relative current initializations). WC is most likely
5

---

Under review as a conference paper at ICLR 2021
A
Standard Training
V1
V2
V4
Brain Predictivity (% of standdad training
100%
Standard Training
Cntical Training
80%
Dgeneveel Copp
Downstream Training
60%
Downstream Training
40%
20%
Critical Training
0%
Dounsample Corv
Number of trained parameters [Million]
Imagenent performance (%% stardard traiinn]
100%
Standard Training
80%
Critical Training
Frozen
Trained
60%
Upsample
Downsample Conv
Down CarEo
40%
Conv
Input
20%
Downstream Training
Sklp
Gate
Add
Number of trained parameters [Milion]
Batchnorm
Non-Linaarit
Figure 3: Training only critical layers reduces the number of updated synapses while maintain- ing high brain predictivity. We could naively reduce the parameters of a fully-trained model by freezing layers from the bottom up, training only the top layers ("Downstream Training DT"; gray box). We instead propose Critical Training (CT) which only trains down-sampling layers (blue box). CORnet-S circuitry. CT only trains critical layers, leaving the rest frozen. C Naively reducing parameters from standard training (black dot, top right) quickly deteriorates brain predictivity scores (DT, gray line) whereas Critical Training reduces parameters while retaining high scores (blue line, CT), Like C, but measuring ImageNet score. CT retains nearly half the score with a fraction of parameters.
not how evolution found the at-birth synaptic connections, but shows that with nearly identical capacity, an alternative initialization distribution leads to networks that are more tiken-like their adult state revealing new space of possibilities (hypotheses) that should be considered.
TRAINING THIN BOWN-SAMPLES LAYERS REDUCES THE NUMBER OF UPDATED SYNAPSES WHILE MAINTAINING HIGH BRAIN PREDICTIVITY
While improved "at-birth" connectivity can reach 54% of a fully-trained model's score, additional visual-experience dependent updates appear necessary reach higher predictivities. With standard training, each iteration simultaneously updates all of the millions of synaptic weights in the neural network, which may be difficult to implement in the brain. Alternatively, learning could take place preferentially in specific components. Cortical circuits are heterogeneous and different neuronal populations undergo distinct plasticity mechanisms, For example, neurons in supragranular and infragranular layers adapt more rapidly than those in layer 4, where inputs from lower areas arrive, as observed in rat somatosensory cortex (Diamond et al., 1994) and primate VI (Schoups et al., 2001).
As proof-of-principle that training reduced set of layers retain high performance, propose a novel thin training technique, which we term Critical Training (CT; Figure 3A). CT updates only the weights in critical layers, instead of updating every single model synapse. In CORnet-S. each of the blocks has one down-son nown-sampling layer to produce an area's final representation (Figure 3B). We explore successive variants of applying CT up block in the architecture and then training the

---

Under review as a conference paper at ICLR 2021
A
B
Standard Training
Mobilenet
Brain Predictivity [percent points
100%
Brain Predictivity standard training)
Resnet
better
100
55% updates = 79% score
80%
1000
11
27
19
15
8
updates 54% score
Labeled Images
10000
11
26
8
-2
-1
-3
60%
50000
-2
40%
100000
11
-14
-11
-19
HMAX
500000
11
-3
-8
-11
-17
-17
20%
better
Piwels
All
-11
-7
supervized updates +CC++C
20 Convergence
10
10t2
1033
10
10th
Supervised synaptic updates
Training Epochs
(training epochs labeled images trained synapses)
Figure 4: High brain predictivity be achieved with few supervised synaptic updates through a combination of training reductions (log x-axis). A By reducing updates with a combination of fewer supervised updates (Figure 1), improved initialization WC (Figure 2), and training only down- sampling layers CT (Figure 3), the resulting models (dark blue dots; fewer supervised updates alone in light blue) maintain high brain predictivity scores while requiring only a fraction of supervised synaptic updates compared to standard CORnet-S (black dot, top right). B Comparison between WC-initialized models trained with CT versus standardly initialized models training all weights, when varying training epochs and labeled images. Colors represent their percent point difference in brain predictivity scores. WC+ CT improve performance in regimes with few epochs and images.
following blocks, e.g. freezing VI, V2, V4 with critical training of the respective down-sampling layers and additional IT training. The final CT ventral stream model is almost completely frozen and only the synapses generating each cortical area's output trained.
We compared Critical Training to a naive approach reducing the trained parameters by freezing entire model blocks, for instance keeping the VI and V2 blocks fixed while training V4 and IT blocks. We term this block-wise freezing and training approach Downstream Training (DT). Compared to standard sock-propagation training all the weights, both CT and DT reduce the number of trained parameters (Figure 3C). However, while the average score with DT (gray) already drops below 65% with over quarter of trained parameters remaining, CT (blue) maintains over 75% with only 1.4 of 52.8 million parameters trained. The choice of which critical layers to train also matters: training the connecting layers between regions- i.c. the last (down-same sampling, default) or first sampling) layer retains most of the performance whereas training layers such as BatchNorm performs worse.
By redueing the number of trained parameters. Critical Training also yields engineering benefits in training time with 30% reduction in the time per epoch at over 80% of the score in brain predictivity and more than 40% of the ImageNet score. The training time reduction less drastic than the parameter reduction because most gradients are still computed for early down-sampling layers.
6 HIGH BRAIN PREDICTIVITY CAN BE ACHIEVED WITH A RELATIVELY SMALL NUMBER OF SUPERVISED SYNAPTIC UPDATES
All three training reduction methods independently minimize the number of supervised synaptic updates required to reach reasonably high brain predictivity score. Reducing the number of supervised updates minimizes required updates by a smaller number of epochs and images (Section 3); Weight Compression (WC) improves the at at-birth synaptic connectivity for high initial scores with no training at all (Section 4); and Critical Training (CT) reduces the number of synapses that are updated during training (Section 5). We now combine these three methods to build novel models that

---

Under review as a conference paper at ICLR 2021
Figure 5: Dissecting training re- ductions. A Transfer to other net- works. We sample from WC ini- tializations determined on CORnet- S, followed by Critical Training of only down-sampling layers. B Ab- solute scores on individual bench- marks of combinations of initializa- tion (KN/WC, Figure 2), and with critical training (CT, Figure 3) tech- niques.
Drain Preductivtty (obol.n))
MobileNet
Resnet50
Brain Predictiiny (accom (atcam]]
21% paramsterss - 91%
57% parameters 578% soore
Behavior
Number of trained parameters [Milion]
only require a small number of supervised synaptic updates to reasonably capture the mechanisms of adult ventral visual stream processing and object recognition behavior.
Figure 4AA shows the average brain predictivity of a range of models with varying numbers of super- vised synaptic updates relative to standard trained CORnet-S (black dot, 3,000 trillion supervised synaptic updates). With a reduced number of supervised updates (training epochs and labeled images) but standard initialization and training all weights (light blue dots), models require 5.2 trillion updates to achieve >50% of the score of a fully trained model and about 100 trillion updates to reach 80% brain score. Adding WC+CT (dark blue dots). the corresponding model already reaches 53% at birth with 0 supervised synaptic updates. At 0.5% the updates of fully trained model (14 trillion vs. 3,000 trillion), models then reach 79% of the score (IZ model with modeling choices marked in Figures to 3). Reference models (gray dots) MobileNet (Howard et al., 2017) and ResNet (He et al., 2016) obtain high scores, but also require many supervised synaptic updates. HMAX (Riesenhuber Poggio, 1999) is fully specified with no updates but lacks in score.
We next examined interactions between the methods by comparing models initialized with WC and trained with CT to models with standard initialization and training all weights, when both are trained with fewer epochs and images. Figure 4B shows the percent point difference between the two model families. WC+CT CT yield strong benefits (green numbers) in regime with few supervised updates, improving by up to 27 percent points when training for only epoch on 1,000 images. With many updates the other hand, WC+CT is actually less advantageous than standard training (red numbers): with all 43 epochs and 1. 1.28M images, the score reduces by 17 percent points. WC+CT therefore most positively interact with small budget of supervised updates, which focus in this work.
DISSECTING TRAINING REDUCTIONS
We asked whether the developed techniques would generalize to architectures other than the CORnet-S architecture that they were based on, perhaps novel way to construct model taxonomies. We therefore applied Weight Compression (WC) and Critical Training (CT) to ResNet-50 (He et al., 2016) and MobileNet (Howard et al., 2017) architectures, both nigh-perffrming models on Brain- Score. We used WC distributions determined on CORnet-S, i.e. we tested transfer without re-fitting. WC+CT maintain 91% of the score in ResNet despite an almost 80% reduction in parameters. When applied to MobileNet, the average score drops by 22% and parameters are reduced less strongly (43%). This difference in retaining the score could be due MobileNet already being very compressed, or having less similar architecture.
With most analyses so far comparing an average score, we dissected the relative contributions of WC and CT to individual benchmarks (Figure 5B). We compared KN to WC initialization, as well as resulting models after critical training (KN+CT and WC+CT). WC initialization improves most over KN in early visual regions VI and V2, while additional training with CT is most beneficial in mid- to level visual cortex V4 and IT, as well as the behavioral benchmark.
DISCUSSION
We developed a range of models with neural and behavioral scores approaching those the current leading model of the adult ventral visual stream, while requiring only fraction of supervised
8

---

Under review as a conference paper at ICLR 2021
synaptic updates. These models were built by complementarily 1) reducing the number of supervised updates, i.c. training epochs and labeled images; 2) improving the "at birth" distribution of synaptic connectivity; and 3) training only critical synapses at the end of each model area. The techniques and resulting models proposed here are first steps to more closely modeling not just adult primate visual processing, but also exploring the underlying mechanisms of evolution and developmental learning.
These proof-of- priiecipleeieeeeeeee demonstrations are far from accounting for the rich information encoded in the genome or the developmental learning that together result in adult mechanisms of visual processing. We here started from CORner-S, which is the current leading model of the adult ventral stream, but does not fully predict all brain measurements. The architecture we based our techniques on might therefore be flawed. We verified favorable transfer to models with similar architectures such as ResNet, but generalization to an already compressed MobileNet was limited (Figure 5A).
Relating to genomic mechanisms, the proposed techniques should generalize to other domains such as auditory processing. With the capacity bottleneck in the genome, mechanisms for wiring up would likely be shared between similar systems. The fact that early visual areas converge earlier during training (Figure 1) and are better predicted than higher areas by WC initialization is consistent with developmental studies of the primate ventral stream. In humans, behaviors that rely on low-level spatial and temporal processing of visual inputs reach adult-like performance considerably earlier than complex visual behaviors that rely on higher cortical regions, such as face perception (Ellemberg et al., 1999; Grill-Spector et al., 2008)
A critical component in more closely modeling primate development is reduce the dependence on labels altogether. Recent unsupervised approaches are starting to rival the classification performance of supervised models (Caron et al., 2018; Hénaff et al., 2019; Zhuang al., 2020) and combining them with the advances presented here could further reduce the number of synaptic updates. More precise biological measurements are required quantify the number of (parallel) experience-dependenn updates. Current unsupervised techniques however still require back proopppgpattionn which is routinely criticized as non-biolog,cal, among others due to the propagation of gradients (Grossberg, 1987; Whittington & Bogaez, 2019; Hunsberger, 2017). Local learning rules might alleviate these concerns and with critical training (Figure 3), could be sufficient to learn in only a subset of layers.
The changes to model initialization and training presented here serve as a proof-of proof-of-principle that models can be changed to more closely align with primate development by reducing training steps with labeled images and improving initialization. It is also possible to achieve high brain predictivity when training only down-sampling layers, but all these models are still far from the actual biolog- ical mechanisms. We expect future work in this direction to further close the gap with improved evolutionarily encoded wiring mechanisms and developmental learning rules.
REFERENCES
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Score: Which artificial neural network for object recognition is most brain-like? bioRxiv, 2018.
Jonas Kubilius, Martin Schrimpf, Ha Hong. Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Rop Kailyn Schmidt, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Like Object Recognition with High- Performing Shallow Recurrent ANNs. In Neural Information Processing Systems (NeurIPS), pp. 12785-127980 12796. 2019.
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D. Cox, and James DiCarlo. Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations. In Neural Information Processing Systems (NeurIPS). 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and L.i Fei-Fei. ImageNet: A large-scale hierarchical image datahase. In IEEF Conference Computer Vision and Pattern Recognition, pp. 248-255. 2009.
Daniel LK Yamins, Ha Hong. Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimizzed hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 11 (1(23):86199866)
2014.
Seyed-Mahdl Saaligh-Razavy and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology, 10(1 (0(11):c1003915 2014.
Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys. Andreas S Tolias, Matthias Bethge, and Alexander S Feker. Deep convolutional models improve predictions of macaque v1 responses to natural images. bioRxiv, pp. 201 764, 2017.

---

Under review as a conference paper at ICLR 2021
Hanlin Tang. Martin Schrimpf. William Lotter, Charlotte Moerman, Ana Paredes, J.O. Josue Ortega Caro, Walter Hardesty, David Cox, and Gabriel Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences (PNAS), 1 115(35):8835 8840, 2018.
Rishi Rajalingham. Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo. Large- scale, resh-resolution comparison of the core visual object recognition behavior of humans, monkeys, and srate-of-the-ar deep artificial neural networks. Journal of Neuroscincce, pp. 7255-7269, 2018
Gary Marcus. The Birth of the Mind: How Tiny Number of Genes Creates the Complexities of Human Thought. 2004.
Stephen Grossberg. A Path Toward Explainable Al and Autonomous Adaptive Intelligence: Deep Learning. Adaptive Resonance, and Models of Perception, Emotion, and Action. 14:36, 2020.
Anthony Zador. A Critique of Pure Learning: What Artificial Neural Networks can Learn from Animal Brains. bioRxiv preprint, 2019.
J. Anthony Movshon and Lynne Kiorpes. Analysis of the development of spatial contrast sensitivity in monkey and human infants. Journal of the Optical Society of America A (JOSA A), 5(1)):2166. 1988.
Lynne Kiorpes and J. Anthony Movshon. Development of sensitivity to visual motion in macaque monkeys. Visual Neuroscience, (16):551-859, 2004.
Darren Seihert. High-level visual ohject representation in juvenile and adult primates. PhD thesis, Massachusetts
Institute of Technology, 2018.
Mathew E. Diamond, Wei Huang, and Ford F. Ebner. Laminar comparison of somatosensory cortical plasticity. Science, (65(5180):18)) 1885-18888 1994.
Anick Schoups, Rufin Vogels, Ning Qian, and Guy Orban. Practising orientation identification improves orientation coding in VI neurons. Nature, 412(6846):5499-5 553, 2001.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv preprint. 2015.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In International Conference on Computer Vision (ICCV), volume nottob, pp. 4793 4801. 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Representation Distillation. arXiv preprint, 2019. Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the Robustness of Convolutional Neural Networks to Internal Architecture and Weight Perturbations. arXiv preprint, 2017.
Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations (ICLR), 2018. Guy Gur-Ari, Daniel A. Roherts, and Ethan Dyer. Gradient Descent Happens in a Tiny Subspace. arXiv preprint, 2018.
Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B. Tenenhaum, and Phillip Isola. Rethinking Few-Shot Image
Classification: Good Embedding Is All You Need? arXiv preprint, 2020.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features CNNs. arXiv preprint, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. The Lottery Ticket
Hypothesis at Scale. arXiv preprint, 2019.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's Hidden in Randomly Weighted Neural Network? arXiv preprint, 2019.
Mathilde Caron, Piotr Bojanowski. Armand Joulin, and Matthijs Douze. Deep Clustering for Unsupervised Learning of Visual Features. In European Conference Computer Vision (ECCV), pp.
2018.
Zhirong Wu, Yuanjun Xiong. Stella X Yu, and Dahua Lin. Unsupervised Feature Learning via -arammeettti Instance Diserimination. Computer Vision and Puttern Recognition (CVPR), pp. 3733-3742, 2018
Chengxu Zhuang, Alex Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In International Conference on Computer Vision (ICCV), pp. 6001-6011 2019.
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-Efficient Image Recognition with Contrastive Predictive Coding. In Computer Vision and Pattern Recognition (CVPR), 2019.
Talía Konkle and George A. Alvarez. Instance-level contrastive learning yields human brain-like representation bioRxiv preprint. 2020.
without
Chengxu Zhuang. Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C. Frank, James DiCarlo, and Daniel
L. K. Yamins. Unsupervised Neural Network Models of the Ventral Visual Stream. hioRxiv preprint, 2020.
Timothy P. Lillicrap. Daniel Cownden, Douglas B. Tweed, and Colin J. Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communications, 7(1):13276, 2016.
Benjamin Scellier and Yoshua Bengio. Equilibrium Propagation: Bridging the Gap between Energy- -Based Models and Backpropagation. Frontiers in Computational Neuroscience, 11:24, 2017.
10

---

Under review as a conference paper at ICLR 2021
Isabella Pozzi, Sander M Bohté, and Pieter R Roelfsema. Attention-Gated Brain Propagation: How the brain can implement reward-basee error backpropagation. In Neural Information Processing Systems (NeurIPS), volume 33, 2020.
Jeremy Freeman, Corey M Ziemba, David Heeger, Eero Simoncelli, and Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature Neuroscience, 16(7):974-981 2013. Najib Majaj, Ha Hong, Ethan A Solomon, and James DiCarlo. Simple learned weighted sumss inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 15(39):13402-13413 2015.
A. L. Yarbus. Eye movements and vision. Plenum Press, 1967.
Agostino Gibaldi and Silvio P. Sabatini. The saccade main sequence revised: A fast and repeatable tool for oculomotor analysis. Behavior Research Methods, pp. 1 1-21, 2020.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human- Level Performance on ImageNet Classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1022 2015.
G Leuba and R Kraftsik. Changes in volume, surface estimate, three-dimensional shape and total number of neurons of the human primary visual cortex from midgestation until old age. Anatomy and Embryology, 190 (4):351-366, 1994.
David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1):106-154, 1962.
P. Jones and L. A. Palmer. The two-dimensional spatial structure simple receptive fields cat striate cortex. Journal of Neurophysiology. 58(6):1187-122) 1987.
Matthew D Zeiler and Rob Forgus. Visualizing and Understanding Convolutional Networks. arXiv preprint, 2013.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 5(3), 2020.
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, and Ludwig Schubert. Thread: Circuits. Distill, 2020.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efffccent Convolutional Neural Networks for Mobile Vision Applications. arXiv preprint, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 2(1 10119, 1999.
Dave Ellemberg. Terri I.. Lewis, Chang Hong Liu, and Daphne Maurer. Development of spatial and temporal vision during childhood. Vision Research. 39(14);2355-233, 1999.
Kalanit Grill-Spector, Golijeh Golarai, and John Gabrieli. Developmental neuroimaging of the human ventral visual cortex. Trends in Cognitive Sciences, 12(4):15--162, 2008.
Stephen Grossberg. Competitive learning: From interactive activation to adaptive resonance. Cognitive Science, 11(1):23-66 63, 1987.
James C.R. Whittington and Rafal Bogacz. Theories Eror Back-Propagation the Brain. Trends Cognitive
Sciences, 23(3):235-250, 2019.
Eric Hunsberger. Spiking Deep Neural Nenworks: Engineered and Biological Approaches to Object Recognition.
PhD thesis, University of Waterloo, 2017.
Evelyn Fix and J. L. Hodges. Discriminatory analysis, nonparametric discrimination. Technical report, United
States Air Force, 1951.
Robert L.. Thorndike. Who belongs in the family? Psychometrika, 18(4):267-276, 1953.
Gideon Schwarz. Estimating the Dimension of a Model. Annals of Statistics, 6(2):4614444 1978.

---

Under review as a conference paper at ICLR 2021
Figure Findings generalize across hyperparameters. Notations like in Figure 1. To test whether fast learn- ing could be achieved by purely chang- ing hyperparameters, we evaluated dif- ferent regularizations (weight decay 0.001 and 0.00001 vs. 0.0001), leamn- ing rates (initial 1.0 and 0.01 0.1 the main manuscript) and learning rate schedules ("Scheduler alternative" decreases learning rate more agres- sively). Trends are qualitatively sim- ilar across all choices, and quantita- tively nearly optimal with respect to fast convergence to high brain predic- tivity for the hyperparameters used in the manuscript.
100%
Brain Predictivity 1% of standard training)
Weight decay 0.001
Weight docay le-5
80%
Learning rate 1.0
60%
Scandard training
Learning rate 0.1
40%
Scheduler alternative
20%
0%
10'
10'
10
10
A BENCHMARK DETAILS
We use the benchmarks as implemented in www www. github. cttp://raiii--oorrrrrraaaa----- at commit 96b0711, and convert base models to brain models with www gittubb.bb..... com/ bra incrsccreereeeeo htmommoolette
at commit 2f778c6. Images were presented degrees without aperture for the VI and V2 benchmarks and at 8 degrees for the V4, IT, and behavior benchmarks. Models committed to an input size 8 degrees visual angle.
B WEIGHT COMPRESSION DETAILS
For all convolutional layers except the first, we cluster kernel weights in a layer using the k-means algorithm (Fix & Hodges, 1951). The number of clusters is determined using elbow (Thorndike, 1953). To capture the relative importance clusters we fit normal distribution NV for each cluster with μ the cluster frequency over kernels and σ the frequency standard deviation. To sample weights for a kernel, we first sample a cluster distribution ~ Nr per kernel and then obtain channel weights by sampling from Gaussian with J7, as the cluster center and the standard deviation o; of clustered weights. In batch normalization layers, we fit one normal distribution each to the weights and biases.
B. COMPRESSING THE FIRST LAYER WITH GABOR PRIOR
The weight compression approach we use in Section 4 is based on different initialization techniques, applied different layers. For the very first layer of size we found Gabor filter most effective following studies in VI (Hubel & Wiesel, 1962; Jones & Palmer, 1987). To generate the Gabor kernels we trained channel weights to a Gabor function
C(2,, 2
exp -0.5((2 + vroel cos (2T + 6)
(1)
where
nl 02 f
Frot = x cos(0) ) + (0)
(2)
(3)
Brot - sin + (0)
Ny By f
Frot and Brot are the orthogonal and parallel orientations relative to the grating, 0 is the angle of the grating orientation, f is the spatial frequency of the grating, is the phase of the grating relative to the Gaussian envelope, σ and o, are the standard deviations of the Gaussian envelope orthogonal and parallel to the grating, which can be defined as multiples (n, and n,y) of the inverse of the grating
12

---

Under review as a conference paper at ICLR 2021
Figure 7: Alternative weight compression methods Compar- ison of different initializations that compress weights, "at birth" i.e. without any training (gray) and after training critical layers (shades of blue) for 6 epochs. Our best clustering- based approach WC achieved similar results as the Mixture Gaussian approach (~3 percent points mean differ- ence) but leads to more diverse clusters. Performance drops when solely sampling weights from ker- nel based normal distributions (Kernel normal) and additionally disabling the Gabor prior (No Ga- bor prior)
100%
Brain Predictivity 1% of standard training]
80%
Cricial train Craaatttiiinnn ng
Criics training
train ing
60%
No training
No traiming
40%
20%
Cluster
Mixture gaussian
Kernel normal
No gabor prior
frequency and C' is scaling factor.
The function is fit per channel, which leads to set Gabor parameter for each of the RGB channels. We then fit multidimennionall mixture of Gaussians the combination of all filter parameter per kernel, resulting in a kernel parameter set. For the three RGB input channels in the first layer and the 8 Gabor parameters components (number distinet Gaussian distributions) based on the Bayesian Information Criterion (Schwarz, 1978) and use 4 components for the first layer of CORnet-S. To generate new kernels sample a kernel parameter set from this mixture distribution and apply them to the described Gabor function that spans the weight values.
therefore fit to 8 27 parameters. We evaluate the best number of
B.2 COMPRESSING BATCHNORM LAYERS
In addition to convolutional layers, models consist of several Batchnorm layers, which contain learnable bias and weight term. To initialize these terms, we fit normal distribution per weight and bias vector of the trained values and sample from this distribution. Note that BatchNorm layers contain running average means and standard deviations for normalization purposes, which are applied at validation time. Those terms are set to when no training has happened, but cause score changes once the model has processed the dataset. During training the mean and standard deviation of the current batch used instead.
B.3 ALTERNATIVE APPROACHES
We have explored a variety of weight compression methods applied to different layers and evaluate their performance "at birth" without training and when trained with critical training.
Figure 7 shows brain predictivities of several alternative compression methods implemented as follows:
WC Weight compression approach with clustering as described in Section using a Gabor prior approach for the first layer, noisy cluster sampling for convolutional layers and fitted normal distributions for Batchnorm layers (4, 166 parameters for CORnet-S).
Mixture Gaussian Instead of sampling weights from cluster centers, this approach multidimensional distributions for convolutional layers with kernel size > 1. We fit mixture Gaussian distribution per layer the weights of channel over all kernels. To sample new kernel, we sample individual channels from this distribution. For convolutional layers with kernel size of we draw weights from a normal distribution adjusted per kernel as described in the next item (428, 1...11.. 14 parameters for CORnet-S).
Kernel normal All weights are sampled based on normal distributions. We mean and standard deviation the weights of one trained kernel and resample kernel from this
13

---

Under review as a conference paper at ICLR 2021
Standad triliig spack Deogge
2005
Droe Prodocaiity (1/ oodted
80%
100
27
Lakded
13
Sandhiind spriptic updiis
Tradking Epechs
Tridking Eanch
Figure 8: Detailed analysis of WC+CT. A When reducing the number of supervised synaptie updates, adding critical training (dark grey) and adding weight compression initialization (dark blue) both improve scores in brain predictivity at the same number of supervised synaptic updates, in comparison to model with standard initialization and training all weights (bright blue). B Brain predictivities for the WC+CT model when trained with a range of epochs and labeled images. C Same as B, but for standardly initialized (KN) model training all weights.
distribution. We do this separately for every kernel to generate whole layer. This approach is similar to the BatchNorm sampling method where we compress BatchNorm weight and bias terms instead of kernels (433,735 parameters for CORnet-S).
No Gabor prior To evaluate the importance of the Gabor prior we use the Kernel normal model and apply the same normal distribution approach to layer one instead of Gabor sampling. Performance drops by 13 percent points without training, and by 9 percent points after critical training. (20,026 parameters for CORnet-S)
C WC INITIALIZED AND CT TRAINED MODEL ANALYSIS
Our best model WC+ CT benefits from a combination of improved initialization through weight compression, and critical training. Figure 8A shows models with standard initialization and training all weights, but with fewer supervised updates (cf. Figure 1), models that only train down-sampling layers (CT), and models that combine critical training with weight compression (WC+CT). A model initialized with weight compression achieves (only WC) 54% brain predictivity score with supervised synaptic updates. Figure 8B and C show detailed brain predicitivity scores, relative to a fully trained model, for models initialized and trained with WC+CT (B) and models initialized with standard Kaiming Normal and training all weights (C) when trained with range of epochs and labeled images. The specific benchmark scores when either training with all labeled images for varying number of epochs (Figure 9A) or when training with fewer labeled images until convergence (Figure 9B) show the benchmarks of early visual achieve the best results, relative to a fully trained model. The V1 score identical over all training states, since we do not train the V1 area.
D DISSECTING TRAINING REDUCTIONS DETAILS
D.1 TRANSFER TO RESNET AND MOBILENET
To show the generalization of our approach we applied the weight compression methods to a ResNet- 50 (He et al., 2016) and a MobileNet (Howard et al., 2017) (version 1, multiplier 1.0, image size 224) architecture. We do not regenerate sampling distributions or clusters based on the new architectures trained weights, but used the CORnet-S based distributions to sample new weights for the different architectures. Since CORnet-S is inspired by ResNet modules, we applied our critical training approach by training all conv3 layers (equivalent down sampling layers) of ResNet50. For MobileNet we explored various layer mappings. When training only the very few layers that result in reduced feature size, which are implemented as depthwise separable convolutional layers and appear three times overall, performance performance dropped close to random. Those layers however are mapped
14

---

Under review as a conference paper at ICLR 2021
100%
2005
Score of stardard training
Scure 1% of standard traitine)
695
Mean
Moan
42%
Behowoo
Dohavior
0%
Training
Images
Figure 9: Individual brain benchmark scores for WC+CT model A Individual brain predictivitiy scores of WC+ models trained with a range of epochs on images. These models score especially high VI, V2 and V4 already after one epoch in comparison training all weights. IT and Behavior benchmarks continuously improve over later epochs as well but fall short of a fully trained model. B Like A, but with models trained until convergence different numbers of labeled images, up to the full dataset of 1. 1.28M images (rightmost points). As in A we see 80% V1, V2 and V4 scores with only 100,000 images. For comparable IT and behavioral scores, more images are required.
model with standard initialization
CORnet-S' conv2 layers due their x 3 kernels whereas critical training CORnet-S trains conv3 down-sampling layers with a kernel size of 1 x 1. To transfer our critical training approach, we therefore additionally train the 1 MobileNet layers corresponding to conv3. This training version allows for more training but still reduces the amount of trained parameters by 43% while maintaining 78% of the original score. For both transfer methods we initialize the first layer using Gaussian distribution. Since the Gabor function is
Gabor method based scalable we can produce Gabor kernels of varying size. Furthermore we disable BatchNorm biases and weights in all transfer models by freezing them to default values. We found that transferring those distributions on new architectures harms brain predictivity scores. Nevertheless, the BatchNorm layers still normalize activations by applying the running average and standard deviation.
CORnet-S's mixture-of- -fauuri-ff-Gaussiann
D.2 COMPARISON OF TECHNIQUES TO REDUCE SUPERVISED SYNAPTIC UPDATES (FIG. 5B)
To analyse the relative contributions of Weight Compression and Critical Training we compare brain predictivitiy scores of different models Figure 5B:
KN A model initialized by standard Kaiming Normal initialization without training.
WC A model initialized by our Weight Compression initialization, described in Section 4, without training-
KN+CT The KN-initialized model trained with Critical Training until convergence, i.e. three downstream layers and the decoder are trained and all other layers remain unchanged.
WC+CT The WC-initialized model with Critical Training. V1 scores do not change because weights in the V1 model area all frozen.
E TRAINING DETAILS
We used Py Torch 0.4.1 and trained the model using the ImageNet 2012 training set Deng et al. (2009). We used a batch size 256 images and trained on a QuadroRTX6000 GPU until convergence. We start with a learning rate of 0.1 and decrease it four times by factor of ten when training loss does not decrease over a period of three epochs. For optimization, we use Stochastic Gradient Descent with weight decay 0.0001, momentum 0.9, and cross-entropy loss between image labels and model logits. We trained all models with these settings except the standard Mobilenet, where we used the pretrained tensorflow model. Since the number of epochs for this model not clearly
15

---

Under review as a conference paper at ICLR 2021
stated, use the published value of 100 training epochs Howard et al. (2017). The training time of full CORnet-S with standard Imagenet dataset for 43 epochs is ~225 days. All variations with less weights/imagess/pochsss trained in shorter time. Reference models trained for 4 days at most under the described settings. If not further specified, we show results of one training run. When showing error bars used seeds 0, 42 and 94.
Code to reproduce our analyses from scratch, including the framework for weight compression and critical training, as well as pre-trained models, will be made available through GitHub.
16


Reviews:
[{"review_id": "x69s-QVkUXe", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------\nThe paper is about ANN being best-known models of developed primate visual systems. However this fact does not yet mean that the way those systems are trained is also similar. This distinction and a step towards answering this question is the main motivation of this work. The authors demonstrate a set of ideas that while drastically reducing the number of updates maintain high Brain Predictability according to the BrainScore. The significance of this result in my opinion largely depends on how well we can map those observations and methods to biological meaning and knowledge on how primate brains are trained (see the discussion point below).\n\n\nCritique, Questions, Discussion\n-------------------------------\n(1) How good the \"match\" between the brain and DCNN is in the first place? For example, if we measure the match in terms of correlation (between responses, or predictions, any metric would work in the context of this question), then 80% of corr=1.0 would be very impressive and significant, while 80% of corr=0.2 (being corr=0.16) could well fall under the noise and while being significant numerically, does not give us the opportunity to say that we have captured 80% of the match between the artificial system and the ventral stream (because what we have actually captured is 80% of corr=0.2, which might as well be almost nothing).\n\n(2) \"squirrels to jump from tree to tree within months of birth\", \"macaques to exhibit adult-like visual representations after months\" -- hoe many synaptic updates happen during those months? Do we know? Maybe it is also in trillions? In which case this portion of the argument would fall apart. Emphasis on \"supervised\" would probably still survive.\n\n(3) \"a child would need to ask one question every second of her life to receive a comparable volume of labeled data\" -- are they not? I would say children get even more data if by \"question\" we will mean not only verbal questions and answers, but also answers that are tactile (\"how this will feel on touch?\"), auditory (\"what does this object sound like\"), visual prediction (\"will this thing now move to the right or to the left?\"), etc. Seen like this I would say that children receive tons of supervised data and \"one per second\" is an underestimation.\n\n(4) How does the \"match\" vary depending on random initialization? Is it consistently 54% or is there a substantial +/-?\n\n(5) How do we know the \"true zero\" in terms of the \"match\"? What would be a model (function? maybe a constant function?) that clearly has zero \"match\"? If we now take this function and run it through your pipeline to get the match%, would the result be indeed 0% or something else? Maybe 54% is the \"true zero\" and not 0%.\n\n(6) Why sampling from CORnet-S-based clusters of parameters is a good way of modeling \"at-birth\" situation? Compared to 54% achieved with this methods, what would be the match% if the network would initialized with vanilla Kaiming Normal? Uniform?\n\n\nRecommendation and justification\n--------------------------------\nMy main concern is with the interpretation of the meaning of this work. BrainScore's metric is a very approximate proxy that weakly reflects the match between models of vision. In this work, however, this metric is taken as a \"gold standard\" and it is assumed that achieving, for example 50% of BrainScore of 0.42 is something biologically meaningful. An ablation experiment that would demonstrate that achieving these 50% (or other numbers presented in the paper) is a non-trivial event which can only happen if the model is indeed becoming more \"brain-like\" would go a long way in making the case of this work strong. I suspect, however, that such an ablation study will show that there are ways to achieve high% of BrainScore using models that are completely dissimilar to the brain. I currently evaluate this submission as borderline, and am looking forward to authors' views on the concerns I have outlined above: do these indeed matter and affect the claims of this work (and how should we see them if that's the case), or are these concern largely irrelevant (and why we can ignore them if that's the case?).\n\n\nAdditional remarks\n------------------\nArguably missing references on modeling of the ventral stream with ANNs: https://www.nature.com/articles/s42003-018-0110-y, https://www.jneurosci.org/content/35/27/10005\n\n\nUPDATE - Nov 30\n-----------------------\nAfter looking at the revised version of the manuscript I am still concerned that the claims made in the abstract (and implied in the main text of the paper) about the match of ANNs to the brain are misleading the reader into assigning greater biological significance to the reported result than it actually holds. While the authors made slight modifications in the text and added a few sentences commenting on the issue, these changes did not constitute a change would make the reader \"extremely aware that when you say \"80% match\" you don't mean \"80% match to the brain\", but \"80% match to the score\"\". I find that a softer claim that would explicitly acknowledge that 5% of \"synaptic\" updates explain 80% of the predictivity score and not 80% of the match to the brain would make this work more scientifically precise and thus more valuable. I am keeping my original assessment of this paper as being borderline.", "extra_scores": {}, "extra_text": {}}, {"review_id": "pk3Rx0YjuPx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarize what the paper claims to contribute. \nPrevious work developed CORnet-S, a biologically inspired network that leads the Brain-Score benchmark of similarity with the primate ventral stream. A limitation of CORnet-S and other deep networks with high Brain-Scores is that they require many more weight updates than seem biologically feasible. In this paper, the number of weight updates used to train CORnet-S is reduced by two order of magnitude, while retaining a fairly high Brain-Score. This is done by combining three approaches, including reduced training, initialization of weights using compact distributions that describe trained weights, and updating only a minority of layers. \n\nList strong and weak points of the paper. \nStrong Points:\n-\tThe paper addresses an important problem that has not been given much attention previously\n-\tThe work builds on the state-of-the-art model in this domain\n-\tThe three approaches to reducing updates are complementary and interesting in different ways; the second and third thought-provoking with respect to their biological relevance\n-\tThe experiments and analysis are thorough\n-\tThe paper is well written\n-\tThe context of the work is clearly described and well referenced\n\nWeak Points: \nI wasn’t able to discern any substantial weaknesses. \n\nClearly state your recommendation (accept or reject) with one or two key reasons for this choice. \nI recommend acceptance. The number of updates needed to learn realistic brain-like representations is a fair criticism of current models, and this paper demonstrates that this number can be greatly reduced, with moderate reduction in Brain-Score. I was surprised that it worked so well. \n\nAsk questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. \n-\tIs the third method (updating only down-sampling layers) meant to be biologically relevant? If so, can anything more specific be said about this, other than that different cortical layers learn at different rates? \n-\tGiven that the brain does everything in parallel, why is the number of weight updates a better metric than the number of network updates? \n\nProvide additional feedback with the aim to improve the paper. \n-\tBottom of pg. 4: I think 37 bits / synapse (Zador, 2019) relates to specification of the target neuron rather than specification of the connection weight. So I’m not sure its obvious how this relates to the weight compression scheme. The target neurons are already fully specified in CORnet-S. \n-\tPg. 5: “The training time reduction is less drastic than the parameter reduction because most gradients are still computed for early down-sampling layers (Discussion).” This seems not to have been revisited in the Discussion (which is fine, just delete “Discussion”).\n-\tFig. 3: Did you experiment with just training the middle Conv layers (as opposed to upsample or downsample layers)? \n-\tFig. 3: Why go to 0 trained parameters for downstream training, but minimum ~1M trained parameters for CT? \n-\tFig. 4: On the color bar, presumably one of the labels should say “worse”. \n-\tSection B.1: How many Gaussian components were used, or how many parameters total? Or if different for each layer, what was the maximum across all layers? \n-\tSection B.3: I wasn’t clear on the numbers of parameters used in each approach. \n-\tD.1: How were CORnet-S clusters mapped to ResNet blocks? I thought different clusters were used in each layer. If not, maybe this could be highlighted in Section 4. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nqpB8mapTJD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an empirical study that elucidates potential mechanisms through which models of adult-like visual streams can \"develop\" from less specific/coarser model instantiations. In particular, the authors consider existing ventral stream models whose internal representations and behavior are most brain-like (amongst several other models) and probe how these fair in impoverished regimes of available labeled data and model plasticity (number of \"trainable\" synapses). They introduce a novel weight initialization mechanism, Weight Compression (WC), that allows their models to retain good performance even at the beginning of training, before any synaptic update. They also explore a particular methodology for fine-tuning, Critical Training (CT), that selectively updates parameters that seem to yield the most benefit. Finally, they explore these methods/algorithms' transfer performance from one ventral stream model (CORnet-S) to two additional models (ResNet-50 and MobileNet).\n\nPros:\nThe problem that the authors present is an interesting one and undoubtedly useful for many applications. Deep neural networks such as the CORnet-S, ResNet-50, and MobileNet are data-hungry, and obtaining labeled data is an expensive process (and perhaps even implausible in many cases). Techniques to condense these models in terms of parameters and alleviate the need for vast amounts of labeled data while maintaining desirable traits (such as brain-like representations) are important for the machine learning community. Though a bit far-fetched at this point, tracking the developmental trajectories of these neural networks can also have other scientific implications in the form of data-driven hypothesis testing.\n\nThe most exciting part of the study is the transfer experiment (from CORnet-S to ResNet and MobileNet). This seems like an interesting and novel way to construct model taxonomies. For instance, sampling from the CORnet-S weight clusters works well for ResNets potentially because these two models can be construed as \"recurrent\" in a way. MobileNets, on the other hand, are purely feedforward and thus are not significantly influenced by knowledge from the CORnet-S weights.\n\nMoreover, the authors conduct a series of numerical experiments to identify \"when\" their proposed methods are most useful. The finding that WC+CT is more advantageous in regimes where data is scarce (as opposed to regimes where data is plenty) is not surprising but a good one to report. I say \"not surprising\" because WT distills knowledge from a fully trained model, and CT only updates a fraction of the parameters (updating more parameters would require more data to prevent overfitting).\n\nCons:\nThe authors take the analogy between \"a developing visual system\" and \"training a model\" a bit too far. They operate under the premise that visual circuitry develops purely via \"supervised\" learning. Is there conclusive evidence for this? It is also surprising that discussions of reinforcement learning mechanisms never feature, given that these are more biologically plausible.\n\nThe novelty (and utility; for ex: Fig 2b) of the proposed initialization technique is marginal. It is not articulated how their method (WC) overcomes the critiques they raise against Frankle et al. 2019. Moreover, claiming that WC achieves decent performance with \"zero\" synaptic updates is not fair. This seems to be closer to restoring pre-trained weights than to random initialization (like KN). \n\nFor CT, the authors choose \"critical\" layers to update. Is there a rationale (or a statistical metric) that justifies choosing these specific layers? \n\nThe WC kernel cluster center visualization analysis (Fig. 5c) seems out of place and poorly discussed. What can be gleaned from the 3x3 kernels shown here?\n\nMinor:\nBy \"supervised updates,\" the authors refer to the number of available labels and not the number of parameter updates that happen. This terminology is non-canonical. \n\nEmploying Gabor priors for the first convolutional layer: Doesn't orientation selectivity emerge in the primary visual areas from experience, rather than structurally hard-coded? \n\nThe authors allude to the possibility of using \"local\" learning rules on a subset of layers identified by CT. However, this is speculation from the point of view of the current manuscript. All the conclusions drawn are from \"global\" gradients.\n\nAmbiguous sentence (Pg. 6, Sec 6): \"Reducing the number of supervised updates minimizes required updates by a smaller number of epochs and images.\"\n\n(Pg. 8) \"synaptic updates primarily take place in higher cortical regions\": Is there evidence for this?\n\nNumerical imprecisions:\n(i) The authors claim that the performance of CORnet-S_wc is 54% (relative to the fully trained model). However, in Fig 2b (mean) and Fig 3c (top) the markings seem to be closer to 50%?\n(ii) (Fig. 4a) The performance of MobileNet seems to be slightly better than CORnet-S, which contradicts the initial claim that CORnet-S is currently the best available model of adult primate visual processing.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6edoeUPGvl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The study starts from the fact that DNNs have been around and popular for a while for modeling the visual system, but that they are not realistic because they are trained via supervised learning approaches with a very large number of parameters and that this is not a feasible model of the development in the visual system.\n\nIn general, although the manuscript presents some interesting ideas, it makes many assumptions without providing clear bases for these assumptions (e.g. compressing the weights of a pretrained network to sample new weights is posed as a realistic approximation of the infant visual brain) and lacks a theoretical foundation for the claims and experiments that are presented. The authors acknowledge that this study is intended as a proof of principle, but given the arbitrary nature of the choices made, I do not see the added significant value of the results.\n\nWhile DNNs are indeed commonly used as models of the primate visual system, in my view, the current study is addressing a somewhat inconsequential problem. This is because to the best of my knowledge, no neuroscientist is claiming that a deep neural network is a complete and accurate model of the (development of the) primate visual system. Furthermore, it is well-known and acknowledged that deep neural networks are not biologically plausible models of (how learning occurs in) the brain. They are currently one of the best computational tools to use to study the sensory (and especially the visual) nervous systems, and that is all that they are. It is not clearly explained why it is necessary to claim that the learning in these models and the development of the brain has to be similar for them to be good models of vision. Of course, we should thrive for better and more accurate models of the brain, but in my view the current study does not serve to this goal.\n\nIn section 4 authors describe an initialization protocol for the network weights which involve compressing a trained model’s weights into clusters and then sampling from these clusters. What is not clear is why the authors assume that this can be a valid model of the infant visual system. At this point their approach sounds like arbitrarily selecting a set of criteria to make the networks perform worse than fully trained networks, and then training them. I could be missing something, but I do not see the relevance or necessity of an approach such as the presented one. A main concern is that no theoretical basis has been established in the paper besides some superficial ideas. For instance, why would an infant brain be made up of a DNN with connections whose weights are initialized with the method authors came up with?\n\nMuch of the methodological details are only included in the appendix. I found it rather odd to not find any information about, for example, the proposed weight initialization method in the paper.\n\nIt is not clear to me what is presented in Figure 1 and why. Why are the authors showing how models from another paper trains?\n\nAnother concern is that nowhere in the results seems to be a test for significance. The improvements of the results could be a coincidence, since the results are heavily dependent on one experiment.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 2 borderline accepts, 1 accept, and 1 reject.

This paper was discussed on the forum and no consensus was reached. The two reviewers who rated the paper as borderline accept emphasized that the biological claims are overblown, that the intellectual contributions (the initialization scheme and partial training) are incremental from a statistical learning perspective, and that the potential applications for the future (like alternate learning rules) are too speculative. I agree with both of these reviewers (and the negative reviewer) that the biological rationale is problematic and the approach is not credible as a model of biology. It is not evaluated as a computer vision model either. And I completely agree with the point raised by several reviewers that there is simply no data about how many synaptic updates to target. Hence, statements regarding % of total synaptic updates and % of brain matches seem empty without a precise target. For all these reasons, I recommend this paper be rejected.
Decision: Reject