Wiring Up Vision: Minimizing Supervised Synaptic Updates Needed to Produce a Primate Ventral Stream

Abstract: After training on large datasets, certain deep neural networks are surprisingly good models of the neural mechanisms of adult primate visual object recognition. Nevertheless, these models are considered poor models of the development of the visual system because they posit millions of sequential, precisely coordinated synaptic updates, each based on a labeled image.  While ongoing research is pursuing the use of unsupervised proxies for labels, we here explore a complementary strategy of reducing the required number of supervised synaptic updates to produce an adult-like ventral visual stream (as judged by the match to V1, V2, V4, IT, and behavior). Such models might require less precise machinery and energy expenditure to coordinate these updates and would thus move us closer to viable neuroscientific hypotheses about how the visual system wires itself up. Relative to standard model training on labeled images in ImageNet, we here demonstrate that the total number of supervised weight updates can be substantially reduced using three complementary strategies: First, we find that only 2% of supervised updates (epochs and images) are needed to achieve 80% of the match to adult ventral stream. Specifically, training benefits predictions of higher visual cortex the most whereas early visual cortex predictions only improve marginally over the course of training. Second, by improving the random distribution of synaptic connectivity, we find that 54% of the brain match can already be achieved “at birth" (i.e. no training at all). Third, we find that, by training only 5% of model synapses, we can still achieve nearly 80% of the match to the ventral stream. This approach further improves on ImageNet performance over previous attempts in computer vision of minimizing trained components without substantially increasing the relative number of trained parameters. These results reflect first steps in modeling not just primate adult visual processing during inference, but also how the ventral visual stream might be "wired up" by evolution (a model's "birth" state) and by developmental learning (a model's updates based on visual experience).

Reviews and author responses:

 We would like to thank all reviewers for taking the time to read our responses and for their updated feedback!

The paper addresses the question of how many weight updates are needed to train a deep network before it takes on biologically realistic representations. The paper uses CORnet-S (a network that has been proposed to resemble primate ventral stream), and BrainScore (a benchmark of how closely related deep network responses are to visual responses in primate ventral stream). Three ways of reducing the numbers of weight updates are explored, each of which is found to vastly reduce updates while moderately reducing BrainScore. First, the network is simply trained for fewer epochs. Second, weights are initialized with clusters of weights found after training. Third, only a subset of layers is updated. A combination of methods leads to 80% of full brain predictivity with 0.5% of the standard number of weight updates.  Strengths: 
- The paper is clear and well written
- The analysis is thorough and supports the authors' arguments
- The method of initializing weights via clusters of trained kernels is interesting  
- The "critical training" method of training only selected layers is also interesting 

Weaknesses: 
- "Recently, such models have been criticized due to how their learning departs from brain development (Marcus, 2004; Grossberg, 2020; Zador, 2019)." This seems to be the motivation for the work, but those authors don't raise the issues that are addressed in this paper. The Marcus book predates deep learning. Grossberg focuses on catastrophic forgetting. Zador focuses on the sparsity of labelled data in life. I am not sure whether the number of weight updates needed to train deep networks presents a substantial independent concern about their biological plausibility, particularly given the possibility of constant unsupervised learning. If the issue has been raised in the literature, more specific references would help here. If not, I think the authors should argue more specifically that it is a problem, before going on to solve it. 
- The cluster-based initialization method seems to be presented as a way of bringing some learning results through a genetic bottleneck, but the biological plausibility isn't obvious. Further discussion of how this relates to and departs from biology would be helpful. 
- It's interesting that critical training worked so well, but in the context of the paper's goals, the biological relevance of this method should also be laid out more clearly. 

Minor: 
- I don't think it was shown how classification performance is affected by the cluster-based initialization method, but it would be nice to see that. 
- "37 bits per synapse (Zador, 2019)" This refers to the binary connectivity pattern (an issue of architecture) rather than the weights (which are addressed in this paper). 
 I think the approach and the results are new and interesting, but I have reservations about the motivation for the work and its connection to biology. 

 I have come around to the authors' argument re. clustering. I also see that the reasoning about limited life experience relates to the approaches involving clustering and training for fewer epochs, if not to the critical training approach (that one seems more likely relevant to biologically plausible backprop than to limited life experience). 

## Updated the score

This paper proposes to address an important research question for connecting biological (BNN) and artificial neural networks (ANN). Although after training, ANN replicates various salient features of BNN, the way they are often trained is biologically implausible and thus, it is hard to argue that ANNs are suitable for modeling BNNs convincingly. In particular, this work focuses on the already existing CORnet who has shown a high Brain-score. The idea of the authors is to show that they can largely reduce the number of updates when using their methods while still retaining a high Brain-score, thus proposing a potential training mechanism for BNNs. ### Strength:

First of all, the question addressed in the paper is essential for both the machine learning and neuroscience community and the author aim at addressing in a well-written piece of work that also clearly contextualizes and motivates the issue at hand. This work uses a state-of-the-art neural architecture, instead of a reduced version of models as is often the case in this type of work. The authors also propose a good analysis and clearly lay out their results, although their interpretation is one of the weaknesses of the paper. 

### Weakness:

Although it is true that this work has many strengths it is hard to overcome the following weaknesses, which question the motivation of the approach undertaken. First, there exists a set of more principled approaches that have been developed over the last 4-5 years for training ANNs in a biologically plausible way.  Second, the results obtained are very subjective to the authors as the result of the improvement of the score is minor. 

Although it is true that we don’t expect the authors to solve how the visual system develops and that this work corresponds to a proof of principle, it is still hard to navigate the plausibility of the assumptions made among all possibilities. On the other hand, there is a large set of work that has been proposed to build biologically plausible deep neural networks starting from clear objective function minimization and enforcing locality of learning rules. Compared to this set of works this one seems more adhoc than principled. This represents an important weakness when addressing the more neuroscience-friendly community. 

Indeed the approach proposed is more of a combination of existing approaches rather than a novel idea: the use of training of a subset of layers, different weights initialization based on the distribution of trained weights. It is not clear how this provides new insight to the statistical learning community. 

 This paper is well-written and addresses an important question possibly helping bridge neuroscience and machine learning. However, this work falls short on both the relevance of the experimental results and the ad-hoc nature of the proposed approach. 

 Thank you for responding to the concerns that I expressed. 
The responses to other reviewers and changes made to the paper have clarified various points, and I am changing my recommendation and increasing the score.

 Thank you for the response! 

Having read other reviews and responses, **I am pretty happy with the paper/changes, so I'm leaving the same score (8).**

> We would argue that even recent unsupervised learning techniques require many experience-dependent inputs (albeit not labeled), so even these forms of unsupervised learning suffer from similar problems.

I fully agree!

 We thank the reviewer for their helpful feedback and the positive evaluation of our paper.

### A biologically plausible method would ideally perform well both under little and large amounts of supervision:
We agree that, ideally, we would not have to choose between learning methods for few supervised updates and methods optimized for many updates. WC+CT indeed performs best in a regime with few labeled examples and epochs compared to standard training.

As somewhat of an intermediate “compromise,” we found that, when we first train with WC+CT to 79% of the score at 0.5% of the updates (Figure 4 star), we can continue with standard training and achieve 100% of the score with only 15 additional epochs (overall corresponding to 1,028 trillion supervised synaptic updates or one third of the fully trained model’s 3,000 trillion updates). We hope future work building on this paper will be able to further close this gap.

### Tying model predictivity to accuracy on another task:
We agree with the reviewer that this paper did not connect its results of various networks’ brain predictivities to another task. Especially the early correspondence of e.g. at-birth models to V1 is not predicted by the (ImageNet) classification task. We leave this up for future work, but added a reference to recent results in the field where robustness to adversarial attacks and image corruptions was found to be predictive of how V1-like model representations are (https://proceedings.neurips.cc/paper/2020/file/98b17f068d5d9b7668e19fb8ae470841-Paper.pdf).

### Opening quotes:
We have fixed all opening quotes to be forward.


 We thank the reviewer for their constructive review and the positive evaluation of our work. Below we respond, point-by-point, to each point and how we updated the paper:

### More detail on Weight Compression:
1. We updated the text to specify that we determine the number of clusters with the elbow method, resulting in 7 clusters for V1, 13 for V2, 16 for V4, and 15 for IT. We also added a table to the supplement detailing the number of clusters for each layer within each region.
2. We added: “This results in 33 KB (4166 parameters) to specify the network initialization, compared to 423 MB (52.8 M parameters) to specify a trained model's weights (assuming 8 byte per parameter).”
3. We have incorporated the relevant reference into the related work.
4. Added “We speculate that increasing the number of clusters would follow logarithmic growth with respect to the performance of the model: i.e. a small number of clusters can already yield useful wiring (as explored here), while adding more clusters will have a positive effect but with decreasing effect size.”

### Generalization to other architectures:
We indeed started from the most ventral-stream-like model at the time, but attempted to generalize our findings and techniques to other architectures: Section 7 and Figure 5A attempt to transfer the identified clusters (Weight Compression) to MobileNet-V1 and ResNet-50 architectures, training only a subset of their layers (Critical Training), reducing the number of trained parameters while retaining high brain predictivity (MobileNet’s score arguably suffers quite a bit, presumably from the network already being rather compact). We have updated the text to reflect that we used MobileNet-V1 and ResNet-50, as suggested.

### Learning rates:
Learning rates were snapshots and not tuned individually. We agree with your intuition and attempted to crank up the learning rate as suggested (rather naively to 1.0), but did not observe any improvements in the trade-off between supervised updates and brain predictivity (Supplement, Figure 6). We have added a sentence that more carefully tuning the LR could further improve this trade-off.

### Applicability of results beyond supervised learning:
We would argue that even recent unsupervised learning techniques require many experience-dependent inputs (albeit not labeled), so even these forms of unsupervised learning suffer from similar problems. We agree that the results here should be applicable in these domains as well, and are not limited to supervised learning.

### 80% of the best performing network:
Indeed, this sentence in the abstract refers to 80% relative to a trained network; perfect performance would indeed be 1.0 Brain-Score. We have updated the sentence as follows: “First, we find that only 2% of supervised updates (epochs and images) are needed to achieve ~80% of _a fully trained model’s_ match to adult ventral stream.”

### Fig. 3C-D caption:
We were unable to expand the caption while still fitting the paper inside of 9 pages, but have detailed critical training in appendix C.


 ### Reducing supervised synaptic updates while retaining high brain score provides insights to the neuroscience community:
We want to emphasize that our goal here was to build models that retain high Brain-Score compared to a fully trained model but require only few updates. We agree with the reviewer that ideas for different initializations and training subsets of layers have been explored by the machine learning community -- in fact, in the related work section, we already refer to network distillation, pruning studies, gradient subspace, BatchNorm training, lottery ticket, unsupervised training, and biologically plausible backpropagation. If we missed key references, please let us know, we would be happy to cite them. 

However, testing models with reduced updates on brain data and building new variants according to the empirical findings, is, to the best of our knowledge, novel and has not been explored before. According to ICLR guidelines (https://iclr.cc/Conferences/2022/CallForPapers), “applications in neuroscience” is a relevant topic worthy of publication at this conference.

Please let us know if we misunderstood your criticism, and if you would like to see certain aspects better explored. If we addressed your concerns, we hope that you might consider increasing your score.


 We thank the reviewer for their feedback on our paper.
We interpreted the “more principled approaches” raised by the reviewer to correspond to attempts at replacing back-propagation, but view this paper as exploring an orthogonal question. We also want to emphasize that this paper’s goal was _not_ to increase the score, but rather to keep it as high as possible while reducing the number of supervised synaptic updates as much as possible.
Below, we respond to these concerns in detail.

### Supervised synaptic updates are orthogonal to back-propagation:
The reviewer refers to “a large set of work that has been proposed to build biologically plausible deep neural networks starting from clear objective function minimization and enforcing locality of learning rules.” In lieu of references (we had asked for clarification, but have so far not gotten an answer), we assumed the reviewer is referring to attempts at replacing back-propagation with biologically plausible learning rules, e.g. https://www.nature.com/articles/s41583-020-0277-3, https://arxiv.org/abs/1609.01596, https://www.nature.com/articles/ncomms13276. In our view, the question explored in our paper is orthogonal to that of back-propagation: while biological systems require only a small amount of experience to perform challenging tasks (e.g. macaques visual system is adult-like after a few months (Seibert et al. 2018), the current most brain-like models of primate ventral stream are trained on dozens of epochs with millions of labeled examples. To our knowledge, it has not been shown that replacing back-propagation leads to more efficient learning. In fact, we here show that even with the same learning algorithm and tweaks to model initialization and trained layers, model correspondence to primate visual ventral stream can be achieved with relatively few supervised synaptic updates.

Since this is, as far as we know, the first attempt to address these concerns, it is by definition more ad-hoc than the body of work focused on biologically plausible backpropagation. We hope that future biological experiments will help to validate and falsify modeling ideas.

We agree that the first version was not clear in stating its assumptions about biological plausibility, and have thus updated:
* The motivation
> Recently, such models have been criticized due to how their learning departs from brain development, requiring many more labeled examples than is reasonable for biological systems' limited waking (visual) experience (Seibert et al. 2018, Zador 2019). For example, all the current top models of the primate ventral stream rely on trillions of supervised synaptic updates, i.e. the training of millions of parameters with millions of labeled examples over dozens of epochs. In biological systems, on the other hand, the at-birth synaptic wiring as encoded by the genome already provides structure that is sufficient for macaques to exhibit adult-like visual representations after a few months (Movshon et al. 1988, Kiorpes et al. 2004, Seibert et al. 2018), which restricts the amount of experience dependent learning. [...]
* Added a new section B.4 to expand on the assumptions of Weight Compression:
> Weight Compression (WC)'s cluster-based initialization generates an "at-birth" network that already captures some useful aspects of the visual inputs in a compressed manner and allows the system to learn faster. The way that these clusters are determined (from a previously trained network) is not biologically plausible and does not correspond to any evolutionary mechanism we are aware of. However, independent of how these clusters are determined, WC shows that it is possible to encode certain priors about a system's wiring diagram in a very compressed manner. Evolution also acquired aspects of the visual inputs (with a different strategy) and encoded them in the genome in a lower information regime.
> 
> For example, we know that there are multiple neuro-developmental mechanisms, such as spontaneous retinal waves and axon guidance cues, that significantly shape the architecture and function of the visual system requiring no (or very little) visual experience (for a review see Huberman et al. 2008 Annu. Rev. Neurosci.). These mechanisms depend on a relatively small number of proteins encoded in a genome and give rise to a highly complex pattern of at-birth synaptic connectivity representing a very large compression of information.
* Made the connection of Critical Training to biology more clear in the manuscript, particularly in the discussion where this question was overlooked, emphasizing that the techniques require further experimental validation. 


 ### Biological-plausibility of the cluster-based initialization (weight compression, WC):

We agree with the reviewer that the way we define the cluster-based initialization method does not match exactly with the evolutionary and developmental mechanisms that give rise to the at-birth connectivity of the visual system. In fact, we write the following in the manuscript (page 6): 

> Weight Compression explores the hypothesis that evolution may have discovered an initialization strategy with improved at-birth representations (relative to current initializations). WC is most likely not how evolution found the at-birth synaptic connections, but shows that with nearly identical capacity, an alternative initialization distribution leads to networks that are more brain-like in their adult state – revealing a new space of possibilities (hypotheses) that should be considered.

We have added a new section (B.4) to the appendix to expand on the biological plausibility:

> Weight Compression (WC)'s cluster-based initialization generates an "at-birth" network that already captures some useful aspects of the visual inputs in a compressed manner and allows the system to learn faster. The way that these clusters are determined (from a previously trained network) is not biologically plausible and does not correspond to any evolutionary mechanism we are aware of. However, independent of how these clusters are determined, WC shows that it is possible to encode certain priors about a system's wiring diagram in a very compressed manner. Evolution also acquired aspects of the visual inputs (with a different strategy) and encoded them in the genome in a lower information regime.
> 
> For example, we know that there are multiple neuro-developmental mechanisms, such as spontaneous retinal waves and axon guidance cues, that significantly shape the architecture and function of the visual system requiring no (or very little) visual experience (for a review see Huberman et al. 2008 Annu. Rev. Neurosci.). These mechanisms depend on a relatively small number of proteins encoded in a genome and give rise to a highly complex pattern of at-birth synaptic connectivity representing a very large compression of information.


### Biological relevance of critical training (CT):

Like weight compression, the critical training method is a proof-of-principle that the brain may employ selective learning in some of its components to allow faster learning. In the manuscript, we wrote the following (pages 6-7):

> With standard training, each iteration simultaneously updates all of the millions of synaptic weights in the neural network, which may be difficult to implement biologically. Alternatively, learning could take place preferentially in specific components. Cortical circuits are heterogeneous and different neuronal populations undergo distinct plasticity mechanisms. For example, neurons in supra- and infra- granular layers adapt more rapidly than those in layer 4, where inputs from lower areas arrive, as observed in rat somatosensory cortex (Diamond et al., 1994) and primate V1 (Schoups et al., 2001).

While the models used do not (yet) explicitly map specific model layers to cortical layers in the brain, our results show that allowing learning only in specific (and critical) components of the model allows the model to learn and approximate the brain with fewer weight updates. This mimics what happens in cortical circuits, where neuronal plasticity is not uniform but varies considerably across cortical layers.

We have made the connection to biology clearer in the manuscript, particularly in the discussion where this question was overlooked, emphasizing that the techniques require further experimental validation. The 9-page limit however prevents us from going into too much more detail in the manuscript, please let us know if (which) other details could be reduced to expand on these questions more. We would like to know if this is a sufficient explanation or if the reviewer would like to see this problem better explored.

### Classification performance with initialization alone is poor:

We tested how well a model initialized with Weight Compression and only a trained linear readout performs on ImageNet and found that its performance is rather poor (~5% top-1 accuracy). This is consistent with Figure 1 in that model consistency with later visual regions such as IT as well as behavioral measurements requires more experience-dependent updates than model consistency with early visual regions such as V1.


 We thank the reviewer for their thoughtful analysis of our manuscript. We agree with the overall feedback that a better justification of the biological-plausibility of several of our proposed approaches will make the paper stronger and have updated the paper accordingly. Below we address, point-by-point, the reviewer’s feedback:

### Number of labeled weight updates is a concern about the biological plausibility of ANN models of vision:

We have updated the text to be more precise about the critique of labeled weight updates. Specifically, we are referring to concerns such as the following:
* Seibert et al. 2018: “our work does constrain models from the perspective of the amount of training data - models of primate visual development ought to exhibit adult-like performance and consistency using only 25 weeks of visual experience. Assuming 16 hours of awake visual experience per day [192] over 25 weeks, our data constrains models of primate visual development to roughly 2,800 hours of waking visual experience.”
* Zador 2019: “For example, supervised learning is the paradigm that allows ANNs to categorize images accurately. However, to ensure generalization, training such networks requires enormous data sets; one visual query system was trained on more than 10^7 “labeled” examples (question-answer pairs). Although the final result of this training is an ANN with a capability that, superficially at least, mimics the human ability to categorize images, the process by which the artificial system learns bears little resemblance to that by which a newborn learns. There are only 10^7 s in a year, so a child would need to ask one question every second of her life to receive a comparable volume of labeled data; and of course, most images encountered by a child are not labeled. There is, thus, a mismatch between the available pool of labeled data and how quickly children learn.”
* Zador 2019: “The survival of an animal requires that it solve the so-called “four Fs”—feeding, fighting, fleeing, and mating—repeatedly, with perhaps only minor tweaks. Each individual is born, and has a very limited time—from a few days to a few years—to figure out how to solve these four problems. If it succeeds, it passes along part of its solution (i.e., half its genome) to the next generation. Consider a species X that achieves at 98% of its mature performance at birth, and its competitor Y that functions only at 50% at birth, requiring a month of learning to achieve mature performance.”

The updated text now reads:
> Recently, such models have been criticized due to how their learning departs from brain development, requiring many more labeled examples than is reasonable for biological systems' limited waking (visual) experience (Seibert et al. 2018, Zador 2019). For example, all the current top models of the primate ventral stream rely on trillions of supervised synaptic updates, i.e. the training of millions of parameters with millions of labeled examples over dozens of epochs. In biological systems, on the other hand, the at-birth synaptic wiring as encoded by the genome already provides structure that is sufficient for macaques to exhibit adult-like visual representations after a few months (Movshon et al. 1988, Kiorpes et al. 2004, Seibert et al. 2018), which restricts the amount of experience dependent learning. [...]



 Hi Reviewer Ekkd,

Thank you for your feedback. Before responding with a full rebuttal, could you please clarify the following:
1. You mentioned "more principled approaches that have been developed over the last 4-5 years for training ANNs in a biologically plausible way" -- are you referring to attempts at replacing back-propagation, e.g. https://www.nature.com/articles/s41583-020-0277-3, https://arxiv.org/abs/1609.01596, https://www.nature.com/articles/ncomms13276?
2. Can you please also clarify which aspects make this other body of work more principled? In other words, what could we change in order to convince you our paper proposes principled methods?

Thank you.

While deep networks are known to be biologically implausible, the strong similarities between the feature maps in such networks and the response characteristics (tuning curves) in different areas of the mammalian visual cortex (in particular the ventral stream) has prompted much work into trying to reconcile the known differences. The authors present work that contributes to such inquiries, demonstrating that at least some of the strong impediments in using such networks as plausible visual stream models may be more easily surmountable than they have thus far appeared to be. The authors present thorough analyses of standard feedforward networks that quantitatively demonstrate that such networks can model the visual stream well while requiring far fewer coordinated parameter changes than occur with traditional training methods.  In particular, they show that very little supervision is required for strong matches to neural responses, that plausible distributions of initial random weights can match neural responses very well, and that weights can be updated very sparsely while still achieving quite high neural response matches. 

Such findings are highly relevant to allowing for such networks to be more plausibly accepted by the neuroscience community as valid models of cortical networks.  The striking finding that well-chosen random weights (Weight Compression) can predict neural responses with more than 50% accuracy is  especially interesting, as it suggests that indeed there may be less training required in mammalian brains compared to randomly initialized neuronal networks. This possibility opens up further avenues of study relating to the ability of real brains to learn with far fewer examples compared to ANNs.  

Figure 4b further demonstrates this possibility, showing that, under the ethologically relevant condition of little supervision, WC + Critical Training, where only a small subset of weights in so-called critical layers are trained, greatly outperforms traditional training.  We note that, in the ideal case, a biologically plausible method might be found that performs well both under little supervision, as WC+CT does here, and under a large amount of supervision, as traditional backprop does, the demonstration of principle for the former case already represents appreciable progress.

However, little in the way of tying model predictivity to accuracy on another task was explored. While tying brain predictivity to network performance is of course a very active area in its own right, we feel that addressing this directly within the context of the present work would facilitate readers' abilities to contextualize the present study with respect to ANN performance.

Overall, we find the authors' presented work to be of great interest to those interested in applying ML models to theoretical questions in neuroscience and to have a thorough set of analyses that both illustrates progress in this area while suggesting future avenues of work.

Minor:
- opening quotes are backwards throughout paper  The authors provide a compelling case that much more biologically plausible sets of initial conditions and sparse, selective supervision and synaptic updates might help substantially close the gap in viewing deep networks as plausible visuocortical models.  

The paper is concerned with closing the gap between the amount of training in deep networks and in developing brains, as the current deep learning models use an unrealistically large number of synaptic updates. The authors address that with three strategies: less training, weight clustering and training in a subset of layers. All methods are tested individually and in combination with each other on primate ventral stream data.   ## Strengths
The paper addresses an important problem in theoretical neuroscience: while trained deep networks appear to be good models of the visual stream, the amount of training they need to get there is incompatible with biology. 

Two of the proposed approaches to reduce the amount of training are somewhat straightforward: training for fewer epochs and freezing the amount of trained layers. However, I think the provided results are important as they show good fit to brain data can be achieved without closely following the training procedures of deep learning.

The last approach, weight clustering, is perhaps more interesting to the theoretical neuro community, as it works in the same way as genomically encoding the network (the “at birth” state). 

Importantly, the proposed ways to reduce the amount of training are tested not only using ImageNet performance, but also ventral stream data, which is arguably more important for neuroscience. 

Overall, the paper presents a novel set of results for an important problem, and supports its claims with a significant number of experiments.
 
## Weaknesses
As mentioned before, I think the weight compression part is the most valuable one, and therefore it should receive more attention in the paper.
1. Clustering should be better explained in the main text. Especially how the number of clusters was chosen.
2. In the beginning of Sec. 4, you discuss how many bits it would take to encode different networks. I think the same should be done for your experiments. E.g. what is the size of CORnet-S vs. the size of a compressed network?
3. I think similar weight compression techniques have been used before, see http://proceedings.mlr.press/v80/wu18h.html for example
4. In general, it’d be interesting to see how at-birth performance changes with the amount of “genetically encoded” information, i.e. the number of clusters passed through compression. E.g., would doubling the amount of clusters you use drastically improve performance? I’m not expecting this experiment to appear in the paper, but perhaps it’s worth discussing.

The majority of results is limited to one architecture, CORnet-S, which is not a very standard deep learning model. However, I think it is OK as repeating the experiments on something like ResNet50 would be very resource-consuming, and CORnet-S appears to be a better model of the visual stream anyway.

## Recommendation
As the paper addresses an important problem with a novel approach and extensive experiments, I recommend to accept the paper.

## Minor points
In Fig.1, were learning rates tuned individually for each length of training, or were short runs just snapshots of the longer ones? If it’s the latter, I’d imagine you can get better results by cranking up the learning rate and later decreasing it before you hit a plateau (e.g., by spacing out the same three LR decreases within a shorter run). I guess doing it for each point in Fig.1A is too much training, but repeating all points in Fig.1B-C should be possible?

> For instance, the fully trained model’s 55 million supervised updates translate to 5.5 updates every second, whereas the model with 1 million updates and 76% relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans

I would say that a more correct conclusion is that supervised learning is a bad concept for development, but I also think your results are applicable beyond supervised learning. Basically, you show that reducing the amount of updates in backprop retains good Brain-Score performance. But that probably tells more about SGD dynamics in deep networks rather than a specific loss function, and might easily translate to, say, more realistic self-supervised learning. (I guess you briefly mention this point in the discussion.)

> Abstract

> First, we find that only 2% of supervised updates (epochs and images) are needed to achieve ∼80% of the match to adult ventral stream.

80% of the best performing network -- currently it reads as 80% of absolute performance, which would presumably be 1.0 Brain-Score?

The caption of Fig.3C-D should better explain what exactly is trained in each setting

In Sec.6, you should specify that you used MobileNet V1 and ResNet-50. The paper provides novel results and important evaluations. There are some methodological issues, but they are not critical to the main point. I therefore favor acceptance. 

Meta-review: This paper experiments with what is required for a deep neural network to be similar to the visual activity in the ventral stream (as judged by the brainscore benchmark). The authors have several interesting contributions, such as showing that a small number of supervised updates are required to predict most of the variance in the brain activity, or that models with randomized synaptic weights can also predict a significant portion of the variance. These different points serve to better connect deep learning to important questions in neuroscience and the presence of the paper at ICLR would create good questions. The discussion between authors and reviewers resulted in a unanimous vote for acceptance, and the authors already made clarifications to the paper. I recommend acceptance.

Decision: Accept (Spotlight)