Wiring Up Vision: Minimizing Supervised Synaptic Updates Needed to Produce a Primate Ventral Stream

1 INTRODUCTION . Particular artificial neural networks ( ANNs ) are the leading mechanistic models of visual processing in the primate visual ventral stream ( Schrimpf et al. , 2020 ; Kubilius et al. , 2019 ; Dapello et al. , 2020 ) . After training on large-scale datasets such as ImageNet ( Deng et al. , 2009 ) by updating weights based on labeled images , internal representations of these ANNs partly match neural representations in the primate visual system from early visual cortex V1 through V2 and V4 to high-level IT ( Yamins et al. , 2014 ; Khaligh-Razavi & Kriegeskorte , 2014 ; Cadena et al. , 2017 ; Tang et al. , 2018 ; Schrimpf et al. , 2018 ; Kubilius et al. , 2019 ) , and model object recognition behavior can partly account for primate object recognition behavior ( Rajalingham et al. , 2018 ; Schrimpf et al. , 2018 ) . Recently , such models have been criticized due to how their learning departs from brain development because they require many more labeled examples than is reasonable for biological systems ’ limited waking ( visual ) experience ( Seibert , 2018 ; Zador , 2019 ) . For example , all the current top models of the primate ventral stream rely on trillions of supervised synaptic updates , i.e . the training of millions of parameters with millions of labeled examples over dozens of epochs . In biological systems , on the other hand , the at-birth synaptic wiring as encoded by the genome already provides structure that is sufficient for macaques to exhibit adult-like visual representations after a few months ( Movshon & Kiorpes , 1988 ; Kiorpes & Movshon , 2004 ; Seibert , 2018 ) , which restricts the amount of experience dependent learning . Furthermore , different neuronal populations in cortical circuits undergo different plasticity mechanisms : neurons in supragranular and infragranular layers adapt more rapidly than those in layer 4 which receives inputs from lower areas ( Diamond et al. , 1994 ; Schoups et al. , 2001 ) , while current artificial synapses , on the other hand , all change under the same plasticity mechanism . While current models provide a basic understanding of the neural mechanisms of adult ventral stream inference , can we start to build models that provide an understanding of how the ventral stream “ wires itself up ” – models of the initial state at birth and how it develops during postnatal life ? Related Work . Several papers have addressed related questions in machine learning : Distilled student networks can be trained on the outputs of a teacher network ( Hinton et al. , 2015 ; Cho & Hariharan , 2019 ; Tian et al. , 2019 ) , and , in pruning studies , networks with knocked out synapses perform reasonably well ( Cheney et al. , 2017 ; Morcos et al. , 2018 ) , demonstrating that models with many trained parameters can be compressed ( Wu et al. , 2018 ) which is further supported by the convergence of training gradients onto a small subspace ( Gur-Ari et al. , 2018 ) . Tian et al . ( 2020 ) show that a pre-trained encoder ’ s fixed features can be used to train a thin decoder with performance close to full fine-tuning and recent theoretically-driven work has found that training only BatchNorm layers ( Frankle et al. , 2021 ) or determining the right parameters from a large pool of weights ( Frankle et al. , 2019 ; Ramanujan et al. , 2019 ) can already achieve high classification accuracy . Unsupervised approaches are also starting to develop useful representations without requiring many labels by inferring internal labels such as clusters or representational similarity ( Caron et al. , 2018 ; Wu et al. , 2018 ; Zhuang et al. , 2019 ; Hénaff et al. , 2019 ; Konkle & Alvarez , 2020 ; Zhuang et al. , 2020 ) . Many attempts are also being made to make the learning algorithms themselves more biologically plausible ( e.g . Lillicrap et al. , 2016 ; Scellier & Bengio , 2017 ; Pozzi et al. , 2020 ) . Nevertheless , all of these approaches require many synaptic updates in the form of labeled samples or precise machinery to determine the right set of weights . In this work , we take first steps of relating findings in machine learning to neuroscience and using such models to explore hypotheses about the product of evolution ( a model ’ s “ birth state ” ) while simultaneously reducing the number of supervised synaptic updates ( a model ’ s visual experience dependent development ) without sacrificing high brain predictivity . Our contributions follow from a framework in which evolution endows the visual system with a well-chosen , yet still largely random “ birth ” pattern of synaptic connectivity ( architecture + initialization ) , and developmental learning corresponds to training a fraction of the synaptic weights using very few supervised labels . We do not view the proposed changes as fully biological models of post-natal development , only that they more concretely correspond to biology than current models . Solving the entire problem of development all at once is too much for one study , but even partial improvements in this direction will likely be informative to further work . Specifically , 1. we build models with a fraction of supervised updates ( training epochs and labeled images ) that retain high similarity to the primate ventral visual stream ( quantified by a brain predictivity score from benchmarks on Brain-Score ( Schrimpf et al. , 2018 ) ) and find that layers corresponding to higher visual regions such as IT are most dependent on training , 2. we improve the “ at-birth ” synaptic connectivity to show that even low-capacity evolutionarily encoded information might lead to reasonable initial representations with no training at all , 3. we propose a thin , “ critical training ” technique which reduces the number of trained synapses while maintaining high brain predictivity and improves over previous computer vision attempts to minimize trained components , 4. we combine these three techniques to build models with two orders of magnitude fewer supervised synaptic updates but high brain predictivity relative to a fully trained model Code and pre-trained models are available through GitHub : https : //anonymous.4open . science/r/anonymous-3A61/ . 2 MODELING PRIMATE VISION . We evaluate all models on a suite of ventral stream benchmarks in Brain-Score ( Schrimpf et al. , 2018 ; 2020 ) , and we base the new models presented here on the CORnet-S architecture , one of the most accurate models of adult primate visual processing ( Kubilius et al. , 2019 ) . Brain-Score benchmarks . To obtain quantified scores for brain-likeness , we use a thorough set of benchmarks from Brain-Score ( Schrimpf et al. , 2018 ) . To keep scores comparable , we only included those neural benchmarks from Brain-Score ( Schrimpf et al. , 2018 ) with the same predictivity metric . All benchmarks feed the same images to a candidate model that were used for primate experiments while “ recording ” activations or measuring behavioral outputs . Specifically , the V1 and V2 benchmarks present 315 images of naturalistic textures and compare model representations to primate single-unit recordings from Freeman et al . ( 2013 ) ( 102 V1 and 103 V2 neurons ) ; the V4 and IT benchmarks present 2,560 naturalistic images and compare models to primate Utah array recordings from Majaj et al . ( 2015 ) ( 88 V4 and 168 IT electrodes ) . A linear regression is fit from model to primate representations in response to 90 % of the images and its prediction score on the held-out 10 % of images is evaluated with Pearson correlation , cross-validated 10 times . The behavioral benchmark presents 240 images and compares model to primate behavioral responses from Rajalingham et al . ( 2018 ) . A logistic classifier is fit on models ’ penultimate representations on 2,160 separate labeled images . The classifier is then used to estimate probabilities for 240 held-out images . Per-image confusion patterns between model and primate are compared with a Pearson correlation . All benchmark scores are normalized by the respective ceiling . We primarily report the average brain predictivity score as the mean of V1 , V2 , V4 , IT , and behavioral scores . We note that the Brain-Score benchmarks in this study are based on limited data and thus present a possible limitation . Nonetheless , they are the most extensive set of primate ventral stream neuronal and behavioral benchmarks that is currently available and the scores generalize to new experiments ( Kubilius et al. , 2019 ) . Brain-Score provides separate sets of data as public benchmarks which we use to determine the type of distribution in Section 4 , and the layer-to-region commitments of reference models . CORnet-S. One of the current best model architectures on the Brain-Score benchmarks is CORnet-S ( Kubilius et al. , 2019 ) , a shallow recurrent model which anatomically commits to ventral stream regions . CORnet-S has four computational areas , analogous to the ventral visual areas V1 , V2 , V4 , and IT , and a linear decoder that maps from neurons in the model ’ s last visual area to its behavioral choices . The recurrent circuitry ( Figure 3B ) uses up- and down-sampling convolutions to process features and is identical in each of the models visual areas ( except for V1COR ) , but varies by the total number of neurons in each area . We base all models developed here on the CORnet-S architecture and use the same hyper-parameters as proposed in ( Kubilius et al. , 2019 ) . Representations are read out at the end of anatomically corresponding areas . 3 HIGH SCORES IN BRAIN PREDICTIVITY CAN BE ACHIEVED WITH FEW SUPERVISED UPDATES . We evaluated the brain predictivity scores of CORnet-S variants that were trained with a combination of fewer epochs and images . Models were trained with an initial learning rate of 0.1 , divided by 10 when loss did not improve over 3 epochs , and stopping after three decrements . Figure 1 shows model scores on neural and behavioral Brain-Score measures , relative to a model trained for 43 epochs on all 1.28M labeled ImageNet images . In Panel A , we compare the average score over the five brain measures of various models to the number of supervised updates that each model was trained with , defined as the number of labeled images times the number of epochs . While a fully trained model reaches an average score of .42 after 55,040,000 supervised updates ( 43 epochs × 1.28M images ) , a model with only 100,000 updates already achieves 50 % of that score , and 1,000,000 updates increase brain predictivity scores to 76 % . Models are close to convergence score after 10,000,000 supervised updates with performance nearly equal to full training ( 97 % ) . Scores grow logarithmically with an approximate 5 % score increase for every order of magnitude more supervised updates . Figures 1B and C show individual neural and behavioral scores of models trained with fewer training epochs or labeled images independently . Early to mid visual representations ( V1 , V2 , and V4 scores ) are especially closely met with only few supervised updates , reaching 50 % of the final trained model in fractions of the first epoch ( Figure 1B ) . After only one full iteration over the training set , V1 , V2 , and V4 scores are close to their final score ( all > 80 % ) while IT requires two epochs to reach a comparable level . Behavioral scores take slightly longer to converge ( > 80 % after 7 epochs ) . Similarly , when training until convergence with fractions of the 1.28M total images , 50,000 images are sufficient to obtain high neural scores ( 80 % of full training in V1 , V2 , V4 , IT ) . Behavioral scores again require more training : half the standard number of labeled images is needed to surpass 80 % . Concretely relating supervised updates to primate ventral stream development , Seibert ( 2018 ) establishes that no more than∼4 months – or 10 million seconds – of waking visual experience is needed to reach adult-level primate IT cortex ( as assessed by its capability to support adult level object recognition ) . From this estimate , we can compute how many supervised updates per second different models in Figure 1A would require ( assuming those updates are evenly distributed over the 10 million seconds ) . For instance , the fully trained model ’ s 55 million supervised updates translate to 5.5 updates every second , whereas the model with 1 million updates and 76 % relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans ( Yarbus , 1967 ; Gibaldi & Sabatini , 2020 ) .

Summary: The paper addresses the question of how many weight updates are needed to train a deep network before it takes on biologically realistic representations. The paper uses CORnet-S (a network that has been proposed to resemble primate ventral stream), and BrainScore (a benchmark of how closely related deep network responses are to visual responses in primate ventral stream). Three ways of reducing the numbers of weight updates are explored, each of which is found to vastly reduce updates while moderately reducing BrainScore. First, the network is simply trained for fewer epochs. Second, weights are initialized with clusters of weights found after training. Third, only a subset of layers is updated. A combination of methods leads to 80% of full brain predictivity with 0.5% of the standard number of weight updates.