Wiring Up Vision: Minimizing Supervised Synaptic Updates Needed to Produce a Primate Ventral Stream

1 INTRODUCTION . Particular artificial neural networks ( ANNs ) are the leading mechanistic models of visual processing in the primate visual ventral stream ( Schrimpf et al. , 2020 ; Kubilius et al. , 2019 ; Dapello et al. , 2020 ) . After training on large-scale datasets such as ImageNet ( Deng et al. , 2009 ) by updating weights based on labeled images , internal representations of these ANNs partly match neural representations in the primate visual system from early visual cortex V1 through V2 and V4 to high-level IT ( Yamins et al. , 2014 ; Khaligh-Razavi & Kriegeskorte , 2014 ; Cadena et al. , 2017 ; Tang et al. , 2018 ; Schrimpf et al. , 2018 ; Kubilius et al. , 2019 ) , and model object recognition behavior can partly account for primate object recognition behavior ( Rajalingham et al. , 2018 ; Schrimpf et al. , 2018 ) . Recently , such models have been criticized due to how their learning departs from brain development because they require many more labeled examples than is reasonable for biological systems ’ limited waking ( visual ) experience ( Seibert , 2018 ; Zador , 2019 ) . For example , all the current top models of the primate ventral stream rely on trillions of supervised synaptic updates , i.e . the training of millions of parameters with millions of labeled examples over dozens of epochs . In biological systems , on the other hand , the at-birth synaptic wiring as encoded by the genome already provides structure that is sufficient for macaques to exhibit adult-like visual representations after a few months ( Movshon & Kiorpes , 1988 ; Kiorpes & Movshon , 2004 ; Seibert , 2018 ) , which restricts the amount of experience dependent learning . Furthermore , different neuronal populations in cortical circuits undergo different plasticity mechanisms : neurons in supragranular and infragranular layers adapt more rapidly than those in layer 4 which receives inputs from lower areas ( Diamond et al. , 1994 ; Schoups et al. , 2001 ) , while current artificial synapses , on the other hand , all change under the same plasticity mechanism . While current models provide a basic understanding of the neural mechanisms of adult ventral stream inference , can we start to build models that provide an understanding of how the ventral stream “ wires itself up ” – models of the initial state at birth and how it develops during postnatal life ? Related Work . Several papers have addressed related questions in machine learning : Distilled student networks can be trained on the outputs of a teacher network ( Hinton et al. , 2015 ; Cho & Hariharan , 2019 ; Tian et al. , 2019 ) , and , in pruning studies , networks with knocked out synapses perform reasonably well ( Cheney et al. , 2017 ; Morcos et al. , 2018 ) , demonstrating that models with many trained parameters can be compressed ( Wu et al. , 2018 ) which is further supported by the convergence of training gradients onto a small subspace ( Gur-Ari et al. , 2018 ) . Tian et al . ( 2020 ) show that a pre-trained encoder ’ s fixed features can be used to train a thin decoder with performance close to full fine-tuning and recent theoretically-driven work has found that training only BatchNorm layers ( Frankle et al. , 2021 ) or determining the right parameters from a large pool of weights ( Frankle et al. , 2019 ; Ramanujan et al. , 2019 ) can already achieve high classification accuracy . Unsupervised approaches are also starting to develop useful representations without requiring many labels by inferring internal labels such as clusters or representational similarity ( Caron et al. , 2018 ; Wu et al. , 2018 ; Zhuang et al. , 2019 ; Hénaff et al. , 2019 ; Konkle & Alvarez , 2020 ; Zhuang et al. , 2020 ) . Many attempts are also being made to make the learning algorithms themselves more biologically plausible ( e.g . Lillicrap et al. , 2016 ; Scellier & Bengio , 2017 ; Pozzi et al. , 2020 ) . Nevertheless , all of these approaches require many synaptic updates in the form of labeled samples or precise machinery to determine the right set of weights . In this work , we take first steps of relating findings in machine learning to neuroscience and using such models to explore hypotheses about the product of evolution ( a model ’ s “ birth state ” ) while simultaneously reducing the number of supervised synaptic updates ( a model ’ s visual experience dependent development ) without sacrificing high brain predictivity . Our contributions follow from a framework in which evolution endows the visual system with a well-chosen , yet still largely random “ birth ” pattern of synaptic connectivity ( architecture + initialization ) , and developmental learning corresponds to training a fraction of the synaptic weights using very few supervised labels . We do not view the proposed changes as fully biological models of post-natal development , only that they more concretely correspond to biology than current models . Solving the entire problem of development all at once is too much for one study , but even partial improvements in this direction will likely be informative to further work . Specifically , 1. we build models with a fraction of supervised updates ( training epochs and labeled images ) that retain high similarity to the primate ventral visual stream ( quantified by a brain predictivity score from benchmarks on Brain-Score ( Schrimpf et al. , 2018 ) ) and find that layers corresponding to higher visual regions such as IT are most dependent on training , 2. we improve the “ at-birth ” synaptic connectivity to show that even low-capacity evolutionarily encoded information might lead to reasonable initial representations with no training at all , 3. we propose a thin , “ critical training ” technique which reduces the number of trained synapses while maintaining high brain predictivity and improves over previous computer vision attempts to minimize trained components , 4. we combine these three techniques to build models with two orders of magnitude fewer supervised synaptic updates but high brain predictivity relative to a fully trained model Code and pre-trained models are available through GitHub : https : //anonymous.4open . science/r/anonymous-3A61/ . 2 MODELING PRIMATE VISION . We evaluate all models on a suite of ventral stream benchmarks in Brain-Score ( Schrimpf et al. , 2018 ; 2020 ) , and we base the new models presented here on the CORnet-S architecture , one of the most accurate models of adult primate visual processing ( Kubilius et al. , 2019 ) . Brain-Score benchmarks . To obtain quantified scores for brain-likeness , we use a thorough set of benchmarks from Brain-Score ( Schrimpf et al. , 2018 ) . To keep scores comparable , we only included those neural benchmarks from Brain-Score ( Schrimpf et al. , 2018 ) with the same predictivity metric . All benchmarks feed the same images to a candidate model that were used for primate experiments while “ recording ” activations or measuring behavioral outputs . Specifically , the V1 and V2 benchmarks present 315 images of naturalistic textures and compare model representations to primate single-unit recordings from Freeman et al . ( 2013 ) ( 102 V1 and 103 V2 neurons ) ; the V4 and IT benchmarks present 2,560 naturalistic images and compare models to primate Utah array recordings from Majaj et al . ( 2015 ) ( 88 V4 and 168 IT electrodes ) . A linear regression is fit from model to primate representations in response to 90 % of the images and its prediction score on the held-out 10 % of images is evaluated with Pearson correlation , cross-validated 10 times . The behavioral benchmark presents 240 images and compares model to primate behavioral responses from Rajalingham et al . ( 2018 ) . A logistic classifier is fit on models ’ penultimate representations on 2,160 separate labeled images . The classifier is then used to estimate probabilities for 240 held-out images . Per-image confusion patterns between model and primate are compared with a Pearson correlation . All benchmark scores are normalized by the respective ceiling . We primarily report the average brain predictivity score as the mean of V1 , V2 , V4 , IT , and behavioral scores . We note that the Brain-Score benchmarks in this study are based on limited data and thus present a possible limitation . Nonetheless , they are the most extensive set of primate ventral stream neuronal and behavioral benchmarks that is currently available and the scores generalize to new experiments ( Kubilius et al. , 2019 ) . Brain-Score provides separate sets of data as public benchmarks which we use to determine the type of distribution in Section 4 , and the layer-to-region commitments of reference models . CORnet-S. One of the current best model architectures on the Brain-Score benchmarks is CORnet-S ( Kubilius et al. , 2019 ) , a shallow recurrent model which anatomically commits to ventral stream regions . CORnet-S has four computational areas , analogous to the ventral visual areas V1 , V2 , V4 , and IT , and a linear decoder that maps from neurons in the model ’ s last visual area to its behavioral choices . The recurrent circuitry ( Figure 3B ) uses up- and down-sampling convolutions to process features and is identical in each of the models visual areas ( except for V1COR ) , but varies by the total number of neurons in each area . We base all models developed here on the CORnet-S architecture and use the same hyper-parameters as proposed in ( Kubilius et al. , 2019 ) . Representations are read out at the end of anatomically corresponding areas . 3 HIGH SCORES IN BRAIN PREDICTIVITY CAN BE ACHIEVED WITH FEW SUPERVISED UPDATES . We evaluated the brain predictivity scores of CORnet-S variants that were trained with a combination of fewer epochs and images . Models were trained with an initial learning rate of 0.1 , divided by 10 when loss did not improve over 3 epochs , and stopping after three decrements . Figure 1 shows model scores on neural and behavioral Brain-Score measures , relative to a model trained for 43 epochs on all 1.28M labeled ImageNet images . In Panel A , we compare the average score over the five brain measures of various models to the number of supervised updates that each model was trained with , defined as the number of labeled images times the number of epochs . While a fully trained model reaches an average score of .42 after 55,040,000 supervised updates ( 43 epochs × 1.28M images ) , a model with only 100,000 updates already achieves 50 % of that score , and 1,000,000 updates increase brain predictivity scores to 76 % . Models are close to convergence score after 10,000,000 supervised updates with performance nearly equal to full training ( 97 % ) . Scores grow logarithmically with an approximate 5 % score increase for every order of magnitude more supervised updates . Figures 1B and C show individual neural and behavioral scores of models trained with fewer training epochs or labeled images independently . Early to mid visual representations ( V1 , V2 , and V4 scores ) are especially closely met with only few supervised updates , reaching 50 % of the final trained model in fractions of the first epoch ( Figure 1B ) . After only one full iteration over the training set , V1 , V2 , and V4 scores are close to their final score ( all > 80 % ) while IT requires two epochs to reach a comparable level . Behavioral scores take slightly longer to converge ( > 80 % after 7 epochs ) . Similarly , when training until convergence with fractions of the 1.28M total images , 50,000 images are sufficient to obtain high neural scores ( 80 % of full training in V1 , V2 , V4 , IT ) . Behavioral scores again require more training : half the standard number of labeled images is needed to surpass 80 % . Concretely relating supervised updates to primate ventral stream development , Seibert ( 2018 ) establishes that no more than∼4 months – or 10 million seconds – of waking visual experience is needed to reach adult-level primate IT cortex ( as assessed by its capability to support adult level object recognition ) . From this estimate , we can compute how many supervised updates per second different models in Figure 1A would require ( assuming those updates are evenly distributed over the 10 million seconds ) . For instance , the fully trained model ’ s 55 million supervised updates translate to 5.5 updates every second , whereas the model with 1 million updates and 76 % relative brain predictivity translates to one labeled image update every 10 seconds which appears more plausible given the upper limit of 2-3 saccades per second in humans ( Yarbus , 1967 ; Gibaldi & Sabatini , 2020 ) .

Summary: This paper presents an empirical study that elucidates potential mechanisms through which models of adult-like visual streams can "develop" from less specific/coarser model instantiations. In particular, the authors consider existing ventral stream models whose internal representations and behavior are most brain-like (amongst several other models) and probe how these fair in impoverished regimes of available labeled data and model plasticity (number of "trainable" synapses). They introduce a novel weight initialization mechanism, Weight Compression (WC), that allows their models to retain good performance even at the beginning of training, before any synaptic update. They also explore a particular methodology for fine-tuning, Critical Training (CT), that selectively updates parameters that seem to yield the most benefit. Finally, they explore these methods/algorithms' transfer performance from one ventral stream model (CORnet-S) to two additional models (ResNet-50 and MobileNet).