Layerwise Learning Rates for Object Features in Unsupervised and Supervised Neural Networks And Consequent Predictions for the Infant Visual System
Authors: Rhodri Cusack
Cliona O'Doherty
Anna Birbeck
Anna Truzzi
Venue:  (2020)
Keywords: deep learning
unsupervised
supervised
infant learning
age of acquisition
DeepCluster
CORnet
AlexNet

Abstract: To understand how object vision develops in infancy and childhood, it will be necessary to develop testable computational models. Deep neural networks (DNNs) have proven valuable as models of adult vision, but it is not yet clear if they have any value as models of development. As a first model, we measured learning in a DNN designed to mimic the architecture and representational geometry of the visual system (CORnet). We quantified the development of explicit object representations at each level of this network through training by freezing the convolutional layers and training an additional linear decoding layer. We evaluate decoding accuracy on the whole ImageNet validation set, and also for individual visual classes. CORnet, however, uses supervised training and because infants have only extremely impoverished access to labels they must instead learn in an unsupervised manner. We therefore also measured learning in a state-of-the-art unsupervised network (DeepCluster). CORnet and DeepCluster differ in both supervision and in the convolutional networks at their heart, thus to isolate the effect of supervision, we ran a control experiment in which we trained the convolutional network from DeepCluster (an AlexNet variant) in a supervised manner. We make predictions on how learning should develop across brain regions in infants. In all three networks, we also tested for a relationship in the order in which infants and machines acquire visual classes, and found only evidence for a counter-intuitive relationship. We discuss the potential reasons for this.

Under review as a conference paper at ICLR 2020
LAYERWISE LEARNING RATES FOR OBJECT FEATURES IN UNSUPERVISED AND SUPERVISED NEURAL NET- WORKS AND CONSEQUENT PREDICTIONS FOR THE IN- FANT VISUAL SYSTEM
Anonymous authors Paper under double-blind review
ABSTRACT
To understand how the brain and mind develop in infancy, it is necessary de- velop testable computational models. In recent years, DNNs have proven valuable models of the adult brain. In the domain of object recognition, for example, convolutional DNNs currently provide the best way to predict the response to a novel stimulus in the ventral visual stream of the adult monkey and adult human. Given this success in modelling the mature brain, we propose them as candidate models for the learning process. We tested if learning different DNNs yields distinct trajectories of development at the large scale that could be testable in fu- ture brain imaging experiments. First, we quantified the development of explicit object representation at each layer in the network at various points in the learning process, by freezing the convolutional weights and training an additional linear de- coding layer. In two DNNs trained in supervised way (CORnet-S and Alexnet), object representation was strongest in the top layer throughout learning. Infants, however, have only extremely impoverished access to labels, and unsupervised learning is likely a better model for their learning process. Infants are known to identify clusters of similar-looking objects, and
evaluated unsupervised DNN with a clustering objective (DeepCluster). Through learning, there was a bottom-up sweep in the development of object representations, with the penulti- mate layer strongest at the end. These results show that different training strategies yield different layerwise trajectories of development, which could be empirically measured in infants with neuroimaging. In all three DNNs, we also tested for a relationship in the order in which infants and machines acquire visual classes. We found different patterns, which could be driven by the different visual features that DNNs and humans exploit, or by differences in the objective function used for learning. Our results show that DNNs can make distinct and testable predictions for infant development. The parallel that we are creating highlights ways in which knowledge of the infant brain and mind could guide the design of new DNNs, such as by providing insight into the selection of auxiliary tasks for unsupervised training, from the myriad available.
INTRODUCTION
1.1
THE NEED FOR MODELS OF DEVELOPMENT IN INFANTS
Humans are helpless for a long time during infancy when compared with most other species. During this helpless period, the brain and mind develop result of genetically programmed maturation in concert with experience from the environment. One way to measure development during the helpless period is by examining infants' capabilities through behaviour. However, despite great ingenuity in experimental design, due to the limited behavioural repertoire of infants, it has proven difficult to obtain more than basic understanding of how the brain and mind developing. Furthermore, some brain systems may undergo extended development prior to any manifestation in behaviour (Tolma & Honzik 1930). Therefore, in addition measuring behaviour new and promising

---

Under review as a conference paper at ICLR 2020
complementary tool is magnetic resonance imaging (MRD, which can measure the structure, wiring and function of the infant brain.
To understand why infants' brains and minds change in the way they do, need to create models of the mechanisms of development. This critical if we are to understand how development is disrupted in infants with brain injury. Infants that are born very preterm, that have potentially negative event during birth are admitted to the neonatal intensive care unit (NICU). At present, a significant proportion of NICU infants develop cognitive, behavioural social impairments later in life. To detect these impairments earlier, and to design effective targeted interventions, we need to create models of brain and mind development in young infants. As is difficult to intuitively imagine what the mind of a developing infant is like, it is necessary to create computational models of infant learning that can be tested against real data.
1.2 DNNs AS A MODEL FOR OBJECT VISION IN ADULTS
promising first brain system for which computational models of infant learning could be built is the object recognition system in the ventral visual stream. This system is relatively well understood in adults and there are proven neuroimaging methods that can be used to probe its representations and to test the hypotheses that will be generated by our computational models in due course. Indeed, in adults, there is substantial work on modelling the ventral visual stream. A key discovery from this work has been that the most straightforward strategy, of first collecting neural data and then creating models to fit it, has not been very effective (Yamins DiCarlo 2016). The issue is that it only feasible to acquire relatively small quantity of neuronal data, but the computational models capable of performing object vision have many millions of parameters. A more effective strategy, therefore, is instead use computational models that optimised to perform object vision to high accuracy, and then to use these as models of neural activity (Yamins & DiCarlo 2016). Using this strategy, has been shown that DNNs designed for the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) can predict patterns of neural activity in the ventral visual stream, as measured with functional magnetic resonance imaging (fMRI) (Khaligh-Razze Razavi Kriegeskorte 2014 Güşlü & van Gerven Wen et al. 2018). electroencephalographh (EEG) and behavioural studies (Cichy et al.) 2016). It is important to note that the claim in these studies is not that there a one-to-one mapping between artificial neurons and biological neurons. Rather, it is that some of the macroscopic aspects of the activity patterns in the DNNs and the brain are similar. For example, the ventral visual stream forms a hierarchy, with visual input from the eyes being passed forwards through a number of brain structures, and gradually being transformed from a representation of the visual input into a representation of semantic category. A quantitatively similar hierarchy of representations is found through the layers of the DNN (Güççlü & van Gerven Wen et al. 2018).
1.3 INFANTS ARE LEARNING, Do THEY LEARN LIKE DNNs?
Some aspects of biological object vision are innate, such as a preference for looking at faces, which is apparent in the first hour after birth (Johnson al. 1991). However, much must be learned through visual experience, as the genetic code alone is limited in capacity (estimated to be 150-750 ME and would be substantially exhausted by the parameters of deep learning models (50-150 MB (Bianco) let al. 2018)). Even if sufficient storage were available, many objects relevant to infants today (e.g.. baby bottle, or a Pokémon (Janini & Konkle 2019)) were invented too recently for innate codes to have evolved for them. A great deal of object vision knowledge, therefore, is learned.
There is evidence that much of this learning is already happening in early infancy. Infants aged 3-4 months old can identify statistical regularities in sequences of pictures (Quinn et al. 1993 French et al. 2004). And, by months, infants start to look at a visual class corresponding to a concurrently spoken label (Bergelson & Swingley 2012 In adults, fMRI of the ventral visual stream has found that there are regions that are selective for partieular visual classes, such as faces, body parts or
places (Kanwisher et al. 1997 Epstein & Kanwisher, 1998. Downing et al.) (00T). In 4-6 month old infants, at least partial selectivity is already present in the ventral visual stream (Deen et al, 2017), although it continues to develop for many years (Gomez et al. (017).
https//www..uura.com/Httt
althouug vocabulary remains very limited until after the first birthday, when it typically hegins to grow rapidly.
2

---

Under review as a conference paper at ICLR 2020
DNNs have provided a useful model of adult object vision. As these DNNs learn from visual "expe- rience" they are therefore candidate models of infant learning. We do not expect them to capture the precise details of the learning process as, for example, the learning rule of back propagation is not biologically plausible (although biologically plausible mechanisms can approximate it (Richards) et al. 2019).. ) Rather, overarching hypothesis is that DNNs might capture some of the broad
macro-scale characteristics of learning.
1.4 AIMS OF THE CURRENT STUDY
In this work, we characterise two macro-scale properties of a number of computational models, with the aim of generating testable measurements for a future large-scale neuroimaging study.
1. Should we expect representations in brain regions of the visual hierarchy to develop simul- taneously or asynchronously? An established principle of infant development is that brain regions underlying simpler functions develop first, and are followed by those underlying more complex functions (Charles A. Nelson in Shonkoff & Phillips (2000) P To provide
an initial prediction of how this might happen in the visual hierarchy we examined how representations developed in different layers of DNNs during training.
2. Are visual classes that are learned earlier by infants also learned earlier by DNNs? In infants, the acquisition of visual classes can be estimated by the onset of the receptive or expressive use of the words for the classes. has been found that, when measured this way, some visual classes are learned before others, e.g., body parts and vehicles precede food and clothing (Braginsky et al.) 2013). The link between the acquisition of words and brain representations in the ventral visual stream is supported by fMRI results examining the vi- sual representations of mental imagery have shown that object classes can decoded from brain representations when participants are tasked with reading concrete noun (Anderson) et af., 2015) Kellenbach et al., 2000). We ask whether some of this ordering is attributable to visual complexity, as reflected in the DNN classification performance.
METHODS
2.1 CHOICE OF NETWORKS AND TRAINING
2. CORNET-S WITH SUPERVISED TRAINING
To date, studies which have shown a parallel between DNNs and the adult brain have used networks trained in supervised manner (Khaligh- Razavi & Kriegeskorte 2014 Güglü & van Gerven Cichy et al. 2016 Wen et al. 2018 Jozwik et al. (018). To remain consistent with these studies, we started with supervised network. Specifically,
used CORnet-S (Kubilius al.) 2018), this was especially designed to meet the Brain-Score benchmark (Schrimpf et al. (018)) by capturing the architectural principles and showing strong predictivity of neural data from the ventral visual stream, while still achieving good classification performance. It has four layers mapping onto regions in the ventral visual stream (V1, V2, V4 and IT)..
2.1.2 DEEPCLUSTER WITH UNSUPERVISED TRAINING
Infants' access to labels is extremely impoverished, making supervised learning an unlikely training curriculum for infant learning. In order to develop computational models that capture the dynamics of infant learning it is necessary to evaluate unsupervised training. One current state-of-the- art unsupervised strategy for learning visual features for object recognition is DeepCluster (Caron al.] 2018), which adjusts convolutional weights to create clusters of images that yield similar patterns of activation. This has a parallel with behavioural results of infant learning, in that they are known to create clusters of similar images, and to be sensitive to deviants from that cluster (Quinn et al.) 1993 French et al.) 2004). DeepCluster uses a simple but elegant technique for self-supervised learning, in which the start of an epoch sees each image (from ImageNet) being passed forward through a convolutional network, and the resulting output activations are then clustered across all of
"This influential work has heen cited more than 7000 times.
3

---

Under review as a conference paper at ICLR 2020
the images using k-means. These clusters are assigned labels, which subsequently used to learn the weights of the convolutional network with stochastic gradient descent on batches images in the typical way.
As in Caron et al. (2118) we used 10,000 clusters. We used AlexNet (Krizhevsky et al. 2012) as the convolutional network, containing five layers each with single convolutional layer. These layers were modified as in |Caron et al. (2018) with the local response normalisation layers removed and batch normalisation used instead (loffe & Szegedy 2015). Also like Caron al.] (2018), a fixed linear transformation based on Sobel filters was used on the input to remove colour and increase local contrast.
2.1.3 ALEXNET WITH SUPERVISED TRAINING
CORnet trained in supervised way, and DeepCluster in unsupervised way. Any difference in the results might be due to this difference supervision. However, the convolutional networks at the heart of these two networks also differ, which could cause further differences. To control for this, we therefore repeated the experiments using the same AlexNet variant as in DeepCluster, but trained supervised way.
2.2 LEARNING TRAJECTORIES
The value for object recognition of the representations was assessed for each of the layers in the net- works (4 layers for CORnet, and 5 layers for DeepCluster and Alexnet), Specifically, we quantified the explicit representation (DiCarlo & Cox 2007) of object class using the method by ZZang et al. (017)) of freezing weights in the convolutional layers, and training a linear decoder on the output of each layer decode the ImageNet categories. This was done across epochs in the learning process, to capture the development of the representations in each of the layers.
2.3 SUMMARISING LEARNING OF VISUAL CLASSES
To summarise the learning curves of the individual classes we fitted the performance with the curve
p A(1 exp(-kt))
where p is top-5 precision, t the epoch, A the asymptotic level of performance, and k the learning rate. Fitting minimised least squares with the addition to the cost function of two regularisation terms equal to k² (to discourage implausibly high learning rates) and (A 100) * (A 100)* to discourage values greater than 100%.
2.4
AGE OF ACQUISITION INFANTS
We also compared the order in which visual classes acquired in infants and machines. Unfor- tunately, there is no data available at present of the order in which infants acquire different visual classes. Therefore, word for the class (Kuperman et al. 2012). While a number of linguistic factors are known to af- feet when words are first used, including the frequency of the word in language and its number of phonemes, critically the second strongest factor is the "concreteness" of the word (Braginsky et al.) 2015). Concreteness the degree which word associated with a perceptual representation, and is typically obtained by asking people to rate it. The concept has been experimentally validated many studies, and concrete words have been shown more readily evoke visual representations in the brain which are capable of being decoded as visual classes using fMRI and EEG (Kellenbach al., 2000) Anderson al.) 2015). Thus, the association of AoA with concreteness suggests that the strength of the visual representation of class may have an effect This might happen due to one clear constraint; child cannot name a visual class before the repre- sentation of that visual class has been developed. So, tested if the visual classes that are labelled earlier in infants are learned more quickly by DNNs.
used proxy, which is the estimate of the age of acquisition (AoA) of the
when its label is acquired.
Using the Natural Language Toolkit (NLTK), the WordNet synsets for the 1000 ImageNet classes were compared to Kuperman et al. (2012) database of 30,000 English words with AoA ratings using Leacock al. (1998) semantic similarity metric. The classes with the highest similarity score were
4

---

Under review as a conference paper at ICLR 2020
DeepCluster
60
Top 5 Precision From Decoding of Layer
80
3
35
50-
30-
60-
40
40-
((4)
20
30
15
0 Layer
20
20
(V2)
0(V1)
Layer
10
Layer
5
10
15
20
25
0
10
15
20
25
10
15
20
25
Epoch
Figure 1:
Explicit representation of visual class measured the three networks during training, as measured by top-5 precision. The corresponding cross-entropy loss is shown in Appendix B, Fig.
considered matching, and manually inspected for any incorrect comparisons or synset definitions. These were deleted, leaving total of 308 classes which further analyses were conducted.
2.5 CLASS CATEGORISATION
To provide visualisation of when different types of classes were learned, we clustered the 308 classes using Leacock et al. (1998)ss metric and then clustering wheearrrhieerrrcyy..o. to yield 20 classes. By visual inspection, we then attached label each of the class clusters.
2.6 IMPLEMENTATION
Training was run on AWS using the Deep Learning AMI version 24.0 on either a p2.8xlarge instance (8 x NVIDIA K80 GPUs with 488 GB of RAM) a p3.8xlarge instance type (4 NVIDIA Tesla V100 GPUs and 244 GB RAM), using Python 3.6 with Pytorch 1.1. Spot instances were used to reduce cost. The three networks were trained from scratch. The ISLVRC 2012 set was used for training and validation. The CORnet-S code was obtained from /ttps: com/ dicarlolab (hcarlolab/CoRnet] CCRneet DeepCluster, Alexnet and the linear classifier implementation were from htttss///csiiss.ccmo/ccccceeerra
RESULTS
3.1 AIM 1: HIERARCHICAL DEVELOPMENT OF REPRESENTATIONS IN DIFFERENT LAYERS
DURING TRAINING
3.1.1 CORNET
Explicit representation of object class in the four layers of the CORnet network during training is shown in Fig. a. The earlier layers in the hierarchy (V1, V2 and V4) reached their asymptotic level quickly (around epoch 1), but IT continued to learn until least epoch 25. However, although IT took longer to reach its asymptote, even after minimal training (epoch 1) it contained greater explicit information than the lower layers.
In human infants, such learning scheme would therefore be be indicated by earlier maturation of lower-order visual processing regions (V1, V2 and V4), than higher-order brain regions (e.g., IT) in the developing brain. However, even early in development, infant IT would be expected to contain stronger explicit representations of object class as was seen for the IT layer of CORnet. It may be thought that this result is inherent to how IT is built to function; was optimised to create explicit representations objects, therefore this signature is found throughout training. However, when examining the results from later models find that this is not the case.
5

---

Under review as a conference paper at ICLR 2020
CORnet
100
Top-5 Top-5 precision
80
80
40
20
40
DeepCluster
100
100
Top-5 predsion
80
60
40
40
50
25
AlexNet
100
Top-5 preasion
80
80
60
40
20
20
20
Epochs of training
Figure 2: Left: Precision for each visual class during training in the most informative layer of the network the top layer for CORnet and AlexNet, and the penultimate layer for DeepCluster. The colour of the curves shows AoA of the class's name for infants (blue to red for low to high AoA). Centre: The learning curves were parameterised with a fit (Eqn. [ shown here. Right: Distributions of the fit parameters. They were correlated for all networks (CORnet r(308)=0.62, p<0.001; DeepCluster r(308)=0.36, p<0.01; AlexNet 1(308)=0.39, p<0.001), showing that classes that were learned more quickly also converging on higher asymptote.
3.1.2 DEEPCLUSTER
In contrast, for DeepCluster, representations mature in "bottom-up" manner (Fig. D Specifically, the explicit representation of object class does not monotonically increase with layer even the end of 60 training epochs, layer 3 contains stronger representations of object class than layer 4. Furthermore, the order of the layers varies through training, with layer 2 stronger than layer 3 early in training, and layer containing stronger representations of object class than layer 4.
Extrapolating to the brain, this more developmentally plausible unsupervised strategy would be indicated not only by higher-order regions in the ventral visual stream developing more slowly, but by earlier regions initially leading in the presence of representations of object class. This hypothesis testable in future fMRI experiments.
In supervised training, object labels are provided at the top layer of the network, and so it is perhaps not surprising that even at early epochs the entire network, including the upper layers, are maturing. This contrasts to unsupervised learning during which the only source of information is the visual input, and so is perhaps not surprising that maturation proceeds in more bottom up manner; until good representations have developed in the early layers, there is poorer information at higher layers. This is consistent with the comple-to-coppler maturation theory (Charles A. Nelson in Shhonooff & Phillips (2000)).

---

Under review as a conference paper at ICLR 2020
3.1.3 ALEXNET
CORnet and DeepCluster are not just different their training strategies, but also in the convolu- tional networks at their heart. To control for this, we repeated training with AlexNet. The results Fig. D show that even when same convolutional network as DeepCluster is used, but instead with a supervised training strategy, the bottom-up learning trajectories of DeepCluster are elimi- nated.
3.2 AIM 2: ACQUISITION OF VISUAL CLASSES
Strikingly, explicit object representation the lower layers AlexNet actually reduced from epochs 3-5 onwards when assessed with top-5 loss (Appendix B, Fig. (5)), effect which also in the supervised CORnet albeit much more weakly. In DeepCluster, this reduction in explicit representation did appear, suggesting that may fact be a feature of supervised learning.
In the second aim, we compared machine and human learning across visual classes. The learning were fit well by the model (Eqn. [ left two columns of Fig. 2 The joint distribution (Fig. 2, right column) showed that classes which were learned quickest were ultimately learned best, as the fit parameters were strongly correlated for all three models (CORnet r(308)=0.62; p<0.001; DeepCluster r(308)=0.36, p<0.01; AlexNet r(308)=0.39, p<0.001)
These parameters were then used to compare the machine with human learning. Paradoxically, classes learned more precisely by the model were if anything learned later by infants (Fig. 3 cor- relation AoA and parameter A, CORnet r(308)=0.1 11 p=0.06; DeepCluster r(308)=0.10 10, p=0.09; AlexNet (330)=0.11) 14. p<0.02). Although classes which were learned more precisely were in gen- eral learned more quickly the model, there was no relationship observed between learning rate parameter (k) and infant AoA.
CDRnet
DeepCluster
[d)
100
Top-5 Precision from Best Layer
100
60
60
40
40
20
Age of Acquisition (years)
made objects
Water and vehicies
Medioval fantzsy
Inseds
Buildings,
Notor vehides
Natural enviorments
Ruls
Birds
vihiides
made ensionaaphaioo
Large
Containers
Cisthes
Dogs
Fish
Figure 3: Relationship between infant age of acquisition and asymptote of machine performance for each visual class in the most informative layer of each network. The colours show different semantic groupings.
IMPLICATIONS, LIMITATIONS AND FUTURE WORK
4.1 POTENTIAL DNNS TO PROVIDE INSIGITT INTO INFANT DEVELOPMENT
In Aim 1, the macroscale distribution of object representations was found to be affected by learning strategy, with supervised training concentrating object representations near the output layer, while unsupervised training yielded distributed representations across layers, peaking in the penultimate layer. Learning strategy also led to layerwise differences the order of learning, with unsuper- vised learning showing bottom-up sweep in the object representation, but supervised learning

---

Under review as a conference paper at ICLR 2020
unchanging order with the top layer leading throughout. A control experiment showed that these effects were due to the learning strategy rather than the DNN architecture. In future work, it will be important to extend the investigations to further DNNs. These could test generalisation variants the strategies, such as local aggregation, a recently proposed unsupervised training objective, which like DeepCluster learns visual emhedding based clustering of images (Zhuang et al. 2019). It would be informative to investigate a wider range of objectives, for example testing DNNs that ex- ploit other structures in visual input such as temporal prediction (Lotter et al. 2016) or cross-modal learning (Wang et al., 2013). Future work could also investigate the benefit of making DNNs more similar brains the implementation level, for example by using an alternative batch normali- sation applicable to online learning These macroscale differences in layerwise learning could then be tested in future infant neuroimaging, to identify which models best approximate human learning-
In Aim 2, we found only a counter-intuitive relationship between the visual classes that are learned best by DNNs and those that are named first by infants. This might be because of the limitations of this measure in infants. The AoA of label is only an approximation to when visual class is acquired, as it is affected by other properties including the number of phonemes in the label. In future work, will measure when infants acquire visual categories. There other factors that influence when infants learn to identify a visual class in addition to the class's 's visual "accessibility"". One is the frequency of the class in the environment. The ImageNet classes are esoteric and unecological, with a high preponderance of dog breeds, for example. More human-like learning will probably require more human-likk like (or baby-like) training sets. Another factor is the innate or learned reward value of the class to the baby (e.g., the mother's s face or their milk bottle may be learned earlier).
4.2 POTENTIAL INSIGHTS FROM INFANT DEVELOPMENT TO GUIDE THE DESIGN OF DNNs
The current state-of-the-art DNNs for visual recognition use supervised training, but there strong interest in developing unsupervised strategies, unlabelled data is more plentiful and cheaper. Diverse unsupervised strategies have been demonstrated, such as cross-channel colour prediction (Zhng et al. (017) and feature counting (Noroozi et al. 2017). Given the enormous space of pos- sible unsupervised learning strategies, it is unclear how to direct investigations in a principled way. Knowledge from the human visual system such the nature of motion and colour representations and how they develop in infancy could guide this search, and the layerwise trajectories of develop- ment could provide a constraint for biomimetic DNNs.
There is recent evidence that DNNs use textures rather than shape to recognise objects, while it is the reverse in humans (Geirhos al. 2018a). The specific features relied upon by DNNs is thought to underlie the lack of robustness to noise, as is evident in adversarial attacks (Geirhos et al. 2018b). This difference in feature weighting might be one cause of the different trajectories of learning for different visual classes between machine and human found in Aim 2. Assessing DNNs in how similar their acquisition order is to humans might therefore provide valuable benchmark to guide the development of more noise-robust DNNs.
4.3 CONCLUSIONS
DNNs were inspired by the brain. Although DNNs learn like humans from large quantities of data, there is little work build formal connections between infant and machine learning. Such connections have the potential to bring considerable insight to both fields but the challenge is to find defining characteristics that can be measured in both systems. This paper has addressed this challenge by measuring two characteristic features in DNNs that can be measured in infants.
4.3.1 ACKNOWLEDGMENTS
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Andrew James Anderson, Elia Bruni, Alessandro Lopopolo, Massimo Poesio, and Marco Baroni. Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text. Neurolmage, 120:309-33 -22, 2015.
Bergelson and Swingley. 6-9 months, human infants know the meanings of many com-
mon nouns. Proceedings of the National Academy of Sciences, 109(9):3253-32553 2012. ISSN 0027-8424. doi: 10.1073/pnas. 113380109. UtLhttp: dwt.:xx.oooo...ooooooooooooo www.:.ooogg...... 1113380109
Simone Bianco, Rémi Cadène, Luigi Celona, and Paolo Napoletano. Benchmark analysis of rep-
resentative deep neural network architectures. CoRR, abs/1810.00736, 2018. URL http:
//a
Mika Braginsky, Daniel Yurovsky. Virginia A Marchman, and Michael C Frank. Developmental changes in the relationship between grammar and the lexicon. CogSci, pp. 256-261, 2015.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep Clustering for Un- supervised Learning of Visual Features. July 2018. URL htt::://aaxxv...gg/aaba..
05520
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva. Comparison of deep neural networks to apatio-tempora cortical dynamics of human visual ob- ject recognition reveals hierarchical correspondence. Scientific Reports, 6:27755, June 2016. ISSN 2045-2322. doi: 10. 1038//sepp27755 htL.http::////
www. www.rattutroomm wwtprre...o/aattciiiioi
sxep27755
Ben Deen, Hilary Richardson, Daniel D Dilks, Atsushi Takahashi, Boris Keil, Lawrence L. Wald, Nancy Kanwisher, and Rebecca Saxe. Organization of high-level visual cortex in human infants. Nature communications, 8:13995, 2017.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends cognitive sciences, 11(8):333-341, 2007.
Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A cortical area selective for visual processing of the human body. Science, 293(5539):240--2777 2001.
Russell Epstein and Nancy Kanwisher. cortical representation of the local visual environment.
Nature, 397(667)):598, 1998.
Robert M French, Denis Mareschal, Martial Mermillod, and Paul C Quinn. The role of bottom-up processing in perceptual categorization by 3- to 4-month-old infants: simulations and data. J. Exp. Psychol. Gen., 133(3):382 -373 September 2004. ISSN 0096-3445 doi: (0.10077009--344 1119666-44..333 133.. do 10 1037/0096 -400099633444444 133. 3...3.38
3.382. htt.ttpp://///
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness, 2018a.
Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Mathhis Bethge, and Felix A. Wichmann. Generalisation in humans and deep neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 31, pp. 7538-7550, Curran Associates, Inc., 2018b.
URL http::/papers. .irrrrii.ppss. nips cc/paper/ humans and
1992-generalisationn
Jesse Gomez, Michael A Bareett, Vaidehi Natu, Aviv Mezer, Nicola Palomero-Gallagher Kevin S
Weiner, Katrin Amunts, Karl Zilles, and Kalanit Grill-Spector. Microstructural proliferation in hu-
cortex is coupled with the development of face processing. Science. 355(e320):66632277777777 2017.
ISSN 0036-8075. doi: 10. he2////ieenc.aag03300 /LL.L//////ww.ooeeeeeeneeaaaaa www.://wwwwbmeee.......... nih.gov/artti ic lerender 1at.:aaatrtddm
URL http:/ www.

---

Under review as a conference paper at ICLR 2020
Umut Güęlü and Marcel A J Gerven. Deep Neural Networks Reveal Gradient in the Com- plexity of Neural Representations across the Ventral Stream. doi: 10.6080KKQN64NG. URL http://dx: doi. wwg//0. 6000KOON6400000
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
Daniel Janini and Talia Konkle. A pokémon-size window into the human brain. Nature human behaviour, 3(6):552, 2019.
Mark Johnson, Suzanne Dziurawiec, Hadyn Ellis, and John Morton. Newborns' prefer- ential tracking of face-like stimuli and its subsequent decline. Cognition, 40(1):1-19, Au- gust 1991. ISSN 0010-0277. doi: 10. 10.101600010-027799 990045-66 URL http://wwee scienceds 1 rect. com sc wwwe//atiiceeeeeeeeeeeeeeeeeseee /
Kamila Maria Jozwik, Nikolaus Kriegeskorte, Radoslaw Martin Cichy, and Marieke Mur. Deep convolutional neural networks, features, and categories perform similarly at explaining primate high-level visual representations. 2018 Conference on Cognitive Computational Neuroscience, Philadelphia, Pennsylvania, USA, 2018, Cognitive Computational Neuroscience, doi: 10.32470/ asp?
CCN.2018.1232-0. URL htt.://pp///cuu.rr.ooo/////////2
PaperNum 1232
Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: module in human extrastriate cortex specialized for face perception. Journal of neuroscience, 17(11):4300 4311, 1997.
Marion I. Kellenbach, Albertus A Wijers, and Gijsbertus Mulder. Visual semantic features are activated during the processing of concrete words: Event-related potential evidence for perceptual semantic priming. Cognitive Brain Research, 10(1-2):67-75, 2000.
S M Khaligh-Razee Razavi and N Kriegeskorte. Deep supervised, but not unsupervised, mod- els may explain IT cortical representation. PLoS Comput. Biol., 2014. ISSN 1553- 734X. URL
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel L. K. Yamins, and James J. DiCarlo. CORnet: Modeling the Neural Mechanisms of Core Object Recognition. bioRxiv, 2018. doi: 1011400/488885. AtLpttts: 04/408385
www
Victor Kuperman, Hans Stadthagen Gonzalezz and Mare Brysbaert. Age-of-acquisition ratings for 30,000 english words. Behavior research methods, 44(4):978-99) 2012.
Claudia Leacock, George A Miller, and Martin Chodorow. Using corpus statistics and wordnet relations for sense identification. Computational Linguistics, 2471):147-165, 1998.
William Lotter, Gabriel Kreiman, and David Cox. Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning. May 2016. /ttp:tttp://aaxiv.rrr//////////p
08104
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning count. 2017 IEEE International Conference on Computer Vision (ICCV), Oct 2017. doi: 10.1 109/icev. 2017.628. URL htt.hhtt:///ee //x www.doo.ogg.oooo /0.g//1... 10. 1109/ ICCV 2017.62. 628
C Quinn, P Eimas, and Rosenkrantz. Evidence for representations of perceptually sim- ilar natural categories by 3-month-old and 4-month-old infants. Perception, 22(4):463 475, 475, 1993. ISSN 0301-0066. doi: 10.10688p220463. URL http: /xxx. dei. org. 1068/ p220463
10

---

Under review as a conference paper at ICLR 2020
Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1611-770, 2019.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, Daniel L K Yamins, and James DiCarlo. Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain- Like? September 2018. URL https: www www.://www..og/ooootttt/aaaaa///
09/ 09/0//070077
Jack P Shonkoff and Deborah A Phillips. From neurons to neighborhoods: The science of early childhood development. ERIC, 2000.
Edward Chace Tolman and Charles H Honzik. Introduction and removal of reward, and maze per- formance in rats. University of Califormia publications in psychology, 1930.
Kaiye Wang, Ran He, Wei Wang, Liang Wang, and Tieniu Tan. Learning coupled feature spaces for cross-modal matching. In Proceedings of the IEEE International Conference on Computer Vision, pp.
2013.
Haiguang Wen, Junxing Shi, Wei Chen, and Zhongming Liu. Deep Residual Network Predicts Cortical Representation and Organization Visual Features for Rapid Categorization. Scientific Reports, 8(1):1-17, February 2018. ISSN 2045-2322. doi: 10. 10.1038/441998-088-- 010---0111222--000 URL
https: www. na ure. com htt///atiiiclleeee wwt..ceee//1111--111111111 http://................././/00..
Daniel L K Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nat. Neurosci., 19(3):55-3665 March 2016. ISSN 1097-6256. doi: 10. 10.1038/mn. 4244.
4244
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local Aggregation for Unsupervised Learnig of Visual Embeddings. arXiv: /903. 12355 [cs], March 2019. URL http:://rxiv..rrt abs/ 12355
arXiv: 12355.
11

---

Under review as a conference paper at ICLR 2020
Epochs of training for convolutional network
20
40
60
Layer
Top- Precision
30
20
10-
Epochs of training for decoder
Figure 4: Top-5 precision as a function of training epochs of the top decoding layer for DeepCluster.
A APPENDIX: DETERMINING NUMBER OF TRAINING EPOCHS FOR THE OBJECT DECODER
Training the object decoders was the most computationally expensive part of this project, one was trained for every layer across many epochs and models. was therefore necessary to use as few training epochs possible. To evaluate how many were needed, trained decoders for epochs on features from a sample of convolutional training epochs (0, 20, 40, 60) and all layers steady increase in decoding performance up (and
(Fig. 4p. It was found that while there presumably beyond) the epochs, the relative performance across different layers, or epochs, was broadly captured by epoch For further analyses we therefore used 2 epochs of training for the decoding layer.
B APPENDIX: FURTHER MEASURES OF CHANGES IN OBJECT REPRESENTATION THROUGH LEARNING
Fig. [ showed the layerwise changes in top-5 precision through learning. Fig. 5 shows the corre- sponding changes in cross-entropy loss.
As DeepCluster learned more slowly than the supervised networks, we extended training 70 epochs (Fig. (6. It can be seen that it was continuing to learn, particularly in the higher layers, but the order of the layers did change within this range.
12

---

Under review as a conference paper at ICLR 2020
(a) CORnet
(b) DeepCluster
stogs
(c) AlexNet Supervised
10
ccoch
Figure 5: Cross-entropy loss during training for each of the three networks, illustrating the change in explicit representation of visual classes.
25
30-
20
30
60
50
60
70
20
30
40
60
Shap
2200
Figure 6: The unsupervised network, DeepCluster, learned more slowly than the supervised net- works (Fig. 55. To test if the layers with the strongest explicit object representation changed over longer period of extended learning, we trained the convolutional layers to 70 epochs.
13


Reviews:
[{"review_id": "HkgYR6zaFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper aims to understand how object vision develops in infancy and childhood by using deep learning models. In particular, it chooses two deep nets, CORnet and DeepCluster to measure learning. CORnet is supervised and is designed to mimic the architecture and representational geometry of the visual system. It tries to quantify the development of explicit object representations at each level of this network through training by freezing the convolutional layers and training an additional linear decoding layer. The paper evaluates the decoding accuracy on the whole ImageNet validation set for individual visual classes. DeepCluster differ in both supervision and in the convolutional networks. To isolate the effect of supervision, it ran a control experiment in which the convolutional network from DeepCluster (an AlexNet variant) is trained in a supervised manner. The paper tries to draw conclusions on how learning should develop across brain regions in infants. In all the networks, it also tested for a relationship in the order in which infants and machines acquire visual classes, and found only evidence for a counter-intuitive relationship. \n\nLimitations\n\nThe topic is extremely interesting and worth intense study. However, the approach is not convincing. CORnet may have some relevances. It is not clear how well it models the representational geometry of the visual system. It is even less clear whether DeepCluster is relevant. Why would it be related to infant learning? \n\nThe whole idea of using DNN to infer biological learning is built on shaky ground given how little we know the learning mechanism of the brain. In particular, back propagation is not widely considered possible in biology. Given the learning mechanism may be very different. What is the basis of using DNN to study the infant learning?\n\nThe findings are also not very surprising and offer much for the community.\n\nGiven the paper lacks rigor and findings, it does not meet the bar of ICLR.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1emeU9CtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to model the development of the human visual system in infants, by training deep neural network architectures inspired by the human visual system on images from ImageNet, and learning a linear decoder on the outputs of each layer (following Zhang et al 2017) to measure how much information useful for distinguishing between classes is contained within each layer in the architecture. The paper measures the amount of class information in each layer over the progress of training.\n\nI agree that deep networks could serve as good models for various parts of the brain, including the visual system especially given that convolutional networks have been inspired from studies of the visual system. However, the paper doesn't seem to provide any evidence for how the training process used for deep neural networks should correspond to the development of the visual system in infants. In particular, backpropagation is considered biologically implausible [1], whereas backpropagation serves as the main method for learning in the neural networks. Furthermore, neural networks have randomly initialized parameters, whereas it seems unlikely that human infants' brains would lack existing organization to such a drastic extent. In order for the results in this paper to hold greater weight, I would expect to see more evidence about how the neural network training process (also including aspects such as batch normalization, and the self-supervised clustering method in DeepCluster) are expected to correlate with learning in human brains.\n\nFor the above reasons, I vote to reject the paper. My conclusions above are based on my surface-level knowledge of neuroscience, so I welcome any clarifications or corrections from the authors about the above points.\n\n[1] https://arxiv.org/abs/1502.04156", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJlwdkiTtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to examine whether DNNs are a good model of infant\nbehavior and development.  The paper is very well written and easy to\nread.\n\nThe goal was to compare the development of object representations\nacross layers with the development in children and to compare the\norder of learning of different object classes.\n\nThe work did show that unsupervised training results in a different\npattern of layer learning than supervised learning, but neither form\nof learning was able to model the development in children.  Perhaps a\nself-supervised multimodal learning system should be tried?\n\n\nThe decision to train the DeepCluster type net in a supervised way for\na control on training method vs architecture type is nice, but it would also\nhave been good to try other kinds of networks.\n\n\nIt is not clear that age of acquisition of the verbal word should be\nrelated to age of acquisition of the visual concept.  The author's\nstate \"A number of linguistic factors are known to affect when words\nare first used, including the frequency of the word in language and\nits number of phonemes, but the second strongest factor is the\n\"concreteness\" of the word (Braginsky et al., 2015). This suggests\nthat the strength of the visual representation of a class has an\neffect on when its label is acquired.\"   It is not clear to me how the concreteness\nof a word relates to the strength of visual representation.\n\nI don't think there is enough new insight gained from this paper for ICLR publication\nat this stage.\n\nMinor comments:\n\nWhat are dashed lines in Figure 2 top left box?\n\nFunding acknowledgement (especially with grant number) should not be\nin an anonymous submission.\n\nmagenetic\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the properties of deep neural networks as they learn, and how they may relate to human visual learning (e.g. how learning develops across regions of the infant brain). The paper received three reviews, all of which recommended Weak Reject. The reviewers generally felt the topic of the paper was very interesting, but overall felt that the insights that the paper revealed were relatively modest, and had concerns about the connections between DNN and human learning (e.g., the extent to which DNNs are biologically plausible -- including back propagation, batch normalization, random initialization, etc. -- and whether this matters for the conclusions of the present study). In response to comments, the authors undertook a significant revision to try to address these points of confusion. However, the reviewers were still skeptical and chose to keep their Weak Reject scores.

The AC agrees with reviewers that investigations of the similarity -- or not! -- between infant and deep neural networks is extremely interesting and, as the authors acknowledge, is a high risk but potentially very high reward research direction. However, in light of the reviews with unanimous Weak Reject decisions, the AC is not able to recommend acceptance at this time. I strongly encourage authors to continue this work and submit to another venue; this would seem to be a perfect match for CogSci conference, for example. We hope the reviews below help authors to improve their manuscript for this next submission.
Decision: Reject