Layerwise Learning Rates for Object Features in Unsupervised and Supervised Neural Networks And Consequent Predictions for the Infant Visual System

1 INTRODUCTION . 1.1 THE NEED FOR MODELS OF DEVELOPMENT IN INFANTS . Humans are helpless for a long time during infancy when compared with most other species . During this helpless period , the brain and mind develop as a result of genetically programmed maturation in concert with experience from the environment . One way to measure development during the helpless period is by examining infants ’ capabilities through behaviour . However , despite great ingenuity in experimental design , due to the limited behavioural repertoire of infants , it has proven difficult to obtain more than a basic understanding of how the brain and mind are developing . Furthermore , some brain systems may undergo extended development prior to any manifestation in behaviour ( Tolman & Honzik , 1930 ) . Therefore , in addition to measuring behaviour a new and promising complementary tool is magnetic resonance imaging ( MRI ) , which can measure the structure , wiring and function of the infant brain . To understand why infants ’ brains and minds change in the way they do , we need to create models of the mechanisms of development . This is critical if we are to understand how development is disrupted in infants with brain injury . Infants that are born very preterm , or that have potentially negative event during birth are admitted to the neonatal intensive care unit ( NICU ) . At present , a significant proportion of NICU infants develop cognitive , behavioural or social impairments later in life . To detect these impairments earlier , and to design effective targeted interventions , we need to create models of brain and mind development in young infants . As it is difficult to intuitively imagine what the mind of a developing infant is like , it is necessary to create computational models of infant learning that can be tested against real data . 1.2 DNNS AS A MODEL FOR OBJECT VISION IN ADULTS . A promising first brain system for which computational models of infant learning could be built is the object recognition system in the ventral visual stream . This system is relatively well understood in adults and there are proven neuroimaging methods that can be used to probe its representations and to test the hypotheses that will be generated by our computational models in due course . Indeed , in adults , there is substantial work on modelling the ventral visual stream . A key discovery from this work has been that the most straightforward strategy , of first collecting neural data and then creating models to fit it , has not been very effective ( Yamins & DiCarlo , 2016 ) . The issue is that it is only feasible to acquire a relatively small quantity of neuronal data , but the computational models capable of performing object vision have many millions of parameters . A more effective strategy , therefore , is to instead use computational models that are optimised to perform object vision to high accuracy , and then to use these as models of neural activity ( Yamins & DiCarlo , 2016 ) . Using this strategy , it has been shown that DNNs designed for the ImageNet Large Scale Visual Recognition Challenge ( ILSVRC ) can predict patterns of neural activity in the ventral visual stream , as measured with functional magnetic resonance imaging ( fMRI ) ( Khaligh-Razavi & Kriegeskorte , 2014 ; Güçlü & van Gerven ; Wen et al. , 2018 ) , electroencephalography ( EEG ) and behavioural studies ( Cichy et al. , 2016 ) . It is important to note that the claim in these studies is not that there is a one-to-one mapping between artificial neurons and biological neurons . Rather , it is that some of the macroscopic aspects of the activity patterns in the DNNs and the brain are similar . For example , the ventral visual stream forms a hierarchy , with visual input from the eyes being passed forwards through a number of brain structures , and gradually being transformed from a representation of the visual input into a representation of semantic category . A quantitatively similar hierarchy of representations is found through the layers of the DNN ( Güçlü & van Gerven ; Wen et al. , 2018 ) . 1.3 INFANTS ARE LEARNING , DO THEY LEARN LIKE DNNS ? . Some aspects of biological object vision are innate , such as a preference for looking at faces , which is apparent in the first hour after birth ( Johnson et al. , 1991 ) . However , much must be learned through visual experience , as the genetic code alone is limited in capacity ( estimated to be 150-750 MB1 ) and would be substantially exhausted by the parameters of deep learning models ( 50-150 MB ( Bianco et al. , 2018 ) ) . Even if sufficient storage were available , many objects relevant to infants today ( e.g. , baby bottle , or a Pokémon ( Janini & Konkle , 2019 ) ) were invented too recently for innate codes to have evolved for them . A great deal of object vision knowledge , therefore , is learned . There is evidence that much of this learning is already happening in early infancy . Infants aged 3-4 months old can identify statistical regularities in sequences of pictures ( Quinn et al. , 1993 ; French et al. , 2004 ) . And , by 6 months , infants start to look at a visual class corresponding to a concurrently spoken label ( Bergelson & Swingley , 2012 ) 2 . In adults , fMRI of the ventral visual stream has found that there are regions that are selective for particular visual classes , such as faces , body parts or places ( Kanwisher et al. , 1997 ; Epstein & Kanwisher , 1998 ; Downing et al. , 2001 ) . In 4-6 month old infants , at least partial selectivity is already present in the ventral visual stream ( Deen et al. , 2017 ) , although it continues to develop for many years ( Gomez et al. , 2017 ) . 1https : //www.quora.com/How-many-bytes-memory-size-is-a-humans-DNA 2although vocabulary remains very limited until after the first birthday , when it typically begins to grow rapidly . DNNs have provided a useful model of adult object vision . As these DNNs learn from visual `` experience '' they are therefore candidate models of infant learning . We do not expect them to capture the precise details of the learning process as , for example , the learning rule of back propagation is not biologically plausible ( although biologically plausible mechanisms can approximate it ( Richards et al. , 2019 ) ) . Rather , our overarching hypothesis is that DNNs might capture some of the broad macro-scale characteristics of learning . 1.4 AIMS OF THE CURRENT STUDY . In this work , we characterise two macro-scale properties of a number of computational models , with the aim of generating testable measurements for a future large-scale neuroimaging study . 1 . Should we expect representations in brain regions of the visual hierarchy to develop simultaneously or asynchronously ? An established principle of infant development is that brain regions underlying simpler functions develop first , and are followed by those underlying more complex functions ( Charles A. Nelson in Shonkoff & Phillips ( 2000 ) ) 3 . To provide an initial prediction of how this might happen in the visual hierarchy we examined how representations developed in different layers of DNNs during training . 2 . Are visual classes that are learned earlier by infants also learned earlier by DNNs ? In infants , the acquisition of visual classes can be estimated by the onset of the receptive or expressive use of the words for the classes . It has been found that , when measured this way , some visual classes are learned before others , e.g. , body parts and vehicles precede food and clothing ( Braginsky et al. , 2015 ) . The link between the acquisition of words and brain representations in the ventral visual stream is supported by fMRI results examining the visual representations of mental imagery have shown that object classes can be decoded from brain representations when participants are tasked with reading a concrete noun ( Anderson et al. , 2015 ; Kellenbach et al. , 2000 ) . We ask whether some of this ordering is attributable to visual complexity , as reflected in the DNN classification performance . 2 METHODS . 2.1 CHOICE OF NETWORKS AND TRAINING . 2.1.1 CORNET-S WITH SUPERVISED TRAINING . To date , studies which have shown a parallel between DNNs and the adult brain have used networks trained in a supervised manner ( Khaligh-Razavi & Kriegeskorte , 2014 ; Güçlü & van Gerven ; Cichy et al. , 2016 ; Wen et al. , 2018 ; Jozwik et al. , 2018 ) . To remain consistent with these studies , we started with a supervised network . Specifically , we used CORnet-S ( Kubilius et al. , 2018 ) , as this was especially designed to meet the Brain-Score benchmark ( Schrimpf et al. , 2018 ) by capturing the architectural principles and showing strong predictivity of neural data from the ventral visual stream , while still achieving good classification performance . It has four layers mapping onto regions in the ventral visual stream ( V1 , V2 , V4 and IT ) . 2.1.2 DEEPCLUSTER WITH UNSUPERVISED TRAINING . Infants ’ access to labels is extremely impoverished , making supervised learning an unlikely training curriculum for infant learning . In order to develop computational models that capture the dynamics of infant learning it is necessary to evaluate unsupervised training . One current state-of-the-art unsupervised strategy for learning visual features for object recognition is DeepCluster ( Caron et al. , 2018 ) , which adjusts convolutional weights to create clusters of images that yield similar patterns of activation . This has a parallel with behavioural results of infant learning , in that they are known to create clusters of similar images , and to be sensitive to deviants from that cluster ( Quinn et al. , 1993 ; French et al. , 2004 ) . DeepCluster uses a simple but elegant technique for self-supervised learning , in which the start of an epoch sees each image ( from ImageNet ) being passed forward through a convolutional network , and the resulting output activations are then clustered across all of 3This influential work has been cited more than 7000 times . the images using k-means . These clusters are assigned labels , which are subsequently used to learn the weights of the convolutional network with stochastic gradient descent on batches of images in the typical way . As in Caron et al . ( 2018 ) we used 10,000 clusters . We used AlexNet ( Krizhevsky et al. , 2012 ) as the convolutional network , containing five layers each with a single convolutional layer . These layers were modified as in Caron et al . ( 2018 ) with the local response normalisation layers removed and batch normalisation used instead ( Ioffe & Szegedy , 2015 ) . Also like Caron et al . ( 2018 ) , a fixed linear transformation based on Sobel filters was used on the input to remove colour and increase local contrast . 2.1.3 ALEXNET WITH SUPERVISED TRAINING . CORnet was trained in a supervised way , and DeepCluster in an unsupervised way . Any difference in the results might be due to this difference in supervision . However , the convolutional networks at the heart of these two networks also differ , which could cause further differences . To control for this , we therefore repeated the experiments using the same AlexNet variant as in DeepCluster , but trained in a supervised way .

Summary: The paper aims to understand how object vision develops in infancy and childhood by using deep learning models. In particular, it chooses two deep nets, CORnet and DeepCluster to measure learning. CORnet is supervised and is designed to mimic the architecture and representational geometry of the visual system. It tries to quantify the development of explicit object representations at each level of this network through training by freezing the convolutional layers and training an additional linear decoding layer. The paper evaluates the decoding accuracy on the whole ImageNet validation set for individual visual classes. DeepCluster differ in both supervision and in the convolutional networks. To isolate the effect of supervision, it ran a control experiment in which the convolutional network from DeepCluster (an AlexNet variant) is trained in a supervised manner. The paper tries to draw conclusions on how learning should develop across brain regions in infants. In all the networks, it also tested for a relationship in the order in which infants and machines acquire visual classes, and found only evidence for a counter-intuitive relationship.