Curriculum Learning With Infant Egocentric Videos
Authors: Saber Sheybani
Himanshu Hansaria
Justin Newell Wood
Linda B. Smith
Zoran Tiganj
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Curriculum learning
Self-supervised learning
Slow changes
Infant development
TL;DR: We show the advantage of training self-supervised learning models of infant egocentric videos using the natural developmental curriculum.

Abstract: Infants possess a remarkable ability to rapidly learn and process visual inputs. As an infant's mobility increases, so does the variety and dynamics of their visual inputs. Is this change in the properties of the visual inputs beneficial or even critical for the proper development of the visual system? To address this question, we used video recordings from infants wearing head-mounted cameras to train a variety of self-supervised learning models. Critically, we separated the infant data by age group and evaluated the importance of training with a curriculum aligned with developmental order. We found that initiating learning with the data from the youngest age group provided the strongest learning signal and led to the best learning outcomes in terms of downstream task performance. We then showed that the benefits of the data from the youngest age group are due to the slowness and simplicity of the visual experience. The results provide strong empirical evidence for the importance of the properties of the early infant experience and developmental progression in training. More broadly, our approach and findings take a noteworthy step towards reverse engineering the learning mechanisms in newborn brains using image-computable models from artificial intelligence.

Curriculum Learning with Infant Egocentric Videos
Saber Sheybani
Himanshu Hansaria
Department of Intelligent Systems Engineering Indiana University Bloomington sheybani @iu. edu
Department of Computer Science Indiana University Bloomington hhansarQiu edu
Justin N. Wood Department of Informatics Indiana University Bloomington woodjnéiug podu.nii... edu
Linda B. Smith
Department of Psychological and Brain Sciences Indiana University Bloomington smi mmithh@iu..c edu
Zoran Tiganj
Department of Computer Science Indiana University Bloomington ztiganj @iu.ieiuuee.eeeee edu
Abstract
Infants possess a remarkable ability to rapidly learn and process visual inputs. an infant's mobility increases, so does the variety and dynamics of their visual inputs. Is this change in the properties of the visual inputs beneficial or even critical for the proper development of the visual system? To address this question, we used video recordings from infants wearing head-mounted cameras to train a variety of self-supervised learning models. Critically, we separated the infant data by age group and evaluated the importance of training with curriculum aligned with developmental order. We found that initiating learning with the data from the youngest age group provided the strongest learning signal and led to the best learning outcomes in terms of downstream task performance. We then showed that the benefits of the data from the youngest age group are due to the slowness and simplicity of the visual experience. The results provide strong empirical evidence for the importance of the properties of the early infant experience and developmental progression training. More broadly, our approach and findings take noteworthy step towards reverse engineering the learning mechanisms in newborn brains using image-computable models from artificial intelligence.
Introduction
common, implicit assumption in the machine-learning literature that the massive scale of daily life, the world presents the same visual statistics to all perceivers. Many researchers therefore believe that it will be possible to mimic biological learning through incremental statistical learning performed over large datasets. However, this assumption may be incorrect because the input received by the visual system depends on the momentary spatial relation of the eyes the world and is thus dependent on the behavior and goals of the perceiver. Over the first year of life. infants' motor abilities, behaviors, and interests change substantially [Lee and Galloway 2012 Largo and Kakebeeke 1994 producing gradually changing datasets for learning. A growing literature on infant egocentric vision also indicates that the semantic content of visual information (faces, hands, objects, scenes) changes systematically with development over the first year post birth product of changing motor skills and interests [Smith et al., 20118] Jayaraman et al., 2017 Kretch et al. 20144] Jayaraman et al. 2015). Experimental studies provide converging evidence of systematic changes in the types of semantic content that infants prefer and choose to look at [Spelke] 1985 Frank et al. 2014 2009
Kwon
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

et al., 2016]. In brief, the human visual system develops in context of an ordered set of visual experiences, curriculum. Does this curriculum matter? Would a randomly ordered aggregation of all the data be just as good? From machine-learning perspective, are there benefits to "developmentally structured" datasets?
The study reported here is motivated by three goals: (1) to determine if the natural order of the developmental curriculum has advantages for machine learning; (2) to determine properties of the ordered data sets that lead to this advantage; and (3) to explore the implications of this self-generated curriculum for building more robust artificial visual systems that can learn continuously from self- supervised training signals. We used the Homeview datase 3 of egocentric videos recorded by head-mounted cameras worn by infants from 2-12 months of age as they went about their daily lives at home and without experimenters present. This dataset provides unique insight into infant visual experience at different stages of development. The entire data set contains videos captured by very young infants with naturally limited postural control (2 month old) to older infants (12 month olds) who crawl or walk and instrumentally use objects. To evaluate the impact of these changing experiences
visual learning, we performed computational experiments by selecting and ordering training videos from the corpus collected by infants at various ages. These videos were then used to train variety of tfateof-thhear self-supervised learning (SSL) models. We found that one key developing property in the input was the duration of self- self-similar inputs, with the youngest infant experiences characterized by very slow change. This finding underscores the integral role of curriculum learning in infant development and provides insights relevant to the continued evolution of self- superssuee learning systems.
2 Related Work
2.1
Learning from child egocentric videos
There is increased recognition of the importance of the structure of the training data in AI and use of egocentric infant and child images and videos as inputs to deep neural networks [Bambach et al.,
2017 2016 2018 Orhan et al. 2020 Zhuang et al. 2021] One key finding is that these egocentric training data are capable of furnishing powerful visual representations [Orhan al. 2020 Orhan and
Lake 2023 and can lead to superior learning over adult egocentric videos |Bambach et al. 2018).
In BBambach et al., 2018), the authors fine-tuned deep convolutional neural network (CNN) using egocentric images recorded by toddlers and adults who were in the same environment during data collection and found that images recorded by toddlers led to better subsequent classification of objects. Analyses of the differences in the training set suggested two statistical properties in the infant corpus that may be critical to learning: (1) the distribution of similarities among images of the same object was extremely skewed such that most images were variations of each other, but the overall range (the long tail) of variability in the infant images was much greater than that in the adult images; and (2) individual objects were much larger and foregrounded in the images and thus easier to segment from the background than were the same objects in the adult images. This result emerges because active toddlers visually attend to objects by decreasing the distance between their sensors (eyes) and the attended object.
In other related work [Orhan et al., 2020 Orhan and Lake 2023), the authors trained self-supervised learning models from scratch the SAYCam dataset [Sullivan et al. 2021]. an egocentric video dataset captured by children between 6-32 months old. They showed that deep SSL models can achieve powerful visual representations and reasonable performance while solely trained on child- egocentric videos. [Orhan et al. 20200 used a temporal classification objective, which can be regarded an implementation of the principle of temporal slowness [Wiskott and Sejnowski] [002] by creating similar embeddings for the frames that were close in time and dissimilar embeddings for the frames that were far apart in time. To provide more robust baseline, [Orhan and Lake 2023 used wider variety of SSL algorithms and architectures and found performance rates as high as 70% of a high-performing ImageNet-trainee model. Furthermore, [Zhuang al. 2021 showed that CNNs trained using contrastive learning on child egocentric videos can produce brain-like like neural responses.
Although the previous work clearly shows that the structure in data sets matters and demonstrates the potential value of taking inspiration from human development, the data sets do not capture or
2

---

exploit the ordering of experiences that is the hallmark of human developmental processes. In the prior cases, the datasets were created by aggregating training videos across large age spans (6 to 32 months), making it unclear what ages or aspects of the training data were responsible for the learning outcomes. In humans, the first year post-birth is a period of substantial developmental change in the cortical visual system, in visual sensitivities, and in object recognition (Jayaamann et al., 2015 Fausey (ct al., 2016). Further, the earliest period- - the first to 4 months constitutes a sensitive period in which disruptions in the expected visual input lead to lifelong disruptions in object recognition and other visual processes [Maurer] 2017].
Accordingly, we take an explicitly developmental approach (Smith et al. 2018 and show that doing so has a profound impact on self-supervised learning. Using a corpus of images from infants 2 to 12 months of age, we show that training that places the natural earliest visual input first optimizes learning. Training with data out of the natural developmental order disrupts learning. To demonstrate the generality of this ordering effect, we used self-superrvsed pre-training across a variety of algorithms and architectures; in all cases, a developmentally ordered visual diet led to the strongest learning outcomes.
2.2 Curriculum Learning in AI
In curriculum learning, models undergo training by initially being exposed to simpler examples and progressing to more difficult ones [Elman] 1993 Bengio et al. 22099] Wu et al. 2020). The approach, common in education, arranges information in an orderly manner to enhance learning (Smith et al. 2018 Avrahami et al.) 1997].
The curriculum contributes to learning by shaping the loss landscape. A good curriculum will result in a smoother learning trajectory. A substantial body of literature in Al focuses on designing ways to order the training samples based on some definition of sample difficulty to allow the models to converge faster with better downstream performance [Wu et al. 2020, Khan et al., 2023 Wang et al.
2019 Guo et al. 2018 Liu et al. 2017]. Unlike human and animal learning [Wilson et al. [019 the extent of the impact of curricula on standard, offline, supervised learning is debated [Wu et al. 2020
Saglietti al. 2022]. However, recent work shows remarkable impact from curricula in online,
seff-supervised learning settings (Saglietti et al. 2022 Joshi and Mirzasoleiman 2023
In this work, we show the impact of the infant visual curriculum, even when using common, offline, self-supervised learning methods. This curriculum not based on an a priori definition of sample difficulty, but rather assumes the natural ordering achieved through human development, an ordering that leads to the full prowess of human object recognition (Smith and Slone 2017 Zaadnoordijk et al. 2022]
2.3
Computational neuroscience work on learning visual features from naturalistic stimuli
Early seminal work by [Olshausen and Field 1996 demonstrated that sparse coding algorithms, when trained on natural images, produced receptive fields similar to those found in the primary visual cortex, suggesting a potential computational principle underlying visual learning in the brain. Similarly, work by (Bell and Sejnowski, 1997) showed how independent component analysis could be used to derive Gabor-like filters, again resembling the response properties of neurons in the visual cortex. More recent work (Karklin and Lewicki, (209) has built on these principles, demonstrating how more complex visual features, like edge co-occurrence statisties, can also be learned from naturalistic stimuli. These studies collectively highlight the importance of naturalistic stimuli in shaping our understanding of the computational principles underlying visual learning in the brain.
In this study, for the first time, we explore the learning of visual features from the embodied data streams produced by infants at different stages of development, including infants younger than six months of age.
Datasets
3.1 Training Dataset
The datasets were created from a larger corpus of 500 hours of head camera videos collected by 101 infants from 2 to 24 months of age (Fausey et al. 2016). From this corpus, we selected videos
3

---

A
2 2-4rno
Objective Functions
Predictier:
SSL Architectures
Contracthe: SinCLii (Reshec)
8lo
training Curricula
10-12mo
Developrne real
And hntt-Bewwippmemma
C
Non-egooentric Action Recogpition (UCF101)
Evalustion
Action Recognition (eemehingomehhngg2)
Stape
Domain-rulated Recognition
(ToyBox)
Evaluatton
Evatuation
Figure Methods overview. A) Sample image sequences from each infant age group in the training dataset. B) Training procedure: Each model is trained in three stages using self-supervised objective function. C) Evaluation: After training, the learned representations are evaluated for their informativeness in three downstream tasks.
collected by infants at 3 age periods who differ in motor skills, visual preferences, and behaviors and, as a consequence, in their statistics of experience. The youngest infants were 2-4 months old (n = 10, total of 41 hours, mean age 2.9 mo, SD 0.75), period characterized by limited postural control; the middle group of infants were 6-8 months (n = 10, total of 42 hours, mean age = 6.9 mo, SD 0.62) and were mastering sitting and crawling. and the instrumental group of infants, 10-12 months (n 9, total of 40 hours, mean age 11.1 mo, SD 0.65), were beginning walk, could sit stably and with increasing skill manipulating objects. In the of the manuscript, we refer to these three age groups as g0, gl, and g2, respectively. See Supplemental Information, Table S3 for recording information per participant.
of objects; and the oldest
We used two additional training sets: (1) Egocentric images collected by adults in the home (18 to 71 years of age, 25 hours of recording) using the same camera, here referred to adult, and (2) random age infant dataset, referred to as rand, constructed by combining random set of hours of recordings from each infant in the three age groups g0, gl, and g2.
video files were subsampled to RGB images at 30 frames per second. Pretraining on a given dataset started by selecting 1.45 million consecutive frames from the given age group (upon concatenating the frames from all subjects within the group). From the created sets of frames, we collected sample sequences of size 16 as inputs for the VidcoMAE model (81,000 samples for the training set and 9,000 samples for the validation set). For JEPA-TT and SimCLR-TT, we used pairs of consecutive frames that are 10-30 seconds apart. The number of samples was kept fixed for all age groups and conditions. allow the data-related variability to be reflected the results, each dataset was split into 3 folds, and each pretraining run performed using different fold.
3.2 Benchmark Datasets
Soomro et al. 2012 is widely used dataset for video- based action recognition research. The dataset consists 13,300 videos of realistic actions collected from YouTube. These videos are divided into 101 action categories, such as playing flute, biking, and horseback riding. Each category contains between 100 to 200 video clips, which are approximately 7 seconds long on average. The wide variety of actions and the realistic setting make the UCF101 dataset challenging benchmark for evaluating the performance video understanding algorithms. This dataset is commonly used for evaluating the quality of spatiotemporal features in vidco models Tran et al. 2015
UCF101
Xie et al., 2017

---

Seneething-Smmetiing V2 (SSv2) Goyal et al. 2017 is an egocentric video dataset of short videos comprising humans performing pre-dehned, basic action with everyday objects. Being an egocentric dataset, SSv2 is more appropriate than UCF101 for evaluating our infant-egocentrie models. In addition, SSv2 emphasizes the importance of dynamic features more than UCF101, which is known to be solvable by only attending to static features of individual frames. Here we evaluate our models on the 10-class subset of the dataset (as suggested in the dataset article).
ToyBox [Wang et al. 2018 is a small but developmentally relevant dataset of 4320 videos, 2-20 seconds each. In each video, object undergoes spatial transformation, such rotation along the x-axis or translation along the z-axis (12 transformations total). There are 12 toy object categories (4 from animals, from houschold, from vehicles), each containing 30 distinct objects. The existence of both object categories and spatial transformations allows us to probe the representations learned by the models terms of both shape-relevant and location/pose-relevee relevant information.
4 Models and Objectives
Evaluating the impact of a developmental visual curriculum requires (1) model with a self-supervised learning objective [Schiappa et al., [222] that can scale up to large corpus of real-world egocentric videos and (2) an architecture that directly learns regularities in videos across both space and time:
spatiotemporal learner. Generative and predictive tasks such as masked modeling (e.g. VideoMAE [Tong et al. 2022] or MaskFeat (Wei et al. (0022) are particularly suitable for this purpose, although other methods based on cross-modal agreement or contrastive learning can be used as well.
4.1 Generative Vision Transformer (MAE)
Transformer-basee -based masked autoencoders are effective and highly scalable methods of representation learning with minimal inductive biases. They are applicable to various sensory modalities, including both images and videos, and are therefore great candidates for a general spatiotemporal learner [Feichtenhoter et al. 2022]. Here we chose VideoMAE as simple adaptation of the masked modeling method for videos. In this learning method, each sample video broken down into sequence of spatiotemporal tubes. During pretraining, a portion of the tokens are randomly chosen and masked, and the pretraining objective is recover the masked tokens. This is achieved using a series of multi-head attention layers [Dosovitskiy et al. 2020) which need to learn the relationship between the tokens in order to successfully predict the masked tokens. From an intuitive perspective, each token can be thought of as a simple dynamic feature (e.g., akin to motion energy features) corresponding to a small area in the visual field; the entire video is the collection of many such features. The mask prediction objective promotes the network to learn the spatiotemporal organization of simple features transforming into complex moving features that occur in the dataset videos.
4.2 Predictive Vision Transformer (JEPA-TT)
I-JEPA [Assran et al., 2023) a recently proposed image-based SSL. algorithm in which the objective is to predict the embedding of one part of the input image (target) using the embedding of another part (context). Therefore unlike masked autoencoders, this objective function does not require pixel level reconstruction. Our JEPA Through Time (JEPA-TT) algorithm is adaptation of I-JEPA, where the context and the target are selected from consecutive video frames rather than the same frame.
4.3 Contrastive CNN (SimCLR-TT)
SimCLR [Chen al. 2020 is well-known image-based SSL algorithm. It works by bringing the embeddings of two augmented versions of an image (positive pair) closer in the embedding space while pushing them away from embeddings randomly selected images from the dataset. Similar to Aubret et al. [022]. our SimCLR-TT algorithm treats consecutive frames augmentations of each other. To test additional network architectures, we use convolutional backbone, ResNet-18 18, for
SimCLR-TT.
5

---

5 Evaluation Criteria
5.1 Downstream Performance
After pretraining each model, the encoder part of the model was evaluated in several downstream tasks using both linear classification and k neerrestneiihhor (kNN) classification. We first collected the encoder responses to all stimuli in the downstream video dataset as feature vectors. For linear classification, we evaluated the usefulness of the features by training and validating linear SGD) classifier. For kNN classification (results shown in the Supplemental Information), for each sample embedding in the validation set, first found the k nearest samples in the training set. We then evaluated whether the class of any of the k nearest neighbors matched the true class of the validation sample, All results in the pretraining and downstream evaluation were generated with seeds per curriculum condition.
5.2 Gradient Norm
In order provide glimpse into the learning process that gives rise the downstream performance results, we report the Stochastic Gradient Descent (SGD) gradient norms across the training stages. We compute gradient norm of a given model layer botchaise Σ Lose ||2 where θ contains the 8e learnable parameters of the layer of interest and the sum goes over all the samples in the batch. Learning in each parameter driven by the magnitude and direction of the noisy, empirical gradient estimate after being averaged across batch samples. This quantity has been used for studying the learning dynamies of SGD [Hochreiter] 1998 SSwartz-ZZiv and Tishby 2017]. Gradient norms can vary as a function of the architecture, optimizer, and objective function and hence, should be interpreted carefully. Here we evaluated the effect of the order of the visual diet on the gradient norm by matching all other system components. We also reproduced the results in various architectures and objective functions.
Results
6.1 Stage 1: Visual inputs from the youngest infants lead to stronger learning signals and better downstream classification
To characterize the impact of training with infant data from different stages of development, we first tracked the gradient norm of the VideoMAE model trained separately with the data from the youngest age group (g0), the oldest age group (g2), the random group (see Section (. and the adult group. Training with the data from the youngest age group provided the largest gradient norm the model (shown for the last encoder layer in Figure 2 panel A)
Next, we evaluated the performance of the VideoMAE pretrained in stage on downstream classification task. As described in Section 3.2 we used the Something-Somettee v2 (10 classes), UCF-101, and Toybox datasets. For Toybox, we compared pretrained models on spatial transformation classification. In all cases, the model pretrained on the data from the youngest age group performed substantially better than other models (Figure 2 panels B-D). These results confirm that models that learn from larger learning signals (gradient norms) during the pretraining stage extract more relevant features from video data. This was true across domains since object classification, spatial transformation classification, and video classification are substantially different tasks.
6.2 Stage 1-3: Developmental curriculum matters
We examined whether the developmental curriculum contributes to larger gradient norms and better downstream classification. After pretraining VideoMAE on stage I using different age groups, we continued the pretraining process with two more stages. In one case, we followed the developmental curriculum by training on data from 2-4 months old infants (g0) in stage 1, data from 6-8 month old infants (g1) in stage 2, and then data from the oldest, 10-12 month, age group (y22 stage Each stage had 5,000 iterations. For comparison, we also pretrained models in an aeti-devellpmenntl curriculum order by training on g2 in stage 1, gl in stage 2, and gO in stage 3. Similar to pretraining on just stage I, we also pretrained the network on the random group, but this time across all three

---

A
B
0.45
Encoder Gradlent Norm
0.40
0.35
0.04
Eveluation
(55v2)
0.30
[Last
0.25
0.20
0.15
Stage 1
Stage 2 Pretraining Stage
Stage
0.30
After Stage
After Stage 2
After Stage 3
Pretraining Stage
C
0.70
0.30
Linear Evaluation Score
Line Evoluation Scove
[TayBox Transformation
0.60
(UCF101)
0.40
0.15
0.30
0.10
0.05
Ei
0.10
After Stage
After Stage
After Stage 3
After Stage 1
After Stage
After Stage
Pretraining Stage
Pretraining Stage
Random
Figure 2: The effect of learning curriculum. A) The norm of the gradient at the last layer of the VideoMAE encoder
training iterations. To increase visibility, the lines have been smoothened by applying a moving average with window length of 100. The average line across 3 seeds is plotted. B-D) The linear evaluation classification tasks of NomethingSomethingVV2 10class UCF101, and ToyBox object transformation. Each black dot indicates the result from one seed. The performance of an untrained VideoMAE model is shown in gray.
of the pretrained VideoMAE models downstream video
stages. Finally, we pretrained on adult data in stage followed by gl and g2 in stage and stage 3, respectively.
As shown in Figure 2 training developmental order (blue bars) led to the highest performance in the subsequent classification tasks. Anti-developmental curriculum (orange bars) led to the lowest downstream performance. This is an important result since data from gO were used in both the developmental and the amti-developmental curricula. However, because g0 was used at the last stage of pretraining for the anti-developmental curriculum, the performance gain was less than when g0 was used in stage I. This underscores the importance of training data arriving in the correct developmental order.
6.3 Developmental advantage comes from the temporal slowness and simplicity of the views acquired by young infants
In this subsection, ask why training with young infant data leads better results. Previous work shows that the visual experience of young infants is substantially slower than that of older infants (Sheybani al. 2002). We hypothesize that the advantage of the developmental curriculum is due to this temporal slowness. A considerable literature also shows that young infants' immature visual system has low acuity [Lewis and Maurer 2005) and a result, laboratory studies, young infants prefer to look at arrays with high contrast and few edges [Kiorpes] 2016). that is scenes that are spatially simple. To test this hypothesis, we created simplicity simpliciry-mathhed
matched subsets of the data from the different age groups: To match temporal complexity, we created samples that consist of only one frame. To match spatial complexity, we selected subset frames from each age group where around 6% of frame pixels were part of an edge, computed using the Canny edge detector. We then repeated our computational experiment in the last section, using the newly created version the developmental and anti-developmental curricula (Figure
7

---

As predicted, reducing the spatial complexity (Matched Spatial and Matched Spatiotemporal) as well as training on single frames (Matched Temporal and Matched Spatiotemporal) increased the gradient norm in both curricula (Figure panel A). This increase in the gradient norms is echoed in the downstream performance of the models within the same stage and curriculum. Comparing the gap between the developmental and anti-developmental curricula, we find both metrics of gradient norm and downstream performance largely reduce as we match the spatial complexity, temporal complexity, both. The aforementioned observations confirm the advantage of early temporal and spatial simplicity in the self-supervised learning models. We finally note that the models trained single frames had diminished performance SSv2, demonstrating the value of the motion information across multiple frames (Figure panel B).
A
B
Encoder Gradient Norm
0.50
0:45
Evaluation
(SSx2
0:43
0.04
0:25
0.02
0.35
0.001
Stage
Stage Protraining Stage
Stage
0.30
Stage Protraining Stage
Stage
C
D
Lintar Evallstion Scora
0:41
9:40
[UCF101)
Eva pollustion
[ToyBoxk
0:10
0.35
0.10
After Stage 1
After Stage Pretraining Stage
After Stage
After Stage
After Stage Protraining Stage
After Stage
Matched Tempprra
Spottol
Matched Spaliotemporall
The effect of the learning curriculum before and after controlling for spatial and temporal complexity. The curricula are indicated by color hue and the conditions are indicated by lightness. A) The norm of the gradient the last layer of the VideoMAE encoder across training iterations. To increase visibility, the lines have been smoothened by applying moving average with a window length of 100. The average line across seeds is plotted. B-D) the linear evaluation score of pretrained VideoMAE models on downstream video classification tasks of SomethingSomething V2- 10elass, UCF101, and ToyBox object classification. The performance of untrained VideoMAE model is shown in gray.
Figure 3:
Curriculum effects hold for other learning objectives
In this subsection, we asked whether the curriculum effect is limited to the generative learning objective or the vision transformer architecture. Several computer vision algorithms have been considered by the computational neuroscience community as potential algorithmic hypotheses for human perceptual learning, with no consensus as yet. We previously focused on VideoMAE due to its direct interaction with the spatial and temporal complexity of the input, without potential difficulties of predictive algorithms, such as representation collapse. Here we reproduce our main result using two more self-supervise learning algorithms: JEPA-TT with vision transformer backbone and SimCLR-TT with convolutional backbone (see the Methods section for implementation details). Both of these algorithms take single images inputs and, therefore, only indirectly interact with the temporal continuity of the input. However, they can still benefit from the spatial simplicity of young infant views and may also indirectly benefit from temporal slowness because of the similarity of the image pairs that are close in time.
Figure 4 shows that the advantage of the developmental curriculum holds for both JEPA-TT and SimCLR-TT, We observe that the effect size smaller than that of VideoMAE, which may be due to the indirect interaction of these algorithms with the spatiotemporal complexity of their data stream.

---

A
JEPA-TT (ViT-Base)
SimCLRIT (ResNet18)
Encoder Gradient Norm
Encoder Gradlent Norm
0.015
0.010
0.005
Stage
Stage Pretraining Stage
Stage 3
Stage
Stage 2
Stage 3
C
D
Pretraining Stage
Lineer Evaluation Score
0.40
Linear Evalustion Score
0.40
0.35
[Toxbax
[Toy Boa
0.30
0.25
0.20
After Stage
After Stage 2 Protraining Stage
After Stage
After Stage
After Stage Protraining Stage
After Stage
E
0.45
Linear Exelicaioon Score
Linear Exal Cvkkation Score
(CHARR3))
0.35
0.35
0.30
0.30
0.25
0.20
After Stage
After Stage
After Stage
After Stage
After Stage 2
After Stage
Pretraining
Pretraining Stage
Dev. Adev.
Figure 4: The effect of the learning curriculum on two CLTT learning systems (JEPA-TT w/ ViT-Base backbone), SimCLR-TT w/ ResNet18 backbone). The curricula are indicated by color hue, with blue for Developmental and orange for Anti- aatii-evelopmental. A,B) The gradient norm the last layer of the encoder. C-F) The linear evaluation score of the pretrained models on downstream image classification tasks. The performance of an untrained model is shown in gray.
6.5 Ablation Analysis
To determine the robustness of the results, we conducted several ablations studies. We showed that
the qualitative pattern of results did not change when we used different optimizer (Figure (S3), or different number of pretraining iterations (Figure(SS), Similar [Wu et al. 2020 Saglietti et al.. 2022], we observed that the importance of curriculum order decreases as the number of training iterations and epochs increases.
7 Discussion
Our findings make three contributions. First, they provide computationally explicit evidence that human infants generate high quality data for visual learning across the first year of development that changes in a developmentally ordered way in its underlying properties. Critically, we show that the order of the data matters. Deep neural networks trained developmentally ordered datasets outperformed those trained on "developmentally incorrect" datasets. Second, the findings indicate that the properties of the initial training set are particularly critical. This result accords with studies showing that the first few months after birth constitute a critical period for human visual development, with disruptions in the expected input leading lifelong consequences for vision and visual cognition [Maurer] 2017). The relative importance of the earliest training set in these computational models offers new insights into human visual development. Critical periods in biological development are typically thought to emerge from changes in the brain rather than from changes in experience. The current findings suggest that the properties of earliest experiences and themselves constrain what can be learned from later experiences in infants and machine learning. Third, the findings indicate potential principle for what constitutes an optimal starting point for visual learning: Simplicity in spatial structure of the images and in the slowness of change in those images.

---

Recently Joshi and Mirzasoleiman [2023] showed that data samples that contribute the most to self-supervised learning are those that have the most similar augmentations the other samples. Such samples are shown to ensure high alignment between the augmented views of the same class, and preserve the centers of class representations. This suggests a possible computational mechanism for how slowness and spatial simplicity benefit SSL: by providing more similar augmentations of the same object (and class). This may be particularly beneficial at the start of training by stabilizing and distinguishing the representation of objects in the high-dimensional space. The constraints of physical space and time on infant experiences mean that real-world experience is extremely skewed to many experiences of just few individual instances. Infants, for example, do not see many instances from a given object class early on, e.g. not many different cups, but rather they see one cup frequently [Clerkin et al. 2017]. Smith et al. [2018] discussed several verbal hypotheses of how this property of the infant experience can facilitate or affect visual learning. A noteworthy account in accordance with (Joshi and Mirzasoleiman 2023 is that early learning in both infants and SSL models can be facilitated by learning to see fewer objects but in many different viewing conditions. We note that the notion of class centroids in [Joshi and Mirzasoleiman, 2023] closely resembles the concept of prototype in the prototype theory of human perceptual categorization [Rosch and Mervis 1975). This another indication that self-supervised deep ANNs can appropriately function continuation of the classic connectionist models in the human perceptual learning literature
[aa 1997 Doerig el al.
2023].
Traditional theories of visual development also do not provide a quantitative account of how newborn visual systems learn from raw visual inputs. As such, these theories do not explain how why newborn visual systems develop the way that they do. By training deep neural networks "through the eyes" of developing humans, can tackle these problems and begin building neurally mechanistic, image-computable models of human newborn visual systems.
7.1 Limitations and Future Directions
Our study focused on showing the effect of the learning curriculum the learning outcomes, which are likely related to the learning dynamics of SGD networks, particularly those learning from self- supervised errors. We did not aim for the models trained here to perform competitively on any benchmark of neural or behavioral match to humans. Nevertheless, a promising future direction for this work is aim for milestone models of development models that develop the same visual skills as infants at a particular stage given the same visual experience. This endeavor involves developing benchmarks of infant behavioral and neural responses in machine learning tasks, similar [Schrimpf) eet al. 2021 Geirhos et al. 2002]] as well as making engineering progress in continual and lifelong learning [Zhuang al. 2022
Upon the remarkable success of the neuroconnectionist research program (Doerig et al. 2023 there have been calls for bridging visual developmental neuroscience and deep learning |Mur 2023 Zaadnoordijj et al. [002]). We argue that work along the aforementioned lines can take a remarkable step towards computational understanding of the developmental trajectory of the human visual system while also providing a road map for training computer vision systems more efficiently.
Acknowledgement
The authors thank Kathryn Bonnen, T. Rowan Candy, David Crandall and the other members of the egocentric vision group at Indiana University for the inspiring discussions that helped develop the ideas used in this project. The authors acknowledge the Indiana University Pervasive Technology Institute [Stewart et al. [017)] for providing supercomputing and storage resources that have con- tributed to the research results reported within this paper. This research was supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute. The collection of the dataset used in this project was supported by NIH grant 1R01EY032897 to Linda Smith and T. Rowan Candy.
10

---

References
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. supervipe learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 15619- 15629, 2023.
Arthur Aubret, Markus R Ernst, Céline Teulière, and Jochen Triesch. Time to augment self-supervised visual representation learning. In The Eleventh International Conference on Learning Representa- tions, 2022.
Judith Avrahami, Yaakov Kareev, Yonatan Bogot, Ruth Caspi, Salomka Dunaevsky, and Sharon Lerner. Teaching by examples: Implications for the process of category acquisition. The Quarterly Journal of Experimental Psychology Section A, 50(3):586-606, 1997.
Sven Bambach, Linda B Smith, David J Crandall, and Chen Yu. Objects in the center: How the infant's s body constrains infant scenes. In 2016 Joint IEEE International Conference Development and Learning and Epigenetic Robotics (ICDL-EpiiRbb) pages 132-137. IEEE, 2016.
Sven Bambach, David J Crandall, Linda B Smith, and Chen Yu. An egocentric perspective on active vision and visual object learning in toddlers. In 2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EEpiRob) pages 290-295. IEEF, 2017.
Sven Bambach, David Crandall, Linda Smith, and Chen Yu. Toddler-inspired visual object learning. Advances in neural information processing systems, 31, 2018.
Anthony J Bell and Terrence J Sejnowski. The "independent components" of natural scenes are edge filters. Vision research, 37(23)3332--3338 1997.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41-48, 2009.
Ari S Benjamin, Ling-QU Qi Zhang, Cheng Qiu, Alan A Stocker, and Konrad P Kording. Efficient neural codes naturally emerge through gradient descent learning. Nature Communications, 13(1):7972. 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
Elizabeth Clerkin, Elizabeth Hart, James Rehg, Chen Yu, and Linda B Smith. Real-world visual statistics and infants' first-learned object names. Philosophical Transactions of the Royal Society B: Biological Sciences, 372(1711 1):201):00555 2017.
Adrien Doerig, Rowan P Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W Lindsay, Konrad P Kording, Talia Konkle, Marcel AJ Van Gerven, Nikolaus Kriegeskorte, et al. The neuroconnectionist research programme. Nature Reviews Neuroscience, pages 1-20, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. 2020.
Jeffrey L Elman. Learning and development neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
Caitlin M Fausey, Swapnaa Jayaraman, and Linda B Smith. From faces hands: Changing visual input in the first two years. Cognition, 152:101-107, 2016.
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958, 2022.
Michael C Frank, Edward Vul, and Scott P Johnson. Development infants' attention to faces during the first year. Cognition, 110(2):100-77) 2009.

---

Michael C Frank, Dima Amso, and Scott P Johnson. Visual search and attention to faces during early infancy. Journal of experimental child psychology, 118:13-26, 2014.
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885-23899, 2021.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitte et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842-5850, 2017.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European conference on computer vision (ECCV), pages 135-150, 2018.
Sepp Hochreiter.
The vanishing gradient problem during learning recurrent neural nets and problem solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Bil -asddd Systems, 6(02): 107-116, 1998.
Swapnaa Jayaraman, Caitlin M Fausey, and Linda B Smith. The faces in infant-perspective scenes change the first year of life. PloS one, (0(5)=e0223880 2015.
Swapnaa Jayaraman, Caitlin M Fausey, and Linda Smith. Why are faces denser in the visual experiences of younger than older infants? Developmental psychology, 53(1):38, 2017.
Siddharth Joshi and Baharan Mirzasoleiman. Data-efficient contrastive supervuee learning: Most beneficial examples for supervised learning contribute the least. International conference on machine learning, 2023.
Yan Karklin and Michael S Lewicki. Emergence of complex cell properties by learning to generalize in natural scenes. Nature, 457(7225):83-86, 2009.
Muhammad Asif Khan, Ridha Hamila, and Hamid Menouar. Clip: Train faster with less data. In 2023 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 39- IEEE, 2023.
Lynne Kiorpes. The puzzle of visual development: behavior and neural limits. Journal of Neuro- science, 138441111344-119999 2016.
Kari S Kretch, John M Franchak, and Karen E Adolph. Crawling and walking infants see the world differently. Child development, 85(4):1503-1515 2014.
Mee-Kyoung Kwon, Mielle Setoodehnia, Jongsoo Baek, Steven Luck, and Lisa M Oakes. The development of visual search in infancy: Attention to faces versus salience. Developmental psychology, 52(4):537, 2016.
Remo H Largo and Tanja H Kakebeeke. Fine manipulative abilities in the first years of life. Motor development in children, pages 33-40, 1994.
Hui-Min Lee and James Cole Galloway. Early intensive postural and movement training advances head control in very young infants. Physical Therapy, 92(7):935 947, 2012.
Terri I. Lewis and Daphne Maurer. Multiple sensitive periods in human visual development: evidence from visually deprived children. Developmental Psychobiology: The Journal of the International Society for Developmental Psychobiology, 1633);163-183, 2005.
Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg. and Le Song. Iterative machine teaching. In International Conference on Machine Learning, pages 2158. PMLR, 2017.
Daphne Maurer. Critical periods re-examined: evidence from children treated for dense cataracts. Cognitive Development, 42:27-36, 2017.
12

---

Marieke Mur. Bridging visual developmental neuroscience and deep learning: challenges and future directions. Journal of Vision, 23(9))4680-4655 2023.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381 (658)).607766) 1996.
A Emin Orhan and Brenden M Lake. What can generic neural networks learn from child's visual experience? arXiv preprint arXiv:2305. 2023.
Emin Orhan, Vaibhav Gupta, and Brenden M Lake. Supe-vupe learning through the eyes of child. Advances in Neural Information Processing Systems, 33:9960-9971. 2020.
Eleanor Rosch and Carolyn B Mervis. Family resemblances: Studies in the internal structure of categories. Cognitive psychology, 7(4):573-605, 1975.
Luca Saglietti. Stefano Mannelli, and Andrew Saxe. An analytical theory of curriculum learning in teacher-student networks. Advances in Neural Information Processing Systems, 35:21113-21127, 2022.
Madeline C Schiappa, Yogesh S Rawat, and Mubarak Shah. Self-supervised learning for videos: A survey. ACM Computing Surveys, 2022.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua Tenenbaum, and Evelina Fedorenko. The neural architecture language: Integrative modeling converges predictive processing. Proceedings of the National Academy of Sciences, 1 (18(45)::210564611 2021.
Saber Sheybani, Zoran Tiganj, Justin N Wood, and Linda B Smith. Slow change: An analysis of infant egocentric visual experience. Journal of Vision, 23(9):4685-466) 2023.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint 1703.v:703300000 2017.
Linda B Smith and Lauren K Slone. A developmental approach machine learning? Frontiers in psychology, page 2124, 2017.
Linda B Smith, Swapnaa Jayaraman, Elizabeth Clerkin, and Chen Yu. The developing infant creates curriculum for statistical learning. Trends in cognitive sciences, 22(4):325-32) 2018.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0402 2012.
Elizabeth S Spelke. Preferential-looking methods as tools for the study of cognition in infancy, Ablex Publishing, 1985.
Craig A Stewart, Von Welch, Beth Plale, Geoffrey Fox, Marlon Pierce, and Thomas Sterling. Indiana university pervasive technology institute. 2017.
Jessica Sullivan, Michelle Mei, Andrew Perfors, Erica Wojcik, and Michael C Frank. Saycam: A large, longitudinal audiovisual dataset recorded from the infant's perspective. Open mind, 5:20-29, 2021.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data- efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203. 2022.
Du Tran, Lubomir Bourdev, Rob Fergus. Lorenzo Torresani. and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489 4497, 2015.
Xiaohan Wang, Tengyu Ma, James Ainooson, Seunghwan Cha, Xiaotian Wang, Azhar Molla, and Maithilee Kunda. The toybox dataset of egocentric visual object transformations, arXiv preprint (XXiv1886.006044 2018.
13

---

Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5017-5026, 2019.
Bileen C Way. Connectionism and conceptual structure. American Behavioral Scientist, 40(6): 729-753, 1997.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14668-14678, 2022.
Robert C Wilson, Amitai Shenhav, Mark Straccia, and Jonathan D Cohen. The eighty five percent rule for optimal learning. Nature communications, 10(1):4646, 2019.
Laurenz. Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pages 38-45, 2020.
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? arXiv preprint arXiv:2012.03107, 2020.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning for video understanding. arXiv preprint arXiv:17/2.0485// 122):, 2017.
Lorijn Zaadnoordijk, Tarek R Besold, and Rhodri Cusack. Lessons from infant learning for unsuper- vised machine learning. Nature Machine Intelligence, 4(6):510-520) 2022.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 20114966 18, 2021.
Chengxu Zhuang, Ziyu Xiang, Yoon Bai, Xiaoxuan Jia, Nicholas Turk- Browne, Kenneth Norman, James J DiCarlo, and Dan Yamins. How well do unsupervised learning algorithms model human real-time and life-long learning? Advances in Neural Information Processing Systems, 35:22628- 22642, 2022.
14


Reviews:
[{"review_id": "Falz038B5i", "reviewer": "Reviewer_L6Nk", "summary": "This paper aims to study how developmental changes impact visual learning. It used infant egocentric videos to pre-train video autoencoders with self-supervised learning loss. It separated the infant data by age group and evaluated the importance of training with a curriculum aligned with developmental order.", "questions": "None.", "limitations": "Limitations of the paper and future studies are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The study goal of this paper is interesting. Few has studied the developmental changes in infant visual experience and characterize how those changes impact visual development.", "weaknesses": "1) The contribution of this paper is limited. The datasets are collected from existing datasets. Only single existing video autoencoder model is used and evaluated. While the goal of the paper is interesting, it is not conducted in a professional way.\n2) The writing and organization of the paper need much improvement. It is not easy to follow the content of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the comments and appreciate the Reviewer's enthusiasm about the goals of the paper. Below we addressed each weakness mentioned by the Reviewer. \n\n**Reviewer:** The contribution of this paper is limited.\n\n**Our response:** We believe that the manuscript delivers several substantial contributions:\nWe demonstrated that not only do data sets matter, the order of data sets matters.\n\nEvolution has taken great pains to control the order of visual experiences for infants, and our results showcase that AI researchers might benefit by providing machines with a common, naturally-ordered learning curriculum.\n\nOur results indicate that machines learn best when they first receive training data that changes very slowly, with a bias to input characterized by high contrast and large simple edges. This “edge-based” learning (V1 in mammalian cortex) is the first step in the formation of high-level visual representations. \n\nThe order of the data matters, regardless of the kind of learner algorithm. This conclusion is further supported by the experiments discussed in the rebuttal. This indicates that our results reflect a general principle about learning. \n\n**Reviewer:** The datasets are collected from existing datasets.\n\n**Our response:** The dataset used here is the only one of its kind ever recorded. It captures the ordered experiences of human infants in the first year of life, a period of marked learning and tuning of the human visual cortex. These are daylong recordings, all daily-life contexts, no experimenters present who might distort the statistics. This manuscript is the first time that the dataset has been used to train artificial neural networks and thus we believe it provides a valuable insight into similarities and differences of human and machine learning.\n\n**Reviewer:** Only a single existing video autoencoder model is used and evaluated.\n\n**Our response:** We conducted additional evaluations using JEPA-TT and SimCLR-TT (see general response for details and Fig. R2 in the rebuttal PDF). Unlike VideoMAE, which is trained on reconstruction, JEPA-TT is trained on prediction using contrastive self-supervised learning through time. The results from JEPA-TT were qualitatively similar to the results from VideoMAE. SimCLR-TT is another approach based on contrastive learning (Aubret et al. 2022), which uses CNN instead of transformer architecture. We found again that the results remained qualitatively similar, supporting our general claim about the importance of the order of training data.\n\nWe also introduced an additional indicator of the learning quality, the gradient norm. This metric provided an insight into the learning process and clearly separated developmental and anti-developmental curricula (see general response for details and Fig. R1A, Fig. R2A and Fig. R2B in the rebuttal PDF). \n\nFurthermore, to gain deeper insight into why learning is better when data comes in developmental order, we conducted an analysis where we controlled for spatial and temporal complexity. This revealed the importance of the temporal properties of the training data, suggesting that training with slower data first will improve learning (see general response for details and Fig. R1 in the rebuttal PDF).\n\n**Reviewer:** The writing and organization of the paper need much improvement.\n\n**Our response:** We will do our best to edit the manuscript to improve its readability. We already changed the visual language in the attached figures in the rebuttal PDF and we plan to apply that visual language to other figures in the manuscript. We welcome any suggestion from the Reviewer regarding the organization of the manuscript."}, {"review_id": "VbioA0Wm88", "reviewer": "Reviewer_KVqq", "summary": "The work is essentially about an experimental evaluation to show the benefit of curriculum learning using ego-centric videos acquired with from head-mounted cameras and very young children. The authors discuss the use of self-supervised learning on the data ordered according to developmental principles and empirically show the improvement that can be observed on down-stream tasks and benchmark datasets.", "questions": "I will try to list some more specific questions that I hope will help to clarify some important points:\n\n- Here and there there are somehow strong statements that would require appropriate citations. For instance: \"A common assumption in the machine-learning literature is that at the massive scale of daily life, the world presents the same visual statistics to all perceivers. Many researchers therefore believe that it will be possible to mimic biological learning through incremental statistical learning performed  over large datasets.\", or also  “An appropriate model and the objective function to assess a developmental curriculum is one with a  self-supervised learning objective  and model architecture that directly learns the regularities in the video dataset across both space and time – a spatiotemporal learner. “\n\n- Has the dataset been acquired by the authors? If yes, why isn’t it a contribution of the work? On the basis of what the videos have been ordered on the basis of the motor abilities? In the paper it is said: “…we selected videos collected by infants at 3 age periods who differ in the motor skills and by hypothesis in the statistics of experience” Who made the selection? I think specific expertise would be needed\n\n- Again on the dataset: In the dataset description is n referring to the number of children or videos? If these are the children, a significant corpus of acquisitions has been discarded from the original 101 to the 29 that have been used. Why? Wouldn’t it be possible to use them as an extra, maybe more complex, test set?\n\n- The two training sets (children and adults) are very imbalanced: 123h for children and 64 for adults. This can negatively affect the results and would deserve some comments\n\n- Row 173: why only edge features?\n\n- Row 187: why a linear classifier?\n\n- Motivations behind the use of video-masked autoencoders?\n\n- Sect. 5.1: is the validation loss computed on a validation set of the same class of data? E.g. is the loss for the model trained on g0 computed on validation from g0? If yes, this may explain the different performance (g0 simpler than others?). If not, it must be more clearly explained\n\n- “…we first tracked the self-supervised loss of the VideoMAE model trained separately with the data from the youngest age group (g0), the data from the oldest age group (g2), random group, and adult group” This is not what we see in Figure 2, where in the legend I see instead, for instance, “Dev. (g0-g1-g2)”. How should we relate the text with the figure? It is not clear to me, I failed to track the results. From what I understand Figure 2 is pertinent only to Sec, 5.2 (maybe also the experiments from 5.1 are there, but too hidden to the appreciated)\n\n- I’m not sure I understood what validation loss is used in Sec. 5.1: is it based on reconstruction? If yes, why it starts from 1?\n\n- “ Our results also provide evidence in favor of the long-standing perspective in  neuroscience that emphasizes the importance of slowly changing sensory signals in visual learning for the emergence of sparse and efficient coding. “ I fail to get this observation from your results, where should it be emerging?", "limitations": "The limitations of the work have been mentioned in a specific section", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I find the idea of coupling a developmental approach with curriculum learning very fascinating, and deserving attention. \nThe main motivations behind the developmental paradigm that inspired the approach are clearly reported.", "weaknesses": "I have some main concerns about the clarity of the presentation and the contributions that make me think the work is not yet ready for publication at this conference:\n\n- On the motivations behind the developmental approach: I find it fascinating, but somehow in contrast with the use of large data sets and complex architectures. On one side there are children with their visual, cognitive and motor abilities under development, on the other these very complex architectures. For the latter, I think a more convincing justification should be provided on why those have been used. \n\n- While the overall inspiration and philosophy of the work are clear (although sometimes maybe not immediately clear for those with no particular knowledge of Cognitive Science) in general, I found a lack in the justifications of the more technical choices. Just as a possible example: from row 157 to row 165, more details in the questions.\n\n- Since the main objective was to show the benefit of the developmental approach, my impression is that the limitation correctly highlighted by the authors of having considered just one architecture is too important to be disregarded. In my opinion, showing that the principles observed in these experiments emerge also when changing the architecture is fundamental to speak in favour of the generality of the results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for detailed comments and for the enthusiasm about our approach. Below are our responses to the Reviewer’s questions. \n\n**Reviewer** asked for a more convincing justification on why we used large data sets and complex architectures. \n\n**Our response:** While we did use advanced architectures, the number of neurons in those architectures is still a fraction of those in a newborn's brain (estimated to be around 100 billion).\n\n**Reviewer** found a lack in the justifications of the technical choices. \n\n**Our response:** With the proposed additions (rebuttal PDF), we are covering state-of-the-art self-supervised architectures and loss functions. In general, we chose self-supervised learning to reflect the nature of learning by human infants.\n\n**Reviewer** stated that ‘’showing that the principles observed in these experiments emerge also when changing the architecture is fundamental to speak in favour of the generality of the results’’.\n\n**Our response:** We agree with this point and we conducted an evaluation using JEPA-TT and SimCLR-TT. For more details, please see general response and Fig. R2 in the rebuttal PDF. \n\n**Reviewer** asked to support with more citations some of the general claims made in the manuscript and pointed out two such statements. \n\n**Our response:** We agree with the Reviewer and we plan to add the following references to support the first statement: Torralba et al. 2003, Felsen et al. 2005, Simoncelli et al. 2001, Lake et al. 2017 and following references to support the second statement: Zaadnoordijk et al. 2022, Orhan et al. 2020, Zhuang et al. 2019. \n\n**Reviewer** asked whether the dataset has been acquired by the authors.\n\n**Our response:** We used the Homeview human-subjects dataset, which is not publicly available due privacy, rights and confidentiality of the participants.\n\n**Reviewer** asked about the basis for ordering of the videos into 3 groups. \n\n**Our response:** The first 12 months are critical for proper development of human visual systems and we choose 3 stages within that period that are characterized by very different properties of the visual inputs. Very young infants, for example, prefer to look at scenes with large, simple high contrast edges, and they are “sticky” lookers [Smith et al. 2018], orienting to and looking at a single object for many seconds. In the following months, there are marked and systematic increases in the control, complexity, purposefulness, speed and precision of body movements that include rapid and more varied changes in head and body orientation, reaching for, handling and performing instrumental acts on objects and self-locomotion through crawling and walking [Kretch et al. 2014]. We provided more details in lines 120-129 in the manuscript.\n\n**Reviewer** asked why not all recorded data from the Homeview dataset was used. \n\n**Our response:** We focused on three stages during the first 12 months and we did not use data from participants outside of those stages (many participants were older than 12 months). \n\n**Reviewer** said that two training sets (children and adults) are very imbalanced: 123h for children and 64 for adults. \n\n**Our response:** To compensate for the imbalance, we bootstrapped the adult data to match the size of the children data. \n\n**Reviewer** asked why we intuit each token in VideoMAE as a small, moving edge feature.\n\n**Our response:** The sentence was intended to provide an intuition to those less familiar with transformer architecture. We understand it might cause confusion, so we will remove the moving edge intuition. \n\n**Reviewer** asked why we used a linear classifier. \n\n**Our response:** Downstream classification using a linear classifier has been a common practice for evaluating self-supervised learning algorithms in recent years [Orhan et al. 2020, Aubret et al. 2022, Assran et al. 2023, Chen et al. 2020]. \n\n**Reviewer** asked about motivations for using video-masked autoencoders.\n\n**Our response:** They are self-supervised spatiotemporal learners with relatively few design choices and hyperparameters. \n\n**Reviewer** asked if the validation loss computed on a validation set of the same class of data and whether that might explain different performance.\n\n**Our response:** A separate validation set was constructed for each age group. Since we observed lower loss for developmental curriculum after each of the three stages, the overall result does not appear to be determined by the properties of the validation set. Furthermore, performance on the downstream classification tasks was independent of the validation set and it was consistently better for the developmental curriculum. \n\n**Reviewer** asked about visualization of the results from Stage 1 shown in Fig. 2. \n\n**Our response:** The beginning of each stage is marked with a dashed line, an arrow and a label. Each stage has 5000 iterations, so the results for Stage 1 are at 5000-th iterations. To avoid confusion, we will replace the visual language in the current figures with the visual language shown in the rebuttal PDF (bar plots).\n\n**Reviewer** asked whether the validation loss in Sec. 5.1 is based on reconstruction and why it starts from 1.\n\n**Our response:** Validation loss is based on reconstruction and was normalized to always start at 1.\n\n**Reviewer** asked about the support for the sentence about slowly changing sensory signals.\n\n**Our response:** Indeed, the difference in learning between developmental and anti-developmental curricula could have emerged due to spatial and/or temporal differences in the data. To better understand the causes of the difference, we performed new analyses in which we controlled for spatial and temporal complexity (see general response for details). As shown in Fig. R1 (rebuttal PDF), temporal complexity was particularly important for driving the difference between developmental and anti-developmental curricula."}, {"review_id": "zH9z8BjkEm", "reviewer": "Reviewer_wvZP", "summary": "The authors aim to explore whether the natural structures and regularities in infant visual experience acquired via video recording when infants wearing head-mounted cameras can facilitate pre-training self-supervised learning representation learners. The authors divide the infant data into 3 groups (according age group) and evaluate the importance of training with a curriculum aligned with developmental order. The other 3 curriculums are designed for comparisons. The experimental results prove the order of the curriculum matter via self-supervised learning tasks and downstream tasks.", "questions": "The questions are listed in the Weakness section. ", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Overall the paper is well-written and is easy to understand the goal of the paper.\n2. In the community, we all know that data is essential. However, we pay less attention to the order of curriculum that infants naturally created for training visual systems. The motivation of the paper is to explore whether the order of the curriculum is essential for visual systems. The authors leverage data curated from a larger corpus of 500 hours of head camera videos collected by infants from 2 to 24 months of age for experiments. They propose a clear experimental protocol and training curriculums for comparisons. In the experiments, they show deep neural networks (i.e., VideoMAE) trained on developmentally-ordered datasets outperformed those trained on “developmentally incorrect” datasets. The discovery of the experiments is valuable to the machine learning community to revisit our training dataset and learning curriculums.\n3. The pre-trained models are finetuned for downstream tasks. Again, the results show a consistent trend that those trained on developmentally-ordered datasets outperformed those trained on “developmentally incorrect” datasets. This indicates that the order in which humans generate data for learning may therefore be an important contributor to achieving adult-like visual processing. This again provides the community to rethink our training protocols.\n4. The reviewer particularly enjoys reading section 5.3. Specifically, the authors propose a methodology to probe the learned visual features using the developmentally-ordered datasets to provide a “peak” of infants’ receptive field. The findings could provide a new tool for developmental psychology researchers to design new experiments to validate the effectiveness of the methodology.", "weaknesses": "Overall, the reviewer enjoys reading the manuscript and learns a ton. I have the following questions, comments, and suggestions:\n\n1. In Sec. 5.3, the authors highlight that it takes 12 components of g0 to explain 95% of the variance; whereas, it takes 33 components for g2. Overall, the larger number for g2 makes sense. The reviewer is surprised by the difference. The reviewer would like to learn from the authors whether the ratio of the two “number of components” could be some indicator. Assuming there is one, the ratio could potentially be used to validate the effectiveness of the learning models. \n\n2. The main “concern” of the work is the paragraph in L287 to L300. Indeed, ViT has made tremendous progress in the vision community. The authors further utilize it on the developmentally-ordered dataset. However, as pointed out in the limitation section, there are other learning systems that could be explored. The reviewer is concerning the masking used in VideoMAE is the appropriate learner, while showing promising results in the work. In addition, the recent LLMs show promising results of autoregressive models (prediction of the next frame) compared to BERT-like approaches (filling masked areas). Moreover, self-attention is one of the important components. The reviewer brings these points to highlight the need to explore other existing learners so that we can provide a more thorough perspective to support the statements in L287 to L300.\n\n3. Implementation details, Training protocol: while the experimental results do empirically prove the order of curriculum is essential, I wonder how the authors determine the number of iterations. In the experiments, 5000 iterations are used. Does the varying number of iterations influence self-supervised pretraining? Does the varying number of iterations influence finetuning downstream tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for detailed comments and for pointing out a number of strengths. Below we addressed each question mentioned by the Reviewer.\n\n**Reviewer** asked whether the ratio of the two “number of components” could potentially be used to validate the effectiveness of the learning models.\n \n**Our response:** As the Reviewer pointed out, the ratio of the number of principal components needed to explain 95% of the variance in g0 and g2 was indeed large. This reflects the dramatic difference in the infant visual inputs across development. We agree with the Reviewer that the ratio could potentially be used to characterize and compare datasets in terms of their similarity to the human visual inputs. \n\n**Reviewer** suggested to explore other existing learners and mentioned prediction of the next frame as a possible learning objective.\n\n**Our response:** We agree that this is important and we conducted additional evaluations using JEPA-TT and SimCLR-TT (see general response for details and Fig. R2 in the rebuttal PDF). Unlike VideoMAE, which is trained on reconstruction (similar to BERT-like approaches also mentioned by the Reviewer), JEPA-TT is trained on prediction using contrastive self-supervised learning through time. The results from JEPA-TT were qualitatively similar to the results from VideoMAE. SimCLR-TT is another approach based on contrastive learning (Aubret et al. 2022), which uses CNN instead of transformer architecture. We found again that the results remained qualitatively similar, supporting our general claim about the importance of the order of training data.\n\n**Reviewer** asked whether the varying number of iterations influence self-supervised pretraining and finetuning downstream tasks.\n\n**Our response:** The choice of 5000 iterations for each of the three stages of training was inspired by the Neyshabur et al. study, which reported the effect of curriculum going away if the number of iterations reached around 20k. Fig. S3 in the manuscript shows the results of the pretraining (panel A) and downstream classification (panel B) for 2000, 5000 and 15000 interactions per stage. Even when training for 15000 interactions, the anti-developmental curriculum did not result in improvement after the first stage (data from the oldest group, g2). Training for 15000 interactions in the anti-developmental curriculum also showed that the last stage (data from the youngest group, g0) made significant progress, but the results of the developmental curriculum were still better than those of the anti-developmental curriculum. \n\nWe would like to add that comparing learning in humans and machines remains methodologically challenging. While there is strong evidence that sensorimotor learning has a large statistical learning component, potentially making use of error gradients, the details of how statistical visual learning is implemented in the brain is largely unknown. Nevertheless, the recent powerful deep learning algorithms provide a method to ask broad questions about the relationship between natural data streams and the learning outcomes, behavioral or neural. Here, our goal was to ask how the properties of the real world experience of infants affect general statistical learners. Along the way, we have done our best to choose the learning system with the goal of minimizing the role of factors that make it less humanlike (e.g., very large batch sizes, many epochs, etc) while still benefiting from the scalability of deep learning systems in learning from relatively large scale real-world data.\n"}, {"review_id": "VxTbfWqpo5", "reviewer": "Reviewer_Bo4Q", "summary": "This paper studies the development of visual intelligence by providing a curriculum for self-supervised video representation learning. The curriculum is aligned with the age order of the infants whose egocentric video data are used as training samples. Experiment results demonstrate that training with data from the youngest infants, who have fewer motor abilities, leads to a faster decrease in both training and validation loss. Additionally, the proposed curriculum improves downstream task performances when evaluated using linear probing.", "questions": "- Is the PCA applied to both the training and validation video data in a specific age group?\n- Figure S6 for CEVR seems to be missing in the supplementary material.", "limitations": "The authors have discussed the limitations of the evaluation method. It would be beneficial for the authors to discuss further the limitations related to the potential biases of the data collection process, or limitations in the egocentric video data recorded from infants.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The use of curriculum learning guided video self-supervised learning to study infant visual experience is a novel and interesting idea.\n- The natural alignment of the curriculum with the age order of infants, based on the egocentric videos recorded, is a commendable aspect of this work.\n- The paper provides a detailed discussion of the contributions and limitations of the research.", "weaknesses": "- Regarding the “slowness” of the video, there lacks a quantitative measurement of it, as well as corresponding experiments regarding the curriculum that is aligned with the slowness of the video rather than the age order.\n\n-  The iteration number 5000 might be not sufficient for models trained on videos with more dynamics to converge. While ablation studies on iteration numbers are presented in the supplementary material, it would be valuable to continue to increase the iteration number, and include a comparison with Random and Adult baselines for a more comprehensive analysis.\n\n- It is recommended to report the video retrieval performance after pre-training with different curricula to provide a more comprehensive evaluation.\n\n- In addition to masked image modeling, exploring contrastive learning-based self-supervised learning methods would enhance the scope of the study.\n\n- Some minor issues include missing section reference number in line 212.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for detailed comments and for highlighting the values of this line of work. Below we addressed each weakness and question mentioned by the Reviewer.\n\n**Reviewer** pointed out the lack of a quantitative measurement for “slowness” of the video, as well as corresponding experiments regarding the curriculum that is aligned with the slowness of the video rather than the age order. \n\n**Our response:** To better understand the role of temporal slowness in our results, we performed new analyses in which we controlled for spatial and temporal complexity (see general response for details). As shown in Fig. R1 (rebuttal PDF), temporal complexity was particularly important for driving the difference between developmental and anti-developmental curricula. Furthermore, we have conducted a detailed analysis of temporal slowness by estimating the autocorrelation function of the videos using the mean pairwise cosine similarity between the pixel representation of the frames at a series of lags. We then used the time constant of the autocorrelation function as a measure of the slowness of the video. We found significant differences between the median time constant across all three age groups, with the youngest age group (g0) being the slowest, followed by the middle age group (g1). The results were further confirmed through a spectral analysis of the embeddings, where the youngest age group had more low-frequency components than the other two age groups. We will add these results to the supplemental of the manuscript.\n\n**Reviewer** suggested to continue to increase the iteration number, and include a comparison with Random and Adult baselines for a more comprehensive analysis.\n\n**Our response:** Fig. S3 in the manuscript shows the pretraining and downstream classification results for up to 15000 interactions per stage. We did not observe qualitative changes in training between 2000, 5000 and 15000 iterations. Following the suggestion from the Reviewer, we are currently conducting experiments for 30000 iterations as well as for Random and Adult baselines. Due to computational complexity, these experiments will take several days to complete. We also note that the choice of 5000 iterations for each of the three stages of training was inspired by the Neyshabur et al. study, which reported the effect of curriculum going away if the total number of iterations reaches around 20k.\n\n**Reviewer** recommended to report the video retrieval performance after pre-training with different curricula.\n\n**Our response:** The reconstruction loss provided in Fig. 2 in the manuscript reflects the video retrieval performance. We apologize if we misunderstood the Reviewer’s suggestion.\n\n**Reviewer** suggested to explore contrastive learning-based self-supervised learning methods to enhance the scope of the study.\n\n**Our response:** We conducted additional evaluations using JEPA-TT and SimCLR-TT (see general response for details and Fig. R2 in the rebuttal PDF). Unlike VideoMAE, which is trained on reconstruction, JEPA-TT is trained on prediction using contrastive self-supervised learning through time. The results from JEPA-TT were qualitatively similar to the results from VideoMAE. SimCLR-TT is another approach based on contrastive learning (Aubret et al. 2022), which uses CNN instead of transformer architecture. We found again that the results remained qualitatively similar, supporting our general claim about the importance of the order of training data.\n\n**Reviewer** asked: Is the PCA applied to both the training and validation video data in a specific age group?\n\n**Our response:** Yes, that is correct. \n\nWe thank the Reviewer for pointing out the missing section reference and that reference to Fig. S6 should point to Fig. S4. \n"}]

Author rebuttal:
Our paper tackles a question at the heart of human and machine learning: How do we compare the learning abilities of human infants and machines? Our study takes an important step in this direction by (1) showing that generic learning algorithms (with no hardcoded knowledge about objects or space) can learn to solve complex real-world vision tasks when the models are trained “through the eyes” of human infants; (2) showing that the order of training data matters, such that machines learn best when they receive training data in the same order as human infants; and (3) showing that slow visual inputs are valuable for kick-starting visual learning. The dominant assumption in AI is that the order of training data does not matter for large-scale computer vision systems, so our results contradict a core assumption in the field. Our study also introduces the first computer vision system that has been trained in a “developmentally correct” order, laying a critical foundation for reverse engineering the learning algorithms in human brains.   

The Reviewers largely agreed with our goals and approach. However, the ratings were ambivalent for four reasons:
 
First, in the original submission, we only tested one ViT model (VideoMAE). To address this critique, we added new experiments testing JEPA-TT models, which learn via contrastive learning through time. As shown in the left column in Fig. R2 (rebuttal PDF), the JEPA-TT model showed similar learning patterns as the VideoMAE model. This result indicates that our conclusions reflect general principles of learning across biological and artificial systems.
 
Second, in the original submission, we did not compare ViTs to CNNs. It was thus unclear whether the spatial inductive bias of CNNs helps or hinders performance. To address this critique, we added new experiments testing CNNs that learn via contrastive learning through time (SimCLR-TT; Aubret et al. 2022). As shown in the right column in Fig R2 (rebuttal PDF), both CNNs and ViTs were sensitive to the order of the training data. This finding shows that the learning effects we reported in our original paper generalize across different architectures (ViTs and CNNs).
 
Third, the Reviewers worried that since we did not provide causal evidence that slow visual inputs drive rapid learning in ViTs, our conclusions were limited in scope. To address this critique, we performed new analyses in which we controlled for spatial and temporal complexity. As shown in Fig. R1 (rebuttal PDF), the learning differences between models trained on developmentally accurate vs. developmentally inaccurate data largely disappeared. These analyses show that slow visual inputs are causally connected in the rapid development of machine vision, akin to slow visual inputs being causally connected to the development of biological vision.    
 
Finally, Reviewers questioned the scope of the project and importance of the results. To address these critiques, we will revise our manuscript to emphasize that our study is the first of its kind, providing a unique opportunity to directly compare the learning abilities of human infants and computer vision models. We would also emphasize that our results have important scientific implications. ViTs have the potential to be powerful image-computable models of human visual systems, but these models will not be accepted if they are thought to learn differently from humans. Our results contradict this assumption, paving the way for high-performing models from AI to serve as image-computable models of visual learning and development.

Below we provide details for the methods used to generate the figures in the rebuttal PDF. 

JEPA-TT 

I-JEPA (Assran et al., 2023) is a recently proposed Image-based SSL algorithm in which the objective is to predict the embedding of one part of the input image (target) using the embedding of another part (context). Our JEPA Through Time (JEPA-TT) algorithm is an adaptation of I-JEPA where the context and the target are selected from consecutive video frames rather than the same frame.

SimCLR-TT

SimCLR is a well-known Image-based SSL algorithm. It works by bringing the embeddings of two augmented versions of an image (positive pair) closer in the embedding space while pushing them away from embeddings of randomly selected images from the dataset. Similar to Aubret et al. 2022, our SimCLR-TT algorithm treats the consecutive frames as augmentations of each other. However, we found this method to overfit our dataset very quickly, therefore we reduced the number of training iterations from 5000 to 1500 (analogous to early stopping).

Controlling for temporal and spatial complexity

To match temporal complexity, we used individual video frames as the model input, instead of a series of frames. To match spatial complexity, we equalized the number of edges (as a measure of image complexity) in images from each of the three age groups. To achieve this, we first extracted the edges using a Canny edge detector and then sampled images to match the number of edge pixels. We found that the gap in the metrics between the developmental and anti-developmental curricula reduced as we matched the spatial complexity, temporal complexity, or both (Fig. R1).

Gradient Norm

We report an additional indicator of the learning process, the gradient norm, computed as $||\frac{1}{batch\ size} \sum \frac{\partial \text{Loss}}{\partial \Theta}||_2$ where $\Theta$ contains the learnable weights of a given layer. This quantity has been used for studying the learning dynamics of Stochastic Gradient Descent (SGD) (Shwartz-Ziv and Tishby, 2017). We found that regardless of the learning objective and architecture, the developmental curriculum had the largest gradient norm in the first stage of training, where it receives the data stream from the youngest infants (g0). Accordingly, the anti-developmental curriculum receives the g0 data at the third stage and has the larger gradient norm.


Meta-review: The paper studied the curriculum learning of visual representation based on inputs gathered from infant's point of view. While the study considered existing techniques in self-supervised learning, the results demonstrated some some intriguing findings about the importance of training with a curriculum aligned with developmental order. 

The initial review ratings were mixed. The reviewers raised concerns on the experiments, and questioned some of the technical details. The authors provided a detailed response that addressed most of the concerns. After the rebuttal, the reviewers applaud the findings and contributions of the paper. The decision is thus to recommend its acceptance.
Decision: Accept (spotlight)