Are Vision Transformers More Data Hungry Than Newborn Visual Systems?
Authors: Lalit Pandey
Samantha Marie Waters Wood
Justin Newell Wood
Venue: NeurIPS 2023 poster (2023)
Keywords: vision transformer
newborn
controlled rearing
object recognition
data hungry
TL;DR: Contrary to popular belief, vision transformers (ViTs) are not more "data hungry" than newborn visual systems.

Abstract: Vision transformers (ViTs) are top-performing models on many computer vision benchmarks and can accurately predict human behavior on object recognition tasks. However, researchers question the value of using ViTs as models of biological learning because ViTs are thought to be more “data hungry” than brains, with ViTs requiring more training data than brains to reach similar levels of performance. To test this assumption, we directly compared the learning abilities of ViTs and animals, by performing parallel controlled-rearing experiments on ViTs and newborn chicks. We first raised chicks in impoverished visual environments containing a single object, then simulated the training data available in those environments by building virtual animal chambers in a video game engine. We recorded the first-person images acquired by agents moving through the virtual chambers and used those images to train self-supervised ViTs that leverage time as a teaching signal, akin to biological visual systems. When ViTs were trained “through the eyes” of newborn chicks, the ViTs solved the same view-invariant object recognition tasks as the chicks. Thus, ViTs were not more data hungry than newborn chicks: both learned view-invariant object representations in impoverished visual environments. The flexible and generic attention-based learning mechanism in ViTs—combined with the embodied data streams available to newborn animals—appears sufficient to drive the development of animal-like object recognition.

Are Vision Transformers More Data Hungry Than Newborn Visual Systems?
Lalit Pandey Department of Informatics Indiana University Bloomington 1pandey@iu. edu
Samantha M. W. Wood Department of Informatics Indiana University Bloomington sw1 Sw113@iu.edu
Justin N. Wood Department of Informatics Indiana University Bloomington woodjn@iu.cm eduu edu
Abstract
Vision transformers (ViTs) are lop-performing models on many computer vision benchmarks and can accurately predict human behavior on object recognition tasks. However, researchers question the value of using ViTs as models of biological learning because ViT's are thought to be more "data hungry" than brains, with ViTs requiring more training data to reach similar levels of performance. To test this assumption, we directly compared the learning abilities of ViTs and animals, by performing parallel controlled-rearing experiments on ViTs and newborn chicks. We first raised chicks in impoverished visual environments containing single object, then simulated the training data available in those environments by building virtual animal chambers in video game engine. We recorded the first-person images acquired by agents moving through the virtual chambers and used those images to train self-supervised ViTs that leverage time a teaching signal, akin to biological visual systems. When ViT's were trained "through the eyes" of newborn chicks, the ViTs solved the same view-invariant object recognition tasks the chicks. Thus, ViT' were not more data hungry than newborn visual systems: both learned view-invariant object representations in impoverished visual environments. The flexible and generic attention-based learning mechanism in coTs-combined with the embodied data streams available to newborn to drive the development of animal-like object recognition.
sufficient
1 Introduction
Vision transformers (ViTs) have emerged as leading models in computer vision, achieving state-of- the-art performance across many tasks, including object recognition [4], scene recognition [5], object segmentation [9], action recognition [70], and visual navigation [14]. Recent studies also suggest that transformers share deep computational similarities with human and animal brains [8, 19, 45, 55). For instance, transformers are closely related to current hippocampus models in computational neuroscience and can reproduce the precisely tuned spatial representations of the hippocampal formation (e.g.. place and grid cells; [55]). ViTs also outperform convolutional neural networks (CNNs) on challenging image classification tasks and are more likely to learn human-like shape biases (and make human-like errors) than CNNs [18, 50].
Despite these strengths, there is lingering worry about using ViTs models of biological vision. The worry relates to the large amount of data needed to train ViTs, compared to the relatively small
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Perform controlledreerrig studies newborn chicks
2
Simllete visual experiences of chicks virtual arimal chambers
Train best VITs on simuleledd visual experiences from chicks
Training
Training
Training
Triining data for VIT coped worper.
Self-supervince transformer (contraziiva through
Test Phase
Noval Ofject
Imprinted Oajecr
Post recagnition across novel views
ViT (trozen)
Linbar clossifor
Mesiaria odject recog vidon performernia HOrOER novel
views
Our design had three steps: (1) Run controlled-rearine experiments testing how view- invariant object recognition develops in newborn chicks.
Figure 1:
(2) Simulate the visual experiences (training data) available to the chicks during the training and test phases, using virtual animal chambers constructed video game engine (Unity 3D). (3) Train and the ViTs on the simulated visual experiences from the chick experiments. For training, used self-supervised ViTs that leverage time as teaching signal. For testing, we froze ViTs and attached linear classifier to the penultimate layer. We then trained & tested the linear classifier with the simulated test images from the chick experiments, in a cross- cross- validated design.
amount training data needed by biological brains. The ViTs that have revolutionized computer vision are trained on massive amounts of data (e.g.. ImageNet, which contains millions of images across over 20,000 object categories). Conversely, animals not require massive amounts object experience solve object perception tasks. For example, even when newborn chicks are reared in impoverished environments containing only a single object (i.c., sparse training data from birth), they still rapidly learn many object perception abilities, including segmenting objects from backgrounds [64], binding colors and shapes into integrated object representations [58], recognizing objects and faces from novel views [57, 66, 65, 68), and remembering objects that have disappeared from view [42]. Thus, ViTs appear to be more "data hungry" than newborn visual systems.
Here, we suggest that the learning gap between animals and ViTs is not as large as it appears. To compare learning across animals and ViTs, we performed parallel controlled-rearing experiments on newborn chicks and ViTs (Figure 1). First, we raised newborn chicks in strictly controlled virtual environments and measured the chicks' view-invariant object recognition performance (data reported in [57]). Second, created digital twins (virtual simulations) of the controlled-rearing chambers in a video game engine, then simulated the chicks' visual training data by recording the first-person images acquired by agent moving through the virtual chambers. Third, trained ViTs the simulated data streams from the virtual chambers and tested the ViTs with the same stimuli used to test the chicks. Consequently, the chicks and ViTs were trained the same data and tested the same task, allowing for direct comparison their learning abilities.
1.1 Related Work
Our study uses a "digital twin" approach for comparing the learning abilities of newborn animals and machines [29, 30]. Digital twin experiments involve first selecting target animal study, then creating digital twins (virtual environments) of the animal environments in a video game engine. We then train and test artificial agents in those virtual environments and compare the agents' behavior with the real animals' behavior the target study. By raising and testing animals and machines in the same environments, we can determine whether they spontaneously learn common abilities when trained on the same data distributions. Digital twin experiments thus resemble prior studies that trained Al algorithms egocentric (first-person) images from human babies and children, acquired from head-

---

A
SIMCLR Projectio
Encoder
Training Phase
"CI
MLP Hoad
Artlaad
As Head
Head!
Aftroedd
lieed
B
leed
Ant lead
Headi
Transfomer
Transformor
255
Flattened Patches
CLS, Token
Postiooal Emberding
Figure 2: (A) View-invariant object recognition task. Newborn chicks and ViTs were trained in an impoverished visual environment (containing one object seen from a single viewpoint range), then tested on their ability to recognize that object across novel views. (B) ViT-CoT leverages the temporal structure of embodied data streams. Learning occurs by making successive views (images seen in 300 ms temporal window) more similar in the embedding space. (C) The VIT architecture. The image is first divided into smaller 8x8 patches and then reshaped into sequence of flattened patches. A learnable positional embedding is added to each flattened patch, and a class token (CLS _oken) is added to the sequence. The resulting embedding then sequentially processed by transformer blocks while also being analyzed in parallel by attention heads, which generate attention filters shown next to each head. The learned representation of the image is adjusted based on a contrastive learning through time loss function.
mounted cameras to capture the child's natural, everyday visual experience [3, 12, 40, 71J. Digital twin studies extend this conceptual approach to newborn animals raised in controlled environments, allowing for tight control over the visual diet available to the animals and machines.
Lee and colleagues [29, 30] used the digital twin approach to compare the learning abilities of newborn chicks and CNNs on a challenging view-invariant object recognition task (Figure 2A). They found that when CNNs were trained "through the eyes" of newborn chicks, the CNNs learned to solve the same hungry than chicks: both could learn robust object features from training data of a single object.
recognition task as the chicks [57]. Thus, CNNs were not more data
One possibility that CNNs can learn robust object features from impoverished visual environments because CNNs have strong architectural inductive bias. CNNs' convolutional operations reflect the spatial structure of natural images, including local connectivity, parameter sharing, and hierarchical structure [28]. This architecture is directly inspired by neurophysiological observations taken from biological visual systems, including a restricted connectivity pattern that resembles the receptive field organization found in the primate visual cortex [16. 24, 69]. CNNs' spatial inductive bias allows them to generalize well from small datasets and learn useful feature hierarchies that capture the structure of visual images. This spatial bias might also explain why CNNs are not more data hungry than newborn chicks (i.e., there a strong inductive bias supporting visual learning).
ViTs do not have this spatial bias. Rather, ViTs learn through flexible (learned) allocation of attention that does not assume any spatial structure. Thus, VITs less constrained by a spatial inductive bias. This flexibility allows ViTs to learn more abstract and generalizable features than CNNs, but might also make ViTs more data hungry than newborn brains. Indeed, transformers are often regarded as "data-driven learners" that rely heavily on large-scale datasets for training [21, 32, 35, 54].
However, it is an open question whether ViTs are more data hungry than newborn visual systems. For example, [6] report that is it possible to train ViTs with limited data, indicating that ViTs might not be data hungry previously thought. That said, the consensus in the field remains that ViTs are data hungry models, especially relative to CNNs [27].
To directly test whether ViT' are more data hungry than newborn visual systems, we performed digital twin experiments on two ViT algorithms. First, we developed new self-supervised ViT algorithm (Vision Transformer with Contrastive Learning through Time: "ViT-CoT") that learns
3

---

by leveraging the temporal structure of natural visual experience, using a time-basee contrastive learning objective (Figure 2B). ViT-CoT learns without image labels (like newborn chicks) and without artificial image augmentations. The algorithm (Figure 2C) contrasts temporally adjacent instances (positive examples) against non-adjacent instances (negative examples), thereby learning representations that capture the ynderlying dynamics, context, and patterns across time. We used this contrastive learning through time approach because it has shown promising results with CNN architectures [1, 44] and because there is extensive evidence that biological vision systems leverage time to build enduring object representations [11, 31, 37, 38, 51-53J.
Second, to investigate whether our results would generalize across different VIT algorithms, we also trained and tested Video Masked Autoencoders (VideoMAE) [15, 23, 49]. VideoMAE (Figure 4A) contains an encoder-decoder architecture and learns temporal correlations between consecutive frames. VideoMAF encodes spatial and temporal dependencies by masking a subset of space-time patches in the training data and learning to reconstruct those patches. Following prior work, we used a high (90%) masking ratio.
Both ViT algorithms use time as teaching signal. This is important because newborn chicks rely heavily on temporal information when parsing objects from backgrounds, binding color and shape features into integrated object representations, building view-invariant object representations, and building view-invariant face representations [36, 60 60-62, 56, 68]. Newbor chicks, like ViT-CoT and VideoMAE, leverage temporal signals to learn how to see.
2 Methods
2.1 Animal experiments & stimuli
We used newborn chicks as an animal model because chicks can be raised in strictly controlled environments from the onset of vision. This allowed us to control all of the visual experiences (training data) acquired by the animal: an essential requirement for directly comparing the learning abilitics of animals and machines. Moreover, chicks can inform our understanding of human vision because the avian brain has similar cells and circuitry mammalian brains [20, 25, 26]. Avian brains also have a similar large-scale organization as mammalian brains, including a hicrarchy of sensory information processing, hippocampal regions, and prefrontal areas.
We focused on the view-invariant object recognition task and behavioral data reported in Wood [57]. In the study, chicks were hatched in darkness, then raised singly in automated controlled-rearing chambers that measured each chick's behavior continuously (24/7) during the first two weeks of life. The chambers were equipped with two display walls (LCD monitors) for displaying object stimuli, As illustrated in Figure (step 1), the chambers did not contain any objects other than the virtual objects projected on the display walls. Thus, the chambers provided full control over all visual object experiences from the onset of vision.
During the training phase, chicks were reared in an environment containing a single 3D object rotating through a 60" viewpoint range. This virtual object was the only object in the chick's visual environment. We modeled the virtual objects after a previous study that tested for invariant object recognition in adult rats [72]. These objects are well designed for studying view-invariant recognition because changing the viewpoint of the objects can produce greater within-object image differences than changing the identity of the object. As a result, recognizing these objects from novel views requires an animal (or machine) to learn invariant object representations that generalize across large, novel, and complex changes in the object's 's appearance. The chicks were raised in this environment for 1 week, allowing the critical period on filial imprinting to close.
During the test phase (second week), the chicks were tested on their ability to recognize their imprinted object across novel viewpoint changes. On each test trial, the imprinted object appeared on one display wall (from a novel view) and an unfamiliar object appeared on the opposite display wall. Test trials were scored as "correct" when the chicks spent a greater proportion of time with their imprinted object and "incorrect" when the chicks spent a greater proportion of time with the unfamiliar object. Wood [57] collected hundreds of test trials from each chick by leveraging automated stimuli presentation and behavioral coding. As a result, the study produced data with a high signal-to -oo--oise ratio [67].

---

The chicks performed well on the task when the object was shown from the familiar view and from all 11 novel views [57]. Despite being raised in impoverished visual environments, the chicks successfully learned view-invariant representations that generalized across substantial variation in the object's appearance. Thus, newborn visual systems can build robust object representations from training data of single object seen from limited range of views.
2.2 Simulating the training data available to chicks
To mimic the visual experiences of the chicks in the controlled-reaue rreaiing experiments, we constructed an image dataset (Figure 1, step 2) by sampling visual observations from an agent moving through a virtual controlled-rearieee engine. The agent received visual input (64x64 pixel resolution images) through camera attached to its head. The agent could move forward or backward and rotate left right. The agent could also move its head along the three axes of rotation (tilt, yaw, and roll) to self-augment data akin to newborn chicks. We collected 80,000 images from each of the four rearing conditions presented to the chicks. We sampled the images at a rate of 10 frames/s.
chamber. The virtual chamber and agent were created with the Unity game
gorward-facing
2.3 Contrastive learning through time in VITs
Studies of newborn perception indicate that newborn animals learn how to see through eef-suueee learning, using time teaching signal [36, 60-62, 56, 64, 68). Thus, to compare the learning abilities of newborn chicks and ViTs, we developed new self-supervised VIT algorithm (ViT-CoT, Figure 2C) that learns through time-based data augmentations (contrastive learning through time). Specifically, for each condition, we initialized the ViT-CoT architecture with three different seeds. Each seed was trained for 100 epochs using batch size of 128. Throughout the training phase, each image (64x64 pixels) was transformed into 8x8 patches, which were then flattened into a single vector. A randomly initialized positional embedding and cls jtskekkenn were then added to this flattened vector before being passed to the ViT encoder. The randomly initialized positional embedding is a trainable parameter that allows the model to learn and capture the spatial information from the input images upon training. The ViT-CoT loss function was:
exp(sim (2,, Z+41)/T) + (1p
loss lllog
2N
exp(sim(z1, =k)/T)
where is the number of samples, 2, Z,+1, and 4,2 are the embeddings of the consecutive frames, and is the temperature parameter. We used the value of 0.5 for the temperature parameter.
To approximate the temporal learning window of biological visual systems, the ViT-CoT models had a learning window of three frames (i.e., 300 ms). This 300-ms learning window is based on the observation that neurons fire for 100-400 ms after the presentation of an image, providing a time window for associating subsequent images [34, 43]. The chicks in Wood [57] were reared in one of four possible environments, so we trained each ViT-CoT digital twin of one of these four environments (Figure 1, step 3). Thus, the models and chicks had access the same training data for learning visual representations.
2.4 VIT-CoT Architecture Variation
To explore whether ViT-CoTs of different architecture sizes are more data hungry than newborn chicks, we systematically varied the number of attention heads and layers in the models. We built ViT-CoTs with 1, 3. 6, or attention heads and layers. See SI Table for architecture details.
2.5 Temporal learning in VideoMAEs
VideoMAEs consider an extra dimension of time (t) to leverage the temporal relationships between consecutive frames. We built small VideoMAE encoder-decoder model with 3 attention heads and layers in the encoder while having attention head and a single layer in the decoder. See Section 6.4 for training details.
5

---

2.6 Temporal learning in CNNs (SimCLR-CLTT)
Finally, to directly compare ViTs with CNNs, we also trained and tested CNNs. We used the same contrastive learning through time objective function as the ViT-CoT model. Specifically, we used the "SimCLR- R-CTT"" CNN algorithm [44] to train 10-layer CNNs under each of the four rearing conditions presented to the chicks and ViT's, See SI Section 6.5 for architecture and training details.
2.7 Training Data Variation
A leading theory of biological learning is that newborn animals learn how by densely sampling the visual environment and encoding statistical regularities in proximal retinal images [13, 22, 47]. By visually exploring the environment and iteratively adjusting connection weights using temporal learning window, animals might gradually build accurate representations of distal scene variables (e.g., objects), without needing hardcoded knowledge of how proximal image features change as a function of distal scene variables. To explore whether dense sampling allows ViT-CoT's to learn view-invariant object representations in impoverished visual environments, we systematically varied the number of images used to train the ViT-CoTs. We trained the different-sized architectures (1, 3, 6, or attention heads/layers) on 0 images (untrained), 1.25k images, 10k images, and 80k images. By training different sized ViT-CoTs on varying numbers of images, we could explore whether larger ViT ToT-CoTs are more data hungry than smaller ViT-CoTss in the embodied visual learning contexts faced by newborn chicks. We used 80k images to train the VideoMAEs and CNNs.
2.8 Supervised linear evaluation
We evaluated the classification performance of the models using the same recognition task that was presented to the chicks. Task performance was assessed by adding single fully connected linear readout layer top of the last layer of each trained backbone and then performing supervised training only on the parameters of that readout layer on the binary object classification task. The linear readout layers were optimized for binary cross-entropy loss.
For the linear evaluation, we collected 11,000 simulated images from an agent moving through the virtual chamber for each of the 24 object/viewpoint combinations (2 objects 12 viewpoints). The object identities were used as the ground-truie labels.
To evaluate whether the learned representations could generalize across novel viewpoints, we sys- tematically varied the number of viewpoint ranges used to train (Nurin)) and test (Nees) the linear classifiers, while holding the number of training images (11,000) constant. We used two different training and test splits, as described below:
Ntrain = 11; Nrest = 1: We first divided the dataset into 12 folds such that each fold contained images of each object rotating through I viewpoint range. The linear classifiers were cross- validated by training on 11 folds (11 viewpoint ranges, 1 1,000 images each) and testing on the held-out fold (1 viewpoint range: 1,000 images).
Ntrain = 1; Ntest 11: In this condition, we inverted the ratio of the training and validation viewpoints. The linear classifiers were trained using only viewpoint range (with 11,000 images) and tested on the remaining 11 viewpoint ranges (1,000 images per viewpoint).
For each linear classifier condition, transfer performance was evaluated by first fitting the parameters of the linear readout layer on the training set and then measuring classification accuracy the held-out test set. We report average cross-validated performance on the held-out images not used to train the linear readout layer.
3 Results
Figure 3A shows the view-invariant object recognition performance of the ViT-CoTs. We also report the performance of the newborn chicks from [57]. All of the ViT-CoTs performed on par or better than chicks when the linear classifiers were trained on viewpoint ranges (Nmin = 11). We observed similar performance for the small, medium, and large ViT-CoTss (3, 6, and 9 attention heads/layers, respectively). However, we observed drop in performance for the smallest ViT-CoT (with single attention head/layer).

---

C
Visualization of embedding space
Architecture attention headsiblocks. training images)
VI-3H
VI-H
VISHH
B
Change rocogrition accuraiy 26
Lundien of numberr trennnn imagges
Thaliigg Sample
Figure 3: (A) Comparing the view-invariant recognition performance of newborn chicks and ViTs. The red horizontal line shows the chicks' performance, and the ribbon shows standard error. The bars show the siew-invariant recognition performance for the four ViT architecture sizes, across the four rearing conditions presented to the chicks. Error bars denote standard error. We used Nurain 11 for the linear probe. (B) Impact of the number of training images on diew-invarian recogmition performance. We trained the VīTs datasets containing different numbers of images sampled from the virtual chamber. For all architecture sizes, recognition performance improved systematically after training larger numbers of images. indicating that VITs can learn view-invarian object representations by densely sampling the visual environment. of the representations (for selected images) in the last layer of a VIT, The features systematically organized, capturing information about object identity and other important latent variables (e.g.. viewing position).
(C) Two-dimensional t-SNE embedding
We also found that the number of training images significantly impacted performance (Figure 3B). The untrained ViT' performed the worst, and performance gradually improved when the ViT-CoTs were trained on larger numbers of first-person images from the environment. We observed nearly identical patterns of improvement across the small, medium, and large architecture sizes, indicating that larger ViT-CoTss were not more data hungry than smaller ViT-CoTs.
In the more sparse linear classifier condition (Nmin = 1), the ViT-CoTs performed on par with the chicks (Figure 1A). This indicates that ViT-CoTs can learn efficient and abstract embedding spaces, which can be leveraged by downstream linear classifiers to support generalization across novel views. Remarkably, the linear classifiers still performed well above chance level even when trained on single viewpoins range (akin to chicks). These results indicate that when ViTs are trained using self- supervised learning, they can form linearly separable and easily accessible riew-invarian features. Like chicks, ViTs can learn accessible -nvvriitr object features impoverished environments.
The VideoMAE algorithm (Figure 4A) also performed well on the task (Figure 4B), matching the performance of newborn chicks. Thus, our conclusions are not limited to ViT-CoT. Rather, different ViTs show the same learning outcomes, spontaneously developing view-invariant object features when trained on the first-person views available to newborn chicks.
Next, we examined how the CNNs performed on the task (Figure 5). When equipped with a biologically plausible time-based learning objective (CLTT, Figure 5A), the CNNs succeeded on the view-invariant object recognition task (Figure 5B). This finding extends prior work showing that CNNs can solve this task whent they learn from static spatial features [29, 30]. The CNNs did outperform the ViTs (ViT-CoT by 7.3%; VideoMAE by 19.6%), which might be due to the strong architectural inductive bias present in CNNs, but not ViTs. Additional experiments are necessary to determine the specific source of this performance gap. We emphasize that, while ViT' performed lower than CNNs, the ViT' still succeeded on the task, learning view-invariant object features the same impoverished visual environments as newborn chicks.
7

---

A
B
Festur
Figure 4: VideoMAEs can learn niew-invarian object representations in impoverished visual worlds. (A) VideoMAE architecture with the same transformer encoder as the ViT-CoT model. (B) View- invariant recognition performance of VideoMAE across the four rearing conditions and two image resolution conditions (64x64 or 224x224). We used Nimin = 11 for the linear probe. The red horizontal line shows the chicks' performance, and the ribbon shows standard error. Error bars denote standard error. ViT-CoT outperformed VideoMAE by 15.8% when trained/tested on 64x64 resolution images and by 6.5% when trained/tested on 224x224 resolution images. (C) t-SNE embedding of the representations in the last layer of a VideoMAE. The features are systematically organized, capturing information about object identity and other important latent variables (e.g., viewing distance shown here).
3.1
Controls for image resolution and temporal learning window
To check whether our conclusions generalize beyond the 3-frame learning window, we repeated the ViT-CoT experiments with 2-frame versus 3-frame learning window. Performance largely the same across the different learning windows (Figure 3B). To check whether our conclusions generalize beyond the 64x64 image resolution, we repeated the experiments with a higher (224x224) image resolution. Performance was largely the same across the ViT-CoT and Video MAE algorithms (Figure 3C), Finally, we tested whether the models would still succeed on the view-invariant recognition task when both the encoder and the linear classifier were unsupervised. We replaced the supervised linear classifier with an unsupervised decoder. The model scored significantly higher than chance level, which shows that purely unsupervised learning model, in which both the encoder (ViT) and decoder are trained without any supervised signals, can learn to solve the same view- hew-invariant recognition task as newborn chicks when trained 'through the eyes' of chicks (see SI Section 6.6 for details).
3.2 Visualizing VITs representations
We visualized the features using t-distributed stochastic neighbor embedding (t-SNE), which does not involve using any labeled images. We used t-SNE to visualize the representations in the last layer of the ViT-CoTss (Figure 3C), VideoMAEs (Figure 4C), and CNNs (Figure SC). The features were systematically organized, capturing information about object identity and other important latent variables (e.g., viewing position). These results are consistent with findings that newborn chicks spontaneously encode information about both the identity and viewpoint of objects [59, 63).
We also created saliency heatmaps from the last layer of the ViT-CoTs (Figure 6). To generate each heatmap, we provided a ViT-CoT with a sample image as input. Then, we determined how much each pixel in the input image produced activation in each attention filter. Finally, for each attention filter, we color-coded each pixel in the input image according to how much it activated the attention filter. Visual inspection indicated that the attention heads became specialized for different features (e.g., whole objects, object parts, floor, depth). The attention heads' specialization varied based on the number of attention heads in the architecture. For models with more attention heads, single attention head responds to fewer features, compared to a model with fewer attention heads (where each attention head responds to more features). This explains the drop in performance for the single attention head network, which used a single attention head to attend to all the features of the environment. These analyses indicate that ViTs can spontaneously learn enduring visual representations when trained on the embodied data streams available to newborn animals.
4 Discussion
We performed parallel controlled-rearin experiments on newborn chicks and self-supervised vision transformers. Using digital twin approach, we trained chicks and ViTs in the same visual environ-
8

---

A
B
C
Figure 5: (A) Diagram of SimCLR-CLTY (CNN) model. (B)
recognition performance
of CNNs across the four rearing conditions and two temporal learning windows (2 frames or 3 frames). We used Miain = 11 for the linear probe. Error bars denote standard error. The CNNs outperformed ViT-CoT by 7.3%. (C) t-SNE embedding of the representations in the last layer of a CNN trained via contrastive learning through time. The features are systematically organized, capturing information about object identity and other latent variables.
ments and tested their object recognition performance with the same stimuli and task. Our results show that ViTs can spontancously learn visual training data chicks. For both chicks and ViTs, impoverished environments (with single object) contain sufficient training data for learning view-invariant object features. These results have important implications for understanding minds and machines alike.
object features when provided with the same
4.1
Are transformers more data hungry than brains?
widely accepted critique of transformers is that they are more data hungry than brains. This critique stems from the observation that transformers are typically trained massive volumes of data, which seems excessive compared to the more sparse experiences available to newborn animals. We suggest that the embodied visual data streams acquired by newborn animals are rich in their own right. During everyday experience, animals spontaneously engage in self- leff-generated data augmentation, acquiring large numbers of unique object images from diverse body positions and orientations. Our results show that self-supervised ViTs can leverage these embodied data streams to learn high-level object features, even in impoverished visual environments containing a single object. Thus, ViTs do not appear to be more data hungry than newborn visual systems, indicating that the gulf between visual learning in brains and transformers may not be as great as previously thought. We support our claim that ViTs are data-efficient learners by showing that two variants of ViTs can both leverage temporal learning to learn like newborn animals.
The field does not have well established procedures for comparing the number of training images across animals and machines, which is why we focused controlling the visual environment available to chicks and ViT's, rather than controlling the number of training images per se. However, while cannot ensure that the number of samples matched between the animals and ViTs, we can make a rough comparison based on the rate of learning in biological systems. Researchers estimate that biological visual systems perform iterative, predictive error-driven learning every 100 ms (corresponding to the 10 Hz alpha frequency originating from deep cortical layers; [39]. If we assume that newborns spend about half their time sleeping, this would correspond 430,000 images in their first day. Thus, biological visual systems have ample opportunity to learn from "big data."
4.2 Origins of object recognition
Our results provide computationally explicit evidence that generic learning mechanism (ViT), paired with a biologically inspired learning objective (contrastive learning through time), is sufficient to reproduce animal-like object recognition when the system is trained on the embodied data streams available to newborn animals. As such, our results inform the classic nature/nurture debate in the mind sciences regarding the sufficiency of different learning mechanisms for driving biological intelligence. Some researchers have proposed that intelligence emerges from a single generic learning mechanism [33, 41, 48), whereas others have proposed that intelligence emerges from collection of innate, domain-specific learning systems [7, 46, 10, 17J. Our results show that-for the case of object recog- nition-a a generic learning system (with no hardcoded knowledge of objects or space) sufficient to learn view-invariant object representations. In fact, ViTs can learn accurate object representations

---

agent stimuli
ViT-9H Attention Heads
H1 H2
H
H4
H
8
Figure 6: Heatmaps generated from the ViT-9H model. We created saliency heatmaps using the ViT-9H model's last transformer block for all nine attention heads (HO H8). The first column shows the agent's position when capturing the first-person image shown in the second column. The subsequent nine columns show the heatmap for each attention head of the ViT-9H model.
in the impoverished environments faced by chicks in cntrrolee-rearingg studies, highlighting both the learning power of ViTs and the rich informational content in embodied data streams.
These results raise an intriguing hypothesis for the origins of visual intelligence: core visual abilities (such as object recognition) might naturally emerge from (1) an attention- baaee sequence learning sys- tem that (2) adapts to the embodied data streams produced by newborn animals. Under this hypothesis, visual intelligence emerges from single innate sequence learning system that rapidly develops core domain-specinf capacities when animals interact with the world during early development. Future re- search could test this 'single system' hypothesis by exploring whether self-supervised ViTs learn other core visual abilities when they are trained "through the eyes" of newborn animals. Ultimately, parallel controlled-rearing studies of newborn animals and machines can determine which learning mech- anisms and experiences are sufficient to produce the rapid and flexible learning of biological systems.
4.3 Limitations
One limitation of the current study that we only tested ViTs across four rearing conditions. Future experiments could test ViTs across a wider range of experimental results, adopting an integrative benchmarking approach used computational neuroscience [45]. second limitation of our study is that the models were trained passively, learning from batches of images in a pre-specified order. This contrasts with the active learning of newborn animals, who interact with their environment to produce their training data. By choosing where to go and what to look biological systems generate their own curriculum to suit their current pedagogical needs. Future research could close this gap between animals and machines by embodying VITs in artificial agents that collect their own training data from the environment.
4.4 Broader Impact
This project could lead to more powerful Al systems. Today's Al technologies are impressive, but their domain specificity and reliance on vast numbers of labeled examples are obvious limitations. Conversely, biological brains capable of flexible and rapid learning from birth. By reverse engineering the core learning mechanisms in newborn brains, we can build "naturally intelligent" learning systems. Naturally intelligent learning algorithms are an untapped goldmine for inspiring the next generation of artificial intelligence and machine learning systems.
10

---

5 Acknowledgements
Funded by NSF CAREER grant (BCS-1351892, JNW), James McDonnell Foundation Understanding Human Cognition Scholar Award (JNW), and Facebook Artificial Intelligence Research award (JNW).
References
[1] Arthur Aubret, Markus Ernst, Céline Teulière, and Jochen Triesch. Time to augment self- supervised visual representation learning. In The Eleventh International Conference on Learning Representations, 2022. doi: 00.48550/arXiv.2207.2 13492..
[2] Vladislav Ayzenberg and Stella Lourenco. Perception of an object's global shape is best described by model of skeletal structure in human infants. elife, 11:e74943, 2022.
[3] Sven Bambach, David Crandall, Linda Smith, and Chen Yu. Toddler-inspired visual object learning. Advances in neural information processing systems, 31, 2018.
[4] Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet-
Ik. arXiv preprint arXiv:2205.01580 2022.
[5] Meiqiao Bi, Minghua Wang, Zhi Li, and Danfeng Hong. Vision transformer with contrastive learning for remote sensing image scene classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remole Semsing, 16:738-749, 2022.
[6] Yun-Har Cao. Hao Yu, and Jianxin Wu. Training vision transformers with only 2040 images. In European Conference on Computer Vision, pages 220-237. Springer, 2022.
[7] Susan Carey. The Origin of Concepts. Oxford University Press, 08 2009. doi: 10. 1093//// 0so/978019536777 000000999668...... w00.66388000...00 000.0011.
[8] Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022.
[9] Wuyang Chen, Xianzhi Du, Fan Yang, Lucas Beyer, Xiaohua Zhai, Tsung- Yi Lin, Huizhong Chen, Jing Li, Xiaodan Song, Zhangyang Wang, et al. A simple single-scale vision transformer for object localization and instance segmentation. arXiv preprint arXiv:2112.09747, 2021.
[10] Noam Chomsky. review of skinner's verbal behavior. Language, 35:26-57, 1959. doi: 10.2307/411334
[11] David D Cox, Philip Meier, Nadja Oertelt, and James J DiCarlo. "breaking position-invariant object recognition. Nature neuroscience, 8(9): 145-1147, 2005.
[12] Guy Davidson, A Emin Orhan, and Brenden M Lake. Spatial relation categorization in infants and deep neural networks. Available at SSRN 4442327, 2023.
[13] Adrien Doerig, Rowan P Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W Lindsay, Konrad P Kording, Talia Konkle, Marcel AJ Van Gerven, Nikolaus Kriegeskorte, et al. The neuroconmectionist research programme. Nature Reviews Neuroscience, pages 1-20, 2023.
[14] Heming Du, Xin Yu, and Liang Zheng. Vtnet: Visual transformer network for object goal
navigation. arXiv preprint arXiv: 1e..21005094447 2021.
[15] Christoph Feichtenhofer, Yanghao Li, and Kaiming He. Masked autoencoders spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958, 2022.
[16] Kunihiko Fukushima. A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biol, Cybern, 36: 193-222, 1980.
[17] Charles R Gallistel. The organization of learning. The MIT Press, 1990.
[18] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885- 23899, 2021.
11

---

[19] Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, al. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):36-3380 2022.
[20] Onur Güntürkün and Thomas Bugnyar. Cognition without cortex. Trends in cognitive sciences, 20(4):291-303, 2016.
[21] A Hassani, S Walton, N Shah, A Abuduweili, J Li, and H Shi. Escaping the big data paradigm with compact transformers. arxiv 2021. arXiv preprint arXiv:2104.05704.
[22] Uri Hasson, Samuel A Nastase, and Ariel Goldstein. Direct fit to nature: an evolutionary perspective on biological and artificial neural networks. Neuron, 105(3):416 2020.
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVE conference on computer vision and puttern recognition, pages 16000-16009, 2022.
[24] David H. Hubel and Torsten N. Wiesel. Receptive fields and functional architecture of monkey striate cortex. Journal of Physiology, 195(1):215-243, March 1968. doi: 10.1 10.p113/jphysiol. 1968. sp008455.
[25] Erich D Jarvis, Onur Güntürkün, Laura Bruce, András Csillag, Harvey Karten, Wayne Kuenzel, Loreta Medina, George Paxinos, David J Perkel, Toru Shimizu, et al. Avian brains and new understanding of vertebrate brain evolution. Nature Reviews Neuroscience, 6(2):151-159, 2005.
[26] Harvey Karten. Neocortical evolution: neuronal circuits arise independently of lamination. Current Biology, 23(1):R12-R15, 2013.
[27] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey, ACM computing surveys (CSUR), 54 (10s):1 -1, 2022.
[28] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 4214555):443-444, 2015.
[29] Donsuk Lee, Pranav Gujarathi, and Justin N Wood. Controlled-rearing studies of newborn chicks and deep neural networks. arXiv preprint arXiv:2112.06106, 2021.
[30] Donsuk Lee, Denizhan Pak, and Justin N Wood. Modeling object recognition newborn chicks using deep neural networks. arXiv preprint arXiv:2106.07/85, 2021.
[31] Nuo Li and James DiCarlo. Unsupervised natural experience rapidly alters invariant object representation visual cortex. science, 15855) 2008
[32] Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco Nadai. Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems, 34:23818 0773818-23833 8-33800 2021.
[33] John Locke. An essay concerning human understanding, 1690. 1948.
[34] Artur Luczak, Bruce L McNaughton, and Kenneth D Harris. Packet-basse communication in the cortex. Nature Reviews Neuroscience, 16(12):745-755, 2015.
[35] Jiawei Mao, Honggu Zhou, Xuesong Yin, Yuanqi Chang Xu, and Binling Nie Rui. Masked au- toencoders is an effective solution to transformer data-hungry. arXiv preprint 17.007221..006677 2022.
[36] Giulio Matteucci and Davide Zoccolan. Unsupervised experience with temporal continuity of the visual environment causally involved in the development of v1 complex cells. Science Advances, 6(22):caba3742, 2020.
[37] Travis Meyer and Carl R Olson. Statistical learning of visual transitions in monkey infer- otemporal cortex. Proceedings of the National Academy of Sciences, 1990499)1400 2011.
12

---

[38] Yasushi Miyashita. Neuronal correlate of visual associative long-term memory in the primate temporal cortex. Nature, 335(6193) 1988.
[39] Randall C O'Reellyy, Jacob L Russin, Maryam Zolfaghar, and John Rohrlich. Deep predictive learning in neocortex and pulvinar. Journal of Cognitive Neuroscience, 13(6))1158-11) 1196, 2021.
[40] Emin Orhan, Vaibhav Gupta, and Brenden M Lake. Self-supervised learning through the eyes of child. Advances in Neural Information Processing Systems, 33:9960- 99760099977,11 2020.
[41] Jean Piaget. The construction of reality in the child. Journal of Consulting Psychology, 19(1): 77, 1955.
[42] Aditya Prasad, Samantha MW Wood, and Justin N Wood. Using automated controlled rearing to explore the origins of object permanence. Developmental science, 22(3)=e1279,, 2019.
[43] Edmund T Rolls and Martin Tovee. Processing speed in the cerebral cortex and the neurophys- iology of visual masking. Proceedings of the Royal Society of London. Series B: Biological Sciences, 257(1148):9--1, 1994.
[44] Felix Schneider, Xia Xu, Markus Ernst, Zhengyang Yu, and Jochen Triesch. Contrastive learning through time. In SVRIIM 2021 Workshop@ NeurIPS, 2021.
[45] Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45)e220564611 18, 2021.
[46] Elizabeth S Spelke. What Babies Know: Core Knowledge and Composition Volume 1, volume 1. Oxford University Press, 2022.
[47] Katherine Storrs, Tim C Kietzmann, Alexander Walther, Johannes Mehrer, and Nikolaus Kriegeskorte. Diverse deep neural networks all predict human inferior temporal cortex well, after training and fitting. Journal of cognitive neuroscience, 33(10);0044-206, 2021.
[48] Esther Thelen and Linda B Smith. A dynamic systems approach to the development of cognition and action. MIT press, 1994.
[49] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 15078-1010,,,, 2022.
[50] S Tuli, I Dasgupta, E Grant, and TL Griffiths. Are convolutional neural networks transformers more like human vision? arxiv 2021. arXiv preprint arXiv:2105.07197.
[51] Quoc C Vuong and Michael Tarr. Rotation direction affects object recognition. Vision Research, 44(14)1717-1730, 2004.
[52] Guy Wallis and Heinrich H Bülthoff. Effects of temporal association on recognition memory. Proceedings of the National Academy of Sciences, 98(8):4800 4804, 2001.
[53] Guy Wallis, Benjamin T Backus, Michael Langer, Gesche Huebner, and Heinrich Bülthoff. Learning illumination-and orientation-invariin representations of objects through temporal association. Journal of vision, 9(7)666,, 2009.
[54] Wen Wang, Jing Zhang, Yang Cao, Yongliang Shen, and Dacheng Tao. Towards data-efficient detection transformers. In European conference on computer vision, pages 88-105. Springer, 2022.
[55] James CR Whittington, David McCaffary, Jacob JW Bakermans, and Timothy EJ Behrens. How to build cognitive map. Nature neuroscience,
2022.
[56] Justin Wood, Aditya Prasad, Jason Goldman, and Samantha Wood. Enhanced learning of natural visual sequences in newborn chicks. Animal Cognition, 19, 07 2016.
13

---

[57] Justin N Wood. Newborn chickens generate invariant object representations at the onset of visual object experience. Proceedings of the National Academy of Sciences, 110:34):1400-14055 2013.
[58] Justin N. Wood. Newly hatched chicks solve the visual binding problem. Psychological Science, 25(7):1475-1493) 2014.
[59] Justin N Wood. Characterizing the information content of a newly hatched chick's first visual object representation. Developmental science, 18(2):194 205, 2015.
[60] Justin N. Wood. A smoothness constraint on the development of object recognition. Cognition, 153:140-145, 2016.
[61] Justin N. Wood and Samantha M. W. Wood. The development of newborn object recognition in fast and slow visual worlds. Proceedings of the Royal Society B: Biological Sciences, 283 (1829), 2016.
[62] Justin N. Wood and Samantha M. W. Wood. The development of invariant object recognition requires visual experience with temporally smooth objects. Cognitive Science, 42(4): 2018.
[63] Justin N Wood and Samantha MW Wood. One-shot learning of view-invariant object represen- tations in newborn chicks. Cognition, 199:104192, 2020.
[64] Samantha M. W. Wood and Justin N. Wood. One-shot object parsing in newborn chicks. Journal of experimental psychology. General, 2021.
[65] Samantha MW Wood and Justin Wood. A chicken model for studying the emergence of invariant object recognition. Frontiers in Neural Circuits, 9:7, 2015.
[66] Samantha MW Wood and Justin N Wood. Face recognition in newly hatched chicks at the onset of vision. Journal of Experimental Psychology: Animal Learning and Cognition, 41(2):206, 2015.
[67] Samantha MW Wood and Justin N Wood. Using automation combat the replication crisis: A case study from controlled-rearing studies of newborn chicks. Infant Behavior and Development, 57:101329, 2019.
[68] Samantha MW Wood and Justin N Wood. Distorting face representations in newborn brains. Cognitive Science. e5 13011, 2021.
[69] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 11 (11)))861198662 2014.
[70] Jiewen Yang, Xingbo Dong, Liujun Liu, Chao Zhang, Jiajun Shen, and Dahai Yu. Recurring the transformer for video action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14063-14000 2022.
[71] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 2001)))2014196111 2021.
[72] Davide Zoccolan, Nadja Oertelt, James J DiCarlo, and David D Cox. rodent model for the study of invariant visual object recognition. Proceedings of the National Academy of Sciences, 106(21):8748-44 8753, 2009.
14

---

Supplementary Material
6.1 Data generation
To simulate the training and testing data available to chicks, we created a virtual animal chamber in the Unity Game Engine. The virtual chamber was equipped with two 19" LCD monitors situated on opposite sides and accompanied by two alternating white walls. The LCD monitors were used (height), positioned the center
to display the virtual objects of size 8 em (length) x white display background. The floor of the virtual chamber was constructed with wire mesh and had transparent holes hold food and water on one side of the chamber. The virtual chamber measured 66 cm (length), 42 cm (width), and 69 cm (height).
We developed virtual chick agent to simulate the visual experiences of newborn chicks. The virtual agent had a height of 3.5 units and a length of 1.2 units. To monitor the agent's 's behavior, the virtual chamber was equipped with two invisible cameras. The first camera, positioned in the chamber's ceiling, captured the top-view the agent, while the second camera, placed in the agent's 's head, recorded first-person RGB images (64 64 resolution) as the agent moved throughout the chamber.
The agent picked a random location inside the chamber and moved that position at the rate of 1.5 units/s. During this movement, the agent directed its visual attention towards the object projected on the LCD monitor, ensuring a centered focus. Once the agent reached the destination point, randomly moved its head along the three axes, rotating 60% on each axis, to naturally augment the data (Figure IB). This complete random sequence of head rotations lasted for 9.5 seconds. The agent repeated this cycle until they had captured 80,000 frst-person images in the rearing condition. We repeated this data simulation cycle for each of the 4 rearing conditions.
The same procedure was repeated to gather the test samples; however, only 11,000 images were captured for each of the 12 viewpoint ranges. These test images were used to train and evaluate the linear classifiers.
6.2 Heatmap generation
To visualize the internal working of the attention heads, we generated saliency heatmaps and heatmap videos from the last block of the transformer. Section 3.2 describes the steps required to generate a heatmap for a sample image, To create heatmap videos, we followed these steps. First, the agent moved inside the chamber and captured images at 10 frames/s. Second, used those images and created saliency heatmaps in a sequential order. Third, we concatenated these heatmaps in a series
A
Objjett Recoontion Pertonneeffoonn peformmncc petfmmrne
120%
Train: viewpoint
B
Tilt Right
Down
Tilt
Tast: novel viewpoints
75%
comeec
50%
rtance Gerfrrrrce
25%
Right
Supplementary Figure I: (A) Comparing the view-invariant recognition performance of newborn chicks and ViTs in the sparse linear classifier condition (1 training viewpoint). The red horizontal line shows the chicks' performance, with the ribbon representing standard error. The bars show the view-invariant recognition performance for the four ViT architecture sizes, across the four rearing conditions presented to the chicks. Error bars denote standard error. In this sparse linear classifier condition, the linear classifiers are trained on the same viewpoint as the ViT encoder and tested on the remaining 11 novel viewpoints. (B) The virtual agent's head rotations, which provided natural data augmentations. Upon reaching the destination point inside the chamber, the agent randomly rotated its head across the three axes (tilt, left/right, and up/down).
15

---

H0
H2
HO
H1
H2
H0
H1
H2
VIT-3H model
HO
H1
H2
H0
H1
H2
Background-centric head (HD)
Object-centric head (H2)
Supplementary Figure 2: Visualizing the attention patterns of individual attention heads using saliency heatmaps derived from the ViT-3H model. The visualizations were generated by using the simulated first-person images captured by the virtual agent. In the VIT-3H model, (HO-H2) correspond to the attention filters of each of the three heads. The blue dotted line highlights that head (HO) largely focuses on background features, while the red dotted line demonstrates that head (H2) largely attends the object features. This indicates that some attention heads become specialized different parts of the environment, such as objects versus backgrounds.
Table I: ViT-CoT with different architecture sizes.
Model
Parameters
Attention Heads
Transformer Blocks
Batch Size
Epochs
VIT-1H
5.8M
1
128
100
VIT-3H
16.9M
3
3
128
100
ViT-6H
36,4M
6
6
128
100
VIT-9H
59.4M
9
9
128
100
using the OpenCV library to produce a heatmap video. We repeated these steps for all the attention heads of all the different architectures. Figure displays the heatmaps generated by the ViT-3H model, showcasing the visual representations of the stimulus captured from various angles. Some heads become specialized to the object while other become specialized to the background features.
6.3 ViT-CoT Architecture
We systematically increased the number of attention heads and layers to create different architecture sizes ranging from smallest (ViT-1H) to largest (ViT-9H) model (Table 1). The ViT-CoT model con- tains two primary components: the transformer layers and SimCLR projection head. The projection head consists of two linear layers and a ReLU activation function. Both of these components are sequentially attached to each other. The projection head receives the output from the final transformer layer as its input and generates an embedding with shape of 128. The embedding is used to adjust the loss function when training the model. During testing, the output is obtained directly from the final transformer layer, without passing through the projection head.
6.4 VideoMAE Training
To train the VideoMAFs, we first sampled 16 frames from a batch with temporal stride 1. Each frame had dimension of 64x64 and channels (16x64x64). Next, randomly masked 90% of the
16

---

A
B
Beengraton scunnry oevee
Wocegatee acesccy
Conditon
Supplementary Figure 3: (A) Diagram showing the 2-frame versus 3-frame learning windows. (B) View-invariant recognition performance of ViT-CoT across the four rearing conditions and two temporal learning windows (2 frames or frames). Error bars denote standard error. We used Nurin 11 for the linear probe. The models with 3-frame learning window performed marginally better than those with a 2-frame learning window. (C) View-invariant recognition performance of ViT-CoT (3-frame learning window) across the four rearing conditions and two image resolutions (64x64 or 224x224), We used Ntrii = 11 for the linear probe. ViT-CoT performs well with both low and high-resolution images.
frames using cube masks. Each mask had a spatial dimension of 8x8 and a temporal dimension of (2x8x8). The VideoMAE encoder then encoded the visible (non-masked patches) into latent space with a dimension of 512. Finally, the VideoMAE decoder combined the encoded features with the masked patches reconstruct the entire sequence of frames. Each VideoMAE model initialized with three different seeds and trained for 100 epochs using a batch size of 64. We used multi-GPU training across 8 NVIDIA A10 GPUs
To ensure that our findings with Video MAE would generalize across different image sizes and mask- ing sizes, we repeated this process with higher image (16x224x224) and larger masks (2x16x16).
6.5 CNN Training
We utilized the default ResNet- 18 architecture but made it more compact by removing the last two residual blocks, resulting in 10-layer CNN architecture. We employed 'Contrastive Learning through Time' as the teaching signal, similar to ViT-CoT, for training these CNN models. Each model was initialized with three different seeds and trained for 100 epochs with batch size of 512.
6.6 Two-alternative forced-choice task
In our main experiments, the VIT (encoder) was trained an unsupervised manner, but we used supervised lincar classifier (decoder) evaluate the features Iearned by the VIT. We also tested whether we would obtain similar results when we used an unsupervised, rather than supervised, decoder to evaluate the ViTs. Specifically, we used modification of the unsupervised technique described by [2], initially developed to test human babies.
The chicks' preferences in [57] can be conceived as a measure of alignment between the test stimuli and the chick's internal representation of their imprinted object. Chicks will approach stimulus they perceive to be the most similar to their internal representation of their imprinted object (i.e., the stimulus that produces less mismatch, or 'error,' between the stimulus and their representation). To approximate this in silica, we converted each trained VIT model into an autoencoder that was "imprinted" to the same stimulus as the chicks and tested on the same stimuli as the chicks. An autoencoder is trained to reconstruct the original stimulus from lower dimensional representation, and the reconstruction loss is higher when there is a larger mismatch between a given stimulus and the internal representation.
We converted the models into autoeneoders by attaching a simple fully connected downstream decoder to the (trained and frozen) ViT; then we performed unsupervised training on the decoder, using the same images that were used to train the VIT encoder. Consequently, both the encoder and decoder were only trained on images of single object shown from single viewpoint range, akin to the chicks. Once the decoder was trained, we used the output from the decoder to quantify how similar each test stimulus was to the VIT's internal representation.
17

---

The chicks were tested using two-alternative forced-choice test (2AFC). To approximate the 2AFC task, we fed two object images into the autoencoder (i.e., ViT decoder), then measured the error signal for each image. If the error signal was smaller for the imprinted object than the novel object, the model was scored 'correct." If the error signal was larger for the imprinted object than the novel object, the model was scored as 'incorrect." We evaluated the models across all 12 of the viewpoint ranges.
The model scored 62.1%. 1%. which is significantly higher than chance level (50%): (1, = 576) 34.03, p .000000005. This result shows that a purely unsupervised learning model, in which both the encoder (ViT) and decoder are trained without any supervised signals, can learn to solve the same view-invariant recognition task newborn chicks when trained 'through the eyes' of chicks.
6.7 Data Availability
The code and data needed to reproduce these findings can be found at the following hht/:////sshhbccommuuuuugnnnnno.
18


Reviews:
[{"review_id": "fQJivM9B9Q", "reviewer": "Reviewer_8hgW", "summary": "The authors present a study showcasing, in one specific scenario, a vision transformer can match the visual learning performance of newborn chicks. The setup is as follows. Newborn chicks are raised in a dark enclosure for a week, and given only one visual stimulus from a variety of angles. Then, the chicks are presented with the visual stimulus from new angles, along with different visual stimuli. The chicks show a preference towards the same stimulus from new angles, which means that they have learned an angle-invariant generalization about the stimulus. Vision transformers are shown the same stimulus in a simulated environment, are trained on the simulated images, and are probed for the ability to learn angle-invariant generalizations too.", "questions": "I tried to put my points in \"weaknesses\" as questions.", "limitations": "Limitations are adequately addressed in my opinion.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Although not entirely novel (similar experiments have been done with CNNs and similar results have been shown with them), I do think that this is interesting work, and I did not find any issues with the experimental setup or results. It extends the previous findings from CNNs to transformers to showcase that transformers do not require more data than a newborn animal to learn the same generalization in one specific scenario.", "weaknesses": "Not sure I really buy the 3-frame learning window idea. Does the algorithm actually not work if you only associate two frames with each other instead of three?\n\nDo you say how many parameters your vision transformers have anywhere? This would be really useful to know.\n\nInstead of a simulator, why not run the test with an actual robot driving/walking around? It would make the work more novel and interesting in my opinion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable questions and feedback, which we address below:\n\n> Not sure I really buy the 3-frame learning window idea. Does the algorithm actually not work if you only associate two frames with each other instead of three?\n\nWe agree that the learning window duration is an interesting hyperparameter. We chose a learning window of 3 frames to approximate the temporal learning window in biological visual systems (around 300ms, which corresponds to 3 of our simulated frames). As the Reviewer points out, the learning window may influence the algorithm’s ability to learn visual features. To test this, we replicated our experiments with a 2-frame learning window. We find that the 2-frame learning window also performs well on the task, demonstrating that the algorithm’s success is not dependent on a learning window of 3 frames.\n\n\n> Do you say how many parameters your vision transformers have anywhere? This would be really useful to know.\n\nOur vision transformers have 5.8M (1-head architecture), 16.9M (3-head architecture), 36.4M (6-head architecture), and 59.4M (9-head architecture) trainable parameters, respectively. We provide details on the number of trainable parameters in Table 1 of the Supplementary Materials.\n\n\n> Instead of a simulator, why not run the test with an actual robot driving/walking around? It would make the work more novel and interesting in my opinion.\n\nThis is an excellent idea for future research! In the present study, we focused on whether ViT models can account for the visual processing abilities of newborn animals (which is why we focus on “disembodied” models of vision that do not produce actions). However, we are currently building a platform for rearing embodied virtual agents in “digital twins” of the controlled rearing chambers, which will allow us to train real and artificial animals with the same data and test them with the same tasks. \n\nWe currently use virtual agents (rather than robots) because of their scalability. First, we can run virtual simulations with dozens of agents simultaneously, at faster than real-time speeds. As a result, we can collect more data and test more models. Second, using virtual agents allows our work to scale beyond our lab. We can make “digital twins” of our chambers available to researchers outside of our lab, so that they can replicate our findings and/or test new models. Finally, once we discover core learning algorithms that develop like newborn animals, we will plug our virtual brains into physical robots and test whether policies learned virtually generalize to the real world. "}, {"review_id": "q4Qv8yd11A", "reviewer": "Reviewer_xTn4", "summary": "This article examines the issues of data-hungry between chicks and ViT. For the sake of experimental accuracy, this paper makes a dark room and controls some variables in terms of organisms. In terms of ViT, ViT-CoT is proposed, and data enhancement, embedding and simple modification of the model are carried out. Finally, a comparative experiment was carried out, which proved the experimental results well.", "questions": "On the (129 line, 4 page) “Thus, to compare the learning abilities of newborn chicks and ViTs, we developed a new self-supervised ViT algorithm...... Specifically, ViT-CoT architecture was initialized with three different seeds ……”. This paper uses time-based data augmentations and studies spatial information from the input images. This paper adds a lot of advantages to the training of vit, but the chick does not. For example, when a chick learns a three-dimensional model of an object, it may be naturally strongly correlated with its spatial variation. The chicks may be confused when it sees three-dimensional objects move but their own position in space does not change.\n\nWe all know that the difference between living organisms and computers is that living organisms may encounter fatigue, illness, or other bad conditions during the learning process, resulting in a decline in learning ability. Even without these interfering factors, it still needs to grow and eat by itself. And it is disturbed by the smell, temperature, feather movement, etc. How to control or reduce the occurrence of such problems in this experiment? If the learning ability of organisms is underestimated because of such problems, how to draw that conclusion of this paper?\n\nIn Figure 3, why does the accuracy of the chick not increase with the training time?\n", "limitations": "The paper lacks discussion on the setting of the proposed task and effects of other training methods.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Clear writing and good figures for easy understanding.\nInteresting perspective on comparing newborn and vit.\n", "weaknesses": "There is a big difference between living organisms and computers. So that some settings in the experiment are not enough to rule out the influence of other aspects. Conclusions are somewhat subjective. For example, the newborn chicken is designed by humans and implemented by human, and as far as I learn from this paper, this design can only imitate limited features of living organisms. Also, why chicken can be represented for the “newborn visual system”.\n\nInadequate verification of the appropriateness of the task for addressing the topic issue. As far as I learn from this paper, the final task for ViT and newborn system is to recognize object across novel views, and the objects are composed of virtual and designed geometry element. The difficulty and appropriateness of this task should be studied thus the conclusion can be convinced.\n\nThis paper proposes a new contrastive loss function to train ViT as self-supervised learning. However, the other powerful self-supervised method, i.e., mask some part of the image and train the model to reconstruct the complete image, is not discussed and tested. This leads to insufficient method design and experiment conduction, since the subject of this paper gives no limitation to the training method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and time. We address your critiques below.\n\n*Reviewer raises two concerns: (1) Why use chicks as a model system for studying newborn vision? (2) Our design can only address some features of animals.*\n\nFor (1), our revision will clarify why chicks are optimal for studying newborn visual learning. We used chicks as a model system because chicks can be raised in controlled environments from the onset of vision. This allows us to control all of the visual experiences (training data) acquired by the animal: an essential feature for directly comparing the learning abilities of animals and machines. Moreover, chicks can inform our understanding of human vision because the avian brain has similar cells and circuitry as mammalian brains (Güntürkün & Bugnyar, 2016; Jarvis et al., 2005; Karten, 2013). Avian brains also have a common large-scale organization as mammalian brains, including a hierarchy of sensory information processing, hippocampal regions, and prefrontal areas.\n\nFor (2), we agree that our study only tests specific learning abilities of animals, but this is an essential first step in this research program. Ultimately, we introduce a powerful new experimental approach (the first of its kind) for directly comparing the learning abilities of animals and transformers. It would be interesting for future research to explore whether ViTs can replicate other features of living organisms (e.g., object parsing, face recognition, motor development, navigation, and audition).\n\n*Reviewer argues that we did not verify the appropriateness of the task for addressing the topic issue.*\n\nOur revision will clarify the appropriateness of the task. We modeled the virtual objects and task after a previous study that tested for invariant object recognition in adult rats (Zoccolan et al., 2009). The objects are well designed for studying view-invariant recognition because changing the view of each object produces a greater within-object image difference than changing the identity of the object. Thus, recognizing these particular objects from novel views requires an animal (or model) to learn abstract object features that can generalize across large, novel, and complex changes in the object’s appearance.\n\n*Reviewer argues that we had an insufficient method design because we did not test other ViTs that use masking approaches.*\n\nWe agree that MAEs are an important candidate for unsupervised visual learning. As suggested, we performed new experiments with the VideoMAE model. We trained/tested VideoMAE using the same approach that we used to train/test ViT-CoT. We found that VideoMAE, like ViT-CoT, can perform the task, showing that vision transformers are sufficient to drive the development of animal-like object recognition.\n\n*Reviewer argues that the temporal learning approach is appropriate for ViTs, but not for chicks.*\n\nThe reviewer is concerned that ViT-CoT uses time-based data augmentation, but that chicks instead use learning mechanisms that are correlated with their own position in space. We will revise the manuscript to emphasize that chicks are highly sensitive to temporal information, thereby justifying our use of models that leverage temporal information to learn. For example, newborn chicks use temporal information when parsing objects from backgrounds (Wood & Wood, 2021a), binding color and shape features into integrated object representations (Wood, 2016), building view-invariant object representations (Wood & Wood, 2016; 2018), and building view-invariant face representations (Wood & Wood, 2021b).\n\n*Reviewer argues that animals have needs (e.g., hunger, fatigue) that are not present in the ViT experiments, making comparisons across chicks and ViTs difficult.*\n\nWe agree; it is inevitable that any comparison between humans/animals and machines will open the possibility for fatigue, illness, hunger, etc. to lead to noisy estimates of biological learning. To minimize the impact of these factors, our design uses automation and long test periods. As reviewed in Wood & Wood (2019), this methodology produces data with a high signal-to-noise ratio and strong test-retest reliability across individual chicks.\n\nWe also note that, while many factors may contribute to differences between animals and machines, the critical factor needed to compare learning across animals and machines is that both are provided with the same training data and tested on the same tasks. Currently, no other method comes close to achieving this goal. Accordingly, while other benchmarks can be used to evaluate whether trained models behave like mature animals, these benchmarks cannot be used to determine whether models use similar learning algorithms as animals (because the animals and models learned from different training data). We therefore argue that a major contribution of our paper is to add a new paradigm to the field that can unambiguously reveal whether animals and machines learn in the same way.\n\n*Reviewer wonders why chick performance did not increase across the training period.*\n\nThank you for pointing out the lack of clarity. Fig. 3 shows the performance of ViT-CoTs (not chicks) trained on various numbers of training samples. The red line showing chick performance is provided so readers can easily see the number of training samples needed to reach chick-level performance. We will clarify this in the revision.\n\n*Reviewer argues that the paper lacks discussion on the setting of the proposed task and effects of other training methods.*\n\nTo make the setting of the task clearer, we will direct readers to Wood (2013) for a detailed description of the task. Moreover, to provide a new dataset to the community (and to help clarify the setting of the machine task), we will provide the full set of training and test images that we simulated in the “digital twin” experiments. Finally, to address the effects of other training methods, we also added new experiments using VideoMAEs and CNNs (see rebuttal PDF)."}, {"review_id": "GtRW8p9b9D", "reviewer": "Reviewer_Au5T", "summary": "This study challenges the notion that Vision Transformers (ViTs), which excel in many computer vision benchmarks, require more training data than biological brains. The study involved controlled experiments on both ViTs and newborn chicks in impoverished visual environments, using a video game engine to simulate the chicks' environments and train self-supervised ViTs that use time as a teaching signal, similar to biological systems. The authors found that when trained in conditions similar to those of newborn chicks, ViTs effectively performed the same object recognition tasks, suggesting that ViTs are not necessarily more \"data hungry\" than biological systems and can develop animal-like object recognition capabilities through generic attention-based learning mechanisms.", "questions": "1.\tIn line 233, it is stated that the “embodied visual data streams acquired by newborn animals are rich in their own right.” How do you ensure that the agent visual stream is matched to that of the animal in terms of number of samples? \n2.\tIn lines 111-112, it is said that “the study produced data with a high signal-to-noise ratio”. What does this mean exactly, and how is it quantified? What would noise be in this context?\n3.\tIn line 122, it is stated that “The agent received visual input (64x64 resolution images)”. Is this matched physically to the visual acuity of newborn chicks? If not, what would the appropriate resolution be, and what happens when you train ViTs at that resolution?\n4.\tIn line 125, it is mentioned there are 4 rearing conditions. What are they? I didn’t seem to see that in the main text but could have mistakenly missed it.\n5.\tIs Figure 3A on heldout test performance, where Ntest = 1? Please clarify in the figure caption. \n6.\tCan you make any statements or predictions about data efficiency of chicks when there is more than one object to be learned? (cf. lines 234-235).\n7.\tMinor: How do temporally-contrastive CNNs match ViTs in this context in terms of number of training samples? It would be good to have some architectures other than ViT to show whether adding more inductive biases increases or decreases training efficiency.\n", "limitations": "Yes, the authors have adequately addressed the limitations of their work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The main strength of this paper is the novelty of training chicks with a regimented visual schedule and ensuring that agents are trained the same way for fair comparison. It is a sorely needed paradigm for claims of data efficiency during organism developmental timescales.", "weaknesses": "There were a couple clarifications regarding the experimental setup and model comparison that can be clarified in the camera ready. I have listed my specific questions in the next section below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback. We address your questions below:\n\n> In line 233, it is stated that the “embodied visual data streams acquired by newborn animals are rich in their own right.” How do you ensure that the agent visual stream is matched to that of the animal in terms of number of samples?\n\nGreat question. Ultimately, the field does not have well established procedures for comparing the number of training images across animals and machines, which is why we focused on controlling the visual environment available to chicks and ViTs, rather than controlling the number of training images per se. However, while we cannot ensure that the number of samples is matched between the animals and ViTs, we can make a rough comparison based on the rate of learning in biological systems. Researchers estimate that biological visual systems perform predictive error-driven learning every 100 ms (corresponding to the 10 Hz alpha frequency originating from deep cortical layers; O’Reilly et al, 2021). If we assume that newborns spend about half their time sleeping, this would correspond to ~430,000 images in their first day. \n\nWe also emphasize that a widely accepted critique of ViTs is that they are more data hungry than animals. This critique stems from the observation that ViTs are typically trained on millions of images across thousands of object categories, which seems excessive compared to the visual environments of newborn animals. Our paper shows that the embodied data streams acquired by newborn animals are rich in their own right. During everyday experience, newborn animals spontaneously engage in self-generated data augmentation, acquiring large numbers of unique object images from diverse body positions and orientations. We show that ViTs, like animals, can leverage these embodied data streams to learn high-level object features in impoverished visual environments (which we will clarify in the Camera Ready). \n\n> In lines 111-112, it is said that “the study produced data with a high signal-to-noise ratio”. What does this mean exactly, and how is it quantified? What would noise be in this context?\n\nBy “noise,” we mean unexplained inter-subject variation, which we can measure as the standard deviation of performance between chicks. By “signal-to-noise ratio,” we mean the size of the effect (the mean difference between chick performance and chance performance) compared to the variability. We quantify the signal-to-noise ratio as Cohen’s d (the “standardized mean difference” = mean difference / standard deviation). Our revision will point readers to Wood & Wood (2019) for a detailed explanation of how this method produces precise measurements of performance with large effect sizes.\n\n> In line 122, it is stated that “The agent received visual input (64x64 resolution images)”. Is this matched physically to the visual acuity of newborn chicks? If not, what would the appropriate resolution be, and what happens when you train ViTs at that resolution?\n\nGreat question! Chick visual acuity is about 25% of human visual acuity, which is why we initially used lower resolution images. However, we also performed new experiments to test whether different image resolutions would impact our results. To do so, we trained both the ViT-CoT and VideoMAE models with 224x224 resolution images. We did not observe a large difference between the small (64x64) and large (224x224) image resolutions, so we do not believe that image resolution strongly impacted our results. We will add these new experiments to our Camera Ready version.\n\n> In line 125, it is mentioned there are 4 rearing conditions. What are they? I didn’t seem to see that in the main text but could have mistakenly missed it.\n\nIn the chick experiments, each chick was raised with one of two objects (Object 1 vs. Object 2) presented from one of two viewpoints (front vs. side), making 4 rearing conditions in total. We will add this information to the Supplemental Materials.\n\n> Is Figure 3A on heldout test performance, where Ntest = 1? Please clarify in the figure caption.\n\nYes, in Figure 3A, performance is held out, cross-validated test performance where Ntest = 1. (The results for Ntest = 11 are in the Supplemental Materials.) We will make this explicit in the Camera Ready version.\n\n> Can you make any statements or predictions about data efficiency of chicks when there is more than one object to be learned?\n\nThis is a great question. At this point, we cannot make any statements about the data efficiency of chicks when there is more than one object to be learned, since all of our chick experiments have focused on how chicks learn their first object representation. In the near future, we will start exploring how chicks learn multiple objects, potentially providing data for distinguishing between candidate image-computable models of newborn visual systems.\n\n> How do temporally-contrastive CNNs match ViTs in this context in terms of number of training samples? It would be good to have some architectures other than ViT to show whether adding more inductive biases increases or decreases training efficiency.\n\nWe agree that a comparison between CNNs and ViTs would be valuable for determining how different inductive biases may impact the development of vision. To do so, we added new experiments comparing CNNs with ViTs, in which we evaluated a temporally contrastive CNN architecture (SimCLR-CLTT) using the same training and test conditions we used for the ViTs (ViT-CoT in original submission and new VideoMAE results added during rebuttal). We found two interesting patterns: (1) both CNNs and ViTs could solve the task (i.e., learn view-invariant object representations from impoverished visual environments) and (2) the stronger inductive biases of the CNNs led to a small but significant bump in performance over ViTs. These findings suggest that starting with a CNN architecture is beneficial, but not necessary, for learning view-invariant object representations.\n"}, {"review_id": "FXMdQlg0Po", "reviewer": "Reviewer_w2HU", "summary": "This work investigates the learning efficiency of vision transformers by comparing their invariant object recognition performance to that of newborn chicks, when being exposed to similar number of images. The authors find that ViTs learn view-invariant representations like chicks and therefore claim that they are not more data hungry than some real animals.", "questions": "-\tAddress the earlier works better about whether ViTs are more data hungry than real animals.\n-\tReplace the supervised linear classification in the test phase with a much simpler classifier that requires no or little training.\n-\tDoes the simulated input during the training phase closely reflect the input visual statistics real chicks can receive?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea behind this paper, which is to train visual models using the same input real organisms can receive and to model the animal development using such models, is a good idea. The paper is also in general well written with clear logic. To confirm that the results are general, the authors tested multiple architectures. The embedding space visualization result shows the potential explanation for why multiple attention-heads outperform single attention-head models.", "weaknesses": "The biggest weakness is that the starting point, that ViTs are thought to be more “data-hungry” than brains, is already proven false in earlier works. A paper published last year (2022) on ECCV, which is on arxiv even earlier, shows that ViTs achieve reasonable performance even when trained on 2048 images [1]. Although the authors repeatedly mention that ViTs are more thought to be more “data-hungry”, they did not provide any citation supporting this point. They also did not discuss why this point is still worth being investigated given this earlier work.\n\nIn addition to this, the linear classification training used during the test phase is also unjustified in the paper. This training requires supervision signal to be given to the model, but this signal does not seem to be available to the real animal. This makes the interpretation of the test phase results potentially wrong. A much simpler classifier that requires very little or no training should be used in this phase, such as a correlation classifier to earlier stored “imprinted” object hidden representations. The current results may very well be an overestimation of the network’s performance under a simpler classifier.\n\nIt is also unclear to me whether the simulated input during the training phase closely reflect the input visual statistics real chicks can receive. The authors designed certain agent movement patterns, is this pattern similar to what real newborn chicks would do? How about the frequency of this “movement cycle”? Is this leading to more or less augmentations compared to the real animal?\n\n[1] Cao, Yun-Hao, Hao Yu, and Jianxin Wu. \"Training vision transformers with only 2040 images.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and time. We address your 3 critiques below.\n\n>#1: The biggest weakness is that the starting point, that ViTs are thought to be more “data-hungry” than brains, is already proven false in earlier works...\n\nWe thank the reviewer for pointing out this citation. This concern highlights an opportunity for us to address this debate in our Related Works section. In the new text, we will discuss that the general consensus in the field is still that ViTs are data hungry models (especially relative to CNNs). For example, a recent (2022) review of vision transformers notes that, “CNNs encode prior knowledge about the images... that reduces the need of data as compared to Transformers that must discover such information from very large-scale data.” (Khan et al., 2022, ACM Computing Surveys). Indeed, even the Cao, Yu, and Wu (2022) paper explains that ViTs, “achieve competitive results with CNNs but the lack of the typical convolutional inductive bias makes them more data-hungry than common CNNs” and that “ViT and variants achieve competitive results with CNNs but require significantly more training data. For instance, ViT performs worse than ResNets with similar capacity when trained on ImageNet (1.28 million images). One possible reason may be that ViT lacks certain desirable properties inherently built into the CNN architecture that make CNNs uniquely suited to solve vision tasks, e.g., locality, translation invariance and hierarchical structure. As a result, ViTs need a lot of data for training, usually more data-hungry than CNNs.”\n\nWe will also point readers to work that has tried to reduce ViTs’ dependence on large scale training data. For example, some researchers have proposed adding a CNN architecture to ViTs to take advantage of the spatial inductive bias inherent in CNNs (e.g., Yuan et al., 2021, ICCV). Another approach from Cao et al. (2022) is to introduce pre-training with artificial augmentations to the dataset (e.g., multi-crop and CutMix).\n\nCritically, however, our approach differs in 3 critical ways from Cao et al. (2022). First, we used a single visual object in our training set with no extra data augmentation, whereas Cao et al. used 2040 images of many different objects with extra data augmentation. Thus, our results demonstrate that ViTs are even less data hungry than previously shown. Second, we used biologically plausible data augmentations that were generated by simulating the visual experiences of an agent moving through virtual replicas of the animal chambers. In contrast, Cao et al. (2022) used artificial augmentations like CutMix (an augmentation that would be impossible for an actual animal to perform). Third, our claim is that ViTs are not more data hungry than newborn visual systems. The only way to address this claim directly is to train ViTs and newborn animals with the same tasks and test ViTs and newborn animals with the same tasks. In this sense, our findings are unique from any existing work on ViTs (including Cao et al., 2022). Moreover, our results have important scientific implications. ViTs have the potential to be powerful image-computable models of newborn visual systems, but these models will not be accepted if they are thought to be more data hungry than brains. Our results directly contradict this widely held assumption, and thus, will shape our growing understanding of the relationship between brains and transformers.\n\n>#2: ...A much simpler classifier that requires very little or no training should be used in this phase, such as a correlation classifier to earlier stored “imprinted” object hidden representations. The current results may very well be an overestimation of the network’s performance under a simpler classifier.\n\nWe used supervised linear classifiers to evaluate the unsupervised ViTs because this is the standard approach in computational neuroscience for quantifying the performance of self-supervised models. While supervised learning was not present in the chick experiments, linear classifiers are simply a formal way of quantifying the degree and form of learned representations. In neuroscience, information that is available directly via a linear readout is generally considered to be explicitly represented by a model or brain region. The linear classifier does not provide the ViT with new information but merely measures the relative placement of different images within the model’s existing feature space. Linear classifiers are also a reasonable approximation of downstream neural computation, since linear classifiers express a plausible rate-code model for downstream decoder neurons (i.e., linear weightings followed by a single threshold value; Hong et al., 2016, Nature Neuroscience).\n\nAccordingly, the most common way to evaluate unsupervised models that are trained on biologically plausible training data (e.g., simulated first-person images from chick experiments) is to test the unsupervised model’s ability to classify objects using a supervised linear readout (c.f., Zhuang et al., 2022, NeurIPS; Zhuang et al., 2021, PNAS; Orhan, Gupta, & Lake, 2020, NeurIPS). \n\n>#3: It is also unclear to me whether the simulated input during the training phase closely reflect the input visual statistics real chicks can receive. The authors designed certain agent movement patterns, is this pattern similar to what real newborn chicks would do? How about the frequency of this “movement cycle”? Is this leading to more or less augmentations compared to the real animal?\n\nWe designed our agents to use the same six degrees of freedom for head movements (roll, pitch, yaw) as newborn chicks. However, it is not possible to affix cameras to a chick’s head, so our simulations may not reflect the movement cycles of the animals. In future research, we plan to characterize the nature of chicks’ actual visual statistics by using tools like DeepLabCut and Unity to yoke the virtual agent to actual chicks during the Input Phase of the experiment.\n"}]

Author rebuttal:
Our paper tackles a question at the heart of biological and artificial intelligence: Are vision transformers (ViTs) more data hungry than newborn visual systems? The answer to this question will have significant implications on (1) how transformers are viewed by AI researchers (e.g., brain-like or not brain-like?) and (2) whether scientists embrace transformers as viable models of biological visual systems. The dominant assumption is that ViTs are more data hungry than visual systems, but we showed that this assumption is incorrect. To demonstrate this, we introduced a novel “digital twin” approach that allowed newborn chicks and ViTs to be trained in the same environments and tested with the same images. This approach made it possible—for the first time—to directly compare the learning abilities of animals and machines. We found that ViTs are not more data hungry than chicks: when ViTs are given the same training data (visual experiences) as chicks, they develop common view-invariant recognition abilities as chicks.  

The Reviewers largely agreed that this is an important topic and that our “digital twin” approach has promise. However, the ratings were ambivalent for three reasons:

First, in the original submission, we only tested one ViT model (ViT-CoT). To address this critique, we added new experiments testing VideoMAE models: state-of-the-art models for temporal ViTs. As shown in Fig. 1 (rebuttal PDF), our ViT-CoT model outperformed VideoMAE by 15%. This result shows that ViT-CoT is particularly strong at learning high-level visual features in impoverished environments, akin to newborn chicks. 

Second, in the original submission, we did not compare ViTs to CNNs. It was thus unclear whether the spatial inductive bias of CNNs helps or hinders performance. To address this critique, we added new experiments testing CNNs that learn via contrastive learning through time (SimCLR-CLTT; Schneider et al., 2021). As shown in Fig. 2 (rebuttal PDF), both CNNs and ViTs can learn high-level object features from impoverished visual environments. This finding shows that the inductive bias of CNNs benefits, but is not necessary for, the development of view-invariant object recognition. 

Third, Reviewers questioned the scope of the project and appropriateness of the task/model system. To address these critiques, we will revise our manuscript to emphasize that our study is the first of its kind, providing a unique opportunity to directly compare the learning abilities of brains and machines. Our revision will also emphasize that newborn chicks are uniquely suited for animal-machine comparisons because chicks are the only animal that can be reared in strictly controlled environments from the onset of vision, which allowed us to fully control their training data and give that same training data to ViTs. Finally, our revision will emphasize that our results have important scientific implications. ViTs have the potential to be powerful image-computable models of newborn visual systems, but these models will not be accepted if they are thought to be more data hungry than brains. Our results directly contradict this widely held assumption, and thus, will shape our growing understanding of the relationship between brains and transformers. 

Meta-review: This was an unconventional submission. To compare model performance against behavioral data is not uncommon, but in this case, the data were derived from newborn chicks reared in a controlled fashion. Regardless, reviewers felt that the ideas were good, the approach was novel and interesting, and the experimental evaluation was solid.

There were initial concerns regarding novelty of the findings, the use of a supervised linear classification step and use of other self-supervised methods. The authors responded extensively, along with additional results (e.g. using VideoMAE). Reviewers generally found that their concerns were addressed and/or this AC judged the rebuttals to be convincing. For instance, the use of linear read-out classifiers is indeed a standard practice in computational neuroscience to gauge the information contained within a set of neurons.

This piece of work is best appreciated as a "scientific" one, where a specific hypothesis is tested. As with any individual scientific paper, the results form part of a larger body of work answering similar questions but under different setups or contexts. In this instance, this piece of work specifically deals with vision transformers and the controlled-rearing setup designed to test view invariance -- and this scope or these limitations should be made more explicit. But the results do add significantly to any previous findings about the data-hunger of models, and it's clearly a unique setup, which certainly adds an extra dimension.

Overall, this is an intriguing piece of work that deserves visibility. I believe the NeurIPS community will find it to be eye-opening and thought-provoking.
Decision: Accept (poster)