SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also found that the number of training images significantly impacted performance (Figure 3B). The untrained ViTs performed the worst, and performance gradually improved when the ViT-CoTs were trained on larger numbers of first-person images from the environment. We observed nearly identical patterns of improvement across the small, medium, and large architecture sizes, indicating that larger ViT-CoTs were not more data hungry than smaller ViT-CoTs.
In the more sparse linear classifier condition  $(\mathrm{N}_{\mathrm{train}} = 1)$ , the ViT-CoTs performed on par with the chicks (Figure 1A). This indicates that ViT-CoTs can learn efficient and abstract embedding spaces, which can be leveraged by downstream linear classifiers to support generalization across novel views. Remarkably, the linear classifiers still performed well above chance level even when trained on a single viewpoint range (akin to chicks). These results indicate that when ViTs are trained using self-supervised learning, they can form linearly separable and easily accessible view-invariant features. Like chicks, ViTs can learn accessible view-invariant object features in impoverished environments.
The VideoMAE algorithm (Figure 4A) also performed well on the task (Figure 4B), matching the performance of newborn chicks. Thus, our conclusions are not limited to ViT-CoT. Rather, different ViTs show the same learning outcomes, spontaneously developing view-invariant object features when trained on the first-person views available to newborn chicks.
Next, we examined how the CNNs performed on the task (Figure 5). When equipped with a biologically plausible time-based learning objective (CLTT, Figure 5A), the CNNs succeeded on the view-invariant object recognition task (Figure 5B). This finding extends prior work showing that CNNs can solve this task when they learn from static spatial features [29, 30]. The CNNs did outperform the ViTs (ViT-CoT by  $7.3\%$ ; VideoMAE by  $19.6\%$ ), which might be due to the strong architectural inductive bias present in CNNs, but not ViTs. Additional experiments are necessary to determine the specific source of this performance gap. We emphasize that, while ViTs performed lower than CNNs, the ViTs still succeeded on the task, learning view-invariant object features in the same impoverished visual environments as newborn chicks.
Figure 4: VideoMAEs can learn view-invariant object representations in impoverished visual worlds. (A) VideoMAE architecture with the same transformer encoder as the ViT-CoT model. (B) View-invariant recognition performance of VideoMAE across the four rearing conditions and two image resolution conditions (64x64 or 224x224). We used  $N_{\mathrm{train}} = 11$  for the linear probe. The red horizontal line shows the chicks' performance, and the ribbon shows standard error. Error bars denote standard error. ViT-CoT outperformed VideoMAE by  $15.8\%$  when trained/tested on 64x64 resolution images and by  $6.5\%$  when trained/tested on 224x224 resolution images. (C) t-SNE embedding of the representations in the last layer of a VideoMAE. The features are systematically organized, capturing information about object identity and other important latent variables (e.g., viewing distance shown here).
3.1 Controls for image resolution and temporal learning window
To check whether our conclusions generalize beyond the 3-frame learning window, we repeated the ViT-CoT experiments with a 2-frame versus 3-frame learning window. Performance was largely the same across the different learning windows (Figure 3B). To check whether our conclusions generalize beyond the  $64 \times 64$  image resolution, we repeated the experiments with a higher (224x224) image resolution. Performance was largely the same across the ViT-CoT and Video MAE algorithms (Figure 3C). Finally, we tested whether the models would still succeed on the view-invariant recognition task when both the encoder and the linear classifier were unsupervised. We replaced the supervised linear classifier with an unsupervised decoder. The model scored significantly higher than chance level, which shows that a purely unsupervised learning model, in which both the encoder (ViT) and decoder are trained without any supervised signals, can learn to solve the same view-invariant recognition task as newborn chicks when trained 'through the eyes' of chicks (see SI Section 6.6 for details).
3.2 Visualizing ViTs representations
We visualized the features using t-distributed stochastic neighbor embedding (t-SNE), which does not involve using any labeled images. We used t-SNE to visualize the representations in the last layer of the ViT-CoTs (Figure 3C), VideoMAEs (Figure 4C), and CNNs (Figure 5C). The features were systematically organized, capturing information about object identity and other important latent variables (e.g., viewing position). These results are consistent with findings that newborn chicks spontaneously encode information about both the identity and viewpoint of objects [59, 63].
We also created saliency heatmaps from the last layer of the ViT-CoTs (Figure 6). To generate each heatmap, we provided a ViT-CoT with a sample image as input. Then, we determined how much each pixel in the input image produced activation in each attention filter. Finally, for each attention filter, we color-coded each pixel in the input image according to how much it activated the attention filter. Visual inspection indicated that the attention heads became specialized for different features (e.g., whole objects, object parts, floor, depth). The attention heads' specialization varied based on the number of attention heads in the architecture. For models with more attention heads, a single attention head responds to fewer features, compared to a model with fewer attention heads (where each attention head responds to more features). This explains the drop in performance for the single attention head network, which used a single attention head to attend to all the features of the environment. These analyses indicate that ViTs can spontaneously learn enduring visual representations when trained on the embodied data streams available to newborn animals.
We performed parallel controlled-rearing experiments on newborn chicks and self-supervised vision transformers. Using a digital twin approach, we trained chicks and ViTs in the same visual environ-
B
Figure 5: (A) Diagram of SimCLR-CLTT (CNN) model. (B) View-invariant recognition performance of CNNs across the four rearing conditions and two temporal learning windows (2 frames or 3 frames). We used  $N_{\mathrm{train}} = 11$  for the linear probe. Error bars denote standard error. The CNNs outperformed ViT-CoT by  $7.3\%$ . (C) t-SNE embedding of the representations in the last layer of a CNN trained via contrastive learning through time. The features are systematically organized, capturing information about object identity and other latent variables.
ments and tested their object recognition performance with the same stimuli and task. Our results show that ViTs can spontaneously learn view-invariant object features when provided with the same visual training data as chicks. For both chicks and ViTs, impoverished environments (with a single object) contain sufficient training data for learning view-invariant object features. These results have important implications for understanding minds and machines alike.
4.1 Are transformers more data hungry than brains?
A widely accepted critique of transformers is that they are more data hungry than brains. This critique stems from the observation that transformers are typically trained on massive volumes of data, which seems excessive compared to the more sparse experiences available to newborn animals. We suggest that the embodied visual data streams acquired by newborn animals are rich in their own right. During everyday experience, animals spontaneously engage in self-generated data augmentation, acquiring large numbers of unique object images from diverse body positions and orientations. Our results show that self-supervised ViTs can leverage these embodied data streams to learn high-level object features, even in impoverished visual environments containing a single object. Thus, ViTs do not appear to be more data hungry than newborn visual systems, indicating that the gulf between visual learning in brains and transformers may not be as great as previously thought. We support our claim that ViTs are data-efficient learners by showing that two variants of ViTs can both leverage temporal learning to learn like newborn animals.
The field does not have well established procedures for comparing the number of training images across animals and machines, which is why we focused on controlling the visual environment available to chicks and ViTs, rather than controlling the number of training images per se. However, while we cannot ensure that the number of samples is matched between the animals and ViTs, we can make a rough comparison based on the rate of learning in biological systems. Researchers estimate that biological visual systems perform iterative, predictive error-driven learning every 100 ms (corresponding to the  $10\mathrm{Hz}$  alpha frequency originating from deep cortical layers; [39]. If we assume that newborns spend about half their time sleeping, this would correspond to 430,000 images in their first day. Thus, biological visual systems have ample opportunity to learn from "big data."
4.2 Origins of object recognition
Our results provide computationally explicit evidence that a generic learning mechanism (ViT), paired with a biologically inspired learning objective (contrastive learning through time), is sufficient to reproduce animal-like object recognition when the system is trained on the embodied data streams available to newborn animals. As such, our results inform the classic nature/nurture debate in the mind sciences regarding the sufficiency of different learning mechanisms for driving biological intelligence. Some researchers have proposed that intelligence emerges from a single generic learning mechanism [33, 41, 48], whereas others have proposed that intelligence emerges from a collection of innate, domain-specific learning systems [7, 46, 10, 17]. Our results show that—for the case of object recognition—a generic learning system (with no hardcoded knowledge of objects or space) is sufficient to learn view-invariant object representations. In fact, ViTs can learn accurate object representations
Figure 6: Heatmaps generated from the ViT-9H model. We created saliency heatmaps using the ViT-9H model's last transformer block for all nine attention heads (H0 - H8). The first column shows the agent's position when capturing the first-person image shown in the second column. The subsequent nine columns show the heatmap for each attention head of the ViT-9H model.
even in the impoverished environments faced by chicks in controlled-rearing studies, highlighting both the learning power of ViTs and the rich informational content in embodied data streams.
These results raise an intriguing hypothesis for the origins of visual intelligence: core visual abilities (such as object recognition) might naturally emerge from (1) an attention-based sequence learning system that (2) adapts to the embodied data streams produced by newborn animals. Under this hypothesis, visual intelligence emerges from a single innate sequence learning system that rapidly develops core domain-specific capacities when animals interact with the world during early development. Future research could test this 'single system' hypothesis by exploring whether self-supervised ViTs learn other core visual abilities when they are trained "through the eyes" of newborn animals. Ultimately, parallel controlled-rearing studies of newborn animals and machines can determine which learning mechanisms and experiences are sufficient to produce the rapid and flexible learning of biological systems.
4.3 Limitations
One limitation of the current study is that we only tested ViTs across four rearing conditions. Future experiments could test ViTs across a wider range of experimental results, adopting an integrative benchmarking approach used in computational neuroscience [45]. A second limitation of our study is that the models were trained passively, learning from batches of images in a pre-specified order. This contrasts with the active learning of newborn animals, who interact with their environment to produce their own training data. By choosing where to go and what to look at, biological systems generate their own curriculum to suit their current pedagogical needs. Future research could close this gap between animals and machines by embodying ViTs in artificial agents that collect their own training data from the environment.
4.4 Broader Impact
This project could lead to more powerful AI systems. Today's AI technologies are impressive, but their domain specificity and reliance on vast numbers of labeled examples are obvious limitations. Conversely, biological brains are capable of flexible and rapid learning from birth. By reverse engineering the core learning mechanisms in newborn brains, we can build "naturally intelligent" learning systems. Naturally intelligent learning algorithms are an untapped goldmine for inspiring the next generation of artificial intelligence and machine learning systems.
Funded by NSF CAREER grant (BCS-1351892, JNW), James McDonnell Foundation Understanding Human Cognition Scholar Award (JNW), and Facebook Artificial Intelligence Research award (JNW).
[57] Justin N Wood. Newborn chickens generate invariant object representations at the onset of visual object experience. Proceedings of the National Academy of Sciences, 110(34):14000-14005, 2013.
[58] Justin N. Wood. Newly hatched chicks solve the visual binding problem. *Psychological Science*, 25(7):1475-1481, 2014.
[59] Justin N Wood. Characterizing the information content of a newly hatched chick's first visual object representation. Developmental science, 18(2):194-205, 2015.
[60] Justin N. Wood. A smoothness constraint on the development of object recognition. Cognition, 153:140-145, 2016.
[61] Justin N. Wood and Samantha M. W. Wood. The development of newborn object recognition in fast and slow visual worlds. Proceedings of the Royal Society B: Biological Sciences, 283 (1829), 2016.
[62] Justin N. Wood and Samantha M. W. Wood. The development of invariant object recognition requires visual experience with temporally smooth objects. Cognitive Science, 42(4):1391-1406, 2018.
[63] Justin N Wood and Samantha MW Wood. One-shot learning of view-invariant object representations in newborn chicks. Cognition, 199:104192, 2020.
[64] Samantha M. W. Wood and Justin N. Wood. One-shot object parsing in newborn chicks. Journal of experimental psychology. General, 2021.
[65] Samantha MW Wood and Justin N Wood. A chicken model for studying the emergence of invariant object recognition. Frontiers in Neural Circuits, 9:7, 2015.
[66] Samantha MW Wood and Justin N Wood. Face recognition in newly hatched chicks at the onset of vision. Journal of Experimental Psychology: Animal Learning and Cognition, 41(2):206, 2015.
[67] Samantha MW Wood and Justin N Wood. Using automation to combat the replication crisis: A case study from controlled-rearing studies of newborn chicks. Infant Behavior and Development, 57:101329, 2019.
[68] Samantha MW Wood and Justin N Wood. Distorting face representations in newborn brains. Cognitive Science, 45(8):e13021, 2021.
[69] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619–8624, 2014.
[70] Jiewen Yang, Xingbo Dong, Liujun Liu, Chao Zhang, Jiajun Shen, and Dahai Yu. Recurring the transformer for video action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14063-14073, 2022.
[71] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3):e2014196118, 2021.
[72] Davide Zoccolan, Nadja Oertelt, James J DiCarlo, and David D Cox. A rodent model for the study of invariant visual object recognition. Proceedings of the National Academy of Sciences, 106(21):8748-8753, 2009.
6.1 Data generation
To simulate the training and testing data available to chicks, we created a virtual animal chamber in the Unity Game Engine. The virtual chamber was equipped with two 19" LCD monitors situated on opposite sides and accompanied by two alternating white walls. The LCD monitors were used to display the virtual objects of size  $8\mathrm{cm}$  (length) x 7 cm (height), positioned at the center on a white display background. The floor of the virtual chamber was constructed with wire mesh and had transparent holes to hold food and water on one side of the chamber. The virtual chamber measured  $66~\mathrm{cm}$  (length),  $42~\mathrm{cm}$  (width), and  $69~\mathrm{cm}$  (height).
We developed a virtual chick agent to simulate the visual experiences of newborn chicks. The virtual agent had a height of 3.5 units and a length of 1.2 units. To monitor the agent's behavior, the virtual chamber was equipped with two invisible cameras. The first camera, positioned in the chamber's ceiling, captured the top-view of the agent, while the second camera, placed in the agent's head, recorded first-person RGB images (64 x 64 resolution) as the agent moved throughout the chamber.
The agent picked a random location inside the chamber and moved to that position at the rate of 1.5 units/s. During this movement, the agent directed its visual attention towards the object projected on the LCD monitor, ensuring a centered focus. Once the agent reached the destination point, it randomly moved its head along the three axes, rotating  $60^{\circ}$  on each axis, to naturally augment the data (Figure 1B). This complete random sequence of head rotations lasted for 9.5 seconds. The agent repeated this cycle until they had captured 80,000 first-person images in the rearing condition. We repeated this data simulation cycle for each of the 4 rearing conditions.
The same procedure was repeated to gather the test samples; however, only 11,000 images were captured for each of the 12 viewpoint ranges. These test images were used to train and evaluate the linear classifiers.
6.2 Heatmap generation
To visualize the internal working of the attention heads, we generated saliency heatmaps and heatmap videos from the last block of the transformer. Section 3.2 describes the steps required to generate a heatmap for a sample image. To create heatmap videos, we followed these steps. First, the agent moved inside the chamber and captured images at 10 frames/s. Second, we used those images and created saliency heatmaps in a sequential order. Third, we concatenated these heatmaps in a series
Supplementary Figure 1: (A) Comparing the view-invariant recognition performance of newborn chicks and ViTs in the sparse linear classifier condition (1 training viewpoint). The red horizontal line shows the chicks' performance, with the ribbon representing standard error. The bars show the view-invariant recognition performance for the four ViT architecture sizes, across the four rearing conditions presented to the chicks. Error bars denote standard error. In this sparse linear classifier condition, the linear classifiers are trained on the same viewpoint as the ViT encoder and tested on the remaining 11 novel viewpoints. (B) The virtual agent's head rotations, which provided natural data augmentations. Upon reaching the destination point inside the chamber, the agent randomly rotated its head across the three axes (tilt, left/right, and up/down).
Supplementary Figure 2: Visualizing the attention patterns of individual attention heads using saliency heatmaps derived from the ViT-3H model. The visualizations were generated by using the simulated first-person images captured by the virtual agent. In the ViT-3H model, (H0-H2) correspond to the attention filters of each of the three heads. The blue dotted line highlights that head (H0) largely focuses on background features, while the red dotted line demonstrates that head (H2) largely attends to the object features. This indicates that some attention heads become specialized to different parts of the environment, such as objects versus backgrounds.
Table 1: ViT-CoT with different architecture sizes.
Model	Parameters	Attention Heads	Transformer Blocks	Batch Size	Epochs
ViT-1H	5.8M	1	1	128	100
ViT-3H	16.9M	3	3	128	100
ViT-6H	36.4M	6	6	128	100
ViT-9H	59.4M	9	9	128	100
using the OpenCV library to produce a heatmap video. We repeated these steps for all the attention heads of all the different architectures. Figure 2 displays the heatmaps generated by the ViT-3H model, showcasing the visual representations of the stimulus captured from various angles. Some heads become specialized to the object while other become specialized to the background features.
6.3 ViT-CoT Architecture
We systematically increased the number of attention heads and layers to create different architecture sizes ranging from smallest (ViT-1H) to largest (ViT-9H) model (Table 1). The ViT-CoT model contains two primary components: the transformer layers and a SimCLR projection head. The projection head consists of two linear layers and a ReLU activation function. Both of these components are sequentially attached to each other. The projection head receives the output from the final transformer layer as its input and generates an embedding  $z$  with a shape of 128. The embedding  $z$  is used to adjust the loss function when training the model. During testing, the output is obtained directly from the final transformer layer, without passing through the projection head.
6.4 VideoMAE Training
To train the VideoMAEs, we first sampled 16 frames from a batch with a temporal stride of 1. Each frame had a dimension of  $64 \times 64$  and 3 channels (16x64x64). Next, we randomly masked  $90\%$  of the
Supplementary Figure 3: (A) Diagram showing the 2-frame versus 3-frame learning windows. (B) View-invariant recognition performance of ViT-CoT across the four rearing conditions and two temporal learning windows (2 frames or 3 frames). Error bars denote standard error. We used  $N_{\mathrm{train}} = 11$  for the linear probe. The models with a 3-frame learning window performed marginally better than those with a 2-frame learning window. (C) View-invariant recognition performance of ViT-CoT (3-frame learning window) across the four rearing conditions and two image resolutions (64x64 or 224x224). We used  $N_{\mathrm{train}} = 11$  for the linear probe. ViT-CoT performs well with both low and high-resolution images.
frames using cube masks. Each mask had a spatial dimension of 8x8 and a temporal dimension of 2 (2x8x8). The VideoMAE encoder then encoded the visible (non-masked patches) into a latent space with a dimension of 512. Finally, the VideoMAE decoder combined the encoded features with the masked patches to reconstruct the entire sequence of frames. Each VideoMAE model was initialized with three different seeds and trained for 100 epochs using a batch size of 64. We used multi-GPU training across 8 NVIDIA A10 GPUs.
To ensure that our findings with Video MAE would generalize across different image sizes and masking sizes, we repeated this process with a higher image (16x224x224) and larger masks (2x16x16).
6.5 CNN Training
We utilized the default ResNet-18 architecture but made it more compact by removing the last two residual blocks, resulting in a 10-layer CNN architecture. We employed 'Contrastive Learning through Time' as the teaching signal, similar to ViT-CoT, for training these CNN models. Each model was initialized with three different seeds and trained for 100 epochs with a batch size of 512.
6.6 Two-alternative forced-choice task