SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lalit Pandey
Department of Informatics Indiana University Bloomington lpandey@iu.edu
Samantha M. W. Wood
Department of Informatics Indiana University Bloomington sw113@iu.edu
Justin N. Wood
Department of Informatics Indiana University Bloomington woodjn@iu.edu
Vision transformers (ViTs) are top-performing models on many computer vision benchmarks and can accurately predict human behavior on object recognition tasks. However, researchers question the value of using ViTs as models of biological learning because ViTs are thought to be more "data hungry" than brains, with ViTs requiring more training data to reach similar levels of performance. To test this assumption, we directly compared the learning abilities of ViTs and animals, by performing parallel controlled-rearing experiments on ViTs and newborn chicks. We first raised chicks in impoverished visual environments containing a single object, then simulated the training data available in those environments by building virtual animal chambers in a video game engine. We recorded the first-person images acquired by agents moving through the virtual chambers and used those images to train self-supervised ViTs that leverage time as a teaching signal, akin to biological visual systems. When ViTs were trained "through the eyes" of newborn chicks, the ViTs solved the same view-invariant object recognition tasks as the chicks. Thus, ViTs were not more data hungry than newborn visual systems: both learned view-invariant object representations in impoverished visual environments. The flexible and generic attention-based learning mechanism in ViTs—combined with the embodied data streams available to newborn animals—appears sufficient to drive the development of animal-like object recognition.
Vision transformers (ViTs) have emerged as leading models in computer vision, achieving state-of-the-art performance across many tasks, including object recognition [4], scene recognition [5], object segmentation [9], action recognition [70], and visual navigation [14]. Recent studies also suggest that transformers share deep computational similarities with human and animal brains [8, 19, 45, 55]. For instance, transformers are closely related to current hippocampus models in computational neuroscience and can reproduce the precisely tuned spatial representations of the hippocampal formation (e.g., place and grid cells; [55]). ViTs also outperform convolutional neural networks (CNNs) on challenging image classification tasks and are more likely to learn human-like shape biases (and make human-like errors) than CNNs [18, 50].
Despite these strengths, there is a lingering worry about using ViTs as models of biological vision. The worry relates to the large amount of data needed to train ViTs, compared to the relatively small
Figure 1: Our design had three steps: (1) Run controlled-rearing experiments testing how view-invariant object recognition develops in newborn chicks. (2) Simulate the visual experiences (training data) available to the chicks during the training and test phases, using virtual animal chambers constructed in a video game engine (Unity 3D). (3) Train and test the ViTs on the simulated visual experiences from the chick experiments. For training, we used self-supervised ViTs that leverage time as a teaching signal. For testing, we froze the ViTs and attached a linear classifier to the penultimate layer. We then trained & tested the linear classifier with the simulated test images from the chick experiments, in a cross-validated design.
amount of training data needed by biological brains. The ViTs that have revolutionized computer vision are trained on massive amounts of data (e.g., ImageNet, which contains millions of images across over 20,000 object categories). Conversely, animals do not require massive amounts of object experience to solve object perception tasks. For example, even when newborn chicks are reared in impoverished environments containing only a single object (i.e., sparse training data from birth), they still rapidly learn many object perception abilities, including segmenting objects from backgrounds [64], binding colors and shapes into integrated object representations [58], recognizing objects and faces from novel views [57, 66, 65, 68], and remembering objects that have disappeared from view [42]. Thus, ViTs appear to be more "data hungry" than newborn visual systems.
Here, we suggest that the learning gap between animals and ViTs is not as large as it appears. To compare learning across animals and ViTs, we performed parallel controlled-rearing experiments on newborn chicks and ViTs (Figure 1). First, we raised newborn chicks in strictly controlled virtual environments and measured the chicks' view-invariant object recognition performance (data reported in [57]). Second, we created digital twins (virtual simulations) of the controlled-rearing chambers in a video game engine, then simulated the chicks' visual training data by recording the first-person images acquired by an agent moving through the virtual chambers. Third, we trained ViTs on the simulated data streams from the virtual chambers and tested the ViTs with the same stimuli used to test the chicks. Consequently, the chicks and ViTs were trained on the same data and tested on the same task, allowing for direct comparison of their learning abilities.
Our study uses a "digital twin" approach for comparing the learning abilities of newborn animals and machines [29, 30]. Digital twin experiments involve first selecting a target animal study, then creating digital twins (virtual environments) of the animal environments in a video game engine. We then train and test artificial agents in those virtual environments and compare the agents' behavior with the real animals' behavior in the target study. By raising and testing animals and machines in the same environments, we can determine whether they spontaneously learn common abilities when trained on the same data distributions. Digital twin experiments thus resemble prior studies that trained AI algorithms on egocentric (first-person) images from human babies and children, acquired from head-
Figure 2: (A) View-invariant object recognition task. Newborn chicks and ViTs were trained in an impoverished visual environment (containing one object seen from a single viewpoint range), then tested on their ability to recognize that object across novel views. (B) ViT-CoT leverages the temporal structure of embodied data streams. Learning occurs by making successive views (images seen in a  $300~\mathrm{ms}$  temporal window) more similar in the embedding space. (C) The ViT architecture. The image is first divided into smaller  $8\times 8$  patches and then reshaped into a sequence of flattened patches. A learnable positional embedding is added to each flattened patch, and a class token (CLS_Token) is added to the sequence. The resulting embedding is then sequentially processed by transformer blocks while also being analyzed in parallel by attention heads, which generate attention filters shown next to each head. The learned representation of the image is adjusted based on a contrastive learning through time loss function.
mounted cameras to capture the child's natural, everyday visual experience [3, 12, 40, 71]. Digital twin studies extend this conceptual approach to newborn animals raised in controlled environments, allowing for tight control over the visual diet available to the animals and machines.
Lee and colleagues [29, 30] used the digital twin approach to compare the learning abilities of newborn chicks and CNNs on a challenging view-invariant object recognition task (Figure 2A). They found that when CNNs were trained "through the eyes" of newborn chicks, the CNNs learned to solve the same view-invariant recognition task as the chicks [57]. Thus, CNNs were not more data hungry than chicks: both could learn robust object features from training data of a single object.
One possibility is that CNNs can learn robust object features from impoverished visual environments because CNNs have a strong architectural inductive bias. CNNs' convolutional operations reflect the spatial structure of natural images, including local connectivity, parameter sharing, and hierarchical structure [28]. This architecture is directly inspired by neurophysiological observations taken from biological visual systems, including a restricted connectivity pattern that resembles the receptive field organization found in the primate visual cortex [16, 24, 69]. CNNs' spatial inductive bias allows them to generalize well from small datasets and learn useful feature hierarchies that capture the structure of visual images. This spatial bias might also explain why CNNs are not more data hungry than newborn chicks (i.e., there is a strong inductive bias supporting visual learning).
ViTs do not have this spatial bias. Rather, ViTs learn through flexible (learned) allocation of attention that does not assume any spatial structure. Thus, ViTs are less constrained by a spatial inductive bias. This flexibility allows ViTs to learn more abstract and generalizable features than CNNs, but it might also make ViTs more data hungry than newborn brains. Indeed, transformers are often regarded as "data-driven learners" that rely heavily on large-scale datasets for training [21, 32, 35, 54].
However, it is an open question whether ViTs are more data hungry than newborn visual systems. For example, [6] report that it is possible to train ViTs with limited data, indicating that ViTs might not be as data hungry as previously thought. That said, the consensus in the field remains that ViTs are data hungry models, especially relative to CNNs [27].
To directly test whether ViTs are more data hungry than newborn visual systems, we performed digital twin experiments on two ViT algorithms. First, we developed a new self-supervised ViT algorithm (Vision Transformer with Contrastive Learning through Time: "ViT-CoT") that learns
by leveraging the temporal structure of natural visual experience, using a time-based contrastive learning objective (Figure 2B). ViT-CoT learns without image labels (like newborn chicks) and without artificial image augmentations. The algorithm (Figure 2C) contrasts temporally adjacent instances (positive examples) against non-adjacent instances (negative examples), thereby learning representations that capture the underlying dynamics, context, and patterns across time. We used this contrastive learning through time approach because it has shown promising results with CNN architectures [1, 44] and because there is extensive evidence that biological vision systems leverage time to build enduring object representations [11, 31, 37, 38, 51–53].
Second, to investigate whether our results would generalize across different ViT algorithms, we also trained and tested Video Masked Autoencoders (VideoMAE) [15, 23, 49]. VideoMAE (Figure 4A) contains an encoder-decoder architecture and learns temporal correlations between consecutive frames. VideoMAE encodes spatial and temporal dependencies by masking a subset of space-time patches in the training data and learning to reconstruct those patches. Following prior work, we used a high  $(90\%)$  masking ratio.
Both ViT algorithms use time as a teaching signal. This is important because newborn chicks rely heavily on temporal information when parsing objects from backgrounds, binding color and shape features into integrated object representations, building view-invariant object representations, and building view-invariant face representations [36, 60-62, 56, 68]. Newborn chicks, like ViT-CoT and VideoMAE, leverage temporal signals to learn how to see.
2.1 Animal experiments & stimuli
We used newborn chicks as an animal model because chicks can be raised in strictly controlled environments from the onset of vision. This allowed us to control all of the visual experiences (training data) acquired by the animal: an essential requirement for directly comparing the learning abilities of animals and machines. Moreover, chicks can inform our understanding of human vision because the avian brain has similar cells and circuitry as mammalian brains [20, 25, 26]. Avian brains also have a similar large-scale organization as mammalian brains, including a hierarchy of sensory information processing, hippocampal regions, and prefrontal areas.
We focused on the view-invariant object recognition task and behavioral data reported in Wood [57]. In the study, chicks were hatched in darkness, then raised singly in automated controlled-rearing chambers that measured each chick's behavior continuously (24/7) during the first two weeks of life. The chambers were equipped with two display walls (LCD monitors) for displaying object stimuli. As illustrated in Figure 1 (step 1), the chambers did not contain any objects other than the virtual objects projected on the display walls. Thus, the chambers provided full control over all visual object experiences from the onset of vision.
During the training phase, chicks were reared in an environment containing a single 3D object rotating through a  $60^{\circ}$  viewpoint range. This virtual object was the only object in the chick's visual environment. We modeled the virtual objects after a previous study that tested for invariant object recognition in adult rats [72]. These objects are well designed for studying view-invariant recognition because changing the viewpoint of the objects can produce greater within-object image differences than changing the identity of the object. As a result, recognizing these objects from novel views requires an animal (or machine) to learn invariant object representations that generalize across large, novel, and complex changes in the object's appearance. The chicks were raised in this environment for 1 week, allowing the critical period on filial imprinting to close.
During the test phase (second week), the chicks were tested on their ability to recognize their imprinted object across novel viewpoint changes. On each test trial, the imprinted object appeared on one display wall (from a novel view) and an unfamiliar object appeared on the opposite display wall. Test trials were scored as "correct" when the chicks spent a greater proportion of time with their imprinted object and "incorrect" when the chicks spent a greater proportion of time with the unfamiliar object. Wood [57] collected hundreds of test trials from each chick by leveraging automated stimuli presentation and behavioral coding. As a result, the study produced data with a high signal-to-noise ratio [67].
The chicks performed well on the task when the object was shown from the familiar view and from all 11 novel views [57]. Despite being raised in impoverished visual environments, the chicks successfully learned view-invariant representations that generalized across substantial variation in the object's appearance. Thus, newborn visual systems can build robust object representations from training data of a single object seen from a limited range of views.
2.2 Simulating the training data available to chicks
To mimic the visual experiences of the chicks in the controlled-rearing experiments, we constructed an image dataset (Figure 1, step 2) by sampling visual observations from an agent moving through a virtual controlled-rearing chamber. The virtual chamber and agent were created with the Unity game engine. The agent received visual input (64×64 pixel resolution images) through a forward-facing camera attached to its head. The agent could move forward or backward and rotate left or right. The agent could also move its head along the three axes of rotation (tilt, yaw, and roll) to self-augment the data akin to newborn chicks. We collected 80,000 images from each of the four rearing conditions presented to the chicks. We sampled the images at a rate of 10 frames/s.
2.3 Contrastive learning through time in ViTs
Studies of newborn perception indicate that newborn animals learn how to see through self-supervised learning, using time as a teaching signal [36, 60-62, 56, 64, 68]. Thus, to compare the learning abilities of newborn chicks and ViTs, we developed a new self-supervised ViT algorithm (ViT-CoT, Figure 2C) that learns through time-based data augmentations (contrastive learning through time). Specifically, for each condition, we initialized the ViT-CoT architecture with three different seeds. Each seed was trained for 100 epochs using a batch size of 128. Throughout the training phase, each image  $(64\times 64$  pixels) was transformed into  $8\times 8$  patches, which were then flattened into a single vector. A randomly initialized positional embedding and acls_token were then added to this flattened vector before being passed to the ViT encoder. The randomly initialized positional embedding is a trainable parameter that allows the model to learn and capture the spatial information from the input images upon training. The ViT-CoT loss function was:
$$
\operatorname {l o s s} _ {z _ {t}} = - \log \frac {\exp \left(\operatorname {s i m} \left(z _ {t} , z _ {t + 1}\right) / \tau\right) + \exp \left(\operatorname {s i m} \left(z _ {t} , z _ {t + 2}\right) / \tau\right)}{\sum_ {k = 1 , k \neq t} ^ {2 N} \exp \left(\operatorname {s i m} \left(z _ {t} , z _ {k}\right) / \tau\right)} \tag {1}
$$
where  $\mathbf{N}$  is the number of samples,  $z_{t}$ ,  $z_{t+1}$ , and  $z_{t+2}$  are the embeddings of the consecutive frames, and  $\tau$  is the temperature parameter. We used the value of 0.5 for the temperature parameter.
To approximate the temporal learning window of biological visual systems, the ViT-CoT models had a learning window of three frames (i.e., 300 ms). This 300-ms learning window is based on the observation that neurons fire for 100-400 ms after the presentation of an image, providing a time window for associating subsequent images [34, 43]. The chicks in Wood [57] were reared in one of four possible environments, so we trained each ViT-CoT in a digital twin of one of these four environments (Figure 1, step 3). Thus, the models and chicks had access to the same training data for learning visual representations.
2.4 ViT-CoT Architecture Variation
To explore whether ViT-CoTs of different architecture sizes are more data hungry than newborn chicks, we systematically varied the number of attention heads and layers in the models. We built ViT-CoTs with 1, 3, 6, or 9 attention heads and layers. See SI Table 1 for architecture details.
2.5 Temporal learning in VideoMAEs
VideoMAEs consider an extra dimension of time (t) to leverage the temporal relationships between consecutive frames. We built a small VideoMAE encoder-decoder model with 3 attention heads and layers in the encoder while having 1 attention head and a single layer in the decoder. See SI Section 6.4 for training details.
2.6 Temporal learning in CNNs (SimCLR-CLTT)
Finally, to directly compare ViTs with CNNs, we also trained and tested CNNs. We used the same contrastive learning through time objective function as the ViT-CoT model. Specifically, we used the "SimCLR-CLTT" CNN algorithm [44] to train 10-layer CNNs under each of the four rearing conditions presented to the chicks and ViTs. See SI Section 6.5 for architecture and training details.
2.7 Training Data Variation
A leading theory of biological learning is that newborn animals learn how to see by densely sampling the visual environment and encoding statistical regularities in proximal retinal images [13, 22, 47]. By visually exploring the environment and iteratively adjusting connection weights using a temporal learning window, animals might gradually build accurate representations of distal scene variables (e.g., objects), without needing hardcoded knowledge of how proximal image features change as a function of distal scene variables. To explore whether dense sampling allows ViT-CoTs to learn view-invariant object representations in impoverished visual environments, we systematically varied the number of images used to train the ViT-CoTs. We trained the different-sized architectures (1, 3, 6, or 9 attention heads/layers) on 0 images (untrained), 1.25k images, 10k images, and 80k images. By training different sized ViT-CoTs on varying numbers of images, we could explore whether larger ViT-CoTs are more data hungry than smaller ViT-CoTs in the embodied visual learning contexts faced by newborn chicks. We used 80k images to train the VideoMAEs and CNNs.
2.8 Supervised linear evaluation
We evaluated the classification performance of the models using the same recognition task that was presented to the chicks. Task performance was assessed by adding a single fully connected linear readout layer on top of the last layer of each trained backbone and then performing supervised training only on the parameters of that readout layer on the binary object classification task. The linear readout layers were optimized for binary cross-entropy loss.
For the linear evaluation, we collected 11,000 simulated images from an agent moving through the virtual chamber for each of the 24 object/viewpoint combinations (2 objects  $\times$  12 viewpoints). The object identities were used as the ground-truth labels.
To evaluate whether the learned representations could generalize across novel viewpoints, we systematically varied the number of viewpoint ranges used to train  $(\mathrm{N}_{\mathrm{train}})$  and test  $(\mathrm{N}_{\mathrm{test}})$  the linear classifiers, while holding the number of training images (11,000) constant. We used two different training and test splits, as described below:
-  $\mathbf{N}_{\mathrm{train}} = 11$ ;  $\mathbf{N}_{\mathrm{test}} = 1$ : We first divided the dataset into 12 folds such that each fold contained images of each object rotating through 1 viewpoint range. The linear classifiers were cross-validated by training on 11 folds (11 viewpoint ranges, 1,000 images each) and testing on the held-out fold (1 viewpoint range: 1,000 images).
-  $\mathbf{N}_{\mathrm{train}} = 1$ ;  $\mathbf{N}_{\mathrm{test}} = 11$ : In this condition, we inverted the ratio of the training and validation viewpoints. The linear classifiers were trained using only 1 viewpoint range (with 11,000 images) and tested on the remaining 11 viewpoint ranges (1,000 images per viewpoint).
For each linear classifier condition, transfer performance was evaluated by first fitting the parameters of the linear readout layer on the training set and then measuring classification accuracy on the held-out test set. We report average cross-validated performance on the held-out images not used to train the linear readout layer.
Figure 3A shows the view-invariant object recognition performance of the ViT-CoTs. We also report the performance of the newborn chicks from [57]. All of the ViT-CoTs performed on par or better than chicks when the linear classifiers were trained on 11 viewpoint ranges  $(\mathrm{N}_{\mathrm{train}} = 11)$ . We observed similar performance for the small, medium, and large ViT-CoTs (3, 6, and 9 attention heads/layers, respectively). However, we observed a drop in performance for the smallest ViT-CoT (with a single attention head/layer).
Figure 3: (A) Comparing the view-invariant recognition performance of newborn chicks and ViTs. The red horizontal line shows the chicks' performance, and the ribbon shows standard error. The bars show the view-invariant recognition performance for the four ViT architecture sizes, across the four rearing conditions presented to the chicks. Error bars denote standard error. We used  $\mathrm{N}_{\mathrm{train}} = 11$  for the linear probe. (B) Impact of the number of training images on view-invariant recognition performance. We trained the ViTs on datasets containing different numbers of images sampled from the virtual chamber. For all architecture sizes, recognition performance improved systematically after training on larger numbers of images, indicating that ViTs can learn view-invariant object representations by densely sampling the visual environment. (C) Two-dimensional t-SNE embedding of the representations (for selected images) in the last layer of a ViT. The features are systematically organized, capturing information about object identity and other important latent variables (e.g., viewing position).