A Newborn Embodied Turing Test for Comparing Object Segmentation Across Animals and Machines
Authors: Manju Garimella
Denizhan Pak
Justin Newell Wood
Samantha Marie Waters Wood
Venue: ICLR 2024 poster (2024)
Keywords: newborn
controlled rearing
object recognition
object segmentation
reinforcement learning
benchmark
Turing test
development
Primary area: datasets and benchmarks

Abstract: Newborn brains rapidly learn to solve challenging object recognition tasks, including segmenting objects from backgrounds and recognizing objects across novel backgrounds and viewpoints. Conversely, modern machine-learning (ML) algorithms are "data hungry," requiring more training data than brains to reach similar performance levels. How do we close this learning gap between brains and machines? Here we introduce a new benchmark—a Newborn Embodied Turing Test (NETT) for object segmentation—in which newborn animals and machines are raised in the same environments and tested with the same tasks, permitting direct comparison of their learning abilities. First, we raised newborn chicks in controlled environments containing a single object rotating on a single background, then tested their ability to recognize that object across new backgrounds and viewpoints. Second, we performed “digital twin” experiments in which we reared and tested artificial chicks in virtual environments that mimicked the rearing and testing conditions of the biological chicks. We inserted a variety of ML “brains” into the artificial chicks and measured whether those algorithms learned common object recognition behavior as biological chicks. All biological chicks solved this one-shot object segmentation task, successfully learning background-invariant object representations that generalized across new backgrounds and viewpoints. In contrast, none of the artificial chicks solved this object segmentation task, instead learning background-dependent representations that failed to generalize across new backgrounds and viewpoints. This digital twin design exposes core limitations in current ML algorithms in achieving brain-like object perception. Our NETT is publicly available for comparing ML algorithms with newborn chicks. Ultimately, we anticipate that NETT benchmarks will allow researchers to build embodied AI systems that learn as efficiently and robustly as newborn brains.

Published as a conference paper at ICLR 2024
A NEWBORN EMBODIED TURING TEST FOR COMPARING OBJECT SEGMENTATION ACROSS ANIMALS AND MACHINES
Manjulata Garimella' Denizhan Pak Justin N. Wood 12,3,4 and Samantha M. W. Wood 13,4
Department of Informatics, School of Informatics, Computing, & Engineering Department of Psychological & Brain Sciences
CCognitive Science Program
*Program in Neuroscience
ABSTRACT
Newborn brains rapidly learn solve challenging object perception tasks, includ- ing segmenting objects from backgrounds and recognizing objects across new viewing situations. Conversely, modern machine learning (ML) algorithms are "data hungry," requiring more training data than brains to reach similar perfor- mance levels. How do we close this learning gap between brains and machines? Here, we introduce a new benchmark-4 Newborn Embodied Turing Test (NETT) for object segmentation- which newborn animals and machines raised in the same environments and tested with the same tasks, permitting direct compari- son of their learning. First, newborn chicks were raised in controlled environments containing a single object rotating on a single background, then their recognition performance was tested across new backgrounds and viewpoints. Second, we per- formed "digital twin" experiments in which artificial agents were reared and tested in virtual environments that mimicked the rearing and testing conditions the chicks. We inserted a variety of MI. "brains" into the artificial agents and mea- sured whether those algorithms learned common object recognition behavior as chicks. All newborn chicks solved this one-shot object segmentation task, suc- object representations that generalized
cessfully learning backkround-invarian across backgrounds and viewpoints. In contrast, none of the artificial agents solved the task, instead learning buckground-dependee generalize across new backgrounds and viewpoints. This digital twin design exposes core limitations in current ML algorithms in developing brain-like object perception. Our NETT publicly available for comparing ML algorithms with newborn chicks. We argue that NETT benchmarks can help researchers build em- bodied systems that learn as efficiently and robustly as newborn brains.
representations that failed
INTRODUCTION
A longstanding goal in artificial intelligence (AI) is to build machines that learn like brains. Early AI research laid the foundation for biologically inspired, neurally mechanistic machine learning (ML) models, and recent progress in deep learning shows that Al algorithms can be used to 're- verse engineer' sensory processing in animals including vision (Yamins & DiCarlo 2016). audi- tion (Kell et al. 2018), and olfaction (Wang et al. 2021 while also informing higher-level abil- incluiinc visually guided action (Michaels et al. 2020), language (Schrimpf et al. 2021). navigation (Whittintoon et al. 2022), and memory (Naycbi al. 2021).
But how will we know when we have succeeded in building machines that learn like brains? We argue that an evaluation platform must meet two requirements: (1) animals and machines must be raised in the same environments; (2) animals and machines must be tested with the same tasks. The first requirement stems from the observation that the intelligence of system depends both its learning algorithms and the training data (experiences) it receives. Any observed differences in behavior across animals and machines could be due differences in learning algorithms, training

---

Published as a conference paper at ICLR 2024
data, some combination of the two factors. Consequently, evaluating whether machines have similar learning algorithms as animals requires giving machines the same training data as animals. The second requirement follows from the observation that evaluations of intelligence and learning are task dependent. Accordingly, biological and artificial systems must be evaluated with the same tasks to ensure that any observed differences not due to differences in the tasks themselves.
While these two requirements may seem straightforward, building experimental platforms that meet both requirements has not yet been possible. Controlling the training data across animals and ma- chines requires performing parallel controlled-rearing experiments on animals and machines. How- ever, most newborn animals cannot be raised in controlled environments from birth, preventing researchers from controlling the training data presented to animals. Accurate comparison between animals and machines also requires measurements with a high signal-to-noise ratio, where a sub- ject's behavior in response to particular stimulus (e-g., an object) can be reliably estimated (Wood) & Wood 2019). However, most studies with newborn subjects have produced noisy data (i.e., high measurement error), researchers lacked the high precision data needed to make accurate com- parisons across newborn animals and machines. Finally, the field lacked an experimental platform for raising machines in the same environments as newborn animals, preventing researchers from matching the training data across animals and machines.
approach- cetth--mathhedeeeeee experience Newborn Embodied Turing Tests (NETTs)-that overcomes these barriers, allowing newborn animals and machines to be raised in the same environments and tested with the same tasks [F NETTs are inspired by Alan Turing's goal of creating tests for determining whether Al systems are capable of human-like intelligence. In the original Turing test, if a person cannot determine whether they are speaking to an AI system or a human, the AI passes the test (Turing. 1950). Turing's original test focused on linguistic competence, and large language models can now pass this test (Brown et al. 2020). Similarly, NETTs assess whether AI systems are capable of mimicking newborn intelligence by performing parallel controlled-rearing studies on animals and machines. NETTs use a "digital twin" design in which artificial animals are raised in virtual environments that mimic the rearing and testing conditions of newborn animals. An artificial animal passes the NETT if its behavioral performance is indistinguishable from that of its biological counterpart. Rather than focusing competence, NETTs focus on building embodied ppielssto-actions' machines that learn the same core competencies as animals when provided with the same experiences animals. NETTs thus focus attention on the early emerging sensorimotor capacities that serve as the foundation for more advanced abilities learned later in life.
Here, we present
linguistic
1.1 CHICKS AS AN ANIMAL MODEL FOR NETTS
NETTs can be created for any animal, provided that it is possible to control the animal's envi- ronment from birth and create digital twins (virtual replicas) of their environments for rearing and testing artificial animals (Lee et al., 2021a by McGGrww et al. 2023 Pak et al. 2023). However, most commonly used animal models in psychology (e.g.. primates, rodents, pigeons), cannot be raised in strictly controlled environments from birth. One species that can independently care for itself at birth is chickens (Gallus gallus). Therefore, is possible to raise newborn chicks in controlled envi- ronments (e.g., environments void of all objects and caregivers; Wood & Wood 2015) 5) and raise/test. machines in virtual replicas of those environments (Pak et al. 2023).
Another advantage of using chicks as a NETT animal model is that researchers can study visual learning in chicks with high precision using automated controlled-rearing methods. Automated controlled-rearing experiments produce data with a high signal-to-noise ratio (Wood & Wood, 2019). which allows researchers to build NETTs with high noise ceilings for comparing learning across animals and machines.
Finally, studies of chicks can inform human development because avian and mammalian brains share many characteristics. On the circuit level, avian and mammalian brains contain homologous cortical circuits for processing sensory input (Karten 2013). While these circuits are organized differently in birds and mammals (nuclear vs. layered organization, respectively). the circuits share similarities in terms of cell morphology, the connectivity pattern of the input and output neurons, gene expression, and function (Calabrese& & Woolley 2015 Dugas-Forr et al., 2012 Jarvis et al.) 2005 Wang et al.) 2010). On the architectural level, avian and mammalian brains share the same large-scale organi- zational principles: both are modular, small-world networks with a connective core of hub nodes
2

---

Published as a conference paper at ICLR 2024
Rabs newbors chécko with ebject on one bookground (haining Phase)
Test whather chicka can recogrize object backgrounds (Test Pheso))
6
Compare perforance of ofneborr chicks and artificall agents
3
Build pepulations artifiiil agents Bridno & bodies)
Newbom Embodied Turing Test
Body
vi
Weappl Heceptbur?
Text artifciall agands with newbom chicks (Test Phans)
4
Rako artiicaal enviroments
as nemborn chicks [Taining Phanel
Figure I: Benchmark for comparing the learning abilities of newborn animals and machines. New- born chicks and artificial agents raised in the environments and tested with the same tasks. This Newborn Embodied Turing Test (NETT) evaluates whether animals and machines develop the same object segmentation abilities when provided with the same visual experiences. NETTs pro- vide closed-loop system for studying the origins of intelligence: researchers build ML models that learn like newborn animals, then use those models to generate predictions for new experiments with animals to validate and refine the models.
that includes visual, auditory, limbic, prefrontal, premotor, and hippocampal structures (Shanahan et al., 2013). Given the similarities between avian and mammalian brains, controlled- studies of newborn chicks can inform our understanding of both avian and mammalian intelligence (see Section A.1 for detailed discussion).
1.2 ONE-SHOT OBJECT SEGMENTATION
In this paper, we created a NETT benchmark for testing whether AI algorithms can mimic the object segmentation and recognition abilities of newborn chicks. We focused on object segmentation be- cause (1) it is required for wal-world vision, with a long history of research in computer vision (Long) cct al. 2015 Redmon et al. 2016 He et al. 2017: Xie et al. 202I). (2) is a complex computational Teat, requiring the integration of many regions with different hue and luminance values into uni- fied object representations (Ostrovsky et al. 2009), and (3) it is one of the earliest high-level visual abilities that emerges in newborn animals (Wood & Wood (021). For example, controlled-rearine studies show that newborn chicks can segment the first object they see from natural backgrounds, even when the chicks have received sparse visual training data (e.g., visual experience of a single object rotating on single background, Wood & [ood]
Our NETT tested whether newborn chicks and artificial agents are capable of one-shot object seg- mentation (Fig. 2). The subjects were presented with a challenging task. During the Training Phase, their visual world contained a single object rotating on single background. As result, from statistical perspective, the object and background had a 100% concurrence rate. During the Test Phase, we tested whether the subjects could recognize the Training Phase object when the object was presented on novel backgrounds and from novel viewpoints.
subjects depend solely on statistical learning to segment and recognize objects, then they should incorrectly bind the object and background features together because the object and background always co-occurred during training. In this case, the subject's object recognition performance would be high when the object is presented the familiar background and low when the object is pre- sented on novel backgrounds. Conversely, if subjects are capable of one-shot object segmentation (i.e.. they can parse objects from backgrounds despite the 100% concurrence rate during training), then their recognition performance should be high, regardless of whether the object is presented on familiar novel backgrounds.
3

---

Published as a conference paper at ICLR 2024
Training Phase
Test Phase
Object Baskground IN)
Farritiar object or background It horsl object beckground
Background va.
Background (1)
Bookgroune IF)
vs
vs
Buckground Beckground
Beckground
Background
Rotation
Rotstion Rotation
Sampll Images frea Training Phose top tng-down view of agenz
Sample Images from Test Phose tup + iop-2ower. of agant corresponding first person ageut
correspandiig Bist peeson agost
Figure 2: (A) One-shot object segmentation task. Newborn chicks and artificial agents are reared in environments containing single object rotating a single background, then tested on their ability recognize that object across new backgrounds and viewpoints (changes in azimuth rotation). (B) Sample images from the Training and Test Phases from the artificial chick experiments.
2
METHODS
2.1 ANIMAL. EXPERIMENTS
We focused on the object segmentation/reconiiiion task and behavioral data from (Wood & Wood
2021) (Exp. 1). All chick experiments and data are originally reported in Wood & Wood (2021) In the study, chicks (n = 35) were hatched in darkness, then raised singly in automated controlled- rearing chambers that measured each chick's s behavior continuously (24/7) during the first 12 days of life. The chambers were equipped with two display walls (LCD monitors) for displaying object stimuli. The chambers did not contain any objects other than the virtual objects projected on the display walls. Thus, the chambers provided full control over all visual object experiences from the onset of vision.
During the Training Phase, chicks were reared in an environment containing a single 3D object lating on a single background image. The object made a full 360 rotation every 15s along the hori- zontal axis. The chicks were reared with one of possible objects, and the object was presented on one of three possible backgrounds. Due to filial imprinting, the chicks were naturally motivated spend time with the object presented during the Training Phase. The chicks were raised this sparse "one object, one background" environment for 5 days, allowing the critical period on filial imprinting to close.
During the Test Phase, the chambers measured the chicks' imprinting behavior and object recog- nition performance. The "imprinting trials" measured whether the chicks developed an imprinting response. During these trials, the familiar (imprinted) object presented one display wall and the other display wall was a blank white screen. If the chicks developed an imprinting response, then they should have spent more time by the familiar object than the blank display. The "object recogni- tion trials" measured the chicks' ability to segment and recognize their familiar object. During these trials, the familiar object was presented one display wall and novel object was presented on the other display wall. Across object recognition trials, the familiar and novel objects were presented on all possible combinations the three background (B) scenes (B1 vs. B1, B1 vs. B2, B1 vs. B3, B2 vs. B2, B2 vs. B3, and B3 vs. B3). For the evaluation, Wood & Wood (02T) grouped the test trials into four conditions (visualized in Fig. bar colors in Fig. 3):
1. Familiar object on familiar background, novel object on novel background (yellow bars);
2. Familiar object on novel background, novel object on familiar background (purple bars);
3. Both objects on the familiar background (blue bars);
4. Both objects novel background (green bars).
4

---

Published as a conference paper at ICLR 2024
The objects were also shown from three possible viewing angles: 0 change in azimuth rotation (same as Training Phase), 30 change in azimuth rotation, and 60 change in azimuth rotation. On each trial, the viewing angle of the imprinted and novel objects were matched. If the chicks developed object segmentation and the ability to recognize objects across novel backgrounds and viewpoints, then they should have spent more time by the display wall with the familiar object versus the novel object. Object recognition trials were scored as the proportion of time spent with the familiar (imprinted) object relative to the total time spent with the familiar and the novel object. Wood& & Wood ( collected hundreds of object recognition trials from each chick by leveraging automated stimuli presentation and behavioral coding. As a result, the study produced data with a high hignal-tonnoise ratio (red noise bands, Fig. Wood & 2019.
2.2 MACHINE LEARNING EXPERIMENTS
Our goal was to raise and test biological and artificial agents under parallel conditions. This required (1) virtual environments for artificial agents that mimicked the environments faced by the chicks, and
(2) artificial agents that could learn from raw sensory inputs, make decisions, and perform actions, akin to chicks.
Virtual environments. To simulate the visual environments of the chicks in Wood & Wood (2021) we created realistic digital twins (virtual replicas) the controlled-rearing chambers, using a video game engine (Unity). The virtual chambers matched the proportions, textures, and colors of the real-world chambers (Section A.2). During the Training Phase and Test Phase, we reared and tested the artificial agents in the virtual chambers, presenting the same stimuli and tasks to the artificial agents that were presented to the chicks.
Artificial agents. To directly compare animals and machines, created artificial agents that performed the same task as all newborn animals: learn from raw sensory data and perform actions in the environment. We created the artificial agents by embodying deep reinforcement learning (RL) algorithms in virtual chick bodies. Each body measured 3.5 units (height) by 1.2 units length (radius) and received visual input (64x64 pixel resolution images) through an invisible forward-facing camera attached to its head. The artificial agents could look up and down, move forward and backwards, rotate left and right, or remain stationary. The actions were represented as three continuous variables: (1) head movement up/down, (2) translation forward/backward, and (3) rotation around the vertical axis. To control the behavior of the artificial agents, we inserted variety of artificial neural networks (ANNs) into the chicks (Section A.3, see Tables [ and 2 in Supplementary Materials for ANN configurations). For each of the six rearing conditions in Wood & Wood (2021), we trained and tested five artificial agents per ML algorithm. All artificial agents in a group had the same network architecture, but each artificial agent's ANN started with a different random initialization of connection weights, and each artificial agent's connection weights were shaped based on its own particular experiences during the Training Phase.
Imprinting Reward. During real-world behavior, newborn chicks spontaneously reduce the dis- tance between themselves and their imprinted object, making the imprinted object larger in their field of view. To encourage the artificial agents imprint, the agents received reward proportional to the size of the imprinted object in their field of view (details in Section A.4). For example, if the object extended through quarter of the agent's 's field of view, then the reward for that step would be 0.25. The artificial agents were trained to optimize the sum of the imprinting reward using Prox- imal Policy Optimization (PPO; /Schulman et al. 2017). As we show in SI Figures 5 and this imprinting reward produces similar imprinting behavior as biological chicks.
At the beginning of each training episode, the artificial agent was spawned at random position and orientation within the chamber. Each training episode lasted 1,000 time steps. We trained the arti- ficial agents for 1,000 episodes. After the Training Phase, the network weights were frozen for the Test Phase (i.c., the artificial agents did not receive any rewards during the Test Phase, and learning was discontinued). This freezing mimics the critical period of filial imprinting in biological chicks, in which chicks stop learning about their imprinted object after the few days of life (Horn) 2004).
During the Test Phase (Section A.5), each artificial agent performed 1,080 object recognition trials (40 trials for each of the 27 background viewooint combinations presented to the chicks). At the beginning of each test episode, the artificial agent was spawned at the middle of the chamber, facing blank wall. Fach object recognition trial consisted of 1,000 time steps. At every time step, we recorded the position of the artificial agent in X,Y coordinates. As with the chicks, we measured
5

---

Published as a conference paper at ICLR 2024
A
Newborn Chicks
B
Small Visual System 3-Layer CNN
Medium Visual System 10-Layer CNN
Large Visual System 18-Layer CNN
Key
E
Dynamic Visual Sensor (DVS)
LSTM Policy Network 3-ayer CNN
10X Training Episodes 3-layor CNN
For
Par dur
Punbiu
Faie
Novel
Novil
Object
Ciject
Novel
Familler
Desigrrunc
Deckgrund
Cechyrund
Dadigmund
Churkegs:
Chalengs:
Challane
Challagge
Solt
backgrued
Datagr.ne
stave oe
sgerat
beckgrunn
bedggrund
Figure 3: Performance of the newborn chicks and artificial agents on the object segmentation and recognition task. During the Test Phase, the familiar and novel objects were presented on familiar and novel backgrounds, in a fully crossed design (Key). (A) The chicks successfully recognized the object across all viewing conditions, regardless of whether the familiar and novel objects were pre- sented on familiar or novel backgrounds. The chicks learned background nuckground-invaiian object represen- tations. (B-G) In contrast, the artificial agents learned with the background features heavily influencing the recognition performance of the agents. To fa- cilitate comparison between the chicks and artificial agents in Panels B-G, mean chick performance is shown as a red line with a "noise band" (red band) reflecting the average deviation of the chicks from the mean chick performance.
object representations,
whether the artificial agent spent a greater proportion of time with the familiar (imprinted) object than the novel object.
3 RESULTS
3.1
BIOLOGICAL CHICK PERFORMANCE
On the imprinting trials, the chicks spent significantly more time by the imprinting stimulus than the blank screen (M 84%, SD 6%, r(30)= 33.87, p < .0001). On the individual level, all of the chicks successfully learned to imprint (range 68% to 93%, chance performance = 50%).
On the object recognition trials, the dependent variable was the proportion of time the chick spent by their familiar object versus the novel object. To compute performance, we used the following formula: time by familiar object / (time by familiar object time by novel object).
For the biological chicks (Fig. 3A), performance was well above chance level (50%) in all four background conditions (one-sample f-tests, all < 10-7, all Cohen's ds > 1.2). Performance was also well above chance level in all three viewpoint conditions (one-sample r-tests, all Ps < 10-10, all Cohen's is ds > 1.7). The chicks successfully recognized their familiar object, regardless of whether the object was presented on familiar novel backgrounds or whether the object was presented from familiar or novel viewpoints. Thus, newborn chicks are capable of one-shot object segmentation.
3.2 ARTIFICIAL CHICK PERFORMANCE
All statistical analyses supporting claims in this section are in Supplementary Methods (Tables 3-5). We visualize the object representations learned by the agents using tSNE (Section A.6, SI Figure 2
Base Agents. First, we examined the object recognition performance of the base agents, each of which was equipped with simple 3-layer CNN visual encoder and PPO policy learning algorithm. During the imprinting trials, all of the base agents learned to detect and approach the familiar object, indicating that all of the artificial agents learned to imprint.

---

Published as a conference paper at ICLR 2024
During the test trials, however, the artificial agents showed entirely different pattern of results from the chicks (Bi While the chicks recognized the familiar object across familiar and novel backgrounds, the artificial agents failed to recognize the familiar object across novel backgrounds. The artificial agents performed at chance level when both objects were shown on novel backgrounds and when both objects were shown on the familiar background. Unlike chicks, whose recognition performance is almost entirely invariant to background changes, the base agents learned background- dependent representations, incorrectly linking object and background features together.
Medium and Large Visual Encoders. The base agent had small visual encoder (3-layer CNN), which may not have been sufficient to disentangle object features from background features during training. To test whether the beekground-dependee learning of the base agents was due to a small visual encoder, we equipped the artificial agents with medium (ResNet- 10) and large (ResNet-15 visual encoders.
During the imprinting trials, all 30 of the agents with medium encoders, and 28 of the 30 agents with large encoders, learned to detect and approach the familiar object. However, despite having larger visual encoders than the base agents, the artificial agents developed the same form of recognition behavior (Fig (C&D). The artificial agents all learned backgrounn-dependeni object representations.
Biologically Inspired Retinal Operations. One way biological visual systems might disentangle object features from background features is through dynamic visual processing in the retina. Retinal ganglion cells compute temporal contrast (i.e., changes in luminance across time; Lichtsteiner et al. 2008 As result, information passed from the retina to LGN is not in the form of static images, but rather visual events. When a still observer watches an object moving along a static background, only the contrast of the object will change. Visual information about the object will be enhanced, and the background will be suppressed.
To test whether retina-like processing can account for newborn chicks' ability segment objects from backgrounds, we added retinal postprocessing to the images before sending the images to the ANN. Our retinal processing mimicked the operations of a Dynamic Visual Sensor (DVS) system (Posch et al. 2014 Gallego et al. 2020). For an image frame at time t: (1) convert pixels from RGB to grayscale/brightnes, (2) apply a Gaussian blur, (3) compute the difference between the resulting matrix and the blurred grayscale matrix from 1-1, (4) apply a threshold to all resulting matrix values.
During the imprinting trials, the artificial agents with DVS "eyes" learned detect and approach the familiar object (26 out of 30 agents). However, despite having dynamic visual sensing, the artificial agents developed the same packground-dependee
recognition behavior as the prior agents 3
Recurrent Learning Algorithms. Another way biological visual systems might parse objects from backgrounds is through recurrent processing. Recurrent neural networks (RNNs) process dynamic, time-series data using feedback loops that allow prior information to persist. This form of short- term memory allows prior input to influence current inputs and outputs. To learn about the temporal dynamics of the visual environment, artificial agents might therefore need recurrent processing. To explore this possibility, we added recurrent processing to our base agents by converting their fully connected policy network into LSTM (long short-term memory) network (Hochreiter & Schmidhuber 1997). The LSTM network contained one LSTM layer with 256 hidden units.
During the imprinting trials, all 30 of the LSTM artificial agents learned to detect and approach the familiar object. However, despite having recurrent processing, the artificial agents developed the same form of background-dependent object recognition as the prior agents (Fig 3
Longer Training. All of the above agents were trained for 1,000 episodes. But some neural net- works learn "grok" a generalizable pattern from the data well after the point of overfitting (Power et al. 2022). To test whether our agents needed longer training period to learn to generalize, we also trained base agents for 10,000 episodes (10x the original base agents).
During the imprinting trials, the artificial agents learned to detect and approach the familiar object (26 out of 30 agents). However, the artificial agents developed the same pattern of background- dependent recognition as the prior agents (Fig
Pre-trained Encoders. We also explored whether pre-trained encoders would help the artificial agents succeed at the task. First, we trained the encoders 'through the eyes' of newborn chicks, by simulating the first-person images acquired by chicks during early visual development (details in
7

---

Published as a conference paper at ICLR 2024
Section A.7). Despite learning from stable (pre-trained) visual encoders, the artificial agents still developed -eeerruundddeeeddennt object recognition (Figure SI Second, we inserted a variety of powerful pre-trained visual encoders into the agents. including Segment Anything (Kirillov et al.) 2023), DINOv2 (Qquab et al. 2023), and Ego4D (Radosavovic et al. 2023) details in Section A.8). These pre-trained models perform well on standard computer vision benchmarks for object segmen- lation; nevertheless, artificial agents equipped with these encoders still failed to solve the one-shot learning task solved by newborn chicks (Figure SI When we visualized the outputs of the pre- trained encoders, we found that some of the attention heads correctly segmented object features from background features. Thus, we speculate that the problem is not discovering useful visual features (pre-trained encoders appear to do this), but rather discovering which features to use to solve the embodied object recognition task.
Anchal-Machine Performance Gap. To quantify the performance gap between the biological and artificial agents, we computed a "noise band" around the mean performance of the chicks (Fig G). The noise band reflects the average deviation of the chicks from mean chick performance. This defines a clear noise ceiling to judge prediction accuracy (Cao & Yamins, 202T). Noise ceilings capture the idea that the accuracy of an artificial agent in predicting chick behavior can only be as good as the accuracy of a chick in predicting chick behavior (e.g.. due to measurement error and individual differences across chicks). If the average artificial agent performance reaches the noise band, then the artificial agent can be said to be "predictively adequate" chick learning (i.e., the MI. algorithms generate similar learning outcomes as the learning algorithms in chicks). This was not the case for this object segmentation NETT: there were large gaps in object recognition performance across the chicks and artificial agents.
4 DISCUSSION
We introduce a matched eatched--epprience Newborn Embodied Turing Test (NETT) for object segmenta- tion. This NETT allows researchers to evaluate whether ML algorithms learn similar object segmen- tation abilities as newborn animals when provided with the same visual diet as animals. To create this NETT, we raised artificial agents in the same environments newhorn chicks from prior controlled- rearing experiments (Wood & Wood [202T] We then tested whether the chicks and artificial agents learned common object recognition behavior. With impoverished training data (one object seen on one background), can ML algorithms match the rapid and robust learning of newborn chicks?
As starting point, we inserted a variety of deep RL algorithms into the artificial agents and mea- sured whether some algorithms can develop newborn-like object perception. We found remarkable consistency across algorithms. All of the algorithms successfully learned to imprint, but none of the algorithms learned background-invariant object representations that could generalize across novel backgrounds and viewpoints. Regardless of whether the artificial agents had small, medium, or large visual encoders, feedforward recurrent policy networks, or dynamic processing in their vi- sual sensors, the artificial agents all learned buckground-dependee features were incorrectly linked to background features. Unlike newborn chicks, modern deep RL algorithms, which largely rely on brute-fore frute-oorce statistical association, are not capable of developing one-shot object segmentation in impoverished visual environments. These results show that there is a mismatch between the learning algorithms that drive biological versus artificial intelligence.
representations, in which object
NETTs: Developmentally Inspired Benchmarks for AI. Until recently, attempts build bridges between artificial and biological intelligence have focused almost entirely on benchmarks from ma- ture humans and animals. While this approach has led to remarkable advances in AI, there is a growing consensus that building "naturally intelligent" learning algorithms will require taking de- velopment seriously. Biological intelligence develops across protracted periods of time, through nested periods of development (Smith et al.] 2018 Adolph et al. (0088). Building animal- like intel- ligence may require allowing AI systems develop under similar conditions. Even developmentally inspired Al approaches that train ANNs with the visual experiences of children (Orhan et al.) 2020) or compare the behavior of MI. models to human infants and children (Kosoy et al. 2020) Gandhi et al., 1021 Shu et al., 202I) still exclude critical learning experiences from the first months or years of life. NETTs uniquely capture the full environmental history of animals, allowing for rigor- ous evaluation of whether embodied ML algorithms learn like newborn animals.
Challenging One-Shot Learning Benchmark for AI Modern ML algorithms can solve object segmentation tasks, but only when trained massive amounts of data (e.g., ., 14 million images from
8

---

Published as a conference paper at ICLR 2024
ImageNet, Deng et al. 2009). When relying solely brute-force statistical learning, ANNs need to see objects across many different contexts to successfully dissociate object and background features, and many still suffer from this "statistical concurrence problem" (Wood & Wood 202I). For ex- ample, when ANNs are trained to recognize objects (e.g.. dumbbells), but the objects are presented in similar scenes across training images (e.g.. the dumbhells are always held by weightlifters), the resulting visual representations incorrectly bind object and background features (e.g., the dumbbell representations include hand and arm features; Mordvintsev al. 2015).
To overcome this statistical concurrence problem, newborn brains use other mechanisms to segment objects from backgrounds. Specifically, motion- segmentation plays a core role in the devel- opment of object perception. Young infants use motion cues to determine the 3-D shape of objects (Arterberry Yonas, 2000; Owsley 1983) and to integrate spatially separated parts into unified object concepts (Johnson et al. 2002 Kellman & Spelke 1983, Kellman et al., 1986). Object seg- mentation through motion cues also develops before the ability to segment objects using static cues
(e.g., color, shape; SSeelke 1990). Likewise, for patients recovering from blindness, motion cues are essential for learning to segment and recognize objects (Ostrovsky et al, 2009). Motion-based segmentation thus appears to be "primitive" of object perception- program of visual learning that enables the brain to assemble fragmented features into unified object representations. Notably, motion-based segmentation allows learning systems overcome the statistical concurrence prob- lem, since motion can act as a selective-gating mechanism, constraining learning to moving features, rather than the whole visual field. Objects move separately from backgrounds, so motion-based seg- mentation can parse objects from backgrounds without the need for diverse visual experiences with objects.
We attempted to implement motion-based segmentation in artificial agents using LSTMs and dy- namic visual sensors, but these components were clearly not sufficient to solve this one-shot seg- mentation task. Moving forward, we anticipate that our NETT will be valuable benchmark for building embodied Al systems that can solve one-shot segmentation tasks, potentially through a brain-inspired motion-based segmentation process.
Limitations. One limitation of the current study is that we tested a limited number of ANNs. We hope that our NETT benchmark will allow Al researchers to plug a variety of learning algorithms into the artificial agents, in a community-wide effort to test whether any existing ML algorithms capable of matching the rapid and robust visual learning of newborn chicks (Section A.9). A second limiting factor in building embodied models of biological intelligence is the availability of algorithms that learn policies with nigh-dimensional action spaces. To avoid this roadblock, we built artificial agents with lower-dimensions action spaces. Future work might close this gap between animals and machines by building more realistic animal bodies, potentially through the use of sensorimotor pretraining (Radosavovic et al. (202).. A third limitation is that our artificial agents only had a single objective: maximize their "imprinting reward." Real chicks, however, have multiple needs (e.g., hunger, thirst, temperature regulation, curiosity), which likely shape learning and behavior important ways. In future studies, it would be interesting to build artificial agents with similar needs as biological chicks, potentially through the use of control systems that capture the homeostatic needs of biological systems (Yoshida et al. 2021) and/or curiosity- rewards (aahak et al. 2017).
Broader Impact. This project could lead to more powerful AI systems. Today's 's AI technologies are impressive, but their domain specificity and reliance on vast numbers of labeled examples are obvi- ous limitations, especially compared to the rapid and flexible learning of newborn brains. NETTs will allow researchers reverse engineer the core learning mechanisms in newborn brains, opening experimental paths towards building "naturally intelligent" ML systems.
5 ACKNOWLEDGEMENT
We thank Bhargav Desai, Zachary Laborde, and Lalit Pandey for many helpful contributions to this project. All three deserve authorship based on contributions provided after the paper was submitted, but due to conference rules, we were not able to add new authors to the paper after submission. This project was funded by NSF CAREER grant (BCS-135189) JNW), James McDonnell Founda- tion Understanding Human Cognition Scholar Award (JNW), and Facebook Artificial Intelligence Research award (JNW).

---

Published as a conference paper at ICLR 2024
REFERENCES
Karen E Adolph, Justine E Hoch, and Whitney G Cole. Development (of walking); 15 suggestions.
Trends Cognitive Sciences, 22(8):699 -7119991111111 2018.
Martha E Arterberry and Albert Yonas.
Perception of three-dimensional shape specified by optic flow by w-week-old infants. Perception & Psychophysics, 62(3):550-556, 2000.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901. 2020.
Ana Calabrese and Sarah MN Woolley. Coding principles the canonical cortical microcircuit in the avian brain. Proceedings of the National Academy of Sciences of the United States of America, 112(11):3517-3222 2015.
Rosa Cao and Daniel Yamins. Explanatory models in neuroscience: Part 1-taking mechanistic abstraction seriously. arXiv preprini 1000002000 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255. IEEE, 2009.
Jennifer Dugas-Ford, Joanna J Rowell, and Clifton W Ragsdale. Cell-type homologies and the origins of the neocortex. Proceedings of the National Academy of Sciences of the United States of America, 109(42):19974-1699)) 2012.
Nathan J Emery. Cognitive ornithology: the evolution of avian intelligence. Philosophical Transac- tions of the Royal Society B: Biological Sciences, 361(1465):33-4, 2006.
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J Davison, Jörg Conradt, Kostas Daniilidis, al. Event-based vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1):154-180, 2020.
Kanishk Gandhi, Gala Stojnic, Brenden M Lake, and Moira R Dillon. Baby intuitions benchmark (bib): Discerning the goals, preferences, and actions of others. Advances in Neural Information Processing Systems, 34:9963-9999 2021.
Onur Güntürkün and Thomas Bugnyar. Cognition without cortex. Trends in Cognitive Sciences, 20 (4):291-303, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2961-2969, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Gabriel Horn. Pathways of the past: the imprint of memory. Nature Reviews Neuroscience, 5(2): 108-120, 2004.
Erich D) Jarvis, Onur Güntürkün, Laura Bruce, András Csillag, Harvey Karten, Wayne Kuenzel, Loreta Medina, George Paxinos, David J Perkel, Toru Shimizu, et Avian brains and new understanding of vertebrate brain evolution. Nature Reviews Neuroscience, 6(2):151-159, 2005.
Scott P Johnson, J Gavin Bremner, Alan M Slater, Uschi C Mason, and Kirsty Foster. Young infants' perception of unity and form in occlusion displays. Journal of Experimental Child Psychology. 81(3):38-337, 2002.
Harvey Karten. The organization of the avian telencephalon and some speculations on the phy- logeny of the amniote telencephalon. Annals of the New York Academy of Sciences, 167(1): 164-179, 1969.
Harvey J Karten. Neocortical evolution: neuronal circuits arise independently of lamination. Current Biology, 23(1):R12-RI1 2013.
10

---

Published as a conference paper at ICLR 2024
Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haignere, Dermott. task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals cortical processing hierarchy. Neuron, 98(3):630-66) 2018.
Haignere, and Josh H Mc-
Philip Kellman and Elizabeth Spelke. Perception of partly occluded objects in infancy, Cognitive Psychology, 1544)4483552, 1983.
Philip J Kellman, Elizabeth S Spelke, and Kenneth R Short. Infant perception of object unity from translatory motion in depth and vertical translation. Child Development, pp. 72-86, 1986.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint (rXiv:2304.02643,, 2023.
Eliza Kosoy, Jasmine Collins, David M Chan, Sandy Huang, Deepak Pathak, Pulkit Agrawal, John Canny, Alison Gopnik, and Jessica Hamrick. Exploring exploration: Comparing children with agents in unified environments. arXiv preprint arXiv:2005 2000::2005.02880, 2020.
Donsuk Lee, Pranav Gujarathi, and Justin N Wood. Controlled-rearing studies of newborn chicks and deep neural networks. In SVRHM 202/ Workshop@@ NewrIPS, 2021a.
Donsuk Lee, Denizhan Pak, and Justin Wood. Modeling object recognition in newborn chicks using deep neural networks. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43, 2021b.
Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck. 128 x 128 120 db 15 jus latency asyn- chronous temporal contrast vision sensor. IEEE Journal of Solid-Star State Circuits, 43(2))566-577, 2008.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3131-334400 2015.
Giulio Matteucci and Davide Zoccolan. Unsupervised experience with temporal continuity of the vi- sual environment is causally involved in the development of v1 complex cells. Science Advances, 6(22)xeaba3742, 2020.
Joshua MeGraw, Donsuk Lee, and Justin Wood. Parallel development of social preferences in fish and machines. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45, 2023.
Jonathan A Michaels, Stefan Schaffelhofer, Andres Agudelo-Toro, and Hansjörg Scherberger. A goal-driven modular neural network predicts parietofrontal neural dynamics during grasping. Pro- ceedings of the National Academy of Sciences of the United States of America, 117(50):32124- 32135, 2020.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. Google research blog, 20:5, 2015.
Aran Nayebi, Alexander Attinger, Malcolm Campbell, Kiah Hardcastle, Isabel Low, Caitlin S Mal- lory, Gabriel Mel, Ben Sorscher, Alex H Williams, Surya Ganguli, al. Explaining heterogeneity in medial entorhinal cortex with task-driven neural networks. Advances in Neural Information Processing Systems, 34:12167-12179) 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fermandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/93, 2023.
Emin Orhan, Vaibhav Gupta, and Brenden M Lake. Self-supervised learning through the eyes of child. Advances Neural Information Processing Systems, 33:9960-9977 2020.
Yuri Ostrovsky, Ethan Meyers, Suma Ganesh, Umang Mathur, and Pawan Sinha. Visual parsing after recovery from blindness. Psychological Science, 40412):1484-1499 2009.

---

Published as a conference paper at ICLR 2024
Cynthia Owsley. The role of motion in infants' perception of solid shape. Perception, 12(6):707- 717, 1983.
Denizhan Pak, Donsuk Lee, Samantha Marie Waters Wood, and Justin N Wood. A newborn embod- ied turing test for view-invariant object recognition. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45, 2023.
Lalit Pandey, Samantha Wood, and Justin Wood. Are vision transformers more data hungry than newborn visual systems? Advances Neural Information Processing Systems, 36, 2023.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In international conference on machine learning, pp. 17359-17371. PMLR, 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, pp. 2778-2787. PMLR, 2017.
Christoph Posch, Teresa
Bernabe Linares- Baarrs-Barrancoo, and Tobi Delbruck. Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proceed- ings of the IEEE,
2014.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177, 2022.
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference on Robot Learning, pp- 416-426. PMLR, 2023.
Antonin Rafinn, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. basble-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http://mmm..cggpppppr/////22
20 3344.ttm
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 779-788, 2016.
A Reiner. A new avian brain nomenclature: why, how and what. Brain Research Bulletin, 66(4-6): 317-331, 2005.
Anton Reiner, David Perkel, Laura L Bruce, Ann B Butler, András Csillag, Wayne Kuenzel, Loreta Medina, George Paxinos, Toru Shimizu, Georg Striedter, et al. Revised nomenclature for avian telencephalon and some related brainstem nuclei. Journal of Comparative Neurology, 473(3): 377-414, 2004.
Felix Schneider, Xia Xu, Markus R Ernst, Zhengyang Yu, and Jochen Triesch, Contrastive learning through time. In SVRHM 202/ Workshop@@ @ NeurlPS, 2021.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: In- tegrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences of the United States of America, 118(45)ee105644111 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.00777. 2017.
Murray Shanahan, Verner P Bingman, Toru Shimizu, Martin Wild, and Onur Güntürkün. Large- scale network organization the avian forebrain: a connectivity matrix and theoretical analysis. Frontiers in Computational Neuroscience, 7:89, 2013.
12

---

Published as a conference paper at ICLR 2024
Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin Smith, Shari Liu, Dan Gutfreund, Eliza- beth Spelke, Joshua Tenenbaum, and Tomer Ullman. Agent: A benchmark for core psychological reasoning. In International Conference Machine Learning, pp. 9614-9623 PMLR, 2021.
Linda B Smith, Swapnaa Jayaraman, Elizabeth Clerkin, and Chen Yu. The developing infant creates a curriculum for statistical learning. Trends in cogmitive sciences, 22(4):325-336, 2018.
Elizabeth S Spelke. Principles of object perception. Cognitive Science, 14(1):29-566 1990.
Martin Stacho, Christina Herold, Noemi Rook, Hermann Wagner, Markus Axer, Katrin Amunts, and Onur Güntürkün. A cortex-like canonical circuit in the avian forebrain. Science, 369(6511); eabc5534, 2020.
Alan Mathison Turing. Computing machinery and intelligence. Mind, 59(23))4433446,, 1950.
Peter Y Wang, Yi Sun, Richard Axel, LF Abbott, and Guangyu Robert Yang. Evolving the olfactory system with machine learning. Neuron, 109(23)33879-333 -892,, 2021.
Yuan Wang, Agnieszka Brzozowska-Prechhn and Harvey J Karten. Laminar and columnar auditory cortex in avian brain. Proceedings of the National Academy of Sciences of the United States of America, 107(28): 12676-12681 2010.
James CR Whittington, David McCaffary, Jacob JW Bakermans, and Timothy EJ Behrens. How to build a cognitive map. Nature Neuroscience, 25510):1257-1222, 2022.
Justin N Wood. A smoothness constraint on the development of object recognition. Cognition, 153: 140-145, 2016.
Justin N Wood and Samantha MW Wood. The development of newborn object recognition fast and slow visual worlds. Proceedings of the Royal Society B: Biological Sciences, 283 218(1)))): 20160166, 2016.
Justin N Wood and Samantha MW Wood. The development of invariant object recognition requires visual experience with temporally smooth objects. Cognitive Science, (2(4):1391-144) 2018.
Samantha MW Wood and Justin N Wood. chicken model for studying the emergence of invariant object recognition. Frontiers in Neural Circuits, 9:7, 2015.
Samantha MW Wood and Justin N Wood. Using automation to combat the replication crisis: A case study from controlled-rearing studies of newborn chicks. Infant Behavior and Development, 57: 101329, 2019.
Samantha MW Wood and Justin N Wood. One-shot object parsing in newborn chicks. Journal of Experimental Psychology: General, 150(11):2408, 2021.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in 2021.
Neural Information Processing Systems,
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 1963)335-335, 2016.
Naoto Yoshida, Tatsuya Daikoku, Yukie Nagai, and Yasuo Kuniyoshi. Embodiment perspective of reward definition for behavioural homeostasis. In Deep RL Workshop NeurIPS 2021, 2021.
13

---

Published as a conference paper at ICLR 2024
A SUPPLEMENTARY MATERIAL
A.1 COMPARING AVIAN AND MAMMALIAN BRAINS
How do avian brains compare to mammalian brains? Historically, scientists believed that avian brains were functionally different from and less sophisticated than mammalian brains (hence the pe- jorative term "bird brain"). Early studies of avian and mammalian brains were methodologically lim- ited to identifying the general shape, appearance, and distribution of cells in brain regions (see Reinnr 2005 for thorough historical review). These studies broadly identified two major telencephalon components in mammalian brains: a layered "neocortex" and a radially organized "basal ganglia." In contrast, the avian telencephalon lacks layered regions, leading early scientists to propose that avian brains were dominated by a basal ganglia that operated largely through instinct rather than learning-
However, this view has been so thoroughly repudiated by the avian neuroscience community that the field completely revised the naming structure for the avian brain to reflect the deep structural and functional homologies between avian and mammalian brains (Reiner et al. (2004). Beginning in the 1960s, more advanced neuroanatomical methods revealed homologs for mammalian visual, audi- tory, somatosensory, and motor cortex in avian brains (Karten 1969). Even though avian neocortex is organized radially rather than in layers, the avian neocortex shares connectivity patterns, physio- logical signatures, and cell types with mammalian neocortex (Stacho et al., 2020). Both mammalian and avian neocortex are organized in canonical circuits, which likely generate similar computational properties across taxa (Stacho et al. 2020).
Given the similarities in both the individual circuits and the large-scale brain organization across birds and mammals, we suspect that avian and mammalian brains are identical at engineering- level of abstraction (i.c., the level at which we build machines and models that learn like animals). This hypothesis is further supported by a wealth of studies showing that some birds, like corvids and parrots, have forebrains of similar size as apes, live in complex social groups, and have long developmental periods before becoming independent (reviewed by |Emery 2006). These birds show like intelligence across wide range of domains, in some cases even outperforming chimpanzees on cognitive tasks (Güntürkün & Bugnyar 2016).
A.2 VIRTUAL ENVIRONMENTS
The animal chambers had two 19" LCD monitors situated on opposite sides of the chamber: the LCD monitors were used to present the virtual objects to the chicks. The other two walls and ceiling were uniform white plastic. The floors of the chambers were constructed with wire mesh and had transparent troughs to hold food and water on one side of the chamber. The animal chambers measured 66 cm (length), 42 cm (width), and 69 cm (height).
To simulate the training and testing conditions of the chicks, created virtual replicas of the animal chambers in the Unity Game Engine. We then trained and tested artificial agents in those chambers. The artificial agents had height of 3.5 units and length of 1.2 units. To monitor the agents' behavior, we positioned an invisible camera at the chamber's 's ceiling, capturing the top-view move- ment of the agent. The agent collected visual input (first-person RGB images, 64 64 resolution) through an invisible camera positioned in the agent's head as the agent moved through the chamber. Each image was passed through an encoder network before being passed to the policy network.
A.3
ARTIFICIAL AGENTS
A.3.1 VISUAL ENCODERS
We used small, medium, and large visual encoders (see Ta) The "Small" encoder is a 3-layer convolutional network that outputs a 512-dimensional encoding vector. The "Medium" encoder is a 10-layer ResNet-10 10 model, incorporating an initial convolutional layer and subsequent ResBlocks that outputs a 128-dimensional encoding vector. The "Large" encoder follows a ResNet- 18 architec- ture. incorporating an initial convolutional layer and four sequential layers of ResBlocks, resulting in din-dimnllal embedding vector.
A.3.2 POLICY NETWORKS
The base PPO Policy network uses the output of the feature extractor followed by a 1-layer MLP for the action and value networks to predict the optimal action. The recurrent policy network added
14

---

Published as a conference paper at ICLR 2024
Model
# Params
Residual blocks
Form
Batch Size
Output Size
Small
600,736
None
CNN
512
512
Medium
7M
2
CNN
512
128
Large
12M
CNN
512
128
2iitv2(vit-s/14))
21M
None
Transformer
128
384
SAM
86M
None
Transformer
128
256
Ego4d
21M
None
Transformer
128
384
SimCLR
7M
CNN
128
512
Vanilla VIT
16.4M
None
Transformer
128
512
Table I: Encoder networks
Hyperparameter
Value
policy
CNN policy
n.steps
2048
learning rate
0.003
n. n.epochs
10
ent_coef
0.5
gae_lambda
0.95
discount factor
0.99
clip-range
0.2
clip.range_vf
0
max grad norm
0.5
target kl
0.01
Table 2: Hyperparameter configuration for policy network
an additional 1-layer LSTM for the action and value networks in addition to the MLP layer. The hyperparameter configuration for the policy networks is described in Table 2
A.4
IMPRINTING REWARD
A.4.1
ONE-EYED AGENT
Newborn chicks are naturally motivated to approach conspicuous, moving objects in their envi- ronment and form social attachment those objects. To mimic the internal, social reward of imprinting. we trained our chick agents using an "imprinting reward." The imprinting reward was designed to motivate the agents spend time with the imprinting object by rewarding the agents for being close to the object and looking at the object.
The imprinting reward based on the percentage of the agent's field of view (FOV) that was filled by the imprinting object. First, we computed the bounds of the object in the FOV. Next, we clamped the object bounds the edge of the FOV so that only the visible parts of the object were included. Then, we used the X and Y coordinates of the object in the FOV to determine the maximum height (in y-coordinates) and width (in x-coordinates) the object covered the FOV. Finally, we calculated the imprinting reward object-height x object-width +
When the agent was close to the object and looking at it, the agent received a high imprinting reward. far away from the object and looking it, the agent received small reward.
When the agent When the agent was not looking at the object (i.e., the object was not in the FOV), the agent received reward, regardless of how close it was the object.
Critically, researchers using this benchmark in the future may "turn off" the imprinting reward in favor of using their own intrinsic reward. After all, newbor animals learn through intrinsic rewards. Building ML algorithms that learn like brains will thus require using biologically plausible intrinsic rewards, like curiosity, mimic the learning capacities of brains.
A.4.2 TWO-EYED AGENT
To compute the imprinting reward for the two-eyed agent, calculate the imprinting reward for each eye (see One-eyed Agent section above). Then we compute the average of those two rewards the two-eyed agent imprinting reward. For visualization of the two-eyed agent, see SI Figure 1
15

---

Published as a conference paper at ICLR 2024
Overhead Chamber View
Loft Eye View
Right Eye View
Supplementary Figure I: The two-eyed version of the NETT. The agent is equipped with two eyes, one on each side of the head, with small degree of overlap. The field view of each eye is designed to match that of newborn chicks. (Left) The overhead view of the chamber, agent, and field of view of each eye (red and blue lines). (Right) The image input to the left (red) and right (blue) eyes. Researchers can concatenate the two images, stitch the images, the images to compute depth, whatever else might be useful for mimicking chicks' rapid and robust learning.
Familar Object Famillar Background
Familer Object Novel Background
Familar Object
Familiar Object Novel Background
Faniller Background
vs. Novel Object Noval Background
Novel Object Background
Novel Object Familiar Background
Novel Object Background
M 72%
M = 51%
M = 50%
M = 52%
SD 20%
SD 21%
SD 8%
8D 11%
Smell (PPO)
(226)
(12))
((27)
(29)
p 10-5
p 10-4
p
p .3
d 1.08
dim 0.92
d 0.05
d 0.18
M 73%
M 27%
= 63%
Mi 48%
Medium Encoder (PPO)
= 6.52
=
x29)= 1.80
[(26] = 0.92
p=100:
10-5
p p
p..37
d-12.12
-3%
M 55%
M- 57%
SD - 20%
SD 20%
SD 12%
SD 17%
Large Encoder (PPO)
(/8) 8.59
((28) 4.60
(28) 2.38
(22) 2.33
p
p
p
d= 1.22
0.06.6
0:44
d= 0.43
M 69%
3
M 61%
M 50%
23%
SD 24%
SD 10%
Encoder (PPO)
(29) 4.338
1(9)) =
x 0.42
1(20 0.29
p 10-3
p 10-3
p8.68
.7.77
d 0.80
0.79
0.08
0.05
53%
SD - 20%
SD 20%
6%
SD 14%
Small Encoder (LSTM)
(/9) 5.84
(29)) 6.92
(29) - 1.62
(22) - 1.15
10.5
M 70%
M 35%
= 52%
= 51%
10k Training Episodes
SD a 18%
SD 20%
SD 10%
8D 7%
Smell (PPO)
=
(129)
(/20) -
(12)) 1.07
p 10-5
p 10-3
p
p 29
d 1.10
dim 0.74
d = 0.22
m 0.20
M 84%
M 34%
= 61%
48%
Training Episodes
Small Encoder (LSTM)
x23)= 3.77
129) 5.16
(29) = 0.48
((29) 2.53
p 10-3
10-4
p 66.6
p= -02
0.69
0:009
Table 3: Results from models with visual encoders that learn alongside the policy network
16

---

Published as a conference paper at ICLR 2024
A.5 TRAINING AND TEST DETAILS
Training Phase. At the beginning of each training episode, the artificial chick was spawned at a random position and orientation within the chamber. Each training episode lasted 1,000 time steps. We trained the artificial agents for 1,000 episodes. For the Small Encoder, Medium Encoder, Large Encoder, DVS Encoder, and 10k Training Episodes agents, the encoder and policy networks were trained together during the Training Phase. After the Training Phase, the network weights were frozen for the Test Phase (i.e., the artificial agents did receive any rewards during the Test Phase, and learning was discontinued).
Test Phase. Each artificial chick performed 1,080 object recognition trials (40 trials for each of the 27
combinations presented to the chicks). At the beginning of each test episode, the artificial chick was spawned at the middle of the chamber, facing blank wall. Each object recognition trial consisted of 1,000 time steps. Performance statistics are provided in SI Table 3
A.6 T-SNE VISUALIZATION OF VISUAL ENCODERS
To visualize the internal working of the encoders, we generated t-SNE plots of the features encoded by the encoder networks. To create each visualization, we first created dataset of images (first- person views from the agent) for each of the six backgrounds. We then extracted the encoder from the frozen model that was saved at the end of the training process. Finally, we ran the images through the encoder to generate feature embeddings. These embeddings were then visualized using t-SNE plots (shown in SI Figure
Tost Animation
Small Visual System 3-Layer CNN
Medium Visual System 10-Layer
Object Background A
Imprinted Object
Imprinted Object
Objoct Background
Object 1, Background C
Object Background
Imprinted Object
Imprinted Object
Y
Objoct Background B
Obycct Background C
Large Visual System 16-Layer CNN
Dynamic Visual (DVS)
LSTM Policy Network 3-layer
Imprinted Object!
Imprinted Object
Imprinted Object
Imprinted Object
Imprinted Object
Object
>>
Supplementary Figure 2: t-SNE visualizations of the feature embeddings for images from the Test Phase. Each gray box shows sample of t-SNE visualizations for a single ANN architecture. Top t-SNEs in each box are from agents imprinted to Object 1, and bottom t-SNEs in each box are from agents imprinted to Object 2. Each selected agent in each box also had a different imprinted back- ground A (left), B (middle), or C (right). For all of the ANN architectures, the feature embeddings clustered based on the background rather than the object in the image.
A.7 TRAINING VISUAL ENCODERS 'THROUGH THE EYES" OF NEWBORN CHICKS
To test whether artificial agents can solve this task when learning from stable visual features, we pre- trained visual encoders using simulated first-person images from inside the virtual controlled-rearing chamber, using the approach described by Lee et al. (2021a) and Pandey et al. (002)). To simulate the visual experiences of the chicks in Wood & Wood (2021). ), we simulated the visual inputs in the chick's 's environment by recording the first firs--person images acquired by an agent moving through the virtual chambers (SI Figure 3). The agent could move forward and backward, turn left and right, and
17

---

Published as a conference paper at ICLR 2024
Stage 1:
Stage
Stage aganti
through the ayaal nomeomm diicka
Trout uncoderr artifiiall apant
Rearing
incied visua system emboded
erwcsserr forr click
Enseder
Aponi
Passhvoly Trained
Enooder
Actively Trained
1. Simulats visual espariences during Training Phose
2. Train CNNs with simulated Images from Training Phase
Vitual cortroled-sering dhamber
R01
Simulatee first person images from Training Phase
Key
Palkcy Natwork: PPO
Polay Network USTM
Supplementary Figure 3: Training visual encoders 'through the eyes' of newborn chicks. In Stage 1, simulated the visual experiences of chicks during the training phase, then trained CNN encoders with those simulated images using a CLTT learning objective. In Stage 2, we froze the encoders and used them visual systems for embodied agents. In Stage 3, we trained and tested the agents (with frozen encoders, but learning policy networks) in the NETT task. Bottom panel shows performance.
rotate its head along the three axes of rotation (yaw, pitch, roll). We collected 80,000 first-person images for each of the six rearing conditions and used those images to train the encoders.
For the encoders, used CNN (ResNet) backbone with 10 layers. We then trained the CNN by exploiting the temporal structure of natural visual experience. There ample evidence that newborn animals (including chicks) time as teaching signal, suggesting that unsupervised temporal learning plays a foundational role in the earliest stages of visual learning (Wood) 2016]] Wood & Wood 2016 2018 2021; Matteucei & Zoccolan 2020). To mimic early temporal learning, we used a contrastive learning objective function that leverages the temporal structure of visual experience, without relying on arbitrary image augmentations labels (Schneider al., 202T). I). The algorithm-Conieee
Learning Through Time (CLTT) contrasts temporally adjacent instances (positive examples) against randomly selected adj-adjacent instances (negative examples), thereby learning representations that capture the underlying dynamics, context, and patterns across time (SI Figure 3). After training the encoders, we inserted them into the artificial agents, then trained and tested the agents in the NETT. The encoders were frozen during the RL training.
The encoders were pre-trained in the same environment which we trained the artificial agents. We reasoned that training the encoder and policy networks sequentially, rather than simultaneously, might help the agents solve the task, since the frozen encoders would provide stable visual features for learning the embodied object recognition task (as demonstrated by Parisi et al. (022). However, like all the other agents we tested, the agents failed to solve the task, largely learning background- dependent object representations (SI Figure
18

---

Published as a conference paper at ICLR 2024
Familir Objecl Background vs.
Familiar Object Novel Background
Familar Object
Familiar Object
Famillar Background
Novel Background
vs.
vs.
va. Noval Object Novel Background
Novel Object Novel Bazkground
Novel
Noval Object Familiar Background
Model
Familiar Background
M 58%
M = 41%
M 50%
= 50%
10%
9%
Prefrained SimCLR (PPO)
(23)) 5.19
((29) - 5.64
((88) - 0.22
[(2]) 0.23
< 10-4
p.x 10-5
p -38
p .82
d 0.95
d 1.03
d = 0.04
0.04
SD 13%
SD = 9%
SD 10%
SD 5%
Pretrained SimCLR (LSTM)
(29)
=
(229= 1:18
[ 0.94
p= -51
p .03
p 25
p
d 0.12
d:40.4
d d=00211
d= 0.17
Table 4: Results of artificial agents with encoders pre-trained 'through the eyes' of newborn chicks
A.8 INSERTING POWERFUL PRE-TRAINED ENCODERS INTO ARTIFICIAL AGENTS
We also tried inserting variety of powerful pre-trained visual encoders into the agents, including DINOv2 (Oquab et al. 2023). Ego4D (Radosavovic et al. 2023), and Segment Anything (Kirillov et al. 2023). The pre- trained encoders were trained millions of images/videos capturing diverse real-world experiences, far exceeding the sparse visual experiences of the chicks. This approach thus violates the spirit of mapceee-e NETT, which involved matching the visual experi- ences acquired by newborn animals and machines. Nevertheless, this approach can provide valuable insight into why artificial agents fail at this task. We found that agents equipped with powerful pre-trained encoders still fail to solve the one-shot learning task solved by chicks. While these pre- trained models perform well on computer vision benchmarks for object segmentation, they are not sufficient to solve the one-shot learning task solved by chicks (SI Figure When we visualized the outputs of the pre-trained encoders, we found that some of the attention heads correctly segmented object features from background features. Thus, we speculate that the problem is not discovering useful visual features (pre-trainee trained encoders appear to do this), but rather discovering which features to use to solve the embodied object recognition task.
Pre-trained Encoders. We used the following pre-trained encoders (SI Table I:
1. Dinov2 Oquab et al. 2023) is self-supervised learning technique applied on Vision Transform- ers developed by Meta Research. The model enables all all-purpose visual features i.e., features that work across image distributions and tasks without finetuning.
Segment Anything Model (SAM) (Kirillov et al. (023) is a foundation model trained on over billion annotations, predominantly for natural images, that is intended segment user-defined ob- jects interest in an interactive manner. The model was developed and released by Meta Research.
3. Ego4D (Radosavovic et al. 2023 is pretrained masked autoencoder (MAE) model with ViT-S architecture trained on the Ego-Soup dataset, which consists of Ego4D and other egocentric human video datasets.
19

---

Published as a conference paper at ICLR 2024
Stage 1:
Stage Traut uncoaler in artiicial aegerd
Stage agenti in
predinined visual emboderr
Pre-trained Encoder
scile Iiiual dell
pre-raieed ancade as vieual
Segrent Ansthngg
Eqo4D Model
Segnent/Aeythingg
DNNOV2
Polacy PPO
Policy: LSTM
Polccy PPO
Policy LBTM
Egu4D Model
Key
Policy: PPO
Policy:
Supplementary Figure 4: Schematic of artificial agents with pre-traine pre-trained visual encoders. In Stage 1, selected set of pre-trained encoders that are noted for their segmentation abilities. In Stage 2, we inserted the pre-trained encoders as visual systems for embodied agents. In Stage 3, we trained and tested these agents (with frozen encoders, but learning policy networks) in the NETT task. Bottom panel shows performance.
Familar Objecl Famiiar Background
Familiar Object
Familar Object
Familia Object Novel Background
Novel Background
Familar Background
vs. Novel Object Novel Background
v5.
vs.
vs. Novel Object Novel Background
Novel Object
Novel Object Femilar Background
Model
Familiar Background
M 62%
M 46%
M 53%
M - 53%
SD 18%
SD 16%
SD 12%
SD 6%
Pretrained SAM (PPO)
x29)=
128) = 1.30
x20)= 1.59
p= 0002
p=
p=..1
.01
d 0.62
di 0.24
d 0.29
d= 0.53
M
46%
= 47%
M 51%
$D 18%
SD 19%
9%
SD 4%
Prefrsined SAM (LSTM)
029) 2.19
122) - 1.12
(220) - 1.71
(12) - 1.21
.4
p0010
244
M 67%
M 33%
M 54%
Mm 50%
SD - 12%
SD 11%
SD = 8%
SD 4%
Pratrained DINOv2 (PPO)
x
1(29) =
1/28)= 2.57
122) 0.19
p 10-7
p
p=
p5.86
d 1.37
d= 1.06
d 0:40
d= 0.03
7171% 71%
M = 34%
M 59%
M 51%
13%
SD 16%
7%
Pretrained DINOv2
(LSTM)
(28)-8.0
(12)) 5.31
(28) 2.67
(26) 1.64
p*
10-1
pm.01
p p1.11
d= 1.64
d.99.9
d 0.50
d= 0.31
M 61%
M 45%
M = 49%
M= 51%
SD 13%
SD 14%
7%
SD
Pratrained Ego4d (PPO)
(28) 4.51
129) =
1/2))= = 0.72
((29) = 1.67
p< 10-4
p= .07
p 48
p=.11
d 0 82
d 0.35
0.13
d 0.31
53%
52%
SD - 18%
SD 20%
SD = 8%
SD - 3%
Pretrained Egold (LSTM)
1/29) 3.26
((2)) = 2.35
1/28) m 2.18
1(29) m 3.57
p= 003
p .011
4/5 0.43
0.40

---

Published as a conference paper at ICLR 2024
Table 5: Results from models with encoders pretrained on massive visual diet
A.9 NETT BENCHMARK FLEXIBILITY
We emphasize that researchers can flexibly control how artificial agents are trained the NETT. Our NETT runs with Stable Baseline 3 (Raffin et al. 2021), researchers vary (among other pa- rameters) the number of training episodes, length the training episodes, batch size, number of test episodes, length of test episodes, learning rate, architecture of visual encoders, architecture of policy networks, and the nature of the intrinsic reward(s). Users can also vary the image resolution of the eyes (cameras) and whether the agent has one eye or two eyes (SI Figure I]. Finally, to help visual- ize and study machine behavior through the same lens as animal behavior, the NETT allows users to record images from an overhead camera and from a camera capturing the agent's first-person views.
A.10 DATA AVAILABILITY
The object segmentation NETT reported in this paper can be accessed the following link: For additional NETTs and details on
https:/github..omm httigghhhub.commbuuudingggggi wwwp://wiiiiiiiieettoooesssaaaee
running this and other NETTs, see: https://orrgins..
luddy.indiana.ce https://orrgisss.udddiiiiaaaaaa
Object
Object 1
Object Background
Object
Object
Object
Background
Background
Background
Background 2
Background
Small Encoder
(PPO)
Medium Encoder (PPO)
Encoder (PPO)
DVS
(PPO)
Encoder (LSTM)
10k Training
Encoder (PPO)
10k Training
Encoder (LSTM)
Supplementary Figure 5: Performance across the training phase for agents whose visual encoders learn along with the policy network.
21

---

Published as a conference paper at ICLR 2024
Object Background
Object 1 Background
Object Background
Object 2 Background
Object 2
Object 2 Background
Background
Pretrained SimCLR (PPO)
SimCLR (LSTM)
Supplementary Figure 6: Performance across the training phase for agents whose visual encoders were pre-trained with simulated visual input from the digital chamber.
Object Background
Object Background
Object Background
Object 2 Background
Object Background 2
Object Background
SAM
(PPO)
SAM
Encoder (LSTM)
DINOv2 Encoder (PPO)
DINOv2 Encoder (LSTM)
EG04D (PPO)
Encoder (LSTM)
Supplementary Figure Performance across the training phase for agents whose visual encoders pretrained on a large and varied visual diet.
22


Reviews:
[{"review_id": "CfKa7fQWhr", "reviewer": "Reviewer_gdnS", "summary": "This paper conducts a comparison study on the one-shot object segmentation ability of real animals and machines. Specifically,  the authors create simulated ‘digital twin’ environments that mimic the rearing condition of real biological newborn chicks. In the simulated environments, the ‘artificial chick’ is trained via deep reinforcement learning to segment objects. The experimental results show that ‘artificial chick’ failed in the one-shot object segmentation task while biological chicks are able to solve it.", "questions": "1. According to section 2, the biological chicken data is from previous work (Wood et al. 2021). However, in the abstract, the authors claim that “we raised newborn chicks in controlled environments …”. This seems contradicting. Could you clarify?    \n\n\n2.  In Introduction, it reads “However, most studies with newborn subjects have produced data with a low signal-to-noise ratio,”. The reviewer has difficulty following the content. Please give citations for ‘some studies’ and elaborate what is the ‘produced data’ here.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "Originality and Significance:    \nThe reviewer found the idea of comparing machine and real animals' learning ability in a strictly controlled environment interesting. This line of study might help us learn more about the similarity and difference of animal brain and deep neural network. \n \n\nQuality:   \nWhile the idea of the paper is interesting, the reviewer found some claims are not well-supported by the experiments. Please see the Weakness section for more details.\n\n\nClarity:    \nThe presentation is mostly clear. However, many improvements, particularly on the figures, are needed.  Please see the Weakness section for more details.", "weaknesses": "1. The technical contribution of the paper is limited. Specifically, the proposed ‘artificial chick’ is a PPO agent trained with ‘imprinting reward’, which encourages the agent to move close to an object. It is not surprising that this simple baseline with a heuristic reward fails to solve the one-shot object segmentation task.   \n\n2. The reviewer found main claims in the paper are not well-supported by the experiments. Specifically, the authors claimed that ‘’... none of the algorithms learned background-invariant object representations that could generalize across novel backgrounds and viewpoints”. However, the ML algorithm the authors used in the paper are just  PPO with different architectures and a heuristic reward. The reviewer thinks those simple methods well-represent the state-of-the-art one-shot object segmentation methods. There are many works on one-shot segmentations [1, 2]. Incorporating those methods in the ‘artificial chick’ might make the experiments more convincing.     \n\n3.  The reviewer has concerns on the significance of the paper. The main finding here is that the RL trained agent failed the one-shot segmentation task. It seems expected. Providing more insights and analysis of why it fails may make the paper more valuable.   \n \n4.  The presentation in the experimental section is unclear. Particularly, the text and numbers in figure 3, which show the only experimental result of the paper, are ineligible.  \n\n\n\n[1] Learning to Segment Rigid Motions from Two Frames, Yang et al., CVPR 2021.   \n[2] One-Shot Learning for Semantic Segmentation, Shaban et al., BMVC 2017", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NuFUZUqQvI", "reviewer": "Reviewer_oTEc", "summary": "## Update\n \nI thank the authors for the answer. I stand by initial review and believe this paper should be accepted.\n\n## Original\n\nThis paper aims to benchmark the intelligence of embodied AI systems against that of biological ones (animals). Towards this end, they propose a new benchmark, the Newborn Embodied Turing Test (NETT). \n\nIn NETT, an agent is born into an environment with a display showing a singular rotating object. The agent must then learn to properly identify that object when presented with distractor objects or when it is placed on novel backgrounds. The biological agents are chicks that learn this skill via filial imprinting and the artificial agents are DRL agents that learn this via an filial-imprinting-like reward.\n\nThe authors find that while biological chicks are well suited to learn this task, their artificial counterparts are not. The authors consider a variety of DNN architectures and find that none perform significantly above chance.", "questions": "### Questions\n\n1. Did the authors perform the biological chick experiments themselves or are those directly from Woods & Woods 2021? (I was unable to get access to this paper to check myself.) If the authors performed this experiment, has it been checked by an Institutional Review Board (IRB) or similar?\n\n### Suggestions for Improvement\n\nThis paper reminded me of \"Exploring Exploration: Comparing Children with RL Agents in Unified Environments\" by Kosoy et al. These two works perform different experiments and have different goals, but the authors may want to mention this work.\n\nSome \"high-water\" marks for artificial chicks would be useful. If one was trained on the test set, how would it do? Ideally it should perfectly solve the task. If one was trained with a singular object but many different backgrounds, how would it do?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper presents a convincing experiment that directly compares the capabilities of biological and artificial chicks. I really like how controlled the setup is. You can't perfectly control for all the differences between biological and artificial systems (as the authors note in their limitation section), but this paper does an admirable job at reducing this gap as much as possible.\n\nThe authors examine a variety of different DNN architectures with the aim to close the gap between biological and artificial chicks.\n\nThe discussion section is well-written and presents a balanced discussion that points out the limitation of the experiments presented and contains interesting pieces of information.", "weaknesses": "Figures in the paper are very low resolution (and possibly heavily compressed with lossy compression). This is particularly noticeable for figure 3, which is nearly illegible.\n\nThe number of trials used for training artificial chicks seems rather small at 1000. Similarly, the episode length of 1000 seems long given how small the environment is. Why were these numbers chosen? Are they similar to the number of times the biological chicks turn towards their object and how long they stay facing it?\n\nThere has been work in deep reinforcement learning algorithms that may be applicable here. One example is as DAAC and IDAAC (\"Decoupling Value and Policy for Generalization in Reinforcement Learning\", Raileanu and Fergus).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CWA6QbLjnf", "reviewer": "Reviewer_FxD2", "summary": "The authors aimed to put both state-of-art vision models and biological vision on similar levels of data diet. They do this by rearing newborn chicks in a very controlled visual environment where they have access to limited visual experience. By exploiting the imprinting ability of chicks, their ability to segment foreground from background is studied. Then vision models are trained with exact same data by instantiating them as the vision modules in a RL agent rewarded to go towards the imprinting object. Finally both the models and chicks are probed for their ability to segment new objects and/or new backgrounds and it was found that the models fare much worse compared to the newborn chicks. This indicates that the state-of-art models lack something that newborn chicks' visual system has.", "questions": "* Why does Figure 1 make it seem like this is a cyclic process? I don't see why it is put on a circle - step 1 does not follow step 6?\n* I wish this was called something other than a Turing test. I think most people (as far as I know) think of the imitation game when they hear Turing test while this is very different. I feel like this is significantly different enough to warrant a different. I almost feel like calling this Turing test is underselling it. Would another name that highlights the \"limited exposure to environment\" aspect of this work be better suited? Turing test says nothing about how much experience the models get to have?\n* Section 2.1 : Was the second display blank (all black) or just a background?\n* Section 2.2 : Do the RL agents need to be called \"artificial chicks\"? I think calling them \"RL agents\" or something is less confusing. \n\n\n\nImprovement to the paper -\n* All the figures need major revamp. The text is barely readable and they need to be of higher resolution. Figure 2 needs to say what \"F\" and \"N\" stand for. Figure 2 could also be made such that the backgrounds and foregrounds are easily distinguishable.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Rarely do I see works with this novelty, quality and significance. \n\n* Studying deep learning models in a setting where the data diet is controlled is timely and needed. \n* The data collected from newborn chicks is compelling and novel. \n* The results showing that current models lack the necessary mechanisms can be very impactful. \n* The data collection procedure and analysis opens up a ton of possibilities for future work. \n* The paper is overall well written.", "weaknesses": "* Perhaps the biggest weakness is that the vision models here are trained in a way that is not very indicative of the state-of-the-art. Most vision models that is used in the community are usually a combination of self-supervised training and supervision with a lot of exemplars. While this might diminish how relevant this work is for the community, it does not take away from the significance of the work since the goal of the work is to study models when they get only as much visual experience as newborn chicks (i.e newborn chicks are not getting a ton of supervision). \n* PPO agents are trained for a lot of episodes (original PPO paper (Schulman, 2017) did 1M episodes) usually while this work only does 1000 episodes. This is probably okay since this setting is much simpler. But I would still train one model for much more steps to rule out strange behaviors (like grokking (Power, 2022))\n* The paper might benefit from an expanded section on the differences between birds and mammal brains, especially since the audience is likely to be more from the computational side. While I do appreciate the effort in section 1.1, it might be helpful to expand it a bit more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zvvjg0cNe8", "reviewer": "Reviewer_ELqk", "summary": "This work introduces a benchmark, the Newborn Embodied Turing Test (NETT), and reveals the limitations of current machine learning algorithms compared to newborn brains in object segmentation tasks. However, the experimental results of this work are insufficient to validate the claimed fairness of the NETT.", "questions": "1.\tThe evaluation of this proposed NETT benchmark is not comprehensive enough. Many variables were not taken into consideration during the data collection process for the biological chicks, such as their binocular vision compared to the single camera of artificial intelligence chicks. Meanwhile, biological chicks might roam around while artificial intelligence chicks do not. These unaccounted variables raise doubts about the effectiveness of the proposed benchmark. The validity of the benchmark may be compromised due to the discrepancies in sensory capabilities and behavior between biological and artificial chicks. Further refinement and consideration of these variables are necessary to ensure the reliability and validity of the benchmark.\n2.\tThe presentation of this work still needs improvement, as the figures are not in vector format. For example, Figure 3 lacks clear labels for the x and y axes, making it difficult to understand the meaning of the figure.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThis work's main contribution is the introduction of the Newborn Embodied Turing Test (NETT) benchmark, which enables a direct comparison of the learning abilities between newborn animals and machines in object segmentation tasks.", "weaknesses": "1.\tThe evaluation of this proposed NETT benchmark is not comprehensive enough. Many variables were not taken into consideration during the data collection process for the biological chicks, such as their binocular vision compared to the single camera of artificial intelligence chicks. Meanwhile, biological chicks might roam around while artificial intelligence chicks do not. These unaccounted variables raise doubts about the effectiveness of the proposed benchmark. The validity of the benchmark may be compromised due to the discrepancies in sensory capabilities and behavior between biological and artificial chicks. Further refinement and consideration of these variables are necessary to ensure the reliability and validity of the benchmark.\n2.\tThe presentation of this work still needs improvement, as the figures are not in vector format. For example, Figure 3 lacks clear labels for the x and y axes, making it difficult to understand the meaning of the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)