Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes
Authors: Aran Nayebi
Rishi Rajalingham
Mehrdad Jazayeri
Guangyu Robert Yang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: neural coding
mental simulation
foundation models
primate frontal cortex
TL;DR: Our findings suggest that the neural mechanisms and behaviors of primate mental simulation are thus far most consistent with being optimized to future predict on reusable visual representations on dynamic scenes.

Abstract: Humans and animals have a rich and flexible understanding of the physical world, which enables them to infer the underlying dynamical trajectories of objects and events, plausible future states, and use that to plan and anticipate the consequences of actions.
However, the neural mechanisms underlying these computations are unclear.
We combine a goal-driven modeling approach with dense neurophysiological data and high-throughput human behavioral readouts that contain thousands of comparisons to directly impinge on this question.
Specifically, we construct and evaluate several classes of sensory-cognitive networks to predict the future state of rich, ethologically-relevant environments, ranging from self-supervised end-to-end models with pixel-wise or object-slot objectives, to models that future predict in the latent space of purely static image-pretrained or dynamic video-pretrained foundation models.
We find that ``scale is \emph{not} all you need'', and that many state-of-the-art machine learning models fail to perform well on our neural and behavioral benchmarks for future prediction.
In fact, only one class of models matches these data well overall.
We find that neural responses are currently best predicted by models trained to predict the future state of their environment in the \emph{latent} space of pretrained foundation models optimized for \emph{dynamic} scenes in a self-supervised manner.
These models also approach the neurons' ability to predict the environmental state variables that are visually hidden from view, despite not being explicitly trained to do so.
Finally, we find that not all foundation model latents are equal.
Notably, models that future predict in the latent space of video foundation models that are optimized to support a \emph{diverse} range of egocentric sensorimotor tasks, reasonably match \emph{both} human behavioral error patterns and neural dynamics across all environmental scenarios that we were able to test.
Overall, these findings suggest that the neural mechanisms and behaviors of primate mental simulation have strong inductive biases associated with them, and are thus far most consistent with being optimized to future predict on \emph{reusable} visual representations that are useful for Embodied AI more generally.

Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes
Aran Nayebi" Rishi Rajalingham' 14, Mehrdad Jazayeri' and Guangyu Robert Yang 12,3
'ceoovern Institute for Brain Research, MIT; Cambridge, MA 02139 "Department of Brain and Cognitive Sciences, MIT; Cambridge, MA 02139 Department of Electrical Engineering and Computer Science, MIT; Cambridge, MA 02139 *Reality Labs, Meta; 390 9th Ave, New York, NY 10001 "Correspondence: anayebi leroiimii...mm edu
Abstract
Humans and animals have rich and flexible understanding of the physical world, which enables them to infer the underlying dynamical trajectories of objects and events. plausible future states, and use that to plan and anticipate the consequences of actions. However, the neural mechanisms underlying these computations are unclear. We combine goal-driven modeling approach with dense neurophysiolog- ical data and high-throughput human behavioral readouts that contain thousands of comparisons directly impinge on this question. Specifically, we construct and evaluate several classes of sensory connnty-c networks to predict the future environments, ranging from self- seff-superiieed
state of rich, ethologically--eeeee end-to-end models with pixel-wise or object-slot objectives, to models that future predict the latent space of purely static image-pretrained or dynamic video- pretrained foundation models. We find that "scale is not all need",, and that many state-of-le the-art art machine learning models fail to perform well on our neural and behavioral benchmarks for future prediction. In fact, only one class of models matches these data well overall. We find that neural responses are currently best predicted by models trained to predict the future state of their environment in the latent space of pretrained foundation models optimized for dynamic scenes in a self- self-supervised manner. These models also approach the neurons' ability to predict the environmental state variables that are visually hidden from view, despite not being explicitly trained to do so. Finally, latents are equal. Notably, models that future predict in the latent space of video foundation models that optimized to support a diverse range of egocentric sen- sorimotor tasks, reasonably match both human behavioral error patterns and neural dynamics across all environmental scenarios that we were able test. Overall, these findings suggest that the neural mechanisms and behaviors of primate mental simulation have strong inductive biases associated with them, and are thus far most consistent with being optimized to future predict on reusable visual representations that are useful for Embodied Al more generally.
find that not all foundation model
1 Introduction
Within the span couple seconds, we are able draw rich inferences and make predictions about novel scenes (Smith and Vul 2013 [Battaglia et al. (2013). A dominant cognitive theory has been that the brain builds mental models of the physical world, using those models to make inferences about the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In the past decade, this hypothesis has been supported by comparisons of human behavior to computational models which predict what will happen next in physical scenarios via forward simulations resembling those of game engines in modern video games [Battaglia et al. 2013 Hamrick et al. 2016, Ullman et al. 2017]. Both neuroimaging work in humans [Zacks] 2008 Fischer et al., 2016, Schwettmann et al. 2019 Pramod et al., [022] and recent electrophysiological work in monkeys [Rajainggam et al., 2022a bJ has further provided evidence for the neurobiological basis of mental simulations in the frontoparietal network (FPN) of primates, a large-scale network consisting of several interacting brain regions. In this work, we make progress towards understanding the neural and behavioral mechanisms of mental simulation by constructing models that perform this behavior in rich, naturalistic environments. Specifically, we aim to determine what inductive biases (in the form of loss function, architecture class, and pretraining environment) enable the brain to generally perform mental simulation across a range of environments and scenarios, from unstructured, continuous sensory inputs. In particular, we assess not only model generalization to novel, high-variation examples within the same environment, but also structural generalization new environments and scenarios altogether.
future state of its environment [Craik] 1943].
Predicting the physical dynamics of environments is also critical to progress in Embodied AL. One common paradigm for learning these dynamics has been as next frame prediction problem via
pixel-wise loss [Villegas et al. 2019 WW et al. 2021 Babaeizadeh et al. 2021 Nash et al. 2022]. These losses emphasize prioritizing accurate prediction of every detail of given scene's dynamics. However, fine grained prediction of upcoming video frames would require near wear-perfect knowledge of the world's physical state (akin to Laplace's ' Demon), which may explain the observation why many of these models tend to underfit in high variation, naturalistic visual environments, with recent efforts aimed at scaling these methods up primarily by increasing their parameter count (Dasari et al. [019 Babaeizadeh et al.) 2021]. It is therefore unclear how much these resultant learned representations are able successfully capture general physical understanding.
Another recent class of approaches involves the design of visual "foundation models" (Bommasani et al, 2021]. trained on large amounts of webscale images and egocentric videos to develop an implicit representation of the world, that can then be deployed to downstream robotic manipulation tasks [Nair et al. 2022 Ma et al. 2023 Majumdar et al.) 2023 Of course, these models are not directly designed to do explicit physical simulation, but we equip them with a forward dynamics model that can be rolled out for an arbitrary number of timesteps. We ask whether such dynamically- equipped foundation models have learned physical knowledge by evaluating their generalization both to new scenarios and environments, and whether their representations bear any similarity to humans and non-human primates performing the same tasks?
In particular, we find strong constraints on primate mental simulation, especially when examining generalization within and across diverse environments. Our core result is that a small class of models best match primate frontal cortex neural dynamics while the animal plays ball interception task in which the ball trajectory is partially occluded prior to hitting the paddle "'MentalPoon"" developed previously by Rajalingham et al. [2022a] Overall, one model class can match both neural response dynamics and human behavioral patterns reasonably well - namely, dynamics that are optimized to future predict in the latent space of VC-1, video foundation model pretrained
the largest variety of egocentric sensorimotor settings overall. We therefore currently observe a tight correspondence between the ability to predict fine-grained neural and behavioral responses for the mental simulation phenomenon, and developing useful representations for Embodied AI more generally.
2 Related Work
Mental simulations have been studied at the level of neural activity only very recently. Prior human neuroimaging studies [Zacks 2008, Fischer et al.) 2016 Pramod et al., [022) in the past decades showed elevated levels of -eaeoooogee--leeldeeeeeeeeee
(BOLD) signal mental simulation, although they do not have the required resolution to verify that these dynamics are actually represented in underlying neural activity. Rajalingham al. [202bb] was the first study to show that neural dynamics recorded from macaque dorsomedial frontal cortex (DMFC), track the occluded ball by comparing these dynamies to Recurrent Neural Networks (RNNs) that simulate the occluded ball's position in Mental-Pong. and finding that they better match these dynamics than RNNs that only perform ball endpoint prediction. However, monkeys can perform these tasks without substantial training, suggesting that they are already equipped with the necessary neural foundations for mental

---

simulation in this environment. Therefore, we aim to also build networks that are not explicitly trained on Mental- -Pnt itself, but are tasked to generalize to this novel setting as test of their general understanding of physical scene dynamics- chiefly developed through three factors: their architecture, optimization objective, and pretraining on a naturalistic environment.
Additionally, we constrain our models by evaluating them against high-throughhee human behavioral data (from Bear al. 2021]) in more naturalistic, 3D environments than Mental-Pong alone, which goes beyond prior behavioral studies that either rely on a narrow range of physical scenarios [Shepard and Metzler 1971, Cooperau and Shepard 1973], such as block towers with several cubes of different colors Groth et al., 2018 Li et al. 2016)] or 2D environments that may not generalize to the real common standard to evaluating the everyday physical scene understanding and neural predictivity of these models, especially since they are usually trained on vastly different scenarios and input types. Towards this end, we require models to operate under similar constraints as the brain, namely (i) to take in unstructured visual inputs across a range of physical phenomena, (ii) to generate physical predictions for each scene (i.e. producing "behavioral outputs"), and (iii) to consist of internal units that can be compared to biological units (i.e. containing "artificial neurons").
world (Bakhtin et al. 2019 A key challenge to addressing these questions
Taken together, these three requirements encompass a large class of functionally reasonable hypothe- ses that we call evensory-cognitic networks", and includes the two broad approaches mentioned in [T] However, they do exclude some approaches for example, particle-based graph neural network
dynamics predictors [Battagia et al., 2016 2018 Li et al., 2019, Ganzaez-Gonzalez et al. 2020)
that take the ground truth simulator state as input (which may not be readily available in real-world
situations, failing to satisfy requirement (i)) or probabilistic programs [Battaglia et al. 2013 Hamrick
et al.) 2016 Ullman et al. 2017] (which fail to satisfy requirements (i) and (iIi).
Nonetheless, we believe these latter approaches from cognitive science are a useful guide for building improved models from pixels that satisfy the three requirements - especially in terms of assessing whether the prediction problem lies at the level of vision, or the dynamics that interfaces with For example, [Bear et al. [021]] Figure 5] demonstrated that particle-based graph neural networks (e.g. DPI-Net [L al., [2019] with access to the ground truth simulator state approach numan--eved physical predictions on the OCP task that we consider in Section 4 a diverse range of scenarios. This observation indicates that the onus mainly rests on learning good visual representation, as these models assume perfect observability of physical dynamics. This further motivates our enforcement of requirement (i) as being a worthwhile challenge, as well as our latent future prediction approach of "factorizing" the mental simulation problem into separately pretrained vision and dynamics modules.
3 Methods
To tackle this question, we took a hypothesis- approach and built sensory-cognitive networks that performed mental simulations of their environment in variety of different ways (schematized
in Figure[] [ Specifically, we tasked models to operate on the Physion dataset [Bear et al.) 202 I]. large-scale video dataset that focuses on everyday physical understanding, consisting of eight different scenarios in a simulated Unity3D-based environment (the ThreeDWorld simulator (Gan et al. 2021) with roughly 2,000 scenes each. Models are pretrained on all eight scenarios of Dominoes, Support, Collide, Contain, Drop, Link, Roll, and Drape; which together cover many scenes involving rigid and soft bodies.
We additionally pretrain subset of models on Kinetics-700 [Carreira et al. 2019 which consists
of over 500,000 training videos from real-world (rather than simulated) scenes from 700 different action categories, in order to assess whether a different dataset of increased scale is beneficial or not, relative to Physion.
We consider models from several sensory-cognitive hypothesis classes:
1. End-to-end self-supervised future prediction:
(a) Pixel-wise: This class of models consists of three parts: encoder, dynamics, and decoder; which are altogether pretrained end-to-end with a pixel-wise loss to predict the next frame. We consider a current state-of-the-ar model in this model family, FitVid (Babaeizadeh et al. 202T], and we pretrain it on Physion (with and without temporal augmentations such as RandAugment (Cubuk et al. 2020 ); along with an
3

---

(A) Model Pretraining
Inputs
Sensory-Cognitive Hypothesis Classes
Ground Truth
Physion
Latent Future Prediction:
2. Dynamics Protraining Stage
Domi
Support
Ego4D, Pretraiining Stage
P
Foundation Model
T+1 Prediction
End-to Future Prediction:
Encoder
Decoder
Pixel wise
Encoder
Object-sfot
Timee
Simulated
(B) Model Evaluations
Murman Behavior: Behhavor:PPyyson Object Contect Prediction (OCP)
Macaque Neurophysiology. Mental-Pong
Unobserved
Exampa
Figure 1: Model Pretraining and Evaluation Pipeline: We built and evaluate sensory-cognitive networks for mental simulation. (A) Models pretrained predict the future state of naturalistic environments in a variety of ways. Their dynamics are then compared to high-throughput human behavioral judgements and dense primate neurophysiological response dynamics. Model pretraining can be (1) two-stage, or (2) end-to-end, For the two-stage latent future prediction models, there is first visual module pretraining (blue), then dynamics module pretraining on Physion (peach). Across all models, the dynamics modules are pretrained on the same dataset (Physion), and then evaluated against neural and behavioral data with model weights fixed. (B) Evaluations are multi-fold: (1)
Comparison to human behavior in OCP on held-out scenes across pretrained Physion scenarios. (2) Comparison neural activity in Mental-Pont an -aito--distriuution environment that none of the models were pretrained on.
earlier variant, SVG Denton and Fergus 2018 that can be additionally trained at larger image scales (128 x 128 rather than 64 x 64 pixels).
(b) Object-slot: This class of models is based on the Contrastive Learning of Structured World Models framework ("C-SWM" [Kipf et al. (0200) which contrastively learns more structured object-slot latent states. It consists of encoder (whose size vary: "small", "medium", and "large") that outputs a fixed number of object slots N 10, and graph neural network forward dynamics module that operates on this representation. Since the dynamics module itself does not have temporal dependencies (only spatial), enable temporal dependencies by passing frames the input channel dimension of the encoder, as a sliding window of temporal dimension of T context frames predict the object-slot latent at timestep T This model family one instantiation of the cognitive theory that humans tend to reason about scenes with regards objects and their relations [Baillargeon al. 1985 Spelke 1990 Spelke and Kinzler 2007]. These Gestalt principles therefore may provide physical knowledge that are better-positione to generalize than more fine-grained (pixel- wise) prediction hypothesis that explicitly evaluate on high-throughput neural and behavioral data, in what follows.
2. Latent self-supervised future prediction: These models consist two parts: an encoder E that produces a latent state space h and dynamics module D that predicts the future state

---

purely in the latent space h of & & is fixed and pretrained to perform a challenging vision task ("foundation model"), through which it learns a partial, implicit representation of the physical world. However, since & is not pretrained to do explicit physical simulation, we train additional dynamics D on Physion, that can later be "rolled out" an arbitrary number of steps. More concretely, given T context frames, & produces the latent sequence hTTT, from which is trained to predict hT+11
relatively simple, being either an LSTM [Hochreiter and Schmidhuber [197]] or a continuous-time RNN (CTRNN) [Miller and Fumarola[[011] or "No Dynamics" " as a control, which always outputs the last context frame latent hT of
We consider a variety of foundation encoders, divided into two primary classes based on their type of large-scale pretraining dataset:
(a) Image Foundation Models (Static scenes):
Standard Convolutional Neural Networks (CNNs) VGG16 (Simonyan and Zis serman 2014] and ResNet-50 [He al. 2016 pretrained ImageNet with supervised categorization objective.
Vision Transformer (ViT) Dosovitskiy et al., 2021 based architectures such as
DeiT [Touvron et al. 2021 and DINO [Crron al. 2021 pretrained on ImageNet
with self-supervised objective.
DINOv2 [Oquab et al. 2023]. which is a very recent ViT pretrained on a larger curated dataset called LVD-142M (142 million images).
CLIP RRafford et al. 2021] which is ViT pretrained on 400 million image-text pairs curated from the Internet (250 times larger than ImageNet).
(b) Video Foundation Models (Dynamic scenes):
R3M Naar et al., 2022]. which is ResNet-50 architecture pretrained with temporally-contrastiee video-language alignment objective on 5 million frames from a subset of the recent large-scale Ego4D human video dataset [Grauman et al.
2022
VIP Ma et 2023], which a ResNet-50 architecture pretrained with goal- conditioned value function objective on 5 million frames from a subset of Ego4D. VC-1 [Majumdar et al. 2023]. which is a very recent ViT pretrained on 7 different egocentric video sources (over 5.6 million frames, including Ego4D) relevant to sensorimotor skills, using a self-supervised masked autoencoding (MAE) [He et al. [2022] objective. While MAE is pretrained on individual frames, the statisties of the frames are informed by egocentric motion, unlike webscale images.
In total, these networks encompass both qualitatively distinct hypotheses (pixel-wise vs. object- slot vs. latent future prediction), alongside several variations within each hypothesis class. This combination of diverse networks allows us to potentially differentiate hypothesis classes across a range of functionally capable instantiations of each hypothesis.
4
Comparison to Human Physical Judgements
4.1 OCP task evaluation
In the object contact prediction (OCP) task [Bear et al. 2021 each evaluation scenario involves a red "agent" object and yellow "patient" object (which did not appear during model pretraining). Both humans and models are tasked to predict the probability that they will come into contact. This prediction requires understanding of the relevant physical phenomenon in the given scenario, corresponding to a higher-order readout of the underlying scene dynamics.
4.2 End-to-end pixel-wise future predictors best predict human behavior in the same environment
The model comparison results are summarized in Figure 2 In Figure 2A, we examine held-out scene accuracy of models across all eight scenarios in the OCP task, where chance performance is 50%. We can see that humans are quite reliable on this task, attaining 74.04% average held-out accuracy across scenarios (grey horizontal line). Furthermore, the best models that approach human accuracy models that are pretrained end-to-end with pixel-wise losses (FitVid and SVG). Despite
5

---

(A)
(B)
Human-to-fummn Consistency
3.55
Human Accuracy
Hymmn Response [Pearzon
Heldout OCP Accuracy
6.79
0:81
2.93-
Roundattion
Video Foundatio Modak
Correlation Average
2.50
GROURINN
KIN
Correlation Averegeh Numen Response g
End-to-ff End-to-End
Latent Future Prediction
(0)
End Ead-tt-End
Latent Future Prediction
Correlation perolationn to Average Human Respansa
0.43
R=00 R=:0.665, p 0.001
0.35
(Pearson''s
(Pearsen'
0.20
0.35
8:12
Video Fosedation Medel.
0.50
0.52
0.54
0.55
0.58
0.50
0.62
2.64
2.45
6.03
Heldout OCP Accuracy
Physion
Figure 2: Model Comparisons Human Physical Judgements: (A) Each model is compared to binary ("yes/no") accuracy on the OCP task (chance is 50%). Grey horizontal bar represents the human accuracy on this task. Mean and s.e.m. across all eight Physion scenarios, weighted by number of scenes in each scenario. Hatched bars represent models pretrained on the Kinetics- 700 dataset, rather than on Physion. (B) Each model is compared to human subject judgement probabilities for the OCP task, via Pearson's 's correlation from the Logistic Regression classifier trained on each of the model dynamics. Grey the correlation of human judgement probabilities to each other. Mean and s.e.m. across all eight Physion scenarios, weighted by number of scenes in each scenario. (C) A model's 's match human judgement probabilities is strongly correlated with its OCP task accuracy. (D) Per scenario model comparisons to human subject judgement probabilities for the OCP task. Mean and s.e.m. across all 41 models.
their differences in architecture, both Vid and SVG are comparable to one another in their OCP test set accuracy (61.12%-66.31 attaining non-significant differences in the distribution of their accuracies across scenarios (Paired f-test, minimum Bonferroni corrected p-value of 0.444 across pairwise comparisons). For a fixed architecture, pretraining with increased image size helps improve accuracy somewhat on the OCP prediction task (SVG 128 x 128, rightmost periwinkle bar, vs. SVG 64 x 64, leftmost one), along with temporal augmentations for a high capacity model (FitVid, rightmost dark green bar vs. leftmost one), but these differences overall sinnsiinnft (Paired t-test p = 0.461 and p = 0.196 within the SVG and FitVid architectures, respectively). However, not all end-to-end models match human accuracy well. In particular, see that the object-slotted C-SWM class of models matches human accuracies least well compared to other model classes, despite varying the encoder size. The class of latent future prediction models overall better matches these behavioral patterns than the more explicitly structured C-SWM models, but there does not appear be strong differentiation across foundation models, and also not much differentiation between having a dynamics module vs. using the encoder latents directly (rightmost two bars vs. leftmost bar in each group). This suggests that at least for the OCP task, either Physion is not sufficiently high-variation enough to pretrain the dynamics module or most of the predictivity comes from the physical scene understanding encapsulated in the foundation model encoder latents. The latter appears to be more likely, since pretraining the dynamics module on larger-scale dataset such as Kinetics-700 700 for subset of the models WVP=LLSTM/CTRNN VC-I+LSTM/CTRNN and R3M+LSTM/CTTRN,; hatched bars) does not improve OCP accuracy over the base encoder latents either, relative to pretraining them on Physion (Paired (-test, minimum Bonferroni corrected p-value of 0.066 across architectures).

---

Additionally, we can look at finer- aner-grained error patterns of the probabilities reported by humans compared to those of models (rather than accuracies alone), summarized in Figure Here we see that despite the metric being more detailed, humans are quite consistent with each other, suggesting that this type of behavioral metric is quite reliable (grey horizontal line). In fact, all models appear to be further from the numan-to-human consistency than the OCP accuracy. However, overall, similar trends appear to hold across models as with the OCP accuracy measure - where the end-to-end pixel-wise models (Fi and SVG) match these consistency scores the best across models, and the object-slotted C-SWM models match them the least well. From an Al perspective, it is actually quite relevant to work towards matching human error patterns, as it is highly correlated with the primarily performance based measure of OCP accuracy, as seen in Figure 2C (R = 0.865, p < 0.001).
And as shown in Figure 2P. all models have the most room to improve to match human error patterns and OCP accuracy (Figure (S4) for soft-body interactions (the "Drape" scenario). On the other hand, models seem to do reasonably well on certain rigid-body scenarios, especially "Support" relations where stacks of objects fall over depending on their shapes and arrangement: "Collide", where pairs of objects collide depending
their trajectories and placement; and "Roll", where objects move across a surface either by rolling or sliding. In these scenarios, the best Physion-pretraine models, namely the pixel-wise future predictors, can attain human consistency scores much higher than what we report in Figures 2 and D of around 0.6 (cf. Figure S5B).
5 Comparison to Dense Neurophysiological Data
To gain more insight into model generalization, we compared the above models to neural dynamics recorded from macaque dorsomedial frontal cortex (DMFC), which was shown by Rajalingham jt al. 2022b) to simulate the ball's position Mental-Pong while behind an occluder, until was intercepted by the paddle. This environment is completely dutto--dittiibution for the models, unlike Physion, and therefore tests structural generalization.
5.1 Inter-animal consistency and neural behavioral decoders
For each of the 79 conditions (different randomized start position of the ball), we present the frames the visible epoch (the time up until the ball reaches the occluder) as context frames to the models, and unroll the model dynamics during the occluded epoch of the condition. We then build detailed, physical mappings (schematized in Figure 3A) from the committed model layer latent dynamics to match every single neuron in the population, and the ground truth ball state while occluded, when mental simulation takes place. We enforce the requirement, established in prior work (Nayebi et al. 2021 Cao and Yamins 2021]. that models are at least as similar to given animal's 's neural responses as two conspecifics' neural responses are to each other. When we perform these mappings, we see that both the inter-animal neural predictivity consistency and ground truth ball state decoding from DMFC is quite high (grey horizontal lines in Figures 33 and C. respectively), indicating that these very reliable neural and behavioral measures of the mental simulation phenomenon.
5.2 Neural response predictivity strongly separates models
When we map model units to DMFC units across 79 conditions (different randomized ball start position). we see in Figure 3 that across all architectures, only dynamically-eqqupped models from the video foundation model class predict DMFC dynamics best (46.34-48.83) 48.% neural predictivity), specifically in the latent space of the VC-1 and R3M encoders. However, the VC-1 and R3M encoder's 's latents alone are insufficient to predict the data (19.03% and 24.79% neural predictivity. respectively), although their latents best predict neural responses compared to other foundation models (cf. Figure (SC). Pretraining end-to-end on Physion with either a pixel-wise loss (SVG and
FitVid; 14.44%-25. 25.35% neural predictivity) or an object-slot loss (C-SWM with varying encoder sizes; 13.69% 23.05% 05% neural predictivity) is not sufficient either, indicating a strong constraint on the neural mechanisms of mental simulation being performed on suitable latent space. However, is not the case that any latent space works, since all dynamics trained on encoder latents pretrained on static images, regardless of their imageset scale, attained at most 29.67% neural predictivity. This is suggestive that for relatively small but time-varying stimulus such as the Mental-Pong Pong ball, models that are pretrained on static images may not be equipped to handle the temporal coherence of a single object moving through time, prior studies have mainly examined generalization of
7

---

(A)
(B)
1.0
L Modch M
Inter-animal Consistency
Encoder
0.8
Neural Predicttvvt
Perfect Simulation Oracle
Oracle
50.6-
Monkay
0.04
Image Poundattn Modeb
0.2
M
(C)
Monkey?
End-to-End
Latent Future Prediction
Ball Position Velocity Predictivity
1.0
(D)
Ball Pasition Velocity Predictivity
R=0,701, <c 0.001
DMFC Predictivity
0.8-
0.6
(Peurson's R))
0.6
(Pearson's R)
0.4
0.2
Draitto Endl
0.2
Veleo Nowedition Stadik
0.0
Sintor Latent Future Prediction
0.4
End-to-End
Neural Predictivity (Pearson's R)
Figure 3: Model Comparisons to Neural Response Dynamies: (A) We measure how similar model
dynamics (red) are each primate's 's DMFC neural dynamics up at least how (linearly) similar primates are to each other, via the linear transform I,N (black arrows). We also assess how well models decode ball state (position and velocity) up how well can be decoded from DMFC via the linear transform L (teal arrows). (B) DMFC test set neural predictivity of each model, averaged across five train-test splits. Median and s.c.m. across 1,889 recorded units. For each of the latent future prediction models, there are three bars: the first corresponding to final context encoder latent itself ("No Dynamics"); and the last bars with LSTM or CTRNN dynamics, respectively. For the VIP. VC-1, and R3M based models, we also pretrain the LSTM and CTRNN dynamics on Kinetics- 700 rather than Physion, and their predictivity is displayed in the last two additional hatched bars. The grey horizontal bar represents how well each primate's 's DMFC can predict the other's s via the linear transform T.N schematized in (A), setting consistency ceiling on model predictivity. Position, velocity, and position + velocity oracles are the ground truth ball state values for those variables while is occluded (and not visible to the primate during game play). Dotted "Perfect Simulation Oracle" line corresponds to the position velocity oracle's 's neural predictivity. (C) Ball position and velocity predictivity of each model, averaged across five train-test splits, from the best DMFC fitting model layer used in (B) via the linear transform L schematized in (A). The grey horizontal bar represents the ball position and velocity predictivity from DMFC units. Median and s.e.m. across four quantities: the ground truth (z. y) position and velocity of the ball while it was occluded (in degrees visual angle). (D) Mental-Pong ball position and velocity predictivity (y-axis) neural predictivity (x-axis). Dotted golden line represents best linear fit to the data.
ImageNet-optimized CNNs novel yet static, single-object scenes (Hong et al., 2016). Moreover, even for video optimized foundation models, we see differentiation at the level of loss function. In particular, goal-conditioned self-supervised learning videos, as instantiated by VIP (which differs from R3M primarily in the choice of objective function, as it shares the same ResNet-50 architecture and is also pretrained on Ego4D), fails match DMFC neural dynamics well (7.37%-18. 12% neural predictivity). We also see parsimony at the level of the architecture and pretraining dataset of the dynamics module, as the VC-I/R3M+CTRNN either attains comparable or improved neural predictivity over the more sophisticated VC- VC-1/R3M+LSTM (middle vs. rightmost bars for each encoder type), even when pretraining on a larger dataset like Kinetics- 700 (Figure This suggests that simple dynamics may be sufficient at predicting neural response dynamics given the appropriate visual encoder and highlights the limits of dataset scale alone without changing inductive biases. and VC-I/R3M-CTRNN models approach the neural predictivity of
Both the the ground truth ball position oracle model 150.74%. leftmost golden bar in Figure 3),, with the joint position velocity oracle performing the best (60.65% neural predictivity, rightmost golden

---

bar, which also corresponds to the "Perfect Simulation Oracle" dotted line). This suggests that substantial amount of the neural response variability is devoted to simulating the ball's state while is occluded, as opposed to other static features of the environment. The remaining gap between the "Perfect Simulation Oracle" and the inter-animal consistency may be due to other factors such as time since stimulus onset, attentional demands/eye movements, and perhaps uncertainty when the ball bounces, which can be explored in future work.
In Figure 3C we see that the most neurally predictive -ynamieally-equipped models generalize to the Mental-Pong task, approaching DMFC's s ability to track the ground truth position and velocity of the ball while it is occluded, despite not being explicitly pretrained in this environment. In particular, there is a linear relationship (R 0.701, p < 0.001) between the model's 's ability to generalize to the Mental-Pons PannallPnng environment and its ability to predict DMFC neural dynamics (Figure This relationship indicates that predicting the underlying neural dynamics is in fact behaviorally relevant to effectively simulating the ball's dynamics, which [Rajalingham et al. 2022a b) judged in humans based on eye tracking data. Furthermore, the dyyamically-equipeee VC- I/R3M-based models can best represent the ball position and velocity separately as well (Figure ( demonstrating that they can independently track these variables.
- Video Foundation Models Can Match Both Human Behavioral and Neural Response Patterns Across Environments
[A]
Batar Vodii
Dall Position Velocity Predicuitty
End to End
Imagee Foundiation Modeb
Video Fousdatioon Models
oe Predicttelt cattetttty [Pestsors
3.8
630
0.66
3.5
12:40
20
C.1
25
Heldout OCP Accuracy
Correlation Average Hyman Response
Figure 4: Rynupiea Video Foundation Models Can Match Both Neural and Behavioral Metrics: (A) Ball state (position and velocity) predictivity in Mental-Pong (y-axis) vs. OCP accuracy in Physion (z-axis). Across models, these two metrics appear to be largely independent. (B) The dynamically-equupped VC-1 models (VC-I+LSTM/CTRNN pretrained on either Physion or Kinetics- 700t can reasonably match neural response dynamics in Mental-Pong and match human error patterns in the OCP task, relative to other models. Dotted grey line is the unity line to indicate that better models will occupy the top right of this scatter plot.
Both the OCP human behavioral error pattern metric and the DMFC Mental- neural predictivity metric are linearly correlated to their corresponding behavior of OCP accuracy and ball position prediction (as seen in Figures C and 3P). which suggests that these more fine-gre fine-grained metrics relevant to the underlying behavioral goal that they represent. But how do they relate to each other, and is there model class that can match both reasonably well?
As we can see in Figure 4A, the two behavioral goals of Mental-Pong ball state predictivity and Physion OCP accuracy do not appear to be very related across models (R s -0.241, R 0.134.. This suggests that held-out accuracy to novel scenes within environment as in OCP, does not imply that the same model will generalize to completely new environmental dynamics as in Mental-Pone Pong. In particular, it is important to consider generalization to new environments, since the end-to-end pixel-wise models such as SVG and FitVid subtly overfit to this environment (attaining the highest OCP held-out scene accuracy), but they fail to generalize as well to the completely out-of-distribution Mental-Pong setting (rightmost periwinkle and dark green points in Figure 4 For example, for FitVid, this failure is visualizable, as it predicts the Mental-Pong ball to be static, regardless of enlarging the ball during evaluation or pretraining with temporal augmentations (Figure S2
Delving into the more challenging, fine-grained measures of DMFC Mental-Pong neural predictivity and correlation to human error patterns, we observe in particular that the celammcally--quippee

---

VC-1-based models (VC-I+CTRN) and VC-I+LSTM in cyan) reasonably match both human error patterns in Physion and Mental-Pong DMFC neural dynamics (Figure 4B).
Just as VC-1 on its own does not universally dominate on every individual sensorimotor task, but instead outperforms the best prior existing visual foundation models on average across the 17 CortexBench tasks MMjuumdar et al. [2023]. our finding that future prediction in the latent space of this model class reasonably matches both human behavioral patterns and neural responses is therefore consistent with this observation. Taken together, our results suggest that future prediction in the latent space of video foundation models for Embodied AI is promising paradigm for developing models of physical dynamics that are both a better match to neural and behavioral recordings, and can structurally generalize across diverse environments and scenarios within.
7 Discussion
Overall, we find that structural generalization to completely new environments and matching dense neurophysiological data to be a strong constraint on sensory-cognitive hypotheses of mental simula- tion. In manner evocative of Yamins et al. (2014)'s ' discovery of then-novel image categorization- optimized CNNs outperforming all prior models in matching primate inferotemporal cortex, our synamically-equipee video foundation models notably outperform all other models in DMFC neural response matching, illustrating the evolving nature of goal-driven models, especially when applied to tigher-connitive brain areas while the animal is performing a task. Going forward, we believe that we are at a crucial turning point whereby foundation models that engage with the visually rich, dynamic scenes that humans and animals naturally interface with, will be jointly critical for progress in Embodied AI and neuroscience, addressing the recent call to action to build Al that can be as grounded in the physical world as animals are [Zador al. 2023 We observe that both the popular machine learning paradigm of pixel-wise future prediction and visual foundation models pretrained on webscale imagesets, typically favored for classic computer vision tasks like image segmentation and classification, underperform here. Instead, future prediction within the latent space of video foundation model, pretrained on diverse egocentric sources, aligns best with high-throughhut neural and behavioral patterns in scenes that it was not originally trained on. Our findings indicate that pri- mate mental simulation harbors robust inductive biases, and so far most consistent with predicting the future state of its environment in a latent space that is reusable across dynamic environments.
On our existing benchmarks, there are a few ways that we envision our current models can be improved. First, we believe that major strides can still be made in the encoder module of the models, namely by better leveraging temporal relationships to build more "factorized" and reusable, object- centric representation. The need for a more factorized, object-centric representation is suggested by the high neural predictivity of the joint, ground truth position velocity oracle in Figure BB, compared to either the position or velocity oracles alone. At the same time, it is crucial to also maintain reusability in this representation since the fixed object-slot C-SWM models do not match these data well, and can be considered an example of an object-centric, but not reusable, representation. Therefore, a couple ideas in this direction would be to employ dynamic object-slots [Traub et al.) 2022. Didolkar et al. 2023 or structured masking [Bear et al. 2023 Yuan et al. 2023 pretrained on Ego4D or even larger egocentric datasets such as CortexBench [Majumdar et al. 2023 to yield object-centric, video foundation models. Additionally, simpler modifications such as differentiating between agent-based egomotion vs. external object motion may likely improve video foundation models by better closing the loop between taking actions in the world vs. simulating it.
The encoder modifications mentioned above may enable the models to learn more explicit representa- tions of objects and their material properties, compared to current self-supervised methods that are more statistical in nature [Balestriero et al.) 2023). This is especially relevant given our finding that all current models struggle most with the soft- body interactions of the "Drape" scenario in Physion, both in terms of accuracy (Figure S4p )) and matching human error patterns (Figure D. yet Ego4D has many examples of deformable objects in it (such as baking, clay, etc). Thus, it is not the pretraining dataset that is impoverished, but rather the current model architectures and loss functions are failing to pick up on these in existing datasets. The dynamics architecture could also benefit from including multiple timescales of hierarchy, with some recent neurobiological evidence of this type of temporal hierarchy already existing in frontal cortex (Sarafyazd and Jazayeri 2019). Such dynamics could sync well with dynamic object-slot encoders, capturing more rapid object state changes at one timescale and slower material property shifts across scenes at a higher-level, more abstracted timescale.
10

---

8 Acknowledgments
We thank the anonymous reviewers for their helpful feedback on the initial manuscript. A.N. thanks Christopher J. Cueva, Leslie P. Kaelbling, Thomas Serre, Kevin A. Smith, Joshua B. Tenenbaum, Rahul Venkatesh, Nicholas Watters, and Chengxu Zhuang for helpful discussions. A.N., M.J., and G.R.Y. acknowledge the generous support of the K. Lisa Yang Integrative Computational Neuroscience (ICoN) Center at MIT. M.J. is also supported by NIH HIMHH-MHH20225) the Simons Foundation, the McKnight Foundation, and the McGovern Institute. R.R. is supported by the Helen Hay Whitney Foundation.
Broader Impact
Almost everything humans and animals do revolves around intuitive understanding of the physical dynamics of the world we are embedded in, enabling to make long-range plans and perform actions that ensure our survival. Furthermore, engineering least this same level of intuitive physical understanding in silico will be critical for progress in robotics, autonomous vehicles, and any other embodied applications that involve safely taking actions the real world. Our work not only provides strong measurement of the degree of alignment between our current best engineered systems to those of humans and animals, but through the differentiation across choices of the architecture, objective function, and pretraining dataset, also provides scientific insight into the evolutionary constraints underlying the neural mechanisms of mental simulation. particular, we quantitatively observe that future prediction in latent space optimized for diverse, dynamic scenes is our current best theory that predicts neural dynamics during mental simulation behavior, compared to popular pixel-wise and object-slot alternatives. This set of observations suggests that making progress in Embodied Al will also correspondingly yield an improved understanding of mental simulation in human and animal brains.
References
M. Babacizadeh, M. Saffar, Nair, S. Levine, C. Finn, and D. Erhan, Fitvid: Overlitting pixel-level video prediction. arXiv preprint arXiv:2106.13195 2021.
R. Baillargeon, E. S. Spelke, and S. Wasserman. Object permanence in five-month-old infants. Cognition, 20(3): 191-208 1985.
A. Bakhtin, I.. van der Maaten, J. Johnson, I.. Gustafson, and R. Girshick. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32, 2019.
R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304. 12210, 2023.
P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, al. Interaction networks for learning about objects. relations and physics. Advances in neural information processing systems, 29, 2016.
W. Battaglia, J. B. Hamrick, and B. Tenenbaum. Simulation as an engine physical scene understanding. Proceedings of the National Academy of Sciences, 110645)): 2013.
W. Battaglia, B. Hamrick, Bapst, A. Sanchez-
Zambaldi, M. Malinowski. A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1at::000000066 2018
D. Bear, E. Wang, D. Mrowca, F. J. Binder, H.-Y. Tung, R. Pramod, C. Holdaway, S. Tao, K. A. Smith, F.-Y. Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. In Fiftty-fffth Conference on Neural Information Processing Systems Datasets and Benchmarky Track (Round I), 2021.
D. M. Bear, Feigelis, H. Chen, W. Lee, R. Venkatesh, K. Kotar, A. Durango, and L. Yamins. Unifying (machine) vision via counterfactual world modeling, arXiv preprint MrXiv:23306.1822 2023.
R. Bommasani, D. A. Hudson, Adeli, R. Altman, Arora, von Arx, M. Bernstein, Bohg, A. Bosselut, E. Brunskill, E. Brynjolfison, S. Buch, D. Card, R. Castellon, N. S. Chatterji. A. S. Chen, K. A. Creel,
J. Davis, D. Demszky. C. Donahue. M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. E. Gillespie, K. Goel, N. D. Goodman, S. Grossman, N. Guha, T. Hashimoto,
11

---

P. Henderson, J. Hewitt, D. E. Ho. J. Hong, K. Hsu, J. Huang. T. F. Ieard, S. Jain, D. Jurafsky, P. Kalluri. S. Karamcheti, G. Keeling, F. Khani, O. Khattab, P. W. Koh, M. S. Krass, R. Krishna, R. Kuditipudi, A. Kumar, Ladhak, M. Lee, Lee, Leskovec, Levent, X. L. Li, X. Li, Ma. A. Malik, D. Manning, S. P. Mirchandani, E. Mitchell, Z. Munyikwa, S. Nair, A. Narayan, D. Narayanan, B. Newman, A. Nic, J. C. Niebles, H. Nilforoshan, J. F. Nyarko, G. Ogut, L. OTT, I. Papadimitriou, J. S. Park, C. Piech, E. Portelance, C. Potts, A. Raghunathan, R. Reich, H. Ren, F. Rong, Y. H. Roohani, C. Ruiz, J. Ryan, C. R'c, D. Sadigh, S. Sagawa, K. Santhanam, A. Shih, K. P. Srinivasan, A. Tamkin, R. Taori, A. W. Thomas, F. Tramèr, R. Wang. W. Wang. B. Wu, J. Wu, Y. Wu, S. M. Xie, M. Yasunaga, J. You, M. A. Zaharia, M. Zhang. T. Zhang, X. Zhang, Y. Zhang, I.. Zheng, K. Zhou, and P. Liang. On the opportunities and risks of foundation models. ArXiv, 2021. htt.ttts:: stttps://crfft stanf ord. edu/ wwwssss/eeperrtttasp pdf
R. Cao and D. Yamins. Explanatory models in neuroscience: Part 1-taking mechanistic abstraction seriously. arXiv preprint arXiv:2104.0 014900 2021.
M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-xupervised vision transformers. Proceedings of the International Conference on Computer Vision (ICCV), 2021.
J. Carreira, E. Noland, C. Hillier, and A. Zisserman. A short note on the kinetics-700 human action datasct. arXiv preprint (xXiv:907.06966 2019.
L. A. Cooperau and R. N. Shepard. The time required to prepare for rotated stimulus. Memory Cognition, 1 (3):246-250, 1973.
K. J. W. Craik. The nature of explanation, volume 445. CUP Archive, 1943.
D. Cubuk, B. Zoph, Shlens, and Q Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper. S. Singh. S. Levine, and C. Finn. Robonet: Large-scale multi-robot learning. Conference Robot Learning (CoRL), arXiv preprint 1110115910.11215. 2019.
Denton R. Fergus. Stochastic video generation with a learned prior. In International conference on machine learning, pages 1174-1183. PMLR, 2018.
A. Didolkar, A. Goyal, and Y. Bengio. Cycle consistency driven object discovery. arXiv preprint 022404 2023.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representations (ICLR), arXiv preprint arXiv:2010. 2021.
Fischer, J. G. Mikhael, J. B. Tenenhaum, and N. Kanwisher. Functional neuroanatomy of intuitive physical inference. Proceedings of the national academy of sciences, (7734)))5577-E50833 2016.
C. Gan, J. Schwartz, S. Alter, D. Mrowca, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, et al. Threedworld: A platform for interactive multi-modal physical simulation. Advanced in Neural Information Processing Systems (NeurlPS) 2021, arXiv preprint arXiv:2007.04954. 2021.
Goyal, P. Dollár, R. Girshick, P. Noordhuis, I.. Wesolowski, Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.0000 7706.026777 2017.
K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of cgocentric vidco. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 18995-19012, 2022
2022.
O. Groth, F. B. Fuchs, I, Posner, and A. Vedaldi. Shapestacks: Learning vision-based physical intuition for generalised object stacking. In Proceedings of the Furopean Conference on Computer Vision (ECCV), pages 702-717, 2018.
J. B. Hamrick, P. W. Battaglia, T. L. Griffiths, and B. Tenenbaum. Inferring mass in complex scenes by mental simulation. Cognition, 157:61-70 -6, 2016.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
12

---

K. He, X. Chen, S. Xie. Y. P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009. 2022.
Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 9(8):7335 178001000.1 1997.
H. Hong, D. L. Yamins, N. Majaj, and J. DiCarlo. Explicit information for category-orthoonal object properties increases along the ventral stream. Nature neuroscience, 1944)613-622, 2016.
D. P. Kingma and Adam: method stochastic optimization. International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1412.694 2015.
Kipf, E. Van der Pol, and M. Welling. Contrastive learning of structured world models. International Conference on Learning Representations (ICLR), arXiv preprint arXiv: 2020.
W. Li, S. Azimi, A. Leonardis, and M. Fritz. To fall or not to fall: A visual approach to physical stability prediction. arXiv preprint ww0Xiv:104.00000 00066, 2016.
Y. Li, Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba, Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. International Conference on Learning Representations (ICLR), arXiv preprint arXiv:1810.01566 2019.
J. Ma, Sodhani, D. Jayaraman, O. Bastami, V. Kumar, and Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. International Conference on Learning Representations (ICLR), arXiv preprint arXiv:2210.0000 00030, 2023.
A. Majumdar, K. Yadav, S. Amaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V.-P Berges, P. Abbeel, J. Malik, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.1 1 18240, 2023.
K. D. Miller and F. Fumarola. Mathematical equivalence of two common forms of firing rate models of neural networks. Neural computation, 24(1):25-31, 2012.
S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. 6th Annual Conference on Robot Learning (CoRL), arXiv:2203. 12601, 2022.
C. Nash, J. Carreira, J. Walker, I. Barr, A. Jaegle, M. Malinowski, and Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.09.94 2022.
A. Nayebi, A. Attinger, M. Camphell, K. Hardcastle, I. Low, C. Mallory, G. Mel, Sorscher, A. H. Williams, S. Ganguli, et al. Explaining heterogeneity in medial entorhinal cortex with task-driven neural networks. Advances in Neural Information P Processing Systems, 34: 10:11116-12111 2021.
2021.
M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafranicc, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El- Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXi::234.007993, 2023.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
R. Pramod, M. A. Cohen, J. B. Tenenbaum, and N. Kanwisher. Invariant representation of physical stability in the human brain. Elife, 1 1::71736, 2022.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 7748-8763. PMLR, 2021.
R. Rajalingham, A. Piccato, and M. Jazayeri. Recurrent neural networks with explicit representation of dynamic latent variables can mimic behavioral patterns in a physical inference task. Nature Communications, 13(1): 5865, 2022a.
R. Rajalingham. H. Sohn, and M. Jazayeri. Dynamic tracking objects the macaque dorsomedial frontal cortex. bioRxiv, pages 2022-06, 2022b.
A. Sanzaez-Gonzalez, J. Godwin, T. Pfaff, R. Ying. J. Leskovec, and P. Battaglia. Learning to simulate complex physics with graph networks. In International conference on machine learning, pages 8459 8468. PMLR, 2020.
13

---

M. Sarafyazd and M. Jazayeri. Hierarchical reasoning by neural circuits in the frontal cortex. Science, 364 (6441))eaav9111, 2019.
S. Schwettmann, J. B. Tenenbaum, and N. Kanwisher. Invariant representations of mass in the human brain. Elife, 8:e46619, 2019.
R. N. Shepard and J. Metzler. Mental rotation of three-dimensional objects. Science, 171(3972:701-700, 1971.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
K. A. Smith and E. Vul. Sources of uncertainty in intuitive physics. Topics in cognitive science, 5(1):185-199 2013.
E. S. Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental science, 10(1):8 96, 2007.
Touvron, Cord, M. Douze, F. Massa, A. Sablayrolles, and Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
M. Traub, S. Otte, T. Menge. M. Karlbauer, J. Thuemmel. and M. Butz. Learning what and where: Disentangling location and identity tracking without supervision. In The Eleventh International Conference on Learning Representations, 2022.
T. D. Ullman, E. Spelke, P. Battaglia, and J. B. Tenenbaum. Mind games: Game engines as architecture for intuitive physics. Trends in cognitive sciences, 21(9):649 665, 2017.
R. Villegas, A. Pathak, H. Kannan, D. Erhan, Q V. Lc, and H. Lec. High fidelity vidco prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019.
B. Wu, S. Nair, R. Martin-Martin, L. Fei-Fei, and C. Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2318-2328, 2021.
D. L. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Scibert, and J. J. DiCarlo. Performance- hierarchical models predict neural responses higher visual cortex. Proceedings of national academy of sciences, (23))8661-86222 2014.
J. Yuan, Chen, B. Li, and Xue. Compositional scene representation learning via reconstruction: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
M. Zacks, Neuroimaging studies of mental rotation: meta-analysis review. Journal of cognitive neuroscience, 20(1):1-19, 2008.
Zador, S. Escola, Richards, Ölveezky, Y. Bengio, K. Boahen, M. Botvinick, Chklovskii, A. Church- land, C. Clopath, et al. Catalyzing next next-generation artificial intelligence through neuroai. Nature Communica- tions, 14( 15577 2023.
14


Reviews:
[{"review_id": "Gn0wkpVGzM", "reviewer": "Reviewer_ZwMf", "summary": "This manuscript compared several classes of deep-learning based sensory-cognitive models in their ability to predict human behavior and monkey neural responses in tasks that require reasoning about physical relationships based on visual inputs. They find that the models that match best to **neural data** are the ones trained to predict future states of environments in the latent space of pre-trained foundation models that are themselves optimized for dynamic scenes in self-supervised way. Within these models, the ones that are trained on diverse ranges of tasks are the best. On the other hand, the models that matched **human behavior** best were the models with pixel-wise end-to-end prediction for future scenes trained on the same dataset.\n\nOverall, I think this is a timely work to help point out the gap of many self-supervised foundation models from the computation potentially being performed in the brain. It emphasizes the fact that pixel-wise prediction overemphasizes details thus losing the ability of extracting useful representation as animal brains can extract. The relative distance from human and neural data across models is useful for AI researchers to explore better models along the \"gradient\" in model space.\n", "questions": "Maybe I am missing it, but I cannot find whether the fine-grain comparison with human or neural data were performed on held-out data or training data, such as in Fig 2B and 3B,C. \nRelated to this, I infer from Line 304 that SVF and FitVid were fitted to the same \"scenario\" of Physion as the scenario being tested on, but Line 301 mentioned they were tested on novel scenario. It might help clarify what exactly “scenario” means, especially given that the same word scenario is also overloaded with the meaning of the 8 different scenarios in the Physion environment.\n\nI understand that there is an urge and publication pressure to make claims such as xx model matches human/neural data. But to me, Fig 2B and 3BC all suggest a non-neglectable gap between the networks and the brain. It is this gap that is more inspiring for future research to fill. I suggest changing some subtitle such as 6. to be a relative statement (e.g., \"provide a better match\") instead of a binary statement of \"can match\".\n\n", "limitations": "Yes. Limitations are well addressed. I want to suggest one additional limitation: a fair comparison between different model architecture/objectives would be training the end-to-end models also on the same data (Ego4D) as those foundation models before the evaluation. Perhaps one reason this cannot be done is due to the limitation of computational resource, which is not the fault of the authors and should not be considered as a negative factor for the decision of this paper, if this is indeed the reason.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The models being compared are comprehensive and represent the states of the art.\nThe metrics being used are reasonable (although with limitation, see my comment for weakness). I appreciate that the authors used correlation of prediction on single stimuli (for human judgement).\nThe environments that the models are evaluated on focus on physical understanding, which is an important gap for the current AI (e.g., chatGPT and many models lack intuitive physics)\nThe datasets used for pretraining foundation models also come from a diverse range, from ImageNet to Ego4D and kinetics, especially Ego4D is an ecologically relevant dataset of egocentric videos.\n", "weaknesses": "After reading, although I have obtained a good understanding of the relative performance of the models in predicting physical contact and a somewhat good picture of the relative similarity to human behavioral judgement or monkey neural data in another game (mental-pong), I still lack a good understanding of where the model perform worse and how they are different from the brain. Sure we see the accuracy is lower as in Fig 2A and they are not correlated well enough to human judgement (Fig 2B, sorry that I am describing it as a half-empty bottle), but readers like me are interested in, at least for the best model, how they are worse than humans: is there any pattern among the stimuli that the models turn to make a wrong judgment? Maybe this question is hard to answer as we may lack a good representational space to start with in which we can say the models perform poorly in certain regions, but maybe some examples to put in supplementary material, or some subjective summary from the authors' observation can be helpful, or a more fine-grained comparison between different scenarios (are some scenarios easier for networks? do networks perform more similarly to the brain in some scenarios). Fig2C may provides some hint that a model with better accuracy to start with is more correlated with human, but it still does not answer anything about the pattern of response of a particular model. The analogy of the desire for this aspect of analysis is what psychophysics has done with visual illusions: by analyzing what stimuli cause illusions (a mistake made by the brain) and what models reproduce illusions, we could come up with hypotheses of the computational principle of the brain. \n\nI think some of the metrics being used might potentially mask certain difference between model and human/neural data. For example, the fine-grain comparison of predicted hit-probability in the Physion Object Contact Prediction (OCP) task is based on Pearson correlation. But correlation may not capture the overall variance of prediction. The predicted probabilities across stimuli by a model may exhibit a much bigger or much smaller variance than those of humans while showing the same level of correlation. Models and humans may also exhibit different levels of average probabilities across stimuli which is also not captured by correlation. I think such information as mean variance of prediction may also help understand in what aspects these models are worse than or different from humans. \n\nI do have some worry that the two tasks tested on human and monkeys reflect different aspects of reasoning. The mental-pong task as displayed does not require any 3D reasoning, while the Physion task, depending on the particular stimuli, should require 3D reasoning to some degrees. I worry that the different patterns of performance between Figure 2 and 3 may partly result from this dataset difference.\n\nOn the other hand, there is another confound, which is the end-to-end models are used differently in Figure 2 and 3. They were trained and tested on the same environments (but novel scenarios) for Figure 2 but tested on a different environment for Figure 3 (the authors did comment about this second confound in Lines 302-307). \n\nGiven the two confounds here, I think the results in Figure 3 are more conclusive than Figure 2 since none of the models being compared were trained on the mental-pong task (the authors did emphasize the result for neural data more than that for the human data in the abstract, which I think is a fair thing to do). But if the authors want to draw some conclusion about model generality (a model trained on a diverse range performs better than those end-to-end models when tested on new tasks), then it seems that an additional comparison in which the end-to-end models are trained on a different dataset than Physion but tested on Physion would be better to be added to Figure 2.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- *is there any pattern among the stimuli that the models turn to make a wrong judgment? …maybe some examples to put in supplementary material, or some subjective summary from the authors' observation can be helpful, or a more fine-grained comparison between different scenarios (are some scenarios easier for networks?*\n\nOne of the main reasons we used correlation-based measures is that 1) they reflect measures used in many prior studies (especially e.g. Bear et al. 2021 where the OCP task is introduced), 2) to be able to compare a large class of models and identify trends across them, and 3) enable comparisons across multiple datasets to understand differences between them (Figure 4A) and promising trends common across them (Figure 4B). Nonetheless, this is a great suggestion, and definitely something that is feasible with the metrics we currently use. In particular, as the reviewer suggests, for the OCP task, we plotted the distribution of per-scenario matches to human error patterns across models. We found a clear pattern emerged, namely that “Drape”, involving a soft material (cloth) draping over other objects by virtue of their shape and the cloth’s material, is generally the hardest scenario for models, whereas “Support”, which involves stacks of objects that may fall over, is the scenario that best matches human error patterns. This is interesting because it suggests that predicting diverse material properties, beyond standard rigid objects (especially those of soft bodies), is a tangible goal for future improved models. We will include this as an additional Figure 2D in the revised manuscript. We also will release our model checkpoints and metrics upon acceptance, so that others may do further analyses on our models. Thank you for making this suggestion.\n\n- *I think some of the metrics being used might potentially mask certain difference between model and human/neural data….But correlation may not capture the overall variance of prediction….I think such information as mean variance of prediction may also help understand in what aspects these models are worse than or different from humans.*\n\nThe correlation based measures reflect measures used in prior studies, like Bear et al. 2021 where OCP is introduced, and enable the identification of trends across large classes of models and stimuli on multiple datasets. Moreover, mean variance of prediction would still be computed across the probabilities for each stimuli (e.g. $(1/d) \\sum_{i=1}^{d} (x_i - y_i)^2$, where d is the number of stimuli), so we are not sure if it would provide more insight in the direction the reviewer is asking for than correlation. Instead, what might be more in line with what the reviewer is suggesting would be looking across individual classes of stimuli. To this end, we have looked at one such intuitive grouping based on scenario and looking at the distribution of human error pattern matching across stimuli in these scenarios across models, in a new Figure 2D. We found that models struggle most with the “Drape” scenario, where cloth drapes over objects due to shape and material, and align best with human errors in the “Support” scenario involving potentially collapsing stacks of objects. This indicates a concrete target for future models, which should aim to better predict diverse material properties, especially for soft bodies. While beyond the scope of our present study, which is more focused on identifying trends of successful models as a starting point for understanding human and animal intuitive physics, there may be other groupings across stimuli that could reveal additional insight; therefore, we will release our model checkpoints upon acceptance so that others may study further measures.\n\n- *I worry that the different patterns of performance between Figure 2 and 3 may partly result from this dataset difference.*\n\nBoth humans and animals can perform Mental-Pong (as shown in Rajalingham et al., 2022a & b), and humans and macaques display 3D reasoning that ensures their survival in the physical world. These general abilities across environments fall under the umbrella of “intuitive physics”. Therefore, while we agree with the reviewer that we expect different environments to potentially recruit different aspects of these abilities (as we quantify in Figure 4A), we want to identify models that can better match **both** fine-grained measures of human and animal intuitive physics well, which we do in Figure 4B in finding that the dynamically-equipped video foundation models can most reasonably do this across model classes. We will emphasize this important point more in Section 6 of the revised manuscript, where Figure 4 is discussed. Thank you for mentioning it.\n\n- *...end-to-end models...were trained and tested on the same environments (but novel scenarios) for Figure 2 but tested on a different environment for Figure 3 (the authors did comment about this second confound in Lines 302-307).*\n\nThe latent future prediction dynamics of all the foundation models were trained on Physion just as the end-to-end models were, and those Physion trained dynamics were evaluated against neural and behavioral data, ultimately outperforming the end-to-end Physion models. Despite our interest, training end-to-end models on datasets larger than Physion exceeds our current computational resources, as evidenced by models like FitVid requiring nearly a month on eight A100 GPUs. To somewhat get around this constraint, we trained the best models, the dynamically equipped video foundation models, on Kinetics 700. This made the encoder and dynamics training datasets different from Physion used in the OCP metrics in Figure 2 (6 hatched bars). The top models, VC-1 and R3M, yielded comparable results to their Physion-trained versions in both human accuracy and neural predictivity. This indicates dataset scale isn’t the sole factor; inductive biases also play a key role. We'll discuss this further in Sec. 4 & Limitations in revision."}, {"review_id": "0dOBMsUs2J", "reviewer": "Reviewer_6vWo", "summary": "The paper compares a rather large variety of DL models that are able to “future predict” environmental states, including pixel-based deep networks, compositional approaches (e.g., slot-wise processing objects), as well as image and video foundation models. In the latter case, the latent space of the foundation model is used to future predict. The models are then evaluated by comparing their prediction accuracy in an object contact prediction (OCP) task with human performance and in a mental pong task with macaque neuro-biological data. ", "questions": "A few more detailed comments: \n\nThe abstract starts very superficial – it took me a while to understand what you are actually after. A full re-write seems necessary. \n\nIn the introduction, the second paragraph emphasizes that “predicting the physical dynamics of environments” is critical… but isn’t this precisely what you emphasize in paragraph 1 of the introduction, that is, that mental models are important? Only mental (intuitive physics-approximating) models allow to predict physical dynamics, isn’t this per definition the case? \n\nOver-general implications are drawn in the introduction – the results are not strong enough in my opinion to come to the put-forward conclusions. Moreover, the written implications stay on a rather superficial level, although the restuls are very particular to the considered two tasks and the methods employed.\n\nFinally, on the methods side, more details would have been useful to compare sizes and understand the exact training procedure to fit the data. \n", "limitations": "not addressed", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper offers a huge data study that reveals model differences when predicting human behavioral performance and macaque neural dynamics data. At the moment, video-based foundation models yield best (but not very good and not much better compared to other models) performance. Generally, the generated data is interesting and should not be lost.", "weaknesses": "On the negative side, the paper focuses on only two tasks, which are not very well-motivated. I have found neither a clear motivation to choose these tasks nor a clear hypothesis what the authors would expect. \n\nFurther, the results that have been achieved are very far from human-to-human consistency / human accuracy and also very far from inter-animal consistency. This implies to me that the results and the performance differences between the models are not really due to actual systematic differences between the models and the model relations to how humans process the data; rather, they may be due to more fundamental design choices. Moreover, particularly in the OCP task, the model performance differences nearly seem random. \n\nSurprising to me is the fact that object-centric models yield worst performance (but all are bad), although the task itself is suitable for an object-centric setup. Maybe the authors did not tap onto the appropriate latent state representation – or it this the case because they do not allow to actually simulate dynamics with the object-centric setup? \n\nThis concern goes hand-in-hand with the fact that the authors do not report an oracle-like performance in the OCP task (as they do in the mental-Pong dataset). From the CogSci work of Tenenbaum, Goodman and others it is well-known that humans do use an internal simulator to approximately solve such tasks… none of the systems the authors consider are able to actually run an internal simulation of the potentially unfolding dynamics. Rather, the authors seem to tap directly into the static latent state of the respective models. \n\nIn the mental-pong task, the authors do provide an oracle comparison, which indeed yields the best neural predictivity performance, followed by VC-1 and R3M, both of which can learn to predict ball position and velocity because they learn to predict video dynamics where entities, for example, fall, fly, and roll around. Thus, I do not really see any insight created by these comparisons, except for that a model is needed that predicts ball positions and velocities… in the end, the oracle should be the baseline, which needs to be beaten by a neural network to come closer to understanding how our brain / macaque brains solve this task. \n\nSeeing this huge study, I am also quite concerned about the compute time (energy) invested here. The authors do not provide enough information about training etc. Thus, it is impossible to even estimate this investment. \n\nThe final discussion stays rather superficial on the actual insights gained (because unfortunately there are no significant insights really). The mere task to match human data / macaque data even better may be interesting for some sub-community along the neuro-DL interface, but I miss both cognitive and structural insights that go beyond the fact that object dynamics (including velocities) as well as object constellations need to be encoded in the considered tasks. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- *…the paper focuses on only two tasks, which are not very well-motivated.*\n\nWe thank the reviewer for suggesting the need for this clarification, and will add it to the Introduction. Specifically, the OCP task (Bear et al. 2021) tests realistic simulations of a wide range of everyday physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, in a realistic 3D environment with 2,000 scenarios per condition, and is therefore more comprehensive than previous behavioral benchmarks so far. Furthermore, Mental Pong is the *first neural* dataset that was shown to involve mental simulation, and is a dense neurophysiological dataset containing almost 2,000 neurons recorded with high temporal precision (Rajalingham et al. 2022b). They are therefore both high-throughput in behavior as OCP is (from 100 human subjects across 16,000 scenarios), or in the neural readout. In addition to these tasks, we benchmark a large variety of models (41 models total), none of which are strawman models – they can functionally perform the task on large-scale datasets and many of them are considered state-of-the-art. Yet, when compared our human behavioral and neural benchmarks, in part due to the high-throughput nature of our benchmarks, they strongly differentiate these functionally reasonable hypotheses.\n\n- *Further, the results that have been achieved are very far from…consistency*\n\nThis is actually a major point of our work. Our work does not claim to have solved intuitive physics, but first shows the limitations of prevalent approaches and offers promising future directions validated against strong neural and behavioral benchmarks. Even though we utilize state-of-the-art machine learning models, they still significantly lag behind primate intuitive physics. As seen in Figures 2C and 3D, aligning with human behavior and neural dynamics directly pertains to solving the AI task in each environment. Notably, the widely-used pixel-wise future predictor excels in familiar settings but overfits in novel ones, like the Pong scenario. This underscores our emphasis on out-of-distribution structural generalizations, rather than the typical ML generalization where training and testing environments are statistically similar.\n\nMoreover, our work *also* points a concrete way forward to better models. In particular, we find that latent future prediction appears to be the most promising paradigm overall, compared to popular alternatives of pixel-wise future prediction, object slots, and the fixed latents alone of foundation models. In other words, the problem of intuitive physics can effectively be “factorized” into a challenging vision problem and then equipping that scene representation with dynamics. And perhaps more crucially, this visual representation needs to be reusable across dynamic scenes. This suggests that reusability is an important factor that future models should incorporate, and points to a move away from the commonly used web-scale imagesets (no matter their scale) that take snapshots with lighting conditions that tend to be unrealistic from the standpoint of egocentric viewpoints humans and animals receive. Furthermore, the dynamics architecture could benefit from better leveraging a more object-based representation of temporally-active state variables (as suggested by the high neural predictivity of the joint, ground truth position + velocity oracle; rightmost bar in Figure 3B). Taken together, these observations suggest that working on self-supervised loss functions that better leverage temporal relationships to learn object-based, video foundation models are a next step, coupled with latent future predicting dynamics on top of these reusable, object-based latent representations. One way toward that goal can be adapting the masked objective in MAE to operate on the future frame rather than the current frame, and train on a more diverse video dataset similar to CortexBench, rather than Ego4D alone.\n\n- *do they not allow to actually simulate dynamics with the object-centric setup?*\n\nOur object-slot models use a learned graph-neural network module on top to simulate the forward dynamics on top of the object-slot latent representation.\n\n- *the authors do not report an oracle-like performance in the OCP task (as they do in the mental-Pong dataset).*\n\nThe CogSci work, as we discussed in Section 2, provides a near-oracle model for the OCP task using ground-truth simulator states, unlike real-world sensory inputs. We will indicate their numerical performance in the revision. With ground-truth inputs that these CogSci models take in, such as in Mental-Pong, the dynamics prediction problem becomes trivial since there is a single object, so our focus is on the much more challenging problem of bridging the gap from raw sensory inputs to behavior.\n\n- *none of the systems the authors consider are able to actually run an internal simulation of the potentially unfolding dynamics.*\n\nActually, the dynamics are additionally trained on top of the latent representation, even in the cases when the visual encoder is fixed. We in fact train multiple dynamics architectures (CTRNN and LSTM) across all foundation models, on both Physion and the larger Kinetics 700 dataset.\n\n- *In the mental-pong task… I do not really see any insight created by these comparisons, since the oracle should be the baseline.*\n\nThe oracle models are not baselines since they possess perfect information inaccessible to animals or pixel-based models, as the ball is occluded. Oracles reveal that using more object-centric priors, which access both ball position & velocity, outperforms relying on just one of these. Taken with Fig. 4B, these indicate that future models should learn reusable *and* object-based representations from egocentric videos.\n\n- *Limitations: not addressed*\n\nThe Limitations section was in the initial submission, in the first paragraph of Supplementary Material."}, {"review_id": "cXcx86uxNs", "reviewer": "Reviewer_tmZS", "summary": "The authors compare \"foundation models\" of vision for mental simulation. They consider several large models including models trained on static scenes and dynamic scenes. It was found that the models optimized with self-supervision on dynamic scenes yielded the best neural predictivity. ", "questions": "*Fixed encoder that is not trained during \"dynamics\" training seems too constraining (also unlike human brains). This might have unequal effects on the different models. For example, it might have hurt imagenet models more than others etc. Have you tried versions where you did update the encoders too?\n\n*There seem to be some models (Fig. 3 D) that are very good in accuracy but not so good in neural predictivity. Do you know why this is happening?", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The work is interesting since mental simulation is relatively understudied but important for embodied models. The paper is clearly written.\n\n* The work is thorough in testing out a lot of models and model families.\n* Results on neural predictivity is strong for the video foundation models. ", "weaknesses": "* Since all the video foundation models are trained with Ego4D and the image foundation models are trained with ImageNet, it is not straightforward to disentangle the effect of dataset vs role of dynamics itself. \n\n* The models only differ in terms of neural predictivity while being similar to each other in accuracy and correlation to human responses. Previous work (Rajalingham et al. [2022b]) seem to support that model behavior should be measured in terms of correlation to human responses to disambiguate between models - with a high correlation suggesting that the model is using the same strategy as humans. In light of this, the results presented seem weaker than as claimed. \n\n* I do not see the contributions of the paper beyond \"large models trained with dynamic scenes data are better at neural predictivity for a dynamic task\" which is not surprising. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- *Since all the video foundation models are trained with Ego4D and the image foundation models are trained with ImageNet, it is not straightforward to disentangle the effect of dataset vs role of dynamics itself.*\n\nWe wanted to understand this question too, and as a result, we included the latents (without any dynamics) in Figures 2-4 (refer to the leftmost bars in each group). Across all nine classes of image and video foundation models that we tried, the most effective latent layers were from the self-supervised video foundation models trained on Ego4D (VC-1 and R3M), as can be seen in Figure 3. However, the latent representations of VC-1 and R3M alone are significantly worse at predicting neural responses than adding dynamics on top of it (remaining bars in each group in Figure 3B). Furthermore, the dynamically-equipped versions of VC-1 excelled in all neural and behavioral metrics (Figure 4B) relative to their latents and those of other models. Thank you for raising this important point.\n\n- *Previous work (Rajalingham et al. [2022b])…support that model behavior should be measured in terms of correlation to human responses… In light of this, the results presented seem weaker than as claimed.*\n\nRajalingham et al. 2022b trains and evaluates models in the same Mental-Pong environment; however, we only (*evaluate* our models in this environment since we are after the more challenging problem of task generalization to multiple diverse scenarios (in the Physion and Mental-Pong environments). For example, as we show in Figures 3C and S2, even state-of-the-art models like FitVid (a pixel-wise future predictor) fail to generalize to Pong after being trained in Physion, unlike our dynamically-equipped video foundation models. The reason why we focus on generalization to Mental-Pong is that monkeys can perform these tasks without substantial training, suggesting that they are already equipped with the necessary neural foundations for mental simulation in this environment. Therefore, we aim to also build networks that are not explicitly trained on Mental-Pong itself, but are tasked to generalize to this novel setting as a test of their general understanding of physical scene dynamics -- chiefly developed through three factors: their architecture, optimization objective, and pretraining on a naturalistic environment.\n\n- *I do not see the contributions of the paper beyond \"large models trained with dynamic scenes data are better at neural predictivity for a dynamic task\" which is not surprising.*\n\nActually, not all large models trained on videos predict neural data well. In fact, a major point of our paper is that many state of the art machine learning models for future prediction fail to match neural and behavioral data well. For example, of the models trained on dynamic scenes, neither the pixel-wise future predictors nor the object-slot models do this well (“End-to-End models in Figure 3B), nor does the video foundation model VIP or its dynamically equipped version do it well either (green bars in Figure 3B). Our work therefore suggests strong constraints on models, specifically *not* through RL, pixel-wise losses, or bespoke object-slots, but rather through self-supervised pretraining from egocentric views that humans and animals naturally receive. Crucially, doing future prediction on this *reusable* latent representation is important (since the fixed latent alone is not sufficient, see the leftmost bars in Figure 3B in each group).\n\n- *Fixed encoder that is not trained during \"dynamics\" training seems too constraining (also unlike human brains)....Have you tried versions where you did update the encoders too?*\n\nThat’s a great question, and one that we have looked into. In fact, our decision to use a fixed encoder during “dynamics” training, despite seeming restrictive, was largely influenced by our experimental findings. For example, prior state-of-the-art models like end-to-end pixel-wise or object-slot predictors did not yield superior results (cf. Figure 3 “End-to-End” bars) and struggled to generalize to Mental-Pong (cf. Figure 3C dark green FitVid bar, and Figure S2), even though their encoders were updated during training. Based on these observations, we saw that generalization to novel environments failed even when the encoders are allowed to update during training, especially if inductive biases such as the loss function and pretraining dataset are not properly chosen. In fact, this approach aligns with insights from more cognitively-inspired graph neural networks (GNNs), which propose to solve intuitive physics tasks by modeling dynamics on meaningful entities, like objects, their material properties, and their relations, rather than raw pixels. Since GNNs can match the OCP human error patterns well (as shown in Bear et al., 2021), they suggest that one can “factorize” the problem of intuitive physics into solving a challenging vision problem (e.g. an understanding of the scene) with dynamics that operates on that representation. These observations together motivated our fixed foundation model encoder with dynamics approach.\n\nGoing forward, resources permitting, we are planning to explore hybrid models that train the encoder with a masked autoencoding (MAE) style objective applied to next-frame prediction on Ego4D and larger datasets like CortexBench. We believe these methods could better leverage temporal relationships to learn reusable *and* object-based latent representations.\n\n- *There seem to be some models (Fig. 3 D) that are very good in accuracy but not so good in neural predictivity. Do you know why this is happening?*\n\nBall position & velocity offer coarser measures than neural predictivity, with far fewer tracked quantities (4 vs. 1889). Thus, many inductive biases are able to track 4 quantities, but far less can track 1889. Our goal is models that excel in **both** metrics, as shown with the dynamically-equipped *video* foundation models in Figure 3D (top right circles)."}, {"review_id": "GWBncFcsrb", "reviewer": "Reviewer_bK8w", "summary": "It has been hypothesized that the brain builds an internal model of its environment, and uses this model to make inferences and plan actions. This works aims to understand the neural mechanisms that are the basis of such computations. To this end, the authors construct several classes of artificial neural network models, each with different inductive biases, and evaluate them on future state prediction tasks in ethologically relevant environments. \n\nThey evaluate each of these models in two ways:    \n\n- comparison to human behavior in an object contact prediction task\n- comparison to neurophysiological data from macaques playing mental-pong, a ball interception task in which the ball is partially occluded\n\nFrom these evaluations, they identify that dynamics modules that predict the future state in the latent space of video foundation models reasonably matches both human behavioral patterns and neural responses across diverse environments. ", "questions": "- The R3M and VC-1 video foundation models (+ dynamics) are the best at neural response predictivity.  However, one is a ResNet and the latter is a transformer based model. Also the training objectives are different for both models. Why is the performance of these two models similar? Is it the scale and/or type of pre-training dataset?\n- For the object-centric models did you try using recurrent networks for the dynamics module? \n- Was any analysis performed to compare the best-fitting latent representations across models? ", "limitations": "Limitations are addressed in the appendix. ", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Understanding the neural basis of mental simulation is an important problem both in neuroscience and AI. \nThe approach taken here is to construct several classes of artificial networks and evaluating them on tasks that require prediction of the future state of the environment. This allows the authors to probe which inductive biases are crucial for mental simulation. \nThe presentation is clear as well. ", "weaknesses": "Not really a weakness, but details about the best fitting model layers would be useful to add. It would also be interesting to see some analysis comparing the best fitting latent layers across different models. \n\nminor:   \n- line 17, 18 in the abstract talks about the mental pong task without introducing it", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- *Not really a weakness, but details about the best fitting model layers would be useful to add. It would also be interesting to see some analysis comparing the best fitting latent layers across different models.*\n\nAbsolutely agree. We actually compared the latent layers in Figures 2-4 across different models (these are the leftmost bars in each group of the nine image and foundation models classes that we tried). We found that the best latents overall were from the video foundation models (VC-1 and R3M) trained in a self-supervised manner on the Ego4D dataset. The dynamically-equipped versions of these models, optimized to predict the future state of the environment in this latent space, were the best across the neural and behavioral metrics we compared (Figure 4B). For clarity, we will include a plot with just the latent layers alone in Figure S3 of the revised manuscript. Thank you for the suggestion.\n\n- *line 17, 18 in the abstract talks about the mental pong task without introducing it*\n\nThank you for pointing this out, we will introduce the Mental Pong task in the revised abstract prior to lines 17-18.\n\n- *The R3M and VC-1 video foundation models (+ dynamics) are the best at neural response predictivity. However, one is a ResNet and the latter is a transformer based model. Also the training objectives are different for both models. Why is the performance of these two models similar? Is it the scale and/or type of pre-training dataset?*\n\nThis is a great question. We believe it is likely a combination of the self-supervised loss function applied to the Ego4D dataset. For example, both R3M and VIP have the same architecture (ResNet-50) trained on Ego4D, but VIP is trained using an offline RL (goal-conditioned) objective predicts neural responses poorly (Figure 3B, dark green bars) relative to R3M (even at the level of latents), whereas R3M is trained with a contrastive, self-supervised objective. Furthermore, when trained with a categorization objective on ImageNet, ResNet-50 performs poorly (Figure 3B, red bars, “Image Foundation Models”), even at the level of the latents (Figure 3B, leftmost red bar, “Image Foundation Models”). Similarly, though somewhat less directly, Transformers like DeIT, DINO, DINOv2, and CLIP are trained with self-supervised objectives, including contrastive ones, on webscale datasets even larger than ImageNet (“Image Foundation Models” bars in Figure 3B), yet perform worse than the self-supervised VC-1 transformer trained on Ego4D. We will include discussion of these points above in Section 5.2 of the revised manuscript. Thank you for bringing it up.\n\n- *For the object-centric models did you try using recurrent networks for the dynamics module?*\n\nYes, the object-slot models are trained with a recurrent graph neural network (GNN) dynamics module, originally proposed by Kipf et al., 2020. We apologize for this not being clear in the text, and we will mention this more clearly in Section 3 when the model is first introduced. Thank you for asking this.\n\n- *Was any analysis performed to compare the best-fitting latent representations across models?*\n\nYes, this is a great question. We actually evaluated the latent layers against our neural and behavioral metrics in Figures 2-4 (refer to the leftmost bars in each group of nine foundation models). The most effective latent representations were from the video foundation models self-supervised on Ego4D (VC-1 and R3M), and were most differentiated on neural responses (new Figure S3, panel C). Their dynamically-equipped versions, optimized to predict the future state of the environment in this latent space, excelled in all neural and behavioral metrics (Figure 4B). We will add a dedicated plot of the latent layers alone in a new Figure S3 in our revised manuscript for better understanding. We appreciate the recommendation."}]

Author rebuttal:
**Global Response:**

We thank the reviewers for their thorough reviews, helpful suggestions, and overall positive enthusiasm about our work.

**For common reference, the core contributions of our work are:**

**1. Dense neurophysiological data strongly constrains hypotheses:** Overall, we find that structural generalization to novel environments and matching dense neurophysiological data to be a strong constraint on models of physical simulation, and many state-of-the-art machine learning models fail to do both of these (cf. Figures 2B, 3B, 3C, and 4A). Yet, monkeys can perform these tasks without substantial training, suggesting that they are already equipped with the necessary neural foundations for mental simulation.

**2. Latent future prediction as a promising paradigm for mental simulation:** More specifically, mental simulation in primates (humans and monkeys) appears to be primarily relevant to *dynamics* that are trained to predict the future state of the environment in a suitable latent space (cf. dynamically-equipped video foundation models in Figures 3B and 4B).

**3. Pre-training dataset scale is not the only factor:** In particular, this latent space is highly constrained, and dataset scale is *not* simply all you need (cf. Figure S1 for within-architecture neural predictivity comparisons across pretraining datasets of high variation). 

**4. Not every latent works:** In fact, this latent space does *not* appear to consist of bespoke object slots or prioritize fine-grained details (e.g. at the level of pixels), or even through supervised tasks on static images (Figure 3B, “End-to-End” and “Image Foundation Models” bars). 

**5. Reusable latents on dynamic scenes as neural models:** Rather, the latent mainly has to be *reusable* across *dynamic* scenes. Taken together, our results observe a correspondence between the ability to predict neural and behavioral responses for the mental simulation phenomena, and developing useful representations for Embodied AI more generally (Figure 4B). This is in contrast to the prior emphasis on classic computer vision tasks such as classification, segmentation, etc (cf. low neural predictivity of “Image Foundation Models” bars in Figures 3 and 4B) either optimized in a supervised or self-supervised manner, but which have up until now been standard models of the primate visual cortex (cf. Schrimpf et al., 2018).

**Computational Resources:**

Reviewers appreciated our thorough study, which we deemed essential to have many functionally reasonable instantiations of these hypotheses and to have the neural and behavioral data strongly separate these hypotheses. The resources used are listed in Supplementary Section A in the initial submission. For all models except FitVid and SVG 128 × 128 (SVG trained on 128 × 128 pixel images), a single NVIDIA A100 GPU was sufficient to train it. Our study suggests that resource-intensive models like FitVid aren't the best for intuitive physics as they don't generalize well (Figure S2). Dataset scale isn't the sole key; inductive biases, as seen in latent future predictions, are crucial, and merely training on larger imagesets, even 250 times bigger than ImageNet, doesn't ensure matching neural dynamics (Figure 3B). We'll release model weights post-publication for use.

**Based on reviewer comments, here we list the major changes we plan to make to our submission if it is accepted:**

**1. Text changes:** With the extra page granted for the camera-ready version, we will enhance the Introduction (Section 1) and Discussion (Section 7) to highlight our paper’s core message: most state-of-the-art machine learning models do *not* meet our rigorous neural and behavioral benchmarks for future prediction. Specifically, many models, including those trained on dynamic scenes like pixel-wise predictors and object-slot models, fall short, as illustrated in Figure 3B. Instead, our findings point towards models that future predict latent representations honed through self-supervised pretraining from egocentric viewpoints that humans and animals naturally receive, as opposed to with RL objectives (VIP bars, Figures 3B and 4B) or webscale imagesets (“Image Foundation Models”). Furthermore, it is pivotal to have trained dynamics on these latent representations, as the fixed latents alone are not adequate (compare the leftmost bars in Figure 3B to the remaining bars in each group, for instance). 

Additionally, Section 2 will emphasize a bit more how prior CogSci results involving graph neural networks (GNNs) that operate on semantic, non-pixel-level features actually motivates our latent future prediction paradigm which factorizes vision & dynamics, while Sections 5.2 and 6 will further stress the importance of model proficiency across multiple metrics.

**2. Additional Figures:** Based on Reviewer ZwMf’s excellent suggestion, we include an additional Figure 2D, to show a more detailed error pattern that consistently emerges across models, revealing that they best match human error patterns overall on “Support” scenarios (along with other rigid body scenarios), and could be most improved on the “Drape” scenario. This suggests a concrete future direction of improving models to predict diverse material properties, in order to better handle soft body interactions.

Also, based on Reviewers 6vWo and bK8w feedback, we are introducing a new Supplementary Figure S3 to specifically highlight latent layers across various models, which were originally included in Figures 2-4 (leftmost bars in each group). The best fixed latents were generally from self-supervised video foundation models, like VC-1 and R3M, which then when dynamically-equipped, excelled in predicting the environment's future state across our metrics. 

**Both new Figures 2D and S3 are included in the one page PDF we have uploaded here.**

**Finally, we reply to each reviewer individually. Due to character limitations, we respond to the major comments of each reviewer.**

Meta-review: There was unanimous agreement among reviewers that the work is important because mental simulation is relatively understudied but important for embodied models. There was also unanimous agreement that the work is thorough – systematically evaluating SOTA models against neural and behavioral data. The reviewers asked for clarifications and raised concerns about possible experimental confounds. There was a substantial discussion between the authors and the reviewers and the rebuttal appears to have addressed all the issues that were raised. The paper appears to make a significant contribution in the establishment of foundation models and neuroscience benchmarks for mental simulation and the AC recommends the paper be accepted.
Decision: Accept (spotlight)