SlowGuess/NeurIPS_2023_OCR huggingface dataset:
And as shown in Figure 2D, all models have the most room to improve to match human error patterns and OCP accuracy (Figure S4) for soft-body interactions (the "Drape" scenario). On the other hand, models seem to do reasonably well on certain rigid-body scenarios, especially "Support" relations where stacks of objects fall over depending on their shapes and arrangement; "Collide", where pairs of objects collide depending on their trajectories and placement; and "Roll", where objects move across a surface either by rolling or sliding. In these scenarios, the best Physion-pretrained models, namely the pixel-wise future predictors, can attain human consistency scores much higher than what we report in Figures 2B and D of around 0.6 (cf. Figure S5B).
5 Comparison to Dense Neurophysiological Data
To gain more insight into model generalization, we compared the above models to neural dynamics recorded from macaque dorsomedial frontal cortex (DMFC), which was shown by Rajalingham et al. [2022b] to simulate the ball's position in Mental-Pong while behind an occluder, until it was intercepted by the paddle. This environment is completely out-of-distribution for the models, unlike Physion, and therefore tests structural generalization.
5.1 Inter-animal consistency and neural behavioral decoders
For each of the 79 conditions (different randomized start position of the ball), we present the frames in the visible epoch (the time up until the ball reaches the occluder) as context frames to the models, and unroll the model dynamics during the occluded epoch of the condition. We then build detailed, physical mappings (schematized in Figure 3A) from the committed model layer latent dynamics to match every single neuron in the population, and the ground truth ball state while occluded, when mental simulation takes place. We enforce the requirement, established in prior work [Nayebi et al., 2021, Cao and Yamins, 2021], that models are at least as similar to a given animal's neural responses as two conspecifics' neural responses are to each other. When we perform these mappings, we see that both the inter-animal neural predictivity consistency and ground truth ball state decoding from DMFC is quite high (grey horizontal lines in Figures 3B and C, respectively), indicating that these are very reliable neural and behavioral measures of the mental simulation phenomenon.
5.2 Neural response predictivity strongly separates models
When we map model units to DMFC units across 79 conditions (different randomized ball start position), we see in Figure 3B that across all architectures, only dynamically-equipped models from the video foundation model class predict DMFC dynamics best (46.34-48.83% neural predictivity), specifically in the latent space of the VC-1 and R3M encoders. However, the VC-1 and R3M encoder's latents alone are insufficient to predict the data (19.03% and 24.79% neural predictivity, respectively), although their latents best predict neural responses compared to other foundation models (cf. Figure S3C). Pretraining end-to-end on Physion with either a pixel-wise loss (SVG and FitVid; 14.44%-25.35% neural predictivity) or an object-slot loss (C-SWM with varying encoder sizes; 13.69%-23.05% neural predictivity) is not sufficient either, indicating a strong constraint on the neural mechanisms of mental simulation being performed on a suitable latent space. However, it is not the case that any latent space works, since all dynamics trained on encoder latents pretrained on static images, regardless of their image set scale, attained at most 29.67% neural predictivity. This is suggestive that for a relatively small but time-varying stimulus such as the Mental-Pong ball, models that are pretrained on static images may not be equipped to handle the temporal coherence of a single object moving through time, as prior studies have mainly examined generalization of
Figure 3: Model Comparisons to Neural Response Dynamics: (A) We measure how similar model dynamics (red) are to each primate's DMFC neural dynamics up to at least how (linearly) similar primates are to each other, via the linear transform  $L^N$  (black arrows). We also assess how well models can decode ball state (position and velocity) up to how well it can be decoded from DMFC via the linear transform  $L^B$  (teal arrows). (B) DMFC test set neural predictivity of each model, averaged across five train-test splits. Median and s.e.m. across 1,889 recorded units. For each of the latent future prediction models, there are three bars: the first corresponding to final context encoder latent itself ("No Dynamics"); and the last two bars are with LSTM or CTRNN dynamics, respectively. For the VIP, VC-1, and R3M based models, we also pretrain the LSTM and CTRNN dynamics on Kinetics-700 rather than Physion, and their predictivity is displayed in the last two additional hatched bars. The grey horizontal bar represents how well each primate's DMFC can predict the other's via the linear transform  $L^N$  schematized in (A), setting a consistency ceiling on model predictivity. Position, velocity, and position + velocity oracles are the ground truth ball state values for those variables while it is occluded (and not visible to the primate during game play). Dotted "Perfect Simulation Oracle" line corresponds to the position + velocity oracle's neural predictivity. (C) Ball position and velocity predictivity of each model, averaged across five train-test splits, from the best DMFC fitting model layer used in (B) via the linear transform  $L^B$  schematized in (A). The grey horizontal bar represents the ball position and velocity predictivity from DMFC units. Median and s.e.m. across four quantities: the ground truth  $(x,y)$  position and velocity of the ball while it was occluded (in degrees of visual angle). (D) Mental-Pong ball position and velocity predictivity  $(y$ -axis) vs. neural predictivity  $(x$ -axis). Dotted golden line represents best linear fit to the data.
ImageNet-optimized CNNs to novel yet static, single-object scenes [Hong et al., 2016]. Moreover, even for video optimized foundation models, we see differentiation at the level of loss function. In particular, goal-conditioned self-supervised learning on videos, as instantiated by VIP (which differs from R3M primarily in the choice of objective function, as it shares the same ResNet-50 architecture and is also pretrained on Ego4D), fails to match DMFC neural dynamics well (7.37%-18.12% neural predictivity). We also see parsimony at the level of the architecture and pretraining dataset of the dynamics module, as the VC-1/R3M+CTRNN either attains comparable or improved neural predictivity over the more sophisticated VC-1/R3M+LSTM (middle vs. rightmost bars for each encoder type), even when pretraining on a larger dataset like Kinetics-700 (Figure S1). This suggests that simple dynamics may be sufficient at predicting neural response dynamics given the appropriate visual encoder and highlights the limits of dataset scale alone without changing inductive biases. Both the VC-1/R3M+LSTM and VC-1/R3M+CTRNN models approach the neural predictivity of the ground truth ball position oracle model (50.74%, leftmost golden bar in Figure 3B), with the joint position + velocity oracle performing the best (60.65% neural predictivity, rightmost golden
bar, which also corresponds to the "Perfect Simulation Oracle" dotted line). This suggests that a substantial amount of the neural response variability is devoted to simulating the ball's state while it is occluded, as opposed to other static features of the environment. The remaining gap between the "Perfect Simulation Oracle" and the inter-animal consistency may be due to other factors such as time since stimulus onset, attentional demands/eye movements, and perhaps uncertainty when the ball bounces, which can be explored in future work.
In Figure 3C, we see that the most neurally predictive dynamically-equipped VC-1/R3M-based models generalize to the Mental-Pong task, approaching DMFC's ability to track the ground truth position and velocity of the ball while it is occluded, despite not being explicitly pretrained in this environment. In particular, there is a linear relationship  $(R\approx 0.701,p\ll 0.001)$  between the model's ability to generalize to the Mental-Pong environment and its ability to predict DMFC neural dynamics (Figure 3D). This relationship indicates that predicting the underlying neural dynamics is in fact behaviorally relevant to effectively simulating the ball's dynamics, which Rajalingham et al. [2022a,b] judged in humans based on eye tracking data. Furthermore, the dynamically-equipped VC1/R3M-based models can best represent the ball position and velocity separately as well (Figure S6), demonstrating that they can independently track these variables.
6 Dynamically-Equipped Video Foundation Models Can Match Both Human Behavioral and Neural Response Patterns Across Environments
Figure 4: Dynamically-Equipped Video Foundation Models Can Match Both Neural and Behavioral Metrics: (A) Ball state (position and velocity) predictivity in Mental-Pong ( $y$ -axis) vs. OCP accuracy in Physion ( $x$ -axis). Across models, these two metrics appear to be largely independent. (B) The dynamically-equipped VC-1 models (VC-1+LSTM/CTRNN pretrained on either Physion or Kinetics-700) can reasonably match neural response dynamics in Mental-Pong and match human error patterns in the OCP task, relative to other models. Dotted grey line is the unity line to indicate that better models will occupy the top right of this scatter plot.
Both the OCP human behavioral error pattern metric and the DMFC Mental-Pong neural predictivity metric are linearly correlated to their corresponding behavior of OCP accuracy and ball position prediction (as seen in Figures 2C and 3D), which suggests that these more fine-grained metrics are relevant to the underlying behavioral goal that they represent. But how do they relate to each other, and is there a model class that can match both reasonably well?
As we can see in Figure 4A, the two behavioral goals of Mental-Pong ball state predictivity and Physion OCP accuracy do not appear to be very related across models ( $R \approx -0.241$ ,  $p \approx 0.134$ ). This suggests that held-out accuracy to novel scenes within environment as in OCP, does not imply that the same model will generalize to completely new environmental dynamics as in Mental-Pong. In particular, it is important to consider generalization to new environments, since the end-to-end pixel-wise models such as SVG and FitVid subtly overfit to this environment (attaining the highest OCP held-out scene accuracy), but they fail to generalize as well to the completely out-of-distribution Mental-Pong setting (rightmost periwinkle and dark green points in Figure 4A). For example, for FitVid, this failure is visualizable, as it predicts the Mental-Pong ball to be static, regardless of enlarging the ball during evaluation or pretraining with temporal augmentations (Figure S2).
Delving into the more challenging, fine-grained measures of DMFC Mental-Pong neural predictivity and correlation to human error patterns, we observe in particular that the dynamically-equipped
VC-1-based models (VC-1+CTRNN and VC-1+LSTM in cyan) reasonably match both human error patterns in Physion and Mental-Pong DMFC neural dynamics (Figure 4B).
Just as VC-1 on its own does not universally dominate on every individual sensorimotor task, but instead outperforms the best prior existing visual foundation models on average across the 17 CortexBench tasks [Majumdar et al., 2023], our finding that future prediction in the latent space of this model class reasonably matches both human behavioral patterns and neural responses is therefore consistent with this observation. Taken together, our results suggest that future prediction in the latent space of video foundation models for Embodied AI is a promising paradigm for developing models of physical dynamics that are both a better match to neural and behavioral recordings, and can structurally generalize across diverse environments and scenarios within.
Overall, we find that structural generalization to completely new environments and matching dense neurophysiological data to be a strong constraint on sensory-cognitive hypotheses of mental simulation. In a manner evocative of Yamins et al. [2014]'s discovery of then-novel image categorization-optimized CNNs outperforming all prior models in matching primate inferotemporal cortex, our dynamically-equipped video foundation models notably outperform all other models in DMFC neural response matching, illustrating the evolving nature of goal-driven models, especially when applied to higher-cognitive brain areas while the animal is performing a task. Going forward, we believe that we are at a crucial turning point whereby foundation models that engage with the visually rich, dynamic scenes that humans and animals naturally interface with, will be jointly critical for progress in Embodied AI and neuroscience, addressing the recent call to action to build AI that can be as grounded in the physical world as animals are [Zador et al., 2023]. We observe that both the popular machine learning paradigm of pixel-wise future prediction and visual foundation models pretrained on webscale imagesets, typically favored for classic computer vision tasks like image segmentation and classification, underperform here. Instead, future prediction within the latent space of a video foundation model, pretrained on diverse egocentric sources, aligns best with high-throughput neural and behavioral patterns in scenes that it was not originally trained on. Our findings indicate that primate mental simulation harbors robust inductive biases, and is so far most consistent with predicting the future state of its environment in a latent space that is reusable across dynamic environments.
On our existing benchmarks, there are a few ways that we envision our current models can be improved. First, we believe that major strides can still be made in the encoder module of the models, namely by better leveraging temporal relationships to build a more "factorized" and reusable, object-centric representation. The need for a more factorized, object-centric representation is suggested by the high neural predictivity of the joint, ground truth position + velocity oracle in Figure 3B, compared to either the position or velocity oracles alone. At the same time, it is crucial to also maintain reusability in this representation since the fixed object-slot C-SWM models do not match these data well, and can be considered an example of an object-centric, but not reusable, representation. Therefore, a couple ideas in this direction would be to employ dynamic object-slots [Traub et al., 2022, Didolkar et al., 2023] or structured masking [Bear et al., 2023, Yuan et al., 2023] pretrained on Ego4D or even larger egocentric datasets such as CortexBench [Majumdar et al., 2023], to yield object-centric, video foundation models. Additionally, simpler modifications such as differentiating between agent-based egomotion vs. external object motion may likely improve video foundation models by better closing the loop between taking actions in the world vs. simulating it.
The encoder modifications mentioned above may enable the models to learn more explicit representations of objects and their material properties, compared to current self-supervised methods that are more statistical in nature [Balestriero et al., 2023]. This is especially relevant given our finding that all current models struggle most with the soft-body interactions of the "Drape" scenario in Physion, both in terms of accuracy (Figure S4D) and matching human error patterns (Figure 2), yet Ego4D has many examples of deformable objects in it (such as baking, clay, etc). Thus, it is not the pretraining dataset that is impoverished, but rather the current model architectures and loss functions are failing to pick up on these in existing datasets. The dynamics architecture could also benefit from including multiple timescales of hierarchy, with some recent neurobiological evidence of this type of temporal hierarchy already existing in frontal cortex [Sarafyazd and Jazayeri, 2019]. Such dynamics could sync well with dynamic object-slot encoders, capturing more rapid object state changes at one timescale and slower material property shifts across scenes at a higher-level, more abstracted timescale.
We thank the anonymous reviewers for their helpful feedback on the initial manuscript. A.N. thanks Christopher J. Cueva, Leslie P. Kaelbling, Thomas Serre, Kevin A. Smith, Joshua B. Tenenbaum, Rahul Venkatesh, Nicholas Watters, and Chengxu Zhuang for helpful discussions. A.N., M.J., and G.R.Y. acknowledge the generous support of the K. Lisa Yang Integrative Computational Neuroscience (ICoN) Center at MIT. M.J. is also supported by NIH (NIMH-MH122025), the Simons Foundation, the McKnight Foundation, and the McGovern Institute. R.R. is supported by the Helen Hay Whitney Foundation.
Almost everything humans and animals do revolves around an intuitive understanding of the physical dynamics of the world we are embedded in, enabling us to make long-range plans and perform actions that ensure our survival. Furthermore, engineering at least this same level of intuitive physical understanding in silico will be critical for progress in robotics, autonomous vehicles, and any other embodied applications that involve safely taking actions in the real world. Our work not only provides a strong measurement of the degree of alignment between our current best engineered systems to those of humans and animals, but through the differentiation across choices of the architecture, objective function, and pretraining dataset, also provides scientific insight into the evolutionary constraints underlying the neural mechanisms of mental simulation. In particular, we quantitatively observe that future prediction in a latent space optimized for diverse, dynamic scenes is our current best theory that predicts neural dynamics during mental simulation behavior, compared to popular pixel-wise and object-slot alternatives. This set of observations suggests that making progress in Embodied AI will also correspondingly yield an improved understanding of mental simulation in human and animal brains.
M. Babaeizadeh, M. T. Saffar, S. Nair, S. Levine, C. Finn, and D. Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106.13195, 2021.
R. Baillargeon, E. S. Spelke, and S. Wasserman. Object permanence in five-month-old infants. Cognition, 20(3): 191-208, 1985.
A. Bakhtin, L. van der Maaten, J. Johnson, L. Gustafson, and R. Girshick. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32, 2019.
R. Balestriero, M. Ibrahim, V. Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mialon, Y. Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210, 2023.
P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
P. W. Battaglia, J. B. Hamrick, and J. B. Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):18327-18332, 2013.
P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
D. Bear, E. Wang, D. Mrowca, F. J. Binder, H.-Y. Tung, R. Pramod, C. Holdaway, S. Tao, K. A. Smith, F.-Y. Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
D. M. Bear, K. Feigelis, H. Chen, W. Lee, R. Venkatesh, K. Kotar, A. Durango, and D. L. Yamins. Unifying (machine) vision via counterfactual world modeling. arXiv preprint arXiv:2306.01828, 2023.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. S. Chatterji, A. S. Chen, K. A. Creel, J. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, J. Etchemendy, K. Ethayarajh, L. Fei-Fei, C. Finn, T. Gale, L. E. Gillespie, K. Goel, N. D. Goodman, S. Grossman, N. Guha, T. Hashimoto,
M. Sarafyazd and M. Jazayeri. Hierarchical reasoning by neural circuits in the frontal cortex. Science, 364 (6441):eaav8911, 2019.
S. Schwettmann, J. B. Tenenbaum, and N. Kanwisher. Invariant representations of mass in the human brain. *Elife*, 8:e46619, 2019.
R. N. Shepard and J. Metzler. Mental rotation of three-dimensional objects. Science, 171(3972):701-703, 1971.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
K. A. Smith and E. Vul. Sources of uncertainty in intuitive physics. Topics in cognitive science, 5(1):185-199, 2013.
E. S. Spelke. Principles of object perception. Cognitive science, 14(1):29-56, 1990.
E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, volume 139, pages 10347-10357, July 2021.
M. Traub, S. Otte, T. Menge, M. Karlbauer, J. Thuemmel, and M. V. Butz. Learning what and where: Disentangling location and identity tracking without supervision. In The Eleventh International Conference on Learning Representations, 2022.
T. D. Ullman, E. Spelke, P. Battaglia, and J. B. Tenenbaum. Mind games: Game engines as an architecture for intuitive physics. Trends in cognitive sciences, 21(9):649-665, 2017.
R. Villegas, A. Pathak, H. Kannan, D. Erhan, Q. V. Le, and H. Lee. High fidelity video prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019.
B. Wu, S. Nair, R. Martin-Martin, L. Fei-Fei, and C. Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2318-2328, 2021.
D. L. Yamins, H. Hong, C. F. Cadieu, E. A. Solomon, D. Seibert, and J. J. DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23):8619-8624, 2014.
J. Yuan, T. Chen, B. Li, and X. Xue. Compositional scene representation learning via reconstruction: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
J. M. Zacks. Neuroimaging studies of mental rotation: a meta-analysis and review. Journal of cognitive neuroscience, 20(1):1-19, 2008.
A. Zador, S. Escola, B. Richards, B. Ölveczky, Y. Bengio, K. Boahren, M. Botvinick, D. Chklovskii, A. Churchland, C. Clopath, et al. Catalyzing next-generation artificial intelligence through neuroai. Nature Communications, 14(1):1597, 2023.