ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
Authors: Qineng Wang
Wenlong Huang
Yu Zhou
Hang Yin
Tianwei Bao
Jianwen Lyu
Weiyu Liu
Ruohan Zhang
Jiajun Wu
Li Fei-Fei
Manling Li
Venue: ICLR 2026 Poster (2026)
Keywords: Embodied Agents
Vision Language Models
Benchmarking
World Modeling
Primary area: datasets and benchmarks
TL;DR: We introduce ENACT, which probes embodied cognition in VLMs via egocentric world modeling. Two permutation tasks (forward/inverse) built from scalable simulator data with online verification.

Abstract: Embodied cognition argues that intelligence arises from continuous sensorimotor interaction with the world. It raises an intriguing question: do modern vision-language models (VLMs), trained largely in a disembodied manner, exhibit signs of embodied cognition? To investigate this, we introduce ENACT, a benchmark that probes this question through world modeling from egocentric interaction. Grounded in a partially observable Markov decision process (POMDP) framework, ENACT comprises two complementary sequence reordering tasks: forward world modeling (predicting an ordered sequence of future states from actions) and inverse world modeling (inferring an ordered sequence of actions from state changes). Correctly solving these tasks indicates that the model has a solid understanding of how the environment will evolve given one's actions. Our scalable dataset contains 8,972 QA pairs derived from diverse, long-horizon household activities in the BEHAVIOR simulator. Experiments reveal a significant performance gap between state-of-the-art VLMs and humans, which widens dramatically as interaction horizons lengthen. We find that models consistently solve the inverse problem better than the forward one and exhibit strong embodied biases, showing a preference for right-handed actions and performance degradation with camera perspectives that deviate from those of human vision.

Published as a conference paper at ICLR 2026
ENACT: EVALUATING EMBODIED COGNITION WITH WORLD MODELING OF EGOCENTRIC INTERACTION
Qineng Wang
Wenlong Huang² Yu Zhou
Hang Yin Tianwei Bao'
Jianwen Lyu'
Weiyu Liu
Ruohan Zhang?t Jiajun Wu a Li Fei-Feift Manling Li
Northwestern University
2 Stanfford University 3 3UCLA
Website
Code
Dataset
ABSTRACT
Embodied cognition argues that intelligence arises from sensorimotor interaction rather than passive observation. It raises an intriguing question: do modern vision- language models (VLMs), trained largely in disembodied manner, exhibit signs of embodied cognition? We introduce ENACT, a benchmark that casts evaluation of embodied cogmition world modeling from egocentric interaction in a visual question answering (VQA) format. Framed as a partially observable Markov decision process (POMDP) whose actions are scene graph changes, ENACT comprises two complementary sequence reordering tasks: forward world modeling (reorder shuffled observations given actions) and inverse world modeling (reorder shuffled actions given observations). While conceptually simple, solving these tasks implicitly demands capabilities central to embodied cognition aegnition-affordancce recognition, action-effect effect reasoning, embodied awareness, and interactive, long- horizon memory from partially observable egocentric input-while avoiding low- level image synthesis that could confound the evaluation. We provide scalable pipeline that synthesizes QA pairs from robotics simulation (BEHAVIOR) and evaluate models 8,972 QA pairs spanning long-horizon home-scale activities. Experiments reveal a performance gap between frontier VLMs and humans that widens with interaction horizon. Models consistently perform better on inverse task than forward one and exhibit anthropocentric biases, including preference for right-handed actions and degradation when camera intrinsics or viewpoints deviate from human vision. Website at enact--embooied-cconiiton www.ilddd.oogiiiiiongiu.b.oooo
affordance
INTRODUCTION
Intelligent behavior in the physical world requires grounding abstract knowledge in interaction with the environment. Embodied cognition argues that intelligence is not passively acquired but enacted through continuous sensorimotor interaction with the world (Smith & Gasser 2005). Recent advances in large foundation models such as Vision- Language Models (VLMs) (OpenAl 2025 2025) DeepMind 2025), although predominantly trained in disembodied fashion, have exhibited promising signs of interactive intelligence. Yet it remains unclear how to measure the extent to which embodied cognition emerges in these models.
Prior work examines complementary slices of embodied cognition: spatial perception in static scenes (Ramakrishnan al. 2024), linguistic reasoning and planning (Li et al. (2024b), and reasoning about interactions between primitive objects (Yi et al. 2019 Gao et al.) 2025). Taxonomies such as Yang et al. (2025) attempt catalogue embodied capabilities for VLMs but ultimately rely subjective criteria. Consequently, a unified objective that tightly couples egocentric perception with embodied interactions in everyday activities remains missing (Frick & Möhring 2016 Thompson
2005 Clark 2006; Barsalou 2020).
To this end, we introduce ENACT, benchmark that studies embodied cognition through world modeling via egocentric interaction in a visual question answering (VQA) framework. Grounded
"Equal contribution.
I Equal advising.

---

Published as a conference paper at ICLR 2026
in partially observable Markov decision process (POMDP, Aström 1965 Figure ??), formulate world modeling (Ha & Schmidhuber, 2018) as the evolution of egocentric visual observations of agent's s actions, which are represented as scene graph (Johnson et al. 2017) changes derived from low-level physics-based simulator state. Specifically, we focus on two forms, forward world modeling and inverse world modeling, and they are formulated as sequence reordering for evaluating embodied cognition for a VLM. This sequence-reorderigg VQA view isolates long-horizon interactive visual reasoning from photorealistic video prediction and forces models to reason about how sequences of embodied actions transform the environment from egocentric observations. In forward world modeling, given a visual observation and a sequence of actions, the model must reorder a shuffled sequence of future visual observations. In inverse world modeling, given an ordered sequence of visual observations, the model must reorder the corresponding shuffled action sequence. Though seemingly a narrow lens, answering these queries implicitly involves a broad set of capabilities central to embodied cognition: affordance recognition, retion-effeed reasoning, and embodied awareness, together with reasoning about contact and other low-level physical consequences encoded in our predicates, all from egocentric input. Under partial observability, it demands integrating observations and actions across extended horizons, posing challenges for interactive and long-term memory. Through the same lens, we also examine factors that inform future VLM data design, including existing biases toward human embodiment such as hagtehhandedness and human- egocentric viewpoints and intrinsics (e.g., FOV, aperture).
the environment conditioned
Leveraging this unified lens, we additionally demonstrate how such evaluation data can be automat- ically and scalably generated within a robotics simulator, such as BEHAVIOR (Li et al. 2024a). Given a robot manipulation trajectory, we extract symbolic scene graphs leveraging ground-truth physics-base based simulation state (e.g., continuous poses and contacts), which is uniquely accessible in simulation compared to the real world. We then select the key- where the scene graph changes and sample subsequences of desired lengths to assemble the state-action sequences for constructing the question-answer (QA) pairs. By additionally leveraging a novel QA sampling scheme, our pipeline can generate up to millions of QAs from single episode, enabling massive scaling across the thousands of trajectories common in robotics datasets. Although we do not train or finetune VLMs here, the resulting data directly supports future embodied decision-making studies (Azzolini et al. 2025).
We report metries at different granularities: Task Accuracy (exact ordering) and Pairwise Accuracy (fraction of adjacent pairs correctly ordered). Our experiments reveal that ENACT is challenging for current VLMs, which lag significantly behind human performance (Figure ??). This performance gap widens as the task horizon increases, where VLM accuracy degrades sharply with trajectory length while human performance remains high. We also find that all models consistently perform better on inverse than on forward world modeling. Furthermore, we uncover two notable biases: VL.Ms show a clear preference for understanding right-handed dynamics, and the performance of representative models like GPT-5 mini drops significantly with non-human-eye-like camera intrinsics or viewpoints.
Overall, our contributions are threefold: (1) We introduce ENACT, benchmark for evaluating embodied cognition via forward and inverse world modeling from egocentric interaction. (2) We provide an easily scalable data generation pipeline leveraging robotics simulation (BEHAVIOR) and provide a dataset of 8,972 QAs synthesized from diverse, long-horizon interactions in everyday environments. (3) Experiments state-of ofate-tf-the-art VLMs reveal widening gap human performance with horizon length, anthropocentric biases (e.g., right-handed priors, human-like camera intrinsics), and real-world evaluations that mirror simulator trends with only limited sim-to-real gap.
ENACT: EGOCENTRIC INTERACTIVE EMBODIED COGNITION TEST
2.1 PROBLEM FORMULATION
We investigate the embodied cognition of VLMs by framing it as a world modeling problem, which we evaluate using egocentric, interactive reasoning tasks. Formally, the benchmark is defined over state space S, whose elements are symbolic scene graphs derived from low-level simulator states G; observation space O C #Xx33 of robot's 's egocentric RGB views of the environment; and an action space A whose elements are scene- scene-graph differences a, = ((s,,),1) A symbolic scene graph is structured data model that represents the objects in scene as symbolic nodes
2

---

Published as a conference paper at ICLR 2026
Mobile Manipulation Demonstrations (30H2)
Filtering and Augmentation
Forward World Modeling QAs
Segmented Frames with Abstract State Changes
Inverse World Modillngg QAo
State
Servadie
Observation
Action a
Curation
Figure 1: Overview of ENACT data curation pipeline. We first obtain aligned scene graphs (states) and RGB observations from mobile manipulation dataset in a robotics simulation (BEHAVIOR). The trajectory is then segmented by identifying key-frames where an abstract state change occurs (i.e.. the scene graph difference is non-empty). From this set of key-frames, we sample multiple key-frame trajectories, which are used to construct the forward and inverse world modeling VQA questions. Here N refers to the number of all sampled trajectories across all step lengths.
(e.g., On ( (fridge)) and their relationships as edges (e.g.. OnTop (pen, desk)). ). We view the underlying embodied task as POMDP. As shown in Figure | we first filter this raw data identify all timestamps where an abstract scene graph state change occurs (i.e., the -cene-graph difference 5(8, 8;-1) * 2). This process yields a smaller, chronologically ordered set of segmented frames, which serve as the candidate pool for our benchmark.
From the pool of segmented frames, we sample R trajectories, each with a chronologically ordered tuple π (No, .i,-1) 1) of L key frames. This initial abstraction into discrete decision epochs similar to a semi-MDP (Sutton et al., 1999). However, we treat each of these final key-frame trajectories as self-contained POMDP instance with scene graphs S, and observations O,, For 2, the action connecting consecutive key frames is the visible scene-get scene-graph graph delta
= 0, ... ag vis 1:t+1: ), where Avis returns the subset of differences in 1 Siz) that visible in both images. Together, these actions form a discrete symbolic action space A. For notation simplicity, relabel indices in for each key-frame trajectory to (0, ....... L 1) and (sk; 0z) (Si,, ).,)
Building on these trajectories, we formalize two tasks. For forward world modeling, given the current image op, the correctly ordered action sequence (ao, 0r and a shuffled list of ob- the model outputs a permutation a E Symm[L 1)) that (ooj----,(((L-))) =
servation images O' = =o'
orders the images to match the actions:
OL- For inverse world modeling, given op. the correctly ordered observation images (o1....,0r. ) and shuffled list of actions A' (ahy...,, d' the model outputs permutation T E Sym( (L 1]) that orders the actions to be consistent with the observation progression: (d'(u)s-.. (t-))) = (ag..... .......,,,..)) 01.-)).
2.2 KEY-FRAME TRAJECTORIES SYNTHESIS FOR SCALABLE DATA GENERATION
Segmented Frames with Abstract State Changes. Raw robot trajectories often contain long stretches with semantic changes (c.g., gripper motion when opening the toolbox in Figure We mark a timestamp f whenever the simulator state makes a minimal abstract state change, such as transitioning from not grasping grasping drill with the right hand. The BEHAVIOR simulator exposes boolean and relational predicates, where flipping one predicate or updating a relation is our atomic state change. A time enters the candidate pool if the scene- graph difference 8(s,, $;-1) is nonempty. To avoid near-duplicate frames, we compare each new change with the last accepted segmented frame: we form predicate-level change signature Cj and keep only if its cosine similarity with the previous signature cj-1is below threshold. This yields chronological set of segmented frames K. = (t1 <... < tM} with (s.,) Thresholds and further details are in the Appendix A.2.1
Key-Frame Trajectories Synthesis. From the segmented M frames, we sample length-L key frame trajectories (fo..... iz-1) with 1 1 i0 < < i--1 < M, so indices do not need to be adjacent. Each candidate is strictly validated: for every h. the visible state change Avis $;,) nonempty, and the edited objects are visible in both images, except for object transitioning events (e.g., pineapple
3

---

Published as a conference paper at ICLR 2026
Diverse Tasks as Data Source
Forward World Modeling QA Example
Inverse World Modeling QA Example
Top cobinet Open
Fridge Open
Right groaping ng top
Tometo Insids of the indpe
Robot -n Nos Right grosping temato Left greaping tomatio
cobinet
Counter orop fridge citting beard
X Shuffie
X
Shuffle
Fg
cornect images?
What the cornect onder actions?
Word
Objects
Ground
VLM Ground Truth
Figure 2: Data sources and QA examples. ENACT is built from diverse, morgoooizoon activities performed by real robots (left). We provide examples for (mid) forward world modeling and (right) inverse world modeling. More QA examples and prompts are available in the Appendix A..33
being diced), where transient occlusion is permitted. We then treat each valid key-frame trajectory as an individual POMDP instance, with Sg and A₂ as defined in the problem formulation. To further scale data generation, we exploit that typically L < M (in practice L < 10 while M X 30), and we use skipping to convert trajectory construction into "seat selection" " combinatorics problem, choosing seats out of M, which yields at most (M) distinct candidates from single trajectory. The detailed algorithm is in the Appendix A.2.2]
World Modeling QA Generation. After obtaining the sequence of key-frame trajectories, we formulate the forward and inverse world modeling as sequence reordering visual-question answering (VQA) tasks. This formulation offers two advantages. First, it yields a clean evaluation signal for long-horizon interactive visual reasoning without conflating performance with photorealistic video prediction. Second, it requires models to reason about how a sequence of embodied actions causally transforms the environment over multiple steps, while maintaining Long-horizon spatial memory from purely egocentric observations in large home-scale scenes.
2.3 DATASET OVERVIEW AND EVALUATION DESIGN
We construct the benchmark from the BEHAVIOR simulator and challenge Li ct al. 2024a). BEHAVIOR Challenge provides diverse Long-horizun home-scale activities, and we select 29 of them, using one trajectory per activity to recover aligned pairs {(st, o.)). Each trajectory is segmented into segmented frames K. then converted into key-frame trajectories and finally into two QA types: forward and inverse world modeling (examples in Figure 2p. Across step lengths L E {3, 10} we sample about 560 items per L for each QA type, yielding 8,972 total questions. The data uses common predicate classes (e.g.. Open, Cooked, Grasping), with distributions shown in Figure 2
Dataset Overview.
full statistics appear in Appendix AA.311 AA.3I
Evaluation Design. Multiple valid answers can exist for a given question, so we use an online verifier that accepts any predicted permutation, or consistent with the input constraints. We report two metrics: Task accuracy captures exact ordering, while Pairwise accuracy grants partial credit for near-correct sequences. Specifically, (1) Task accuracy measures success at the question level: a question receives score 1 if the verifier accepts the full prediction and 0 otherwise, and the dataset score is TA- (I/[D)) Exer 1 1(accepted(z:). ( (2) Pairwise accuracy measures stepwise consistency: for a question with length L, we count adjacent pairs passing the verifier's 's local check sstate-attion for forward; action state for inverse) and divide by L, then report the micro- averrge PA = (E, #correct pairs in <>)((,,, T.z), equivalent to averaging per-item pairwise scores when T is fixed. Implementation details are in Appendix A.3.2
EXPERIMENTS AND ANALYSIS
3.1 WORLD MODELING AS A PROXY FOR EVALUATING EMBODIED COGNITION
Experimental Setup. (1) VLM evaluation setup. We evaluate ENACT with 7 proprictary VLMs
from 3 families (OpenAl 2025 Deep 2025 Anthropic 2025) and 23 open-weight models from
4

---

Published as a conference paper at ICLR 2026
Forward World Modeling
Inverse World Modeling
Model
4
7
10
4
7
10
Proprietary Models
GPT-5
84,62
75.26
69.90
64.18
57.48
52.16
69.4
16.93
96.28
80.37
76.09
68.78
65.71
62.13
57.12
55.33
GPT-5 mini
67.50
76.25
70.65
63.41
58.14
52.38
46.65
44.11
85.05
76.77
75.43
67.67
63.79
57.04
55.04
50,02
GPT-5
50.29
38.61
30.35
25.97
17.59
53.95
42.48
31.40
28.20
20.33
Gemini 2.5 Pro
86.10
[6.42]
69.83
60.80
53.26
48.12
50.12
36.98
87.94
81.18
75.
20.03
06.00
62.99
57.78
56.62
Gemini 2.5 Flash
81.64
67.33 94
54.17
43.78
37.43
92.73
20 88
78.07
62.77
72.18
60.83
58.19
53. 14
51.78
47.59
44.96
Gemiai 2.5 Flash-LLtte
27.60
46.04
23.16
Claude Sonnet
45.82
30.52
26.61
20. 16
73.25
37.00
32.71
30.50
28.49
Open-Weighh HeigHWiggh Models
GLM-4.5VV
74.30
99
47.6
16.78
50.83
25.669
60
19.67
80 59
69.22
57 57.04
51.53
146.9
41.68
33.36
17.93
72.47
25.89
25.80
75.79
[62.
50.83
85.85
57 54
$2.89 88
27 85
25.55
62.
70.09
60
$5.38
45. 90
39.35
34.12
30.55
Gemma 3- i-
63.29
11..00
32.04
25.82
22.11
19,50
16.74
16.24
64.995
18.337
40.04
33.86
28.53
23.63
21.74
19.36
QVQ-72B-Preo Pee
69.14
52.96
40.83
36.27
33.16
30.63
26. 30
24.76
71.33
58.77
48.43
44.36
40.33 10.26
39.30
36.66
36.58
Qwen2.5 5 V2- las
78.15
00:05
49.857
41.92
36.77
31.73
28.00
25.07
77.30
65.85
53.
48.19
44.07
37.57
33.76
36.27
Qwen2.5. VLL-3-5---
-55.4 46
44.35
35.25
27.52
26.42
18.07
63.55
59.20
51.01
36
47.17
41.47
40.16
Ovis2.5.9g
42.51
31.08
24.61
20.78
18.11
64.96
51.24
35.47
30.93
23.70
23.25
MiniCPM--V.e V.4.5
60.75
38.73
25.47
24 81
21 21
56
18.35
60.33
$3.08
47.35
39.55
34
30.65
27.05
25.71
Beftes--SB-LLaooe
60.23
36.99
$1.83
24.25
21.29
20.80
20.4 46
17.71
47.38
33.86
27 27.26
23.48
19:87
18.50
17.04
15.16
Cosmoss Reason
56.28
41. 86
34.75
78. 40
26 46
26 26.49
25. 41
24.88
58. 58.30
45.93
44
50
35. 72
34.56
31.50
28.64
RAGEL
30.24
40. 19
79 65
25.37
22.75
19.45
17 84
15.87
36.73
85
40.09
35.44
79. 67
24. 79
28. 70
18.91
Human Performance
93.62
95.30
95.04
93,87
95.43
95.41
94.75
95.13
92.05
93.56
94.35
94.25
95.96
97.74
96.30
96.29
Table 1: Evaluation on ENACT (Pairwise Accuracy). Dark gray indicates the best result within each category (Proprietary Open-Weight Models), and Light gray denotes the second-best result within the category. Complete results are in Table 9 (Task Acc.) and Table 10 (Pairwise Acc.).
Team et al. 2025 Lu et al.) 2025 Yao et al. 2024| Azzolimi et al. [2025] Team 2024 Deng et al. 2025). For input, all images are resized to 512 x 512, and we use unified prompt template per QA type. Models are instructed return a parsable Python list encoding a permutation of indices. We apply the online verifier in Section 2.3 and report Task Accuracy and Pairwise Accuracy. (2) Human evaluation setup. We also recruit trained annotators to answer the benchmark under the same instructions as the models. For inter-annotator agreement (IAA), we uniformly stratify 240 items over QA type and step length and collect independent labels from three annotators. Krippendorff's a = 0.83 indicates strong agreement. Full details are in Appendix BB2.1 and B.1.1
11 families (Wang et al. 2025b, Bai et al. 2025; Hong et al. 2025 MetaAl 2025
We visualize Task Accuracy for GPT-5 and human annotators in Figure ??. Since many models collapse long horizons (L = 8 8-10, near-zero task success), we focus the more informative Pairwise Accuracy. The main results appear in Table
Is inverse world modeling easier than forward? Across families and step lengths, inverse consis- tently outperforms forward, with the margin widening as I. grows. For example, GPT-5 and Gem- ini 2.5 Pro maintain gaps at T. 6, and open-weight models such as GLM-4.5V and Qwen2.5-VL. also show higher inverse scores than forward for most L (see Table This asymmetry suggests that models handle retrospective textual reasoning better than the prospective visual simulation required for forward planning.
How does performance change with step length? Accuracy decreases monotonically with L for nearly every model. proprietary and open-weight alike. Shorter tasks (L < 4) are manageable for several VLMs. while longer tasks (L 2 8) are challenging even for the strongest models. Pairwise Accuracy softens but does not alter this downward trend. This sharp performance decay reveals that VLMs struggle to track evolving physical states.
Can VLMs achieve near-human performance? Human performance is far better than any evaluated VLM. SOTA VLMs such as GPT-5 and Gemini-2.5 Pro are comparable to humans only at step length L= 3; their performance drops sharply as the horizon grows. This vast gap confirms that, compared with humans, VLMs still struggle with interactive embodied world modeling tasks.
What is the performance comparison among VLMs? GPT-5 and Gemini 2.5 Pro are the strongest overall in both forward and inverse settings. Several open-weight VLMs are competi- tive: Intern VL3.5-241BB- 2-2BBBBB8B GLM-4.5V, and Qwen2.5-VI often close much of the gap and even surpass Claude 4 Sonnet in multiple settings. GPT-5 mini is also highly competitive, achieving the best score at several short and mid horizons (e.g., forward at L = 3, 7))
Does Reason,, trained on embodied data, outperform other similar-sized models? We compare Cosmos- Reanoss--eannnn---e and other similar-sized VLMs in Table 10 and Figure 17 For
5

---

Published as a conference paper at ICLR 2026
similar-sizze sized models, Saammos-eesson-755 exhibits more stable and generally better performance than other VLMs when the interaction horizon exceeds 5 steps.
Key Takeaways: World Modeling as Proxy for Evaluating Embodied Cognition
Higher performance on the inverse task than the forward one highlights stronger language- based retrospection than action- cetion-oondiiooedd visual reasoning.
Long-horizon degradation reveals limited interactive, spatial memory under partial observ- ability.
The human-moder gap shows that current VLMs are still far from robust embodied world models in mobile manipulation settings.
Beyond these trends, we also confirm that augmenting key-frame selection with contact-based predicates derived from low-level physics yields qualitatively similar behavior (inverse >> forward, strong long-horize hong-horizon degradation); detailed results are provided in Appendix B Table
3.2 SENSITIVITY TO IMAGE REALISM
Since ENACT is generated in the BEHAVIOR simulator, we ask whether VLMs are sensitive to image realism and whether sim-to-real gap appears. To investigate this, collect additional real-world videos and manually annotate them. Furthermore, leveraging the automated ENACT pipeline with simulation, we conduct an in-depth and larger-scale study of how different rendering configurations in the simulator affect performance.
Woal-World Experimental Setup. To assess whether the simulator findings transfer to real im- agery, we construct a real-world benchmark over three everyday scenes (kitchen, dinner table, and workspace). From videos recorded each scene, we manually select segmented frames and annotate scene graph for every frame. Applying the ENACT pipeline to these annotations yields 960 real-world QA pairs, on which we evaluate Intern Mtt.mVL3..--411B A28B. Examples from the three scenes shown in Figure ??.
Simulation Experimental Setup. (1) Configuration. We use GPT-5 mini as the base model for SOTA VLMs due its strong cost-performance balance shown in Table For diversity, we also evaluate Intern VL.3552241 and report its performance across all settings in Figure 16 We evaluate step lengths LE {3, 6, 9}. For each L and each QA type (forward, inverse),
sample 50 items, yielding 300
total QAs. We report, for each setting, the Pairwise Accuracy difference A = PAtaseline - PAvariamt and two-tailed unpaired p-values versus the baseline. | < 0.05 will be considered
small change.
(2) Image realism implementation. BEHAVIOR uses Isaac Sim (NVIDIA (205), our baseline uses Ray Tracing (NVIDIA) 2021) with default global effects. We evaluate three alternatives on realism spectrum: Realistic (segmented frames translated to a real-world style using GPT-image- OpenA] (2025). Path Tracing (higher-fiellit rendering, Kajiyal (1986)), and Ray Tracing Only (Ray Tracing with global effects such as reflections and stage lights disabled). Detailed setup, prompts, and examples are in the Appendix B.4 Results are summarized in Figure (panel A).
Does real-world data change performance? As summarized in Table InternVL3.5-2411B. exhibits similar absolute accuracy on real-world data as the simulator, On real videos, inverse queries consistently outperform forward ones, and accuracy drops sharply as the temporal horizon increases. Overall, real-world evaluation does not reveal a significant sim-to-real gap, and the empirical trends are fully consistent with those observed in the simulator. This consistency validates the simulation as a faithful proxy for evaluating real-world embodied cognition.
Forward (Real- World)
Inverse (Real- World)
Metric
9
3
8
10
Task Accuracy
50.00 33.33 13.33 8.33
0.00
90.00
85.00
$5.00
21.67 13.33
3.33
Pairwise Accuracy 80.00 67.78 61.25 49.00
38.89
37.14
26.88
25.74
90.00
88.33
72.92 57.67 44.17
43.57
31.04 26.85
Table 2: Real-workh evaluation of Intern 241.--2411B-2881 Task and pairwise accuracies (%) on 960 QA pairs generated from real-world manipulation videos. We report performance across different interaction horizons for both forward and inverse world modeling.

---

Published as a conference paper at ICLR 2026
Does rendering realism change performance? We find no statistically significant degradation or improvement across the spectrum. All settings have p 2 0.2 relative to the baseline, and observed deltas small across both QA types and all step lengths (Figure A; Figure T6] A), suggesting that the evaluated VLMs are robust to rendering variations in world modeling tasks.
Key Takeaways: Image Realism
Real-world evaluation mirrors our simulation findings with minimal sim-to-real gup, while simulation offers a more controlled and reproducible testbed.
Insensitivity to rendering variations indicates bottlenecks in multi-step interaction reasoning ruther than low-level image realism.
3.3 Do VLMS EXHIBIT ANTIROPOOCNTRES BIAS ON HUMAN VISION?
VLMs are mostly trained on RGB images that mirror how humans typically see the world. However, different embodiments may have diverse camera configurations. We therefore test whether VLM performance is sensitive to camera configuration, i.e., if dataset bias is present.
Experimental Setup. (1) Configuration. We reuse the setup from Section 3.2 We use GPT-5 mini as the base VLM, and report Intern VL3.5-241B in the Appendix [B.5] (2) Camera FOV. The baseline Aperture 40. We examine Aperture 30, 60, 80, and Fisheye. Rendering and all other parameters are held fixed. (3) Camera Height. The baseline is (1.75 m) high for eye-level view used in Behavior replays. We test High (+0.5 m) and Low (-0.25 m). We choose (-0.25 m) since a lower height will consistently make relevant objects invisible. Examples are in the Appendix B.5 Results are summarized in Figure (panels B. and B.2).
Does field of view matter? Figure 3 (B.1) shows that small change to Aperture 30 shows no significant difference from baseline (p 0.1). Larger deviations hurt performance: Aperture 60, 80, and Fisheye consistently and significantly worse than baseline across QA types and step lengths (p S 0.01), suggesting that the model performs better with human-like intrinsics.
Does camera height matter? As shown in Figure 3 (B.2), increasing the camera height (High) significantly reduces GPT-5 mini's accuracy in the forward setting with = -0.13. By contrast, the High inverse setting shows no statistically significant change, though with performance drop = 0.06. For the Low camera, both forward and inverse are not significantly different from the baseline, likely because the -0.25 shift remains within normal human height variation.
Key Takeaways: Anthropocentric Bias on Human Vision
Performance degradation on non-standard views implies VLMs are biased towards human- like egocentric viewpoints and intrinsics.
Reliance on numan-centric visual priors limits generalization to diverse robotic embodiments with non standard optics.
3.4 Do VLMS EXHIBIT ANTHROPOCENTRIC BIAS ON EMBODIMENT?
To further understand VLM embodiment, we investigate two potential biases: self-awareness regarding the robot's own body and handedness asymmetry. a common trait in humans.
Experimental Setup. We study these two aspects using distinct experimental setups. (1) Robot Appearance. To test for self-awareness, we assess whether VLMs can recognize their embodiment regardless of its appearance. We reuse the experimental configuration from Section 3.2] with GPT- 5 mini as the base model. The bascline is the default black-and andc--andwhiit robot appearance. We test three variants: White Color, Random Color (robot color is randomized at each frame), and Skin Color (robot is rendered with human-like skin tone). (2) Handedness Asymmetry. Inspired by human motor control, where approximately 89% of the population is right-handed (Papadatou- -Pattou et al.) 2020), we investigate if VLMs exhibit a similar "dominant hand". We analyze this configuration with a predicate-lewe error analysis of all tested VLMs and report GPT-5 mini in Figure 3 We isolate all errors related to the Left Srrassnng ng and wigghrrrriinnn predicates. Using the framework described in Section (35)) we frame our metrics in terms of Precision and Recall. We also report Mixing Rate, which measures the proportion of ground-truth state differences for one hand that the model
7

---

Published as a conference paper at ICLR 2026
A: Image Realism
B.1: Camera FOV
Forward
Forward
p=0.43 A= 0..00.03
p 42
p-0.78
0.18 A=. 0 04
p 02 0.0.
p-0.00 A= 0.12
p-0.00
A= 0.03
A= 0.01
0.10.11
(sig)
0.01
Inverse
p=0.91 åm-0.00
p=0
p=0.27 4=+0.00 04
Inverso
p.8.8 d=+0.00
p=0.02 4m-0.09
p=0.01
p=0.00 Am-0.13
0.001
Am-0.03
Am-0.11
p-value
Tracing 8.2: Camera Height
Realistic
Ray Trading Only
Aperture
Fisheye C.2: Robot Handedness Analysis
Aperture
C.1: Robot Appearance
Forward
p-0.00 A=10.13
p-0.10 0-05.
Forward
p-0.29 0.03
P-0.76
p-0.88
Hand Precision(
Recall
Mixing
4=+0.01
4=+0.01
Forward
Tafh
0.4485
0,4067 Joanss
0.0638
Right 0.4976 0.4958
0.0487
Inverse
Inverse
p=0.11
p=0.54 dm-0.02
p=0.62 6--0.01
p=0.24 4m-0.04
p=0.92 4=+0000
Left
0 0.400
0.1858
Am-0.06
Right 0.4618...... 0.4618.
High
Low
White Color Random Color
Skin Color
Figure 3: Evaluations on image realism and anthropocentric bias on human vision through ENACT. Heatmaps show two-tailed unpaired t-test results against the baseline, using Pairwise Accurucy. < 0.05 is considered significant. Darker red means more significant. A is the performance change from the baseline. If significant and A < O, the setting worse than the the left- and right-hand predicates, where Mixing is
baseline. C.2 reports the robot's performance the proportion of ground truth left or right cases that are predicted as the other hand (i.c., mixing one hand into the other). + means standard error.
incorrectly attributes to the other. Higher precision and recall with lower mixing indicate greater proficiency. Appearance examples and handedness analysis are in the Appendix B.6.1 and B.6.2
Are VLMs aware of their own embodiment, and is this awareness robust to changes in their visual appearance? As shown in Figure and Figure (panel C.1), altering the robot's appearance has no statistically significant impact on performance for both GPT-5 mini and Intem VL.3.5224 1.1.5-211B B. For all variants (White, Random, Skin Color), the performance deltas small (A) 0.05) and non-significant (all > 0.10), suggesting that the model's 's understanding of its interaction with the world not tied specific visual representation of its body.
Do VLMs exhibit a handedness asymmetry in their interactions with the world? Our analysis of hand-related errors, summarized in Figure (panel C.2), reveals consistent and strong asymmetry (complete еттог results are shown in Figure 38a and 38b). For both forward and inverse tasks, the right hand consistently outperforms the left hand across metries. Precision and recall are substantially higher for the right hand, while the mixing rate is significantly lower. For instance, in the forward task, 9.38% true left-hand changes were incorrectly identified as right-hand changes, whereas only 4.67% % of right-hand changes were misattributed to the left. Full analysis is in Appendix C.I
Key Takeaways: Anthropocentric bias on Embodiment
VLMs are robust to the embodiment appearance variations.
VLMs exhibit strong right- bias, which is consistent with human handedness distribu- tion.
3.5
ERROR ANALYSIS
3.5.1
EXPERIMENTAL SETUP
To gain deeper insight into the reasoning failures of VLMs, we design a systematic error analysis framework. Evaluating errors directly from output permutations (e.g., comparing predicted order [3, 1] to ground truth [2. 3. 1]) is difficult and often uninformative about the underlying cognitive mistakes. Exploiting the fact ground-truth symbolic graphs readily accessible in simulation, instead
8

---

Published as a conference paper at ICLR 2026
convert the model's output into a format that allows for structured, fine-grained comparison with the ground truth. For the forward world modeling task, we take the model'ss predicted permutation of images (d=(1)...... 1)(L-1))
(o1..... ,r.-1) and compute the corresponding sequence of actions (i.e., visible state differences) that this ordering implies: 0д == Avis (* (1++1)> (a))) This yields predicted action sequence (ão. 22- For the inverse world modeling task, the model directly outputs predicted action sequence.
With both predicted and a ground-truth action sequence, we can perform a pairwise comparison at each step Each action og is set of atomic state differences (e.g.. {Add Open (fridge), Remove Inside (basket, cabinet) H. By comparing the predicted set âş with the grounded-truth set dk, we can categorize each atomic state difference. This comparison, simi- lar to analyzing Venn diagram, yields three primary outcomes for each ground-truth state difference: (1) Correct: The state difference is present in both the ground-trute and predicted sets. (2) Omission: The state difference is in the ground-truth set but missing from the prediction. (3) Hallucination: The state difference
the predicted set but not in the ground truth. Detailed setup is in Appendix C We assume each state difference is independent event and aggregate these counts across all actions and all questions in the dataset. Based on this framework, we classify errors into five main categories:
1. Entity Substitution. The model correctly identifies the state change predicate but applies it to the wrong object(s).
2. Polarity Inversion. The model correctly identifies both the object(s) and the predicate, but reverses the polarity of the change (e.g., 'remove' instead of 'add').
3. Predicate Substitution. The model correctly identifies the object(s) involved but describes the state change with an incorrect predicate.
4. Hallucination. The model predicts state change that did not occur in the ground truth.
5. Omission. The model fails to predict ground-truth state change that occurred.
3.5.2 ERROR DISTRIBUTION ANALYSIS
Our error analysis for GPT-5, shown in Figure reveals that the vast majority errors fall into two main categories: Omission and Hallucination. For the forward task, these two error types account for a combined 81% of all failures. This figure even higher for the inverse task, where they make up nearly 84% of errors. This indicates that the model's 's primary challenge is not misinterpreting the specifics of a known state change, but correctly identifying which changes occurred and which did not.
While Omission and Hallucination errors are dominant in both settings, their distribution shifts between tasks. In forward modeling, Hal- lucination is the most common error 43.9%, followed by Omission at 37.1%. 1.. Remarkably, in the inverse task, these two errors are per- fectly balanced, each accounting for exactly 41.8% of all failures. Other error types are far less frequent. Polarity Inversion is more common the forward setting (12.4%) than the inverse (9.2%). Entity Substitution is also slightly more prevalent in the forward task (6.3% 5.4%). Finally, Predicate Substitution mains the rarest error type, though it is more pronounced in the inverse setting (1.9%) compared to the forward task (0.3%). Detailed analysis is in Appendix C
Forward
Inverse
75%
Entity Substitution Predicate Substitution 00mssion
Inversion Hallactnaaion
Error Distribution, broken down by forward and inverse tasks, evaluated on GPT-5,
Figure
Key Takeaways: Structural Error Analysis
The dominance of hallucinations suggests models rely on learned textual priors rather than faithful visual grounding.
High omission rates reveal a fundamental deficit in tracking object persistence across complex state transitions under egocentric partial observation.

---

Published as a conference paper at ICLR 2026
4 RELATED WORK
Embodied Cognition. The theory of Embodied Cognition argues that intelligence arises from an agent's s sensorimotor interaction within environment, grounding abstract knowledge in perception
and action (Gibson 2014, Varela et al. 2017 Clark 1998. Brooks, 1991; O O'regan & Noë [000]]
Barsalou 1999; Lakoff & Johnson| 2008). Grounded in the POMDP framework for decision
making (Kaelbling et al. 1998), work focuses how spatial perception, physical interaction. and linguistic abstraction jointly support embodied cognition (Frick & Möhring 2016: Thompson
2005 Clark 2006 Barsalou 2020). Rather than enumerating individual capabilities, we adopt unified lens of world modeling through egocentric interaction, using controlled, reproducible, and scalable simulation to contrast the behavior of current VLMs with humans.
World Modeling. World models learn action coiiioncondiiooned dynamics for imagination and planning
(Ha & Schmidhuber 2018: Hafner et al. 2019),, achieving scalable gains from counterfactual rollouts
(Hafner et al.] 2023] Bruce et al. 2024; Agarwal et al., 2025) Janner et al. 2022] Wan et al.
2025). Many recent systems emphasize generative fidelity and long-horizon prediction in video
or latent space (Bruce et al., [024|] Agarwal et al. 2025 Finn & Levine| 2017 Ebert et al. 2018).
Complementary benchmarks evaluate control and prediction quality in visually rich settings iTian et al. 2023 Chi et al. 20244] Yue et al., 2025). or study physical and scene understanding from
non-interactive visual data (Bakhtin al. 2019 Yi al., 2019 Girdhar et al. 2020 Bear et al., 2021
Tung et al. 2023 Li et al. 2024a Dang al. 2025 Yang et al., 2025). Others focus sequence-
level temporal coherence and ordering (Qin et al. 2024; Chen et al. (025). Aurora-Bench (Qiu et al.,
2025) focuses short-horizon and ceneerl-purpose video forward and inverse world modeling. As argued by Xing et al. (2025). a world model should serve as a sandbox for reasoning. Our benchmark is therefore designed to study forward and inverse ordering with explicitly defined action space and scalable construction.
VLMs in Embodied AI. VLMs are central to embodied agents, acting as high-level planners (Huang
cet al. 2022a Ahn et al.
2022
Huang et al.) 2023b 2022b] Liang et al. [2022a; Huang et al.
2023a 20244] Jiang al. 2024 Wang et al. 2025a) or end-to-end policies (Zitkovich al.) 2023
Kim et al. [024]] Team et al.) 2024; Driess el al.) 2023). However, current applications are often
confined settings where involved reasoning is not required (Lynch al. 2023). Correspondingly.
many benchmarks emphasize instruction-followei
and goal- conditind control (Das et al. 2018
Padmakumar et al., 2022; Mees et al.] 2022 Fan et al. 2022, Sermanet et al., 2024| Li et al.
2024b Yang et al.) 2025 Gao et al.) 2025), with less focus on the multi-step, consequence-aware
reasoning essential for long-horizon interaction. We address this gap with a benchmark that uses egocentric interaction to evaluate agent's forward and inverse world modeling, requiring long- range interactive reasoning. GVL (Ma et al., 2024) casts value estimation as a reordering task over observation sequences, whereas ENACT focuses transition modeling over interleaved observation- action sequences with an explicitly defined scene graph action space that applies to interactions of any quality.
5 CONCLUSIONS
In this work, we introduced ENACT, a benchmark designed to evaluate the extent to which embodied cognition emerges in VLMs trained on passive datasets. By framing evaluation forward and inverse world modeling from egocentric interaction, ENACT assesses a model's understanding of environmental dynamics and the consequences of its actions. Grounded in a POMDP, we cast this as two sequence-rorderiig tasks: forward world modeling, which predicts an ordered sequence of future states from actions, and inverse world modeling, which infers an ordered sequence of actions from state changes. Our extensive experiments reveal significant performance gap between state- of-the-art VLMS and humans, gap that widens dramatically as the interaction horizon increases. We consistently found that models solve the inverse problem more effectively than the forward one. Furthermore, our analysis uncovered strong embodied biases within these models, including a preference for right-handed actions and significant performance drop with non-human-like camera perspectives. An in-depth error analysis showed that reasoning failures are primarily driven by the omission and hallucination of state changes. ENACT provides scalable and insightful tool for charting a course toward more genuinely embodied artificial intelligence. A detailed discussion of limitations is provided in Appendix
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
The ENACT benchmark was generated in the BEHAVIOR simulator to avoid the privacy risks associated with real-world human data; it contains no human subjects or personally identifiable information. All human annotators hired for evaluation were compensated at rates significantly exceeding their local minimum wage and were not exposed to any sensitive content.
We acknowledge that the simulator may not fully capture the complexity of real-world environments, which can introduce biases and limit the generalizability of our findings. Furthermore, the large-scale models we evaluate carry significant computational and environmental cost. While ENACT is intended for academic research, we recognize that the technologies helps develop could have dual-use applications.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility, our complete codebase is available at our Github Repository This repository contains all scripts for data generation using the BEHAVIOR simulator (Li ct al. (024a), evaluation of all Vision-Lannuage Models, and analysis. Our implementation includes the automated verifier, prompt templates, and the code replicate main experiments, controlled ablation studies (Sections 3.2 3.3 and 3,4), and human baseline evaluation. The full ENACT dataset is also publicly available.
ACKNOWLEDGMENTS
This work is in part supported by the Stanford Institute for Gluann-Centered AI (HAD), ONR MURI N00014-22-1-2740, ONR MURI N00014-24-1-2748, and ONR MURI N00014-21-1-2801.
REFERENCES
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501 250.:25001.3355, 2025.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.016/0 2022.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and- laiio-and-language navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3674-3683, 2018.
Anthropic. Claude sonnet 4 model card. https::/www.aattrroii...com www.:///....ooeecceeeeeaeeeeeeee sonnet May 2025. Release date: May 22, 2025.
Karl Johan Áström. Optimal control of markov processes with incomplete state information Journal of mathematical analysis and applications, 10:174-205 1965.
Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, al. Cosmos-reasonl: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503. 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2502. 2025.
Anton Bakhtin, Laurens der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems, 32, 2019.
11

---

Published as a conference paper at ICLR 2026
Lawrence W Barsalou. Perceptual symbol systems. Behavioral and brain sciences, 22(4):577-660, 1999.
Lawrence W Barsalou. Challenges and opportunities for grounding cognition. Journal of Cogmition. 3(1):31, 2020.
Daniel M Bear, Elias Wang, Damian Mrowca, Felix J Binder, Hsiao- Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Yun Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2 2021.
Rodney A Brooks. Intelligence without representation. Artificial intelligence, 47(1-3):13--15, 1991.
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. Forty-first International Conference on Machine Learning, 2024.
Delong Chen, Willy Chung, Yejin Bang. Ziwei Ji, and Pascale Fung. Worldprediction: A bench- mark for high-level world modeling and long-horizon procedural planning. arXiv preprint arXiv:2506. 04363, 2025.
Lunjun Chen, Ming Yang, and Huazhe Xu. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint arXiv:2202.0948) 2022.
Xiaowei Chi, Chun-Kai Fan, Hengyuan Zhang, Xingqun Qi, Rongyu Zhang, Anthony Chen, Chi-min Chan, Wei Xue, Qifeng Liu, Shanghang Zhang, al. Eva: An embodied world model for future video anticipation. arXiv preprini arXiv:2410. 24ti::2410.5561,,, 2024.
Andy Clark. Being there: Putting brain, body, and world together again. MIT press, 1998.
Andy Clark. Language, embodiment, and the cognitive niche. Trends cognitive sciences, 10(8): 370-374, 2006.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference computer vision (ECCV). pp. 720-736, 2018
Ronghao Dang, Yuqian Yuan, Wenqi Zhang, Yifei Xin, Boqiang Zhang, Long Li, Liuyi Wang. Qinyang Zeng, Xin Li, and Lidong Bing. Ecbench: Can multi modal foundation models understand the egocentric world? a holistic embodied cognition benchmark. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24593-24602, 2025.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 10 2018
Knut De Swert. Calculating inter-coder reliability in media content analysis using krippendorff's alpha. Center for Politics and Communication, 15(1-15):3, 2012.
Google DeepMind. Gemini 2.5 pro model card. |https:/ wwwpaageegoogeaaggg www.gollgii.....oooo com/ /gm/gniiiiiiii. ii 1 2.11-225ppp... 50-ppo.pppp. June 2025. Updated: June 27, 2025;
I hww.rrdd/ddooudeeeeeeeo /ttttt///mmmmi/
released to General Availability on June 2025.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie. Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 14683, 2025.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.033: 2023.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model- deep reinforcement learning for vision-bar based robotic control. arXiv preprint 18/ 00568. 2018.
12

---

Published as a conference paper at ICLR 2026
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar, Minedojo: Building open-ender embodied agents with intemet-scale knowledge. Advances in Neural Information Processing Systems, 35: 18343-18362, 2022.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE international conference on robotics and automation (ICRA), pp-
IEEE, 2017.
Andrea Frick and Wenke Möhring. A matter of balance: Motor control is related to children's spatial and proportional reasoning skills. Frontiers in Psychology, 6:2049, 2016.
Qiyue Gao, Xinyu Pi, Kevin Liu, Junrong Chen, Ruolan Yang. Xinqi Huang, Xinyu Fang, Lu Sun, Gautham Kishore, Bo Ai, et al. Do vision-language models have internal world models? towards an atomic evaluation. arXiv preprint arXiv:2506.21870 2025.
Gioele Gavazzi, Ambra Bisio, and Thierry Pozzo. Time perception of visual motion is tuned by the motor representation of human actions. Scientific reports, 3(1): 1668, 2013.
James J Gibson. The ecological approach to visual perception: classic edition. Psychology press, 2014.
Rohit Girdhar, Deva Ramanan, and Abhinav Gupta. Cater: A diagnostic dataset for compositional actions and temporal reasoning. In International Conference on Learning Representations, 2020.
Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 18995-19012, 2022.
David Ha and Jürgen Schmidhuber. World models. arXiv preprint 1803. 2(3), 2018.
Danijar Hafner, Timothy Lillicrap, lan Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555-2565. PMLR, 2019.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301. 040104 2023.
Andrew Hayes and Klaus Krippendorff. Answering the call for a standard reliability measure for coding data. Communication methods and measures, 1(1):77-89, 2007.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv e-prints, pp. trXiv-2501) 2025.
Siyuan Huang, Zhengkai Jiang, Hao Dong, Yu Qiao, Peng Gao, and Hongsheng Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model. arXiv preprint arXiv:2305.1 1 1176,, 2023a.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference machine learning, pp. 9118-9147 9147. PMLR, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.0566 2022b.
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint M.Xi:22007.05973. 2023b.
Wenlong Huang, Chen Wang, Yunzhu Li. Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal rea- soning of relational keypoint constraints for robotic manipulation. arXiv preprint XxXiv22099006552 2024.
13

---

Published as a conference paper at ICLR 2026
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint wrXiv:2205.0999) 2022.
Hanxiao Jiang, Binghao Huang, Ruihai Wu, Zhuoran Li, Shubham Garg. Hooshang Nayyeri, Shen- long Wang, and Yunzhu Li. Roboexp: Action-conditioned scene graph via interactive exploration for robotic manipulation. arXiv preprint arXiv:2402.15487 2024.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-2910, 2017.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99134, 1998.
James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques, pp. 143-150, 1986.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-actoon model. arXiv preprini arXiv:2406.09246 2024.
Klaus Krippendorff. Content analysis: An introduction to its methodology. (No Tile), 1999.
Klaus Krippendorff. Computing krippendorff's alpha-reliability. 2011.
George Lakoff and Mark Johnson. Metaphors we live by. University of Chicago press, 2008.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang. Gabrael Levine, Wensi Ai, Benjamin Martinez, Hang Yin, Michael Lingelbach, Minjune Hwang, Ayano Hiranaka, Sujay Garlanka, Arman Aydin, Sharon Lee, Jiankai Sun, Mona Anvari, Manasi Sharma, Dhruva Bansal, Samuel Hunter, Young Kim, Alan Lou, Caleb R Matthews, Ivan Vella-Renteria, Jerry Huayang Tang, Claire Tang. Fei Xia, Yunzhu Li, Silvio Savarese, Hyowon Gweon, C. Karen Liu, Jiajun Wu, and Li Fei-Fei. Behavior- 1k: A human- centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint arXivv2403.0922 2024a.
Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Erran Li Li, Ruohan Zhang, et al. Embodied agent interface: Benchmarking llms for embodied decision making. Advances in Neural Information Processing Systems, 37:100428- 100534, 2024b.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman. Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. arXiv preprint 20X.0:2209.07753. 2022a.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint (XXx:222/1099111 10, 2022b.
Shiyin Lu, Yang Li, Yu Xia, Yuwei Hu, Shanshan Zhao, Yanqing Ma, Zhichao Wei, Yinglun Li, Lunhao Duan, Jianshan Zhao, al. Ovis2. 5 technical report. arXiv preprint 1177 2025.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
Yecheng Jason Ma, Joey Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, et al. Vision language models are in-context value learners. In The Thirteenth International Conference Learning Representations, 2024.
14

---

Published as a conference paper at ICLR 2026
Oier Mees, Lukas Hermann, Erick Rosete-Best and Wolfram Burgard. Calvin: A benchmark for Ianguage-conditoons policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters. 7(3):7327-7334 2022.
MetaAl Llama 4 model card, April 2025.
URL
1lama- 1000-44mui--- www.iiimmmmliteeelleleeeee
NVIDIA. Nvidia rtx ray tracing, 2021. URL httpaattss///a...pe............. Ac- cessed: 2025-09-17.
NVIDIA. Isaac 2025. 5.0.0, Apache-2.0 License.
htt.tt/tt//ggghhh.c..o///aaaaaaa Version
Version
OpenAl. Gpt-5 system card. gust 2025. Accessed: 2025-09-16.
Au-
OpenAl. Introducing our latest image generation model in the api. OpenAl Blog, 2025. URLhttps //openai www.///nww/////coocooo/cc/eeeeee hww.://aagggegrrroooo------eee and API release.
"gpt-image-I" I" model capabilities
J Kevin O' O'regan and Alva Noë. A sensorimotor account of vision and visual consciousness. Behav- ioral and brain sciences, 24(5):939-973, 2001.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pp. 2017-2025, 2022.
Marietta Pupadatou- Pastou, Eleni Ntolka, Judith Schmitz, Maryanne Martin, Marcus Munafö, Sebastian Ocklenburg, and Silvia Paracchini. Human handedness: A meta-analysis. Psychological bulletin, 146(6):41,, 2020.
Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, al. Worldsimbench: Towards video generation models as world simulators. arXiv preprint arXiv:2410. 24f..287... 2024.
Yifu Qiu, Yftah Ziser, Anna Korhonen, Shay Cohen, and Edoardo M Ponti. Bootstrapping world models from dynamics models in multimodal foundation models. arXiv preprint arXiv:2506.06000 2025.
Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, and Vladlen Koltun. Does spatial cognition emerge in frontier models? arXiv preprint arXiv:2410.066 06468, 2024.
Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arold, Sharath Maddineni, Nikhil Joshi, et al. Robovqa: Multimodal long-horizon reasoning for robotics. In 2024 IEEE International Conference Robotics and Automation (ICRA), pp. 645-652. IEEE, 2024.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10740-10749, 2020.
Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-22 2005.
Richard S Sutton, Doina Precup. and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, (12(1-2):181-1111 1999.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
15

---

Published as a conference paper at ICLR 2026
Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12211. 2024.
Qwen Team. Qvq: To see the world with wisdom, December 2024. URL Sttps://cwen.i github. http://///oooggg//----p------
Evan Thompson. Sensorimotor surperittiisssiiiiiiiiii and the enactive approach to experience. Phenomenology and the cognitive sciences, 4(4):407-427, 2005.
Stephen Tian, Chelsea Finn, and Jiajun Wu. A control-centric benchmark for video prediction. arXiv preprint arXiv:2304.13725, 2023.
Hsiao- Yu Tung, Mingyu Ding, Zhenfang Chen, Daniel Bear, Chuang Gan, Josh Tenenbaum, Dan Yamins, Judith Fan, and Kevin Smith. Physion++: Evaluating physical scene understanding that requires online inference different physical properties. Advances in Neural Information Processing Systems, 36:67048-67068, 2023.
Francisco J Varela, Evan Thompson, and Eleanor Rosch. The embodied mind, revised edition: Cognitive science and human experience. MIT press, 2017.
Chi Wan, Kangrui Wang, Yuan Si, Pingyue Zhang, Huang Huang, and Manling Li. Worldagen: Unified state-action prediction with test-time world model training. In NeurlPS 2025 Workshop on Bridging Language, Agent, and World Models for Reasoning and Planning, 2025. URL https:/ /ttp:///epeeeveeeeeeeeeee net fww.m?id=egoooooooo.ooo
Kangrui Wang, Pingyue Zhang, Zihan Wang, Yaning Gao, Linjie Li, Qineng Wang, Hanyang Chen, Chi Wan, Yiping Lu, Zhengyuan Yang, et al. Vagen: Reinforeing world model reasoning for multi-turn vlm agents. arXiv preprint arXiv:2510. 169907 2025a.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265, 2025b.
Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, al. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073, 2025c.
Brad Wyble, Howard Bowman, and Mark Nieuwenstein. The attentional blink provides episodic distinctiveness: sparing at cost. Journal of experimental psychology: Human perception and performance, 35)7)787, 2009.
Eric Xing, Mingkai Deng, Jinyu Hou, and Zhiting Hu. Critiques of world models. arXiv preprint arXiv:2507.05 2025.
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang. Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al. Embodiedhench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. arXiv preprint ar 250.00500009500 2025.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint a4Xiv:2408. 01000, 2024.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv: 2019.
Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, et al. Spatial mental modeling from limited views. arXiv preprint McXiv:2506.21455 2025.
16

---

Published as a conference paper at ICLR 2026
Hu Yue, Siyuan Huang, Yue Liao, Shengcong Chen, Pengfei Zhou, Liliang Chen, Maoqing Yao, and Guanghui Ren. Ewmbench: Evaluating scene, motion, and semantic quality in embodied world models. arXiv preprint arXhvv2505.09694 2025.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pp. 2165-2183, PMLR, 2023.
17

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
ENACT: Egocentric Interactive Embodied Cognition Test
19
A
Notations
19
A.2
Key- Koa-Frame Trajectories Synthesis for Scalable Data Generation
19
A.3
Dataset Statistics and Evaluation Design
22
B
Experiments and Analysis
24
B.
Human Annotation
24
B.2
World Modeling as Proxy for Evaluating Embodied Cognition
29
B.3
Additional Ablation Experiments Common Sctup
34
B.
Sensitivity to Image Realism
36
B.S
Sensitivity to Camera Configurations
39
Do VLMs Have Emhodied Biases?
40
Error Analysis
40
Methodology for Error Calculations
40
2
Structural Error Analysis
45
Semantic Eror Analysis
45
D Additional Related Work
55
E
The Use Large Language Models
F
Limitations
56
18

---

Published as a conference paper at ICLR 2026
A
ENACT: EGOCENTRIC INTERACTIVE EMBODIED COGNITION TEST
A.1 NOTATIONS
We list all the notations used across the entire paper in the following two tables.
Notation
Short description
Notation
Short description
T
# frames in replay
W H,
image height and width
[M]
index set {1, ..,,, ,....,,... M)
M
# segmented key frames
K
segmented timestamps [4 < ... < lm)
(oL,
RGB & scene graph at timestamp 4 scene graph time
On
RGB image time L space of scene graphs # sampled trajectories successors of node i
8t
G
L
target trajectory length (steps)
R
adjaceney matrix on [M] (DAG) # edges in the DAG
Adj())
[b]
865)
scene-ge geen-ggraph difference (see long)
Vis(-)
visibility predicate (see long) action space
Ava(s))
visible-change change extractor (see long)
A
ai
local action Si+1
ai++
action from to state sequence along of sampled trajectories end-node weight DP s] weighted discrete distribution datasets
key- sey-frame trajectory (see long) action sequence along # paths of length & ending at predecessor set in backtracking Iverson bracket (true=1, false=0) component in signature sig The entity involved in component
St
A-
II
W;
P
Categorical(u))
1(}
7, transition
The operation key in component
The predicate in component
Notation
Longer description
T= ((o,, 8t)
Raw replay trajectory with RGB observations E Hxxxxx x and scene graphs G.
85)
difference operator over scene graphs summarizing semantic changes (ob- jects, relations, attributes) between frames i and j.
Vis(ō(s;,
Prodicate returning iff the semantic difference is visually verifiable; induces an edge i when < and the predicate Action-level representation extracted from o(;;,;;); may be atomic or com- posite and can be empty when no visible semantic change exists.
(frame skipping allowed).
Avia 85) E AU(())
J (i.....,áz)
Key-frame trajectory: strictly increasing indices with valid edges E for all & 1...., L 1. Sequences induced by (a.g-9121... ai 0::-1-9ig)>) Base: DP = 1. Recurrence: DP = - DPPe - 1, Eys for 2.......
8x, A.
S,
(81)...,,
An
DP[C, recurrence
M, = ((si,), fai P)
Deterministic finite-horizon fragment induced by with transition
P(sie (ig- 88g11'
sig 0i
Signature corresponding to an action a,, transformed from natural language to predicate-based structural format.
Table 5: Notation used throughout the paper.
A.2 KEY-FRAME TRAJECTORIES SYNTHESIS SCALABLE DATA GENERATION
A.2.1 SEGMENTED FRAMES WITH ABSTRACT STATE CHANGES
We provide examples of a scene graph and our scene graph differences for two adjacent segmented frames, shown in Figure 5 and
19

---

Published as a conference paper at ICLR 2026
Our frame selection process is iterative. For previously selected key-frame time f search for the earliest subsequent frame ty that satisfies a set of criteria designed to ensure semantic significance and visual clarity.
First, to handle discrepancies where the rule-based simulator updates the scene graph before a change is visually apparent (e.g., registering object "OnTop' upon initial contact), introduce temporal stability filter: a state change only considered a candidate the resulting new state persists for at least 40 frames. At our simulator's 30Hz rate, this corresponds R I.33,, which is consistent with cognitive science findings that humans update attentional sub-events on the order of ~I (Wyble al. 2009; Gavazzi al. 2013). and empirically yields reliable keyframe segmentation for our home-scale manipulation tasks. This value is tunable hyperparameter rather than a hard constraint. be adjusted for other environments or model classes within the same automated ENACT pipeline.
Second, to prevent the recording of minor, oscillatory state changes, such as those that might oceur from vibrations when robot carries object (e.g., plate with pizza), we employ filtering algorithm to suppress these small fluctuations in the scene graph.
Finally, to ensure that each selected key-frame represents sufficiently distinct change from the previous one, we implement similarity check. We convert the scene graph difference between the last selected frame 1 and candidate frame fę into a one-hot vector, which serves unique signature for that state change. We then compute the cosine similarity between the signature of the change at ta and the signature of the previously accepted change at f4-1. We aim to find balance between maximizing the number of segmented frames and ensuring each frame depicts a clearly visible state change. Through empirical evaluation, we determined cosine similarity threshold of 0.97. A candidate frame la is accepted only if its change signature's 's similarity to the previous one is below this threshold. This method effectively filters out duer-dupl frames while retaining rich. sequential set of key-frames that clearly chronicle the task's progression.
Scene Graph Example
'nodes':
['name' 'robot_r1', 'category': 'agent', 'states': []],
['name" "plate_94"" 'categony": 'plate', 'states': [[],
''name": 'plate_93', 'categozy"s 'plate', 'stales' []],
1' "naame "bowl_92', 'category': "boxl', 'states': []],
[' "name' 'bowl_91", 'category': "bexl' 'states': []],
I'name' 'pizza_90', 'categozy': 'pizzs', 'states'; []],
I'name' 'pizza_83', 'cstegory's 'pizzs', 'states'; [1],
I'name' "floors_<qjivm__"" 'category': 'floors', 'atatea':
I'name'
"category'; " 'breaxtast_table 'states'; Uh
(' 'naame "Esidge_petcxe_0", 'category'; 'fridge', 'stazes': ['Open'1),
I' I'name "drop_in__snk_Ikxiiq... "category'; 'drop_in_siik'', 'szates': [1),
[' 'name' sstraighs_chaii_uooti
'estegory'; 'straight_chair" 'states': []),
["namer tboctom_cabinetrrhhaee
reategory'; botttomcabiieet', "states's []]
1,
"Rdges
['Coon' 'cobot_rl', 1tol: 'giate_93', 'stales'; ['RightGraaping"
('Erom! : 'plate_94', 'so': 'pizza_30', 'states'; ['Under']],
['from' : 'plate_94', 'zo": "breakffstt_table__ttta xitckkkxttkk_0 0', 'stazes': ['OnTop' ]],
['from': 'boul_92', "tof: "breakfaat_table__rff bttbbleex_ttii__"______ 'states': ['ortopl }),
["Iron': "bowl_91', "tofs "DreakCast_table_kka Ltb "states'; ['OnTop'] }},
['from' 'pizza_90', "to': "p'att_94', 'states': ['CaTop']],
['Eron' "piaza_89r, "tol: "pLate_93', 'states'; ['CnTop']),
(' ('con' "breakfast_table_ee
'tef: 'plate_94', 'states'; ('Under']) ]
(' "Erom"
"te': "floora_agjivee 'scates': ['Oniopf]]] 1)),
If ''rom'' ottraigh:_choiis_woffe "tal: "floora_aqjkum_"" 'states' : ["Cntopp
Figure 5: A scene graph representation detailing the entities (nodes) and their semantic physical connections (edges) within the BEHAVIOR (Li et al. 2024a) environment.
20

---

Published as a conference paper at ICLR 2026
Scene Graph Difference Example
:2442":
"type": rdirE', 'add'; 4
"nodes': [],
'edges':
1 ffromm: "robot_ri", 'tof: 'plate_93', 'states'; ["RightSrespinng)
remove';
"odds": [],
'edges':
1f ('from's 'plate_93', (tef: "pizza_89', 'states'; ('Under']
[ffrrm 'plate_93', 'tof 'breakfast_table xftakii "states"; ('Ontop"1)
Figure 6: An example of a scene graph difference, representing a state change by specifying the added and removed edges between objects.
Algorithm 1: KFTS: Key-Frame Trajectory Sampling
Input: Segmented frames {(;;, s;) step length L 2 2, samples Rt predicate Vis
Output: Set of key-frame trajectories II
Build DAG: for 1 < < < M do
L Eg [Vis(8 s,))1
DP counting: initialize DP[1, i] <- 1; for l L do
for 1..M do
L DP(C < Ejei DP(e - 1, Eji
Weights: w; + DP[L, :|: if E. w, then
L return 0
Weighted backtracking sampling: + 0; sample end-nodes of Categorical(w)) for r 1..R do
π -- 1-(1 cur + 7
for l - L..2 do
P + { < cur Ej,eur = 1 A 1 >
if P 0 then
break
sample j P with prob DP[ 1, 5]; prepend jr to cur +- j*
if π = L then
L add to II
return II
A.2.2 KEY-FRAME TRAIECTORIES SYNTHESIS
Given the set of M segmented frames from the previous stage, the goal of Key-Frame Trajectory Synthesis (KFTS, see Algorithm is to efficiently sample large number of valid trajectories of fixed length L. trajectory defined as a sequence of indices π (i1....,4r. such that 1 1 i < i2 < < iL < M. The key constraint is that for any two consecutive frames is and 4++1 in the trajectory, the state change between them must be semantically meaningful and visually verifiable. The KFTS algorithm, detailed in Algorithm[1 I accomplishes this efficiently by converting the problem into path sampling on a Directed Acyclic Graph (DAG) and using dynamic programming. The process consists of three main stages:
1. Directed Acyelic Graph (DAG) Construction: We first model the relationships between all segmented frames. The M frames are treated nodes in graph. A directed edge exists from
21

---

Published as a conference paper at ICLR 2026
frame i to frame (where i < j) if and only if the state difference ((s,, constitutes a valid, visible transition. This validity is determined by a predicate Vis(-), which checks if the objects involved in the state change are clearly visible in both frames, as described in Section 2.2 This process results in an adjacency matrix E for a DAG, where Fag = indicates a valid one-step transition from frame i
2. Dynamic Programming Path Counting: Instead of enumerating all possible (1) combinations, we use dynamic programming (DP) to efficiently count the number of valid trajectories. We build DP table where DP i| stores the total number of valid trajectories of length e that terminate at frame i. The base case is DP i] = 1 for all frames i, as any single frame is a valid path of length one. The table is filled using the recurrence:
DP = -E
1. Egs
jei
This equation sums the number of valid paths of length & - 1 ending at any valid predecessor of frame i. After filling the table up to length L, the entry D DP i| gives the exact number of distinct, valid, length-/. trajectories that end at frame i.
3. Weighted Backtracking Sampling: With the DP table computed, we can sample trajectories efficiently without bias. To generate one trajectory, we first sample an end-node ig from all possible frames {1, ...... M). The sampling is weighted, with the probability of selecting frame being proportional to its weight more trajectories are more likely to chosen as endpoints.
DP[L, i]. This ensures that frames that can be part of
Once the end-node i, is selected, we reconstruct the path backwards. To select the previous node i consider all valid predecessors j of in (i.e., all < ig where Ej,2 = 1). We sample the predecessor j" with a probability proportional to DP - 1, This process is repeated iteratively: to find node ig. we sample from the predecessors of f with probabilities proportional to the values in the DPlk, row. This weighted backtracking ensures that every valid trajectory of length I. has chance of being sampled, and the likelihood of sampling any specific path is uniform across all valid paths. We repeat this procedure R times to generate the desired number of trajectories.
This DP-based approach is highly scalable its complexity is polynomial in M and L,, making it far more efficient than a brute-force combinatorial search, especially when M is large.
Motivation of Scene Graph State-Action Spaces. We opt for symbolic scene graph representation of state and action, and while it may not capture the fine-grained details of low-level dynamics, this abstraction advantageous for our objectives for two primary reasons. First, our focus is on detecting semantic changes within a scene, a task that naturally aligns with the semantic abstraction of VLMs rather than the continuous motor trajectories. Second, the symbolic predicates we employ are grounded in practical robotics applications. This is demonstrated through our egocentric real- world experiments, confirming the real-world relevance of our chosen states. The feasibility of this symbolic approach is further substantiated by its use in guiding the data collection for the BEHAVIOR benchmark, where these predicates defined the goal conditions for simulated activities and enabled human annotators clearly verify whether task states were successfully achieved.
A.3 DATASET STATISTICS AND EVALUATION DESIGN
Table 6: The 11 predicate classes used to define abstract state changes in our benchmark.
Predicate Classes
RightGrasping Inside Covered
LeftGrasping Under Open
OnTop Contains ToggledOn
Cooked
Transition
A.3.1 DATASET STATISTICS
To ensure a comprehensive evaluation of models' reasoning capabilities across different time horizons, the sampled problems feature trajectory lengths varying from to 10 steps. As illustrated in Figure]
22

---

Published as a conference paper at ICLR 2026
Forward World Modeling
Inverse World Modeling
10 Stopst 11.19
Steppc 12.8%
10 Stops: 11.1%
Stepac 12.8%
Stepe: 12.2%
8 Stepes 12.2%
Stupac 12.88 12.85
Stupac 12.85 12.8%
Steps:
Stepa:
Stupar 12.0%
Stupst 12.0%
Stepp: 12.8%
Stepps
Steps:
12.8%
Steps: 12.8%
12.89
Figure 7: The distribution of problems by the number of steps in our ENACT benchmark dataset is shown for both forward (left) and inverse (right) world modeling tasks. The dataset is balanced, with nearly uniform distribution of problems ranging from 3 to 10 steps.
this dataset intentionally balanced, featuring near-unin unir-un distribution of problems for each step length across both task types. This balance ensures that our evaluation is not biased towards shorter or longer-term reasoning.
The abstract state changes that define the actions in our benchmark are grounded in a set of 11 symbolic predicates. These predicates describe relationships between the agent and objects, as well as changes in object states. The complete list of predicates is detailed in Table
A.3.2 EVALLATION DESIGN
From indices to dynamics. We grade what changes, not just which index. Each adjacent state pair yields action signafure adi (s;-1; semantics (operation 7 on entity and predicate p). For the reference sequence, we compute (i) the visible subset C; and (ii) the full set F,. For prediction, we compute C; (full diff). This uses state differences the model's proxy answer and avoids brittle numeric matching.
fc (7, e, p)). turning scene-graph deltas into compact
Online verifier. Forward dynamics. After reconstructing the shuffled storyboard, we compare the ground-truth index sequence and the prediction Exact acceptance: Semantic acceptance (when lengths match): for all steps i,
C S Č,
Intuition: the predicted step must cover the reference's visible change. The overall decision is mat (exact OR semanticy, length-mismatched predictions are not accepted (but still get pairwise credit below).
Inverse dynamics. The model orders actions. Exact acceptance: indices match. Semantic acceptance (equal length): for all
Č C F..
ie., the predicted action description can concise subset the full reference transition at that position. Again, match (exact OR semantic).
Metrics. Task accuracy (TA). Score 1 iff the verifier accepts the full prediction, else 0; average the split:
23

---

Published as a conference paper at ICLR 2026
TA
Σ 1 (accepted(2)}
P
RED
Pairwise accuracy (PA). Measures stepwise consistency. If lengths match,
Σ 1[ S C2 (forward) or C₁ C Fi (inverse) }. PA(x) = L
1=1
Accepted predictions have (A 1. If lengths differ, we compute PA via monotone align- ment between reference and predicted steps that maximizes the number of subset-satisfying pairs (forward/inverse rule above). We report the micro-average.
Ex #correct pairs
PA
Σx L
Summary. Multiple valid answers are allowed via the subset rules: forward requires reference-visible S predicted, inverse requires predicted reference-full. TA captures all-or- nothing acceptance; PA gives graded credit for near-correct dynamics.
A.3.3 ENACT EXAMPLES
Forward World Modeling 3 Steps
Inverse World Modeling 3 Steps
## Actions Order
## Shuffled Actions
[Action 1] top cabinet transitions open. The robot r1 stopped being using the right gripper to grasp the top cabinet.
[Action 1]
turned on. The
washer stopped being open.
The teddy bear changes to be inside the washer. The robat r1 no longer using the left gripper grasp the teddy bear.
[Action 2]
[Action 2] The top cabinet is no longer open.
##l Ground Truth Order for Shuffled Next States
Ground Truth Order for Shuffled Actions
(2. 1]
[2, 1)
Figure 8: 3-Step Forward World Modeling (left) and Inverse World Modeling (right) samples.
B EXPERIMENTS AND ANALYSIS
B.1 HUMAN ANNOTATION
B. ANNOTATION INTERFACE & HUMAN PERFORMANCE EVALUATION
To establish an empirical upper bound on performance for the ENACT benchmark, we recruited three trained annotators to complete the same set of tasks assigned to the Vision-Language Models (VLMs). The anmotators interacted with our customized human annotation interfaces implemented in Gradio, which are illustrated in Figure II for the Forward World Modeling tasks and in Figure 12 for the Inverse World Modeling tasks. Importantly, annotators followed exactly the same instructions and task prompts those provided to the VLMs,, ensuring fair and consistent comparison between
24

---

Published as a conference paper at ICLR 2026
Forward World Modeling 6 Steps
Inverse World Modeling 6 Steps
Far GONTH
Mucii Stades
Saccsmatt 8
State
State de
Meat Stat
Jua 30min
Saut Riota 3
## Actions in Order
## Shuffled Actions
[Action 1] The fridge becomes
[Action The robot 11 becomes using the right gripper grosp the head cabbage.
[Action 2] The cutting board now becomes covered by the diced steak. The steak now becomes the diced steak.
[Action The robot becomes using the left The robot
gripper grosp the frying stopped being using the left gripper to grasp the The frying pan stopped being on top touching the countertop.
drop-in containing the diced pineapple. The robot r1 changes to be using the right gripper to grasp the carving knife. The cutting board changes to be covered by the diced pineapple. The carving knife longer on
[Action
changes
[Action The robot transitions right gripper to grasp the carving knife. The robot stopped being using the right gripper grasp the cutting board. The carving knife is no longer top and touching countertop.
using the
and touching the countertop. The pineapple now becomes the diced pineapple.
[Action 4] The carving knife now becomes on top and touching the countertop. The robot rl no longer using the right gripper to grasp the carving knife.
[Action The fridge changes be open. The robot longer using the right gripper grasp the fridge.
[Action 5) The frying now becomes top of and touching the burner. The frying pan changes to be containing the diced head cabbage. The frying pan becomes covered by the diced head cabbage. The robot gripper to grasp the frying pan.
[Action 6] The robot transitions to be using the right gripper grasp the bowl. The drop-in sink stopped being containing the diced pineapple. The cutting board stopped being pineapple. The bowl stopped being inside the drop-in sink.
by the diced
longer using the left
## Ground Truth Order for Shuffled Next States
#t Ground Truth Order for Shuffled Actions
4. 1,
[4.
Figure 9: 6-Step Forward World Modeling (left) and Inverse World Modeling (right) samples.
25

---

Published as a conference paper at ICLR 2026
Forward World Modeling 9 Steps
Inverse World Modeling 9 Steps
Stari
Stnil
Next Statt
Stale
Stare
Next Statio
Next Staten
Stards
draba Poeds
## Actions in Order
## Shuffled Actions
[Action 1] The cutting board becomes covered the diced steak.
The robot r1 becomes using right gripper to
[Action 1]
the beefsteak tomato.
[Action 2) The carving knife becomes of and touching the countertop. The robot r1 is longer using right gripper grasp carving knife.
[Action 2] The robot r1 now becomes using the left gripper grasp beefsteak tomato. The beefsteak tomato is longer inside the paper bag.
[Action 3] The bowl transitions and touching the drap-in sink.
[Action 3] The fridge longer open.
[Action 4] The paper bag changes to be on top and touching breakfast table. The robot longer using the gripper grasp the paper bag.
[Action 4] The robot #1 changes to be using the right gripper to grasp the carving knife. The carving knife stopped being top touching the countertop.
[Action 5) The cutting board longer under the pineapple. The pineapple now becomes the pineapple.
[Action 5] The paper bag longer top and touching the breakfast table.
[Action 6] The robot r1 becomes using the right gripper grasp the paper bag.
[Action 6) The robot r1 transitions to be using the right gripper grasp bowl. cutting board transitions pineapple. The robot right gripper grasp the carving knife.
be covered by the diced longer using the
The robot r1 is no longer using the right gripper to grasp the paper bag.
[Action 7]
[Action 8] The fridge changes be open. The robot r1 changes to be using the right gripper to grasp the paper bag. The beefsteak tomato becomes inside fridge. longer using the right gripper grasp the beefsteak tomato. The robot r1 is no longer using the gripper to grasp the beefsteak
[Action 7] bowl transitions
drop-in sink. The robot r1 stopped being using the right gripper to grasp the bowl. The bowl stopped being top countertap.
touching the
[Action 8] The gripper to grasp the half pineapple. The bowl now becomes containing cutting board stopped being covered by the diced pineapple.
r1 becomes using left
pineapole. The
Ground Truth Order for Shuffled Next States
#M Ground Truth Order for Shuffled Actions [1, 7.
Figure 10: 9-Step Forward World Modeling (left) and Inverse World Modeling (right) samples.
26

---

Published as a conference paper at ICLR 2026
knear, Fi sec.
Old (aaaacd AOL wwe.nnieg
Statt Aesume
Pogeos
Your Task
will proided othh single Cament irape
sure imugss dbowed 1. doar minehel somd side muat filou anqua adiens
Curmen Staee
actontion the fectora aetei totte.
teulthe Poture Ireegriiu makhre the oxterne attio
Beceroaedd mmage mum ng sutts
adtand
beenedded
Actions Order
wwwkeegninng 1ith grpper to proptto rogroa
open (er thiheegn
in teghte phypint
Srop trutattnran on kop bouling the bank (Aatir temggn shupppd bet jacter stopot so0..
Action The Themetn
Actioe
Image Placement Controls
Next State 1
Thotlnge ronnationn
Poatoor111
Add Pou
Bemonee from Poattonn
Sabmit:
The annotation interface used for evaluating human performance on Forward World Modeling problems. Annotators are presented with a "Current State" image (top left) and ordered list of textual actions. The main task is to fill the "Next State" slots by selecting the correct image from the shuffled Candidate Image Library on the right. The annotator must follow the sequence of actions, using the result of the previous action as the starting point for the next, to determine the correct chronological order of all future states.
Figure 11:
27

---

Published as a conference paper at ICLR 2026
Lagis theasea bugis
Coffee Unforelhllosstee
Progoo
Sample
chepping wesd 27506/40005512
ordoing h723560
Sho Sidcbar
Your Task
tmaggo
Anniya manaitor from the rustiagg secoeo Oveemine preiffc vau change that coursec.
procus a indauqarr aa gas paoced thid, bounh, sto) und you tuee carvate untend action.
Shuffled Actions
hectmmm mouch othvaves chopping steck to toranger inder stoped truh ing chep inprew becsm (estin
cre gightrsson ruction
halleg juston choping becomes under Theret aopelaan touch othoway Juston agthelee be
Gation Flentrdder
Sebinn Pleccheterr
Nac aucars
Action Placement Controls
Landiory Actiant le
Atlan broc
dripp brodd Iha biconiu
B aiupet gari
Actron Flechhlder
Arlen The
Acten ser
thinghhe git
Targer
Add Position
Paition
Poguoe
Lamene Pa
Sak Rerooraa
The annotation interface used for evaluating human performance on Inverse World Modeling problems. Annotators are shown an ordered sequence of state transitions, displayed as pairs of "Current State" and "Next State" images. For each transition, their task is to select the correct action description from shuffled Candidate Action Library that caused the visual change between the two states.
Figure 12:
28

---

Published as a conference paper at ICLR 2026
human and model performance. This setup allows to quantify the extent to which current VLMs approach human-level competence on the ENACT benchmark.
B.1.2
AGREEMENT ANALYSIS
To ensure the reliability of our human-generated labels, we conducted rigorous Inter-Annotator Agreement (IAA) analysis. The initial dataset was annotated by three annotators working on disjoint, non-overlapping subsets, which precluded direct agreement measurement. We therefore implemented systematic cross-annotation protocol. For each task type (forward and inverse) and for each of the eight step-length categories (from 3 to 10 steps), we randomly sampled five questions from each annotator's 's original assignment. This process created balanced IAA evaluation set totaling 240 unique questions. Each of these sampled questions was then reassigned to the two annotators who had not performed the original annotation. For example, the five items sampled from Annotator work for a given category were re-annotated independently by Annotator B and Annotator C.
Following this protocol, we assessed the resulting annotations using Krippendorff's Alpha (a) (Krip- pendorfi 2011), robust statistical measure that is well-suited for this analysis as it accommodates multiple annotators and is resilient to missing data. Given that our annotation task involved ordering. we configured the analysis for ordinal data. The alpha coefficient is calculated based on the observed and expected disagreement among annotators, according to the formula:
Do De
1
Here, Do represents the observed disagreement, which is calculated from the pairwise differences between all annotations for each item. De the expected disagreement, which represents the disagreement that would occur by chance, derived from the marginal distribution of all annotations. This balanced design ensured that every question the IAA set received three independent labels, allowing for robust pairwise agreement calculation across all three pairs of annotators (A vs. B, A vs. C, and B vs. C) for each condition. To assess the stability of our a coefficient, we computed 95 % confidence intervals (CI) using the bootstrap percentile method with 1,000 resamples of the 240 evaluation items.
Our analysis yielded an overall Krippendorff's s Alpha of a 0.8320, with 95% bootstrap confi- dence interval of (077879, 0.8682]. Given the established standard (Krippendorfi 1999 Hayes & Krippendorfi 2007 De Swert 2012), alpha value above 0.80 indicates high level of reliability. This strong result confirms that our annotation guidelines are clear and consistently applied by the annotators.
Pairwise agreement scores were also consistently high, further validating the reliability between individual annotators:
Annotator A01 B02: 0.8180
Annotator A01 C03: 0.8265
Annotator B02 C03: 0.8518
In addition to the chance-corrected alpha metric, we found that the annotators were in perfect agree- ment on 184 of the 240 selected questions, resulting a agreement rate of 76.67 %. Collectively, these strong agreement metrics validate the reliability of our annotation process and the high quality of the resulting dataset.
B.2 WORLD MODELING AS A PROXY FOR EVALUATING EMBODIED COGNITION
B.2.1 EXPERIMENTAL SETUP
To ensure fair and consistent comparison across all models, employed standardized evaluation protocol. For each task type (forward and inverse world modeling), unified question prompt template was used. All input images were resized uniform resolution of 512 512 pixels before being
29

---

Published as a conference paper at ICLR 2026
Task
Action-Effect Reasoning Causal Inference
Affordance Recognition Embodied Awareness Temporal Abstraction
Forward Inverse
Table 7: We posit that ENACT reflects embodied world modeling reasoning rather than simple temporal correlation. Unlike passive video prediction, our formulation explicitly conditions state transitions actions (Forward World Modeling) and infers actions from state changes (Inverse World Modeling), thereby evaluating VLMs as transition models. Based on your suggestion, we include a table that relates ENACT's evaluation tasks specifie cognitive constructs.
passed to the models. To ensure deterministic and reproducible outputs, the decoding temperature for all models was set O. Models were instructed return their answers parsable Python list representing the permutation of indices, as shown in Figure 13 and Figure [4] A comprehensive list of the specific models used in our evaluation is provided prompt template across all experiments because we follow the design choice mentioned in Liang et al. (202bb), that the models should adapt to users' input, instead of the reverse case.
Table g
We deliberately choose one
Organization
Pipeline
Proprietery
GPT-5
2105-08
2025
-2025-08-77
OpenAl API
GPI-5-0an0
2025
OpenAl API
Genini 2.5 Pro
2025 06
Sowini-2,5-pp
Gentni API
Google
Genuni 2,5 Flasth
2025 06
genini-2, 5-
Gentii A51
Genini 2.5 Flads-L.te
2025-06
gemimi www.babb-ibbttt labe-iibe
Genini AP1
Authimpic
Caadi
22005.25
Glauce--seen wonnet-------- 3-322000
Arthmupi
Open- Model
Zhipo
GL.M4 (V-hhiingg
G1M--4
G1M-6.5V
Flagg
Zhips loundation Open
Thips
Open
Meta
Liamo S-Soom 166
Limo- 4-ao--6----0000 170- 178-1240-tos
2015-04
20255
Taana/
ModelSScope
MedelSSope API
IneenVL.35 2418 A288
2025 08
aaniiaaa/Inncat.... (hnncrnnE33..41 5-2416-1288
Insem APt
Shanghui Al Lab
2025-08
2025-08
Opent//iet/zeteen/l..
Hugzing Faxe Transfomen
Hugging Fra: Tranafomens
IncnVLL35558
2025-08
Hugging Fuce Trarsfomers
2005-08
Google
googla/gemm-11-22--
grogle/gemm-3--00-
QVD Freview
2024-12
Deca/@fC-1aa-Feeviee
MedelSScpe API
Qwen! 5-V 7215-ins Ins
2025-01
Oven/Gwen2
ModeltScope
Aliaba
Qwen2.5-V1 5-V1--VV-222 22B-322-320 lnn
2025-01
Oven/
ModelScope API
Quen2.5 VL. Iao
2025-01
Qwen/gven2.2
0osu://ve22.5VV---1 Nn-VL-18-Inataaa
Hugging Fuce Tratsfumens
Quen2.5 VL. Ins
2025 01
Secn/Gven2.3-VV 10t.----------traa
Hugging Fuce Transfomens
MDD
0.5s2.3.0g
Dute2 3-28
2025-08
AIDC-AI/Ovia2...
2T00-2T/0c1e2
5-28
Hugging Transfomers
Bugaing Face Trassormens
OpenBMB
MinCPM
1005-08
opertibb/Mi
Sperbn6//ki
iugginng
Hugging Transforees
Higging
Idetici Stl Liamed
2024 06
Huggiing wwwee///eeeeeeeeaa/ea
loggaing Transfomers
Nvidia
Cosmos Ressoal
2025 05
wridia/Cosnos-
Hugging Fuce Tratsfomens
Table 8: Details of Vision Language Models (VLMs) assessed in this study.
Forward World Modeling Prompt
You. are a capeble agent designed to infer multi step forward dynamics transitions in enbodied decision-making. your goal predict the correct sequence of future states that result from applying a giver series of accions to an initial state.
#4 Your Task
be provided kxFuture State Imagesvi (labeled i, 2, 3, etc.). To deteraine their correct order, you. must follow the sequence of actions provided below.
single v*Current State imagess
shuffled
1. Start with the +*Current State Tmagest. Apply the *ffirst action.
the 'Actions in Order" 1ist this state.
3. Find the Futuure State imageve that matches the outcome of this action. is the Next, apply the sasecond actionss to the state
state
correct sequance,
jus: ideatified.
5.
sind the cerresponding image among the remaining future states. Continue this process until ail actions have been applied and all future have been ordered.
* Output Format
30

---

Published as a conference paper at ICLR 2026
Your zeeponse +*must be onlyr a Python 1ist of inLegera representing the correct chronological order of future image labels. include any text, reasoning, explanation.
**xxample:**
If you detarmins the corract sequence is 'Next State 3' 'Next 2
<Next State Your output must be: '[2, 3, 21
44 Actiona in Order CHANGE)
Now, please provide your answer in the requested format.
The prompt used evaluate VLMs on the multi-step Forward World Modeling task. The model must order shuffled future state images by reasoning over given action sequence.
Figure 13:
Inverse World Modeling Prompt
capable agent dasigned infer multi-step inverse dynamics transitions embodted ed decision on-making. Your goal is to determire the correct chrono ogfoal order sequence images.
actions that caused
Lransitions
Your
You: will ba given an ordered sequence of images that show scane evolving over time, along with a shuffled list of the actions that caused these changes.
To solve this, you muat:
Analyze transition visual change that occurred.
image second. Determine specific provided below, idantify the single action that
vxShuffled Actionsux
beat deacribes this change.
3. Sepest this precess for all subsequent pairs of images (second to third, third to fourth,
until you have correct ordered al the acti
Output Format
Your respense esmust be onlyvs a Python list of integers representing correct ordar the sction labals.
Do not include any other text, reasoning, explanationa, code formatting,
*xExmple::*
If the correct sequence is (Action 2] > [Action 3] [Acsion 11, your output
(2, 3, 11
44 Shuffled Actions CONUFLEDDACTIONEE
Now, please provide your in the requested format.
Figure 14: The prompt used to evaluate VLMs on the multi-step Inverse World Modeling task. The model must order a set of shuffled actions by reasoning over an ordered sequence of state images.
B.2.2
DETAILED RESULTS
detailed examination of the full experimental results are presented Table D (Task Accuracy) and Table 10 (Pairwise Accuracy).
B.2.3 CONTACT EXPERIMENT
To verify that our conclusions are not an artifact of using only "semantic scene graph" predicates in ENACT, we add an ablation which key frames are also gated on changes binary contact relations between objects (e.g., touch / no-touch). Concretely, we augment the symbolic predicate set so that both state changes and contact changes trigger key-frame sampling, while keeping the rest of the pipeline unchanged, and re-evaluate Vn3.5mVL3.5-244
the resulting trajectories.
As shown in Table II the qualitative trends remain the same as our main results. Inverse world modeling consistently outperforms forward modeling across all horizons (e.g., 86.67% vs. 90.00% task accuracy at 3 steps, and 3. 3.45% vs. 16.67% at steps), and both task and pairwise accuracies still drop substantially the number of interaction steps increases for both directions. This suggests
31

---

Published as a conference paper at ICLR 2026
Forward World Modeling
Inverse World Mudeling
Model
8
10
8
10
Proprictury Models
GPES5
80,59
62.72
47.13
33,62
86.19
72.65
59.65
43.75
33.08
24.04
17.15
133000
GPE-5 mini
83.39
62.72
15,22
31.71
19.02
9.12
5.29
2,80
84.79
67.42
56.09
41.11
29.67
18,07
13.50
6.60
GPT-5 nime
56.57
3.83
0.00
0.00
17.22
8.19
1.05
0.00
Gemini 2.5 Pro
81.99
62.72
47.30
29.79
17.90
10.00
3.28
3.60
87.76
23.52
58.51
43.35
33.51
23.65
15.88
14.40
Gemint Flash
75.52
30.52
25.22
14.29
6.28
2.95
1,28
0.20
82.52
61.15
36.96
27.70
17.95
13.86
6.20
3.80
Gemini Flat--Lite
27.18
3.63
1.40
0.18
42.35
8.54
0.88
0.00
Clande Sonnet 4
56.29
24.91
8.52
2,96
0.00
0.00
0.00
72.73
42.16
24.17
13.59
6.95
2.63
1.46
1.00
Open: Weighr Models
GLM-4.VV
66.05
40.77
18,09
8.54
0.35
0.11
19.55
57.53
32.52
20.58
11,40
5.44
1.64
0.40
GLMM-.1V
57.52
28.40
11.30
2.25
0.15
0.90
0.00
13:43
12:00
4.33
0.87
0.53
0.00
0.00
Lhua- 4- -Soou-- 17B- Ins
58.74
21.43
5,04
1,74
0.20
0.18
0.00
0.00
61.36
36.32
10:26
2.96
1:25
0.00
0.15
0,00
Mamaa-4Mav-178--11
63.99
32.58
14.78
4,36
1.57
0.35
0.00
0.00
71.50
491
20
24.35
11.85
4.19
1.58
0.55
0..00
67.83
43.38
12.00
1.05
0.36
59.
40.38
24.25
15.18
1:57
2.00
46.33
14.81
3,45
1.05
0.00
0.00
0.00
0.00
6644
45.12
2365
11.85
5.93
1.95
1.28
0,40
54.72
25.09
5.39
1,05
1.22
0.18
0.00
0.00
63.99
40.2
20,00
6.79
3.40
0.55
0.36
0.20
W.e.5V....5548
$4.55
22.15
6:43
2.09
0.52
0.00
0.00
0.00
63.64
32.05
16.00
5.75
2.27
0.53
0.18
0.00
Gemmm--32200-t
53.15
22.82
0.87
0.18
0.00
0.00
31 88.88
1461
505
1-57
0,35
0.60
Gemma-3 3-12----112--1
51.22
21.78
6.09
1.05
0.17
0.00
0.00
0.00
52,80
27.55
9:74
2.70
1:25
0.35
0.00
0.00
Gemma- 3-4h-11
$2.80
20.55
1.57
0.17
0.70
0.00
0.00
0.00
$2.45
18.12
1.83
1:92
0.17
0.00
0.00
0.00
QVQ 720 Preview
60.84
29.79
8.17
2.09
0.70
0.00
0.00
0.00
66.96
40.24
16.87
6.97
3.84
1.23
0.55
0.00
Qwee7 --
71.68
40.42
18.96
7.84
3.52
0.00
0.00
75.87
53.48
29.74
17:77
11.52
4.74
1.45
0.40
Qwen2.5 V2B las
51,40
32.75
10.09
3.48
0.52
0.00
0.00
0.00
39.34
33:45
19.13
8.89
6.11
2,11
0.91
0.00
Qwen 5- V-- Ins
22.73
23.17
5.39
0.52
0.17
0.00
0.00
0.00
70.10
41.11
16.52
5.23
1.05
0.00
0.00
0.00
Qwes2.5. Dee.2.5.VE.33B-B
45.98
13.76
5.91
0.70
0.17
0.00
0.30
0.00
56.64
32.75
13.39
5.75
1,05
0.15
0.00
0.30
Ovas2.5.90 9B
47.55
23,00
10.51
2,96
1,05
0.18
0.30
0.00
62.76
35.54
16,90
6.27
1.75
0.35
0.00
0.90
Dva2.5528
17.77
0.87
0.00
0.00
23.87
1.57
0.00
0.00
MiniCPM-V-4.5
48.43
19.16
1,92
0.18
0.90
68.01
37.98
22,09
9,41
3.66
1,75
0.18
0.20
MiniCPM-o--.6
26.05
17.07
5:23
1.57
0.00
0.00
0.00
0.00
38.64
27.35
11,30
2,44
0.52
0.18
0.00
0.00
Idelios Jama?
48.08
16.20
2.26
0.52
0.17
0.00
0.00
0.00
46.33
16.72
2.96
1.57
0.00
0.00
0.00
0.00
Cosmus- Reasonn
45.45
21.43
5.04
0.52
0.17
0.00
0.00
0.00
51.92
29. 00
12,02
3.31
0.52
0.18
0.00
0.00
BAGEL
25,87
17.77
3.83
2,09
0.17
0.00
0,00
0.00
36.20
35.6 89.89
14,61
7.14
2.97
0.18
0.18
0.00
Human Performance
90.38
92.16
89.74
85.71
88.31
87.02
85.58
84,00
91.78
90.34
88.70
88.15
89.53
92.28
87.73
85,00
Table 9: Evaluation on ENACT (Task Accuracy). Dark gray indicates the best result within each category (Proprietary or Open- Weight Models), and Light gray denotes the second-best result within the category.
Forward World Modeling
Inverse World Modeling
Modd
3
5
7
8
10
4
5
7
8
10
Proprietary Models
GPT5
84.62
69.96
64.18
57.48
52.16
46.93
86.28
80.37
76.09
68.79
68.21
62.13
57.12
55.33
GPT:5 mini
87,50
76.25
70.65
63.41
58.14
52.35
46.65
44.11
85.05
76.77
75.43
67.67
63.79
$7.04
55.04
50,02
GPT:5 nan
67.85
30.35
25.97
21.90
16.84
72.81
53.95
42.48
31,68
28.20
24.11
20.33
Gemini 2.5 Pro
86.10
76.43
09.85
611.00
53.26
48.12
40.12
36.9%
87.94
81.18
75.
20.00
66.05
62.991
57.78
56.62
Gemin 2.5 Flash
81.64
07.94
54.17
43.35
37.43
32.75
29.35
25.07
82.75
72.18
04..5
58.19
53.14
51.75
47.59
44.95
Gemini 2.5 Flach-l Lite
64.34
49.07
38. 70
33.87
27.81
25.
23.31
30.31
69.
57.55
46.00
30.00
34,06
30 18
27.51
23.16
Claude Sonnet
65.65
45.82
36.65
30.52
26.61
22.78
21.49
20.16
73.25
56.85
48.87
43.07
37.00
32.71
30.50
28.49
Open-Wrigho Models
GLM-4.5V
74,30
59.99
47.65
38.75
30.83
25.69
21.00
19.67
M0.59
69.28
57.04
51.53
46.95
41.08
37.36
37.95
GI TiV-iiii
67.31
49.48
38.4
31.20
25.80
31.50
20.14
18.75
75:35
56.37
46.57
36.79
29.61
24.56
25.01
25.
Lhmm--4Soooe Scout- 17B- 16E-Ins
68,
42.63
34.30
30.52
28.50
26.57
25.94
20
66.00
50.00
41.30
37.04
29.73
25.611
22.45
26.54
1.mm-4-Mav-170 17--1--
72.47
$2.00
43.87
$5.30
20 90
25.85
22.79
30. 49
72.55
60
50.
45.10
55.17
31 31.68
28.10
25.80 80
IntemVVL3.522411
75.79
62.25
50.83
27.85
24
82
22.26
60.61
39.35
34.12
30.56
JitenVL3.5-148
54.90
36.53
27.87
25.47
22.02
19.75
18.29
20.60
69
59.52
45.45
37.61
32.25
20.11
28.58
1ntem\113.558R
64.42
44.81
48
24.32
23.62
21.50
30
15.47
65.03
10
45 35.35
37.67
35.02
70
26.41
23,60
3
63.11
42.04
30.26
26. 23
21.73
20.28
19 64
21.9%
64.95
50.12
41.61
35.78
20.00
26.57
27.55
24/04
Gemmu-3-277-
63.29
68.66
32.04
25.82
11
19.50
16.74
16.29
64.95
48.37
40.04
33.87
25.55
23.63
21.74
19.36
Gemma-22 3-120-1
02.35
43.55
32.78
25.68
22.45
20.40
17.30
16.71
53.25
43.79
34,05
29550
25.57
22.51
21.00
19.16
Gemma- 1-40-in
61.98
41.17
35.70
15.16
50.51
76.
17
26.73
25.80
$3.06
36.41
20.5
26 26.38
22. 66
74.44
55.71
31.62
QVQ-720 Preview
69.14
52.96
40.83
36.27
33.16
30.63
26.30
24.76
71.33
55.77
48.43
44.36
40.26
39.30
36.66
36.58
Qwen2.5. VL-72----------
78.15
60:05
49:87
41.92
36.77
31.75
26.03
25.007
77.50
63.85
53.30
48.19
44.07
37.57
33.76
36.27
Qwen2.5 2enn2.5-V1-322--11
67.83
44.35
27.52
18.07
63.55
59.70
54.57
49.36
40.16
QwenZ VL-7-aass
26.84
43.90
32.00
23,07
19.66
16.69
11.82
11.31
70.54
56.45
42.43
32,89
25.07
19.52
16.72
17.42
Qwen2.5 VL-3B- Las
58.22
35.31
30.57
24,05
20.36
17.64
14:57
15.07
$7.43
49.13
40.45
34.88
28.33
26.14
22.97
20.51
Ovis2.5.98
58.39
42.51
34.96
31.08
24.61
20.78
18.11
16.96
64.86
$1.74
41.65
35.47
30.95
26.55
26.64
23.70
23.25
Ovis2.5.20 26
46.94
38.85
32.65
26.86
25.63
22.21
22.19
24.87
54.28
44.06
35.43
29.06
27.84
25.56
27.62
28.29
MinCCMM-V-45 V-4.5
00.75
38.73
33.65
25.47
24.81
21.40
21.56
18.35
69.25
53.08
47.35
39.55
34.87
30.63
27.05
25.71
MissCPMM---2.6
35.31
39.37
29.48
31.78
27.66
26.39
24.59
27.4
$4.11
45.26
44.70
40.00
38.28
36.12
33.23
31.71
Mectcs5-4B-Laamm3
08.25
36.99
31.85
24.25
21.29
20.80
46
17.71
47.58
33.86
27.26
25.48
19.87
18.50
17.04
15.16
Caemos- Remson!
56.28
41.86
14.75
78.40
46
26. 49
25.41
24.8%
58. 30
45.03
44.25
38 50
35.72
34.56
51.50
28.64
BAGEL
30.24
40.19
29.65
25.37
22.75
19.45
17.54
15,87
50.73
52.85
40.09
35.44
29.67
24.39
25.70
18.91
Homan Performance
93.62
95.30
95.04
93.87
95.43
95.41
94.75
95.13
92.05
93.56
94.35
94.25
95.96
97.74
96.30
96.29
Table 10: Evaluation on ENACT (Pairwise Accuracy). Dark gray indicates the best result within
each category (Proprietary or Open- Weight Models), and Light gray denotes the second-best result within the category.
32

---

Published as a conference paper at ICLR 2026
Forward (with contact changes)
Inverse (with contact changes)
Metric
4
7
8
9
10
3
5
6
7
8
10
Task Accuracy
86.67 43,33 36.67
20.00 3.45
3.33
0.00 0.00
90.00
73.33 30,00 26.67
16.67
3.33
6.67
0.00
Pairwise Accuracy 90.00 72.22 60.00 53.33
48.28
42.38
31.67
34.44
90.00 82.22
55.83
57.33
46.11
32.86
38.75
27.78
Table 11: Effect of including contact changes in key-frame selection. Task and pairwise accuracies (%) of Intern
when key frames are triggered by both state changes and contact changes.
that our findings are robust to the choice of symbolic key-frame criteria and are not driven by sparsity introduced by state-change-only sampling.
B.2.4 How DOES ACTION REPRESENTATION AFFECT VLMs' PERFORMANCE?
Forward
Inverse
Metric
4
5
7
9
10
3
4
5
7
8
10
Vinilla (Natural Language)
Task Accuracy
68.97 35.17
27.59
8.97
6.90
2.07
0.69
0.00
83.45
60.69
44, 14
24.83
13.79
7.59
4.14
0.00
Pairwise Accuracy 76.21 57.70
57.41
41.24
38.85
30.34
29.05
26.52
83.45
69.66
61.72
53.10 47.01
40.39
34.40
27.25
Symbolic Predicates
Task Accuracy
67.59
40,00
20.00
9.66
4.14
0.69
1.38
0.00
79.86
51.03 40.00
23.45 8.97
5.52
2.76
0.73
Pairwise Accuracy 74.48
61.61 48,97 41.52
35.75
32,12
30.34
25.68
79.86
62.99 58.62 51.59 39.43
36.85
32.07
24.33
Style Encodings
Task Accuracy 65.52 44.14
18.62
11.03
8.28
0.69
0.69
0.00
77.24
48. 28 35.17 23.45
12.41
7.59
2.76
0.00
Pairwise Accuracy 73.10 64.60 48.62
39.45 38.05 27.49 26.72
77.93
61.38 59.14 45.38 40.80 36.06
27.16
25.30
Table 12: Effect of action representation on Intern 2L3.5-241B. Task and pairwise accuracies (%) on a 2,304-QA subset of ENACT under three action encodings.
Our primary goal is to evaluate VLMs under the standard interface of natural-language actions, but this leaves open whether the inverse advantage is merely a consequence of language priors, i.e.,
models being better at mapping visuals to familiar verbs than to unfamiliar symbolic actions. To test this, construct a subset of ENACT trajectories with 2,304 QAs and compare three action-predicate encodings while keeping the underlying videos and questions fixed: (1) the original natural-language descriptions ("vanilla"), (2) structured symbolic predicates, and (3) emoji-style encodings. Table [2] reports task and pairwise accuracies of Intern VL3.5-241B across horizons for all three settings. Across all representations, we observe the same qualitative pattern as in our main results: Inverse world modeling consistently outperforms forward modeling at comparable horizons, and performance for both directions degrades sharply as the number of interaction steps increases. While absolute accuracies vary slightly across encodings, the inverse > forward gap is preserved even with purely symbolic or emoji-style actions, suggesting that our conclusions are not driven solely by natural- language priors.
B.2.5 HOW OFTEN DO ACCEPTED PREDICTIONS OMIT PARTS OF TRANSITIONS?
Our semantic verifier operates with subset inclusion: a predicted transition is accepted semantically correct if its predicate set is a subset of the ground-truth transition and does not contain any predicates that contradict the ground truth. In other words, allow partial correctness (omitting some true predicates), but never accept hallucinated predicates that conflict with the annotated transition. To quantify how often such omissions occur among accepted predictions, we measure mismatch ratios at two granularities: (i) a data-level mismatch ratio, computed per QA as the fraction of semantically accepted predictions whose predicate set is a strict subset of the ground truth: and (ii) a pair-level mismatch ratio, computed per ordered pair in the reordering task as the fraction of accepted pairs where at least one element a strict subset of the corresponding ground truth transition. Results for GPT-5, Intern V13.5-243. and human annotators are shown in Table At the data level, all models (and humans) exhibit very low mismatch rates 1=-),, indicating that most semantically accepted predictions recover the full transition. At the pair level, models show higher mismatch ratios than humans (around 14-15% vs. 3.6%), reflecting that they occasionally capture only a subset of the true transition when comparing two candidate steps.
Overall, this suggests that while our verifier does grant some partial credit, such cases are relatively rare the QA level and do not dominate the evaluation.
33

---

The drop in sink Published as a conference paper at ICLR 2026
Model
Data-Level Mismatch (%)
Pair-Level Mismatch (%)
GPT-5
1.65
15.31
Human
2.26
3.62
Intern VL3.5-241B
1,49
13.96
Table 13: Mismatch rates among semantically accepted predictions. Data-level and pair-level mismatch ratios (%) for cases where the semantic verifier accepts prediction but its predicate set is strict subset of the ground-truth transition.
B.3 ADDITIONAL ABLATION EXPERIMENTS COMMON SETUP
To gain deeper insights into model sensitivities, we conducted a series of controlled ablation ex- periments. This section outlines the common experimental framework that applies to our analyses of Image Realism (Section B.4) Camera Configurations (Section B.5 and Robot Appearance
(S.6..on/B.6.]).
For these experiments, we selected two representative models. Given its strong balance of perfor- mance and computational cost in our main results, we chose GPT-5 mini as our primary model to represent state ffate-of-the-ee proprietary VLMs. To include strong open-weight counterpart, also selected Intern VarrnVILS VtternLL3..52411.-- L33.--241B--2888 which demonstrated robust performance among open models.
Forward 3-Step Example- Baseline
Inverse 3-Step Example- Baseline
dext Sciee
Sinle
Next State
Cext State
## Actions in Order
## Shuffled Actions
[Action 1] The drop-in sink now becomes under the cutting board. and touching the drop-in sink. The countertap now becomes under the cutting board. The cutting board is longer top of and touching the countertop. The robat rl is na longer using the left gripper grasp the cutling board.
[Action 1] The cutting board changes to be on top touching drop-in
cutting
transitions to be under the cutting board. The countertop changes to be under the cutting board. The robot grasp the cutting board.
longer using the left gripper to
[Action 2| The robot r1 stopped being using the
[Action The carving knife transitions top of and touching the countertop. The robot #1 is no longer using the right gripper to grasp carving knife.
right gripper grasp
i# Ground Truth Order for Shuffled Next States [2, 1]
tt Ground Truth Order for Shuffled Actions [2, 1)
Figure 15: Illustrative trajectories of Forward World Modeling and Inverse World Modeling for representative baseline question.
In our summary heatmaps (Figure for GPT-5 mini and Figure [6 for Intern we use A to visualize the performance difference between variant and the baseline, To assess the statistical significance of these differences, we perform two-tailed unpaired Welch's t-test. unpaired test is appropriate each question is evaluated in an independent session. We specifically use Welch's t-test as does not assume equal variance between the two groups being compared Obaseline vs. variant). We report the p-value for each comparison and consider a result to be statistically significant if < 0.05. We qualitatively classify any performance change where |A] < 0.05 as small change. We show one baseline question and its images for both forward and inverse settings in Figure 1 and for other settings, only show their images, as they all share the same question text and answers.
34

---

Published as a conference paper at ICLR 2026
A: Image Realism
B.1: Camera FOV
Forward
p-0.62
D-0.76
Forward
p-o
A=-0.01
4=00.02
4=-0.09
0.05 (sig)
nverse
nverse
p=0.85
pm0.39 A- 0.04
p=0.76 0.01
p=0.61 0.02
p=0.44 A= 0.04
p=0.03
p 08
0.001
A=+0.01
0.10
A-
Path Tracing B.2: Camera Height
Realistic
Ray Tracing Only
Aperture C.1: Robot Appearance
Aperture 60
Aperture 80
Fisheye
C.2: Robot Handedness Analysis
Forward
Forward
Hand Precision(
Recall(
Mixing(
p=0.10 6--0,07
0=0.38 A--0.04
p=0.8 Á- 0.01
0=9.99 A--0.00
p=1.00
4-00.00
Inweree Forward
Left
0.2901
0.2626
0.1333
Right
0.3465
0.3427
Inverse
p=0.39
p=0.89 4m-0.01
Inverse
p=0.58 A- 0.03
p=0.96
p=0.92
Left:
0.9821
0,2570
0.217
6m-0.04
Am-0.01
4m-0.00
0.320
High
Low
White Color Random Color
Skin Color
Figure 16: Ablation experiment results with Intern VL1.52LLL355221B ENACT. Heatmaps show two-tailed unpaired p-values against the baseline, using Pairwise Accuracy. p < 0.05 is considered significant. Darker red means more significant. A the performance change from the baseline. If significant and A < 0, the setting is worse than the baseline. C.2 reports the robot's s performance the left- and right-hand predicates, where Mixing the proportion of ground truth left or right cases that are predicted as the other hand (i.e., mixing one hand into the other hand). Note that, although Intern VL.3.5-2413 2----411--2888 performance is less significant than GPT-5 mini, the | across unnatural camera configurations still remains high (> 0.05) when the same settings are significant for GPT-5 mini.
35

---

Published as a conference paper at ICLR 2026
B.4 SENSITIVITY TO IMAGE REALISM
Although the BEHAVIOR simulator is designed to be photo-realistic, we were curious whether "sim-to-real"" gap might still exist due subtle differences rendering quality. Specifically, we sought to investigate if such a gap affects performance on our world modeling tasks and to quantify the impact of rendering fidelity on the reasoning capabilities of state-of-the-art Vision- Lasgun-Language Models, such as GPT-5 mini. In the following sections, we detail the experimental setup for evaluating model performance across various levels of image realism.
Forward World Modeling
Inverse World Modeling
70
60
Pairwise Accuracy (%)
60
50
Pairwise Accuracy
40
40
30
30
20
Number Objects
Objects
Cosmas Cosmas-Reason Reason1-78
Qwen2.5wvg 5-VL-BB-Ins
Ovis2.5-98 MiniCPM-V-4.5888
lacrics3-BB-Liama3
IntemV...550 3.5-88
Gemma-3-1226
Figure 17: Comparison between Cosmos-Reasonl and other similar-sized models.
B.4.1 REALISTIC: GENERATED IMAGES AS REAL WORLD PROXY
Since activities diverse and complex, reproducing simulator outputs in the real world one-to-one basis would incur prohibitively high costs. However, with the advent of powerful image generation models with the ability of image-scale reproduction (e.g., GPT-image-1) 1), is feasible to use them as a real-world proxy to convert frames rendered by simulator into realistic styles, which provides a cost-effective and well-aligned alternative.
Constructing prompts for high-accuracy style transfer poses several challenges. First, since our segmented frames are extracted from a replayed robot trajectory, the generated realistic frames corresponding to the trajectory must preserve consistent content and style, including object shapes and appearances, lighting conditions, material properties, and camera parameters. Second, image generation models often demonstrate instability and errors in understanding fine- fine-grained structures of robotic arms (particularly the gripper) and in interpreting robotic actions. To mitigate these issues, establish detailed set of rules and incorporate them into the prompt design (Figure 20), which improves both stability and fidelity in the generated outputs.
Project Instruction and Prompt for Image Generation
# Below are the instructions and regul ionss, treat them as the sole, global reference all image generations you going to pertorm.
Coze objective
Convest simulstor screenshots inte photoreslistic PER images. Change style only; do not change content.
#4 Content Took (Content-rooked)
Presezve count, position, size, geometry, pose objecta. The robot hand and knife sagies, shapes, and actions must match exactly.
#4 Camera crop and viewpoint must remain unchanged.
The outdocr scene must remain daytime; tree and fence silhouettes must not change. If realiam conflicts with content,
fidelity takes precadence.
Style Requicements Lighting: Warm under-cabinet tungsten (320C-3600 x) soft window daylight fill.
36

---

Published as a conference paper at ICLR 2026
Image Realism
Forward
Image Realism Pair Accuracy Inverse
GPTS
GPTS 1
0.9
0.9
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
Realstic
Path Tracing
Tracing (Baseline)
Tracing Only
0.2
Realstic
Path Tracing
Tracing (Baseline)
Ray Tracing Only
(a)
(b)
Figure 18: Ablating image realism with GPT-5 and InternVL3.5-241B-A333 (a) Forward dynamics; (b) Inverse dynamics. Bar plots report Pairwise Accuracy across four rendering set- tings-Realister Realistic, Path Tracing, Ray Tracing (Baseline), and Ray Tracing Only. Error bars denote +SEM. The baseline x-tick is bolded.
Forward - Realistic Frame
Inverse Realistic Frame
Juvaat
Rexicotate
Saate
Figure 19: Examples simulator frames converted into realistic styles for both Forward World Modeling (left) and Inverse World Modeling (right) trajectories.
37

---

Published as a conference paper at ICLR 2026
Filmic contraat, smooth highlight zoll-off, no cruahed blacka oz blown highlights.
Tone:
Cameras approximately 35 f/2,8-4, shallow DOP; aubject sharp with gently blurred shadows, and ambient occlusion.
background.
Shadows: Reslistic soft shadows,
Materiala:
Metal knife and trims: Brushed, anisotropic metal.
Robot: Matte polymer.
Cutting board and countertop: Sealed/olled wood grain.
Glass/walls: Glossy glass with reslistic reflections and refrections.
grain; light vigrette. Frohibited: Cartoonish look, plastic sheen, blooe, oversaturation, hard out l sharpening, ligh:ing
#ost-precesaing: Subtle
Acceptance
Edge aligument: S8IM >- 0.95 (along object boundsries).
Segmentation: 1OU >= 0.93 for zebot, knife, cutting board, outdoors. 3", dolta px.
Color difference: delta
Knite shape error: <= px.
silhouetle ercor:
Implementation Suggestions
Use low denoi se strength 9.20-0.35, CFG 4-6. Negative prompt: forbid new objects, gecmetry changes, cartoonish/ aetsoonnsh//veeee htt////rraturaee//// Detail paas: add
naterial detail light filt. grain.
Now, review and summarize what have learzed from thess instructions.
Following the instructions have learned, transform the given image into realistic photograph styla.
Figure 20: The prompt used to generate realistic photographic style images from segmented frames (of replayed robot trajectory).
B.4.2 PATH TRACING SETUP
To generate the highest-fidelity images for our analysis of image realism, we utilized path tracing. This was achieved directly through the built-in, real-time path tracing engine provided by the NVIDIA Isaac Sim simulator. An example can be seen in Figure 21
Forward - Path Tracing
Inverse Path Tracing
Slmic
Figure 21: The figure illustrates Forward World Modeling (left) and Inverse World Modeling (right) trajectories rendered using the path tracing engine in NVIDIA Isaac Sim.
B.4.3 RAY TRACING ONLY SETUP
This setup was designed to represent intermediate rendering quality (representing 'unrealistic") While still utilizes the ray tracing pipeline as its foundation, we manually disabled several advanced lighting and post-processing effects to reduce visual fidelity. Specifically, turned off the following features: reflections. DLSS, ambient occlusion, sampled lighting, ambient light, and flow. The resulting visual style, which lacks these richer effects, can be
Figure 22
B.4.4 ROBUSTNESS TO NOISE IN PREDICATE DELTAS
To assess whether conclusions sensitive noise in the abstract transitions, study on a subset of 2,304 QAs sampled from ENACT. For each trajectory, we randomly perturb the symbolic predicates in the abstract deltas and re-run both forward and inverse evaluations for
robustness
38

---

Published as a conference paper at ICLR 2026
Forward Ray Tracing Only
Inverse - Ray Tracing Only
Next
Figure 22: Examples of an intermediate rendering style created with a simplified ray tracing pipeline for Forward World Modeling (left) and Inverse World Modeling (right) trajectories.
Metric
10
Forward (with random perturbations)
Task Accuracy 68.51 = 1.44 37.17
2.11
25.75
1.74
10.80
+ 1.59
1.60
= 2.11
1.11 = 0.80
0.92
0.40
0.00
0.00
Pairwise Accuracy 76.02 = 1.19 59.77
1.82
64.48
2.64
42.25
1:30
37.36
1.30
30.90 = 0.59
28.76
0.43
26.98
1.01
Inverse (with random perturbations)
Accuracy
83.45
58.39
3.40
13.79
4.14
0.73
1.26
Pairwise Accuracy 83.45 + 0.69 68.74 1:40
61.49
19
0.72 52.46
+ 0.69
45.39 = 1.23 39.70 = 1.64 35.00 + 2.57 29.01 = 1.64
Table 14: Robustness to random perturbations in predicate deltas. Mean + standard deviation (%) over three random seeds on 2,304-QA subset of ENACT for Intern
over three random seeds. Table 14 reports the mean = standard deviation across
seeds.
Across all horizons, we observe only very small standard deviations, and the qualitative trends remain unchanged: Inverse world modeling consistently outperforms forward world modeling at comparable horizons, and performance for both directions degrades sharply the number of steps increases. These results indicate that our findings are stable across activities and robust to random perturbations in the predicate deltas, rather than being driven by few particularly clean or favorable trajectories.
B.5 SENSITIVITY TO CAMERA CONFIGURATIONS
B.5.1
CAMERA APERTURE SETUP
Our default baseline is aperture 40. We also investigate apertures 30, 60, and 80. Examples can refer
to Figure 23
24 and 25
Forward Aperture 30
Inverse-Apertie 30
SuESI St
Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured with camera aperture of 30.
Figure 23:
B.5.2 FISHEYF LENS SETUP
Isaac Sim provides the fisheye lens settings. We choose fisheyePol hytoooolooomll..mmmmmmm which the most similar to daily fisheye lens, such as GoPro, as our evaluated target. The effect can be seen in the example Figure 27
B.5.3
CAMERA HEIGHT SETUP
The default setting height 1.75 m, we also investigate the high (+0.5m) setting and low 1-0.25m)setting and the examples shown in Figure 28 and 29
39

---

Published as a conference paper at ICLR 2026
Forward Aperture 60
Inverse Anerteree 60 NeXt
Next
Figure 24: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured with camera aperture of 60.
Forward Aperture
Inverse - Apperure 80
SuIrI Stad
Netespic
Next State
Gurrent State
Sexie
Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured with a camera aperture of 80.
Figure 25:
B.6 Do VLMs HAVE EMBODIED BIASES?
B.6.1 ROBOT APPEARANCE
We test three variants: White Color, Random Color (robot color randomized at each frame), and Skin Color (robot is rendered with human- timan-like skin tone). Examples can be referred to Figure 31 32
and 33
B.6.2 HANDEDNESS
Based on our experimental setup C.I We further examine whether predictions involving agent interactions reflect real-world handedness asymmetry (typically favoring the right hand). In both humans and models, and across both task types, right-hand precision and recall systematically exceed those the left (Figures 37). Furthermore, left-to-right mixing rate (ground-truth left-hand components wrongly predicted right-hand ones) substantially exceeds the reverse (Figure 38).
C ERROR ANALYSIS
C.I METHODOLOGY ERROR CALCULATIONS
Signature Modeling from Scene Graph-level Differences For error analysis, it is found hard to recognize predicate-level or semantic-level errors through natural language- based actions, i.c. visible differences between consecutive states. Hence, parse the raw (natural language) action predicates sig
signature into sequence of unique state-change signature (ag 00 :...).
G = (1,6), |
(7, s, | (transition, e, Pfrom - Pto), Y E {add, remove)
To further structure these signatures, each signature a;
sig is then modeled as finite set of components {ci, c2.... ). Each component c; represents an atomic unit of state change. We distinguish three categories of components: edge components (addition removal ? of predicates p between two entities e and e2), node components (addition or removal 7 of the predicate of entity e), and node transition components (transition from the previous predicate frrm e).
predicate Pto of an entity
Error Modeling from Signatures We categorize errors from two perspectives: structural and semantic. Structural errors concern the form of actions and include entity substitution (object replacement), predicate substitution (relation/attribute replacement), polarity inversion (add, remove transition), omission, and hallucination. Semantic errors concern interpretation and are grouped
40

---

Published as a conference paper at ICLR 2026
Camera FOV Pair Forward
Camera FOV Pair Accuracy Inverse
0.7
0.5
0.5
0.3
0.3
0.2
Aperure 30
Aperture 40 Aperture 60 (Baseline)
Apeture 60
Fisheye
0.2
Aperture 30 Aperture 40
Aperture 60 Aperture 80 (Saseline)
Fisheye
(a
(b)
Figure 26: Ablating camera field-of-view (FOV) with GPT-5 and Intern (a) Forward dynamics; (b) Inverse dynamics. Bar plots report Pairwise Accuracy across five lens Battings-Aperturr 30, Aperture 40 (Baseline), Aperture 60, Aperture 80, and Fisheye. Error bars denote +SEM; the baseline tick is bolded.
F
Inverse - Fisheye
OBEA Sinee
Mard SIINe
Figure 27: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured with fisheye-style camera.
Forward
Inverse High
Current State
dex Silee
Cunco
Next State
Figure 28: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured from camera height of 2.25 m.
Forward - Low
Inverse Low
Alon StAte
Staree
Next
Figure 29: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), captured from camera height of 1.5
into spatial relations (misplaced object positions), functional states (incorrect functionalities or status), material states (wrong physical properties), and agent interaction (misattributed agent actions).
41

---

Published as a conference paper at ICLR 2026
Camera Height Pair Accuracy Forward
Camera Height- Pair Accuracy Inverse
0.5
0.5
0.3
0.3
0.2
High
Mid (Baseline)
Low 0.2
High
Mid (Baseline)
Low
(a)
(b)
Figure 30: Ablating camera height with GPT-5 and WLe..VLLL-55-211----A22 (a) Forward dynamics; (b) Inverse dynamics. Bar plots report Pairwise Accuracy across three viewpoints (High, Mid baseline, and Low). Error bars denote SEM: the baseline tick is bolded.
Forward - White Color
Inverse White Color
State
Figure 31: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), with the robot gripper rendered in white.
Forward
Random Color
Inverse Random Color
Corvart
Siate
stone
Next
Figure 32: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), with the robot gripper rendered in random color at each frame.
Forward - Skin Tone
Inverse- Skin Tone
State
Next
Aext
Figure 33: Example trajectories of Forward World Modeling (left) and Inverse World Modeling (right), with the robot gripper rendered in human skin-liks color.
42

---

Published as conference paper at ICLR 2026
Robot Appearance Pair Forward
Robot Appearance Pair Inverse
0.9
mww.L-35-2414228
0.7
0.7
0.5
0.5
0.3
0.3
0.2
White Color
Fancy (Baseline)
Random Color
Skin Color
0.2
White Color
Fancy (Basaline)
Random Color
Skin Color
(a)
(b)
Figure 34: Ablating robot appearance with GPT-5 and Intem
Forward
dynamics; (b) Inverse dynamics, Bar plots report Pairwise Accuracy across four styles-White White Color, Fancy (Baseline), Random Color, and Skin Color. Error bars denote #SEM; the baseline tick is bolded.
GPT-5 Forward Hand Signatures
GPT-5 Inverse Hand Signatures
Left Ground Truth
Lefioo
Miking
Right Ground Truth
Left RtM Right
Masched
fland
Left Hand
Right Hand
Matched
Left Prediction
Left Prediction
Retiit
Miking
Right
Mikng
Right Prediction
(b)
Figure 35: Illustration of overlaps between ground truth and GPT-5 predictions sets for left and right hands related signatures in forward and inverse tasks. The size of ellipses project the total counts of signatures, and overlaps denote matched signatures (center regions) or mixing errors (cross-hand overlaps).
Both perspectives are based on comparing component sets of paired ground-truth and predicted signatures. For each pair, we compute set-level differences and classify components into missing (in ground truth only), matched (in both), and hallucinated (in prediction only). To support this categorization, we preprocess the signature dataset into structured data with these three groups of components, as outlined in Algorithm
To categorize structural errors, we define criteria for each component type (edge, node, transition node). Entity Substitution occurs when entities differ while other fields match; Predicate Substitution when the predicate differs; and Polarity Inversion when only the operation (add/remove) differs. After pairwise classification, remaining unmatched ground-truth components categorized to Omission. and unmatched predicted components as Hallucination.
After structural error categorization, each component is further labeled by semantic error type: Spatial Relations, Functional States, Material States, or Agent Interactions. Labeling uses a predefined mapping table that links all observed predicates their semantic categories. When a component contains listed predicate, the table is consulted to assign its semantic label. The overall workflow of error detection and categorization illustrated Algorithm
Handedness Asymmetry Error Modeling To systematically capture handedness asymmetry, compute for left- and right-hand components: precision (correct matches predicted), recall
43

---

Published as a conference paper at ICLR 2026
Forward Precision
Inverse Precision
0.7
Left Hand
Left Hand
Right Hand
Right Hand
0.0
0.8
0.5
0.5
0.4
0.3
0.2
Gemin2.5Pro
CPT-5
GPT- mini
InDerVLL...
Human
0.2
Gemiin2.SPro
GP1-
GPT-5 mini
IntemVL3..
Human
2418-A280
2418-A260
Figure 36: Precision of left/right hand related components prediction in (a) forward and (b) inverse tasks, with models Gemini2.5Pro, GPT-5, GPT-5 mini, Intern VL3.5-2411-B.------- and Human. Error bars indicate the standard error (SE).
Forward
Inverse
0.7
0.7
Left Hand
Left Hand Right
Right
0.6
0.6
0.5
0.3
InterVLL3
InterVL3
2418
2419 A288
(a)
(b)
Figure 37: Recall left/right hand related components prediction in forward task, with models Gemini2.5Pro. GPT-5, GPT-5 mini, VL3.5VLL..24BBB444 and Human. Error bars indicate the standard error (SE).
Forward Hand
Right
0.12
Right Left Blas
0.20
Right Left Blas
0.08
0.10
0.04
0.00
Gemini2.5PM 5Pm
GPT-5
GPTS mini
Internvi 3.5- 2418-A288
Human
0.00
Gemini2.5Pm
GPT-5
GPTS mini
Internvi 3.5-
Human
2418-A288
(b)
Figure 38: Hand-mixing rate, i.e.the ratio of left/right hand-mixing all ground truth left/right and components (a) forward and (b) inverse task, with models Remini2.5Pro, GPT-5, GPT-5 mini, --2.555---1--22---5 and Human. Error bars indicate the standard emor (SE).
44

---

Published as a conference paper at ICLR 2026
Total Counts Forward
Total Counts Inverse
35.000
45,000
37,000
33,000
35,000
25.000
15,000
Gen 5Pm
6915
Ruran
Gswrlz 5P
6915
Rurian
(a)
(b)
Figure 39: The amount of total errors made by Gemini2.5PPr GPT-5, GPT-5 mini, Intern VL3.5- 241B-A28B and Human, under (a) forward tasks and (b) inverse tasks.
(correct matches over ground truth), and the hand-mixing rate (the fraction of ground-truth left-hand components predicted right, vice versa).
For computing the hand-mixing rate, we define left-right mixing at the level of each signature-level difference (with missing, matched, and hallucinated components). If the missing set contains left- (or right-) hand usage. while the hallucinated set lacks the same hand but includes the opposite one, then all missing components involving that hand are counted as left-to-right (or right-to-left) mixing, as outlined in Algorithm 4
C.2 STRUCTURAL ERROR ANALYSIS
We compared error patterns in forward and inverse tasks across Gemini-2.5 Pro, GPT-5, GPT-5 mini, IntemVL-335-211B-222333 and human predictions (Figures 39).
C.3 SEMANTIC ERROR ANALYSIS
In semantic analysis (Figure 46 all systems-Geminie Gemini-2.5 Pro, GPT-5, GPT-5 mini, MMterVLL335-241B--AAA and humans-shnn show similar pattern: errors are concentrated in Spatial Relations and Agent Interaction, reflecting difficulties reasoning about object positions and agent actions (e.g., leftright-haad grasping). A task-dependent asymmetry also appears: spatial-relations errors are more common in forward tasks, while agent-interaction errors higher inverse tasks. For illustration, we sample representative GPT-5 cases for each semantic category under both settings (Figures
50
45

---

Published as a conference paper at ICLR 2026
GPT-5 Forward
GPT-5 Inverse
GPT-5 mini Forward
GPT-5 mini Inverse
Gemini2.5Pro Forward
Gemini2.5Pro Inverse
InternVL3.5... Forward
InternVL3.5... Inverse
Human Forward
Human Inverse
0%
25%
50%
75%
100%
Entity Substitution
Polarity Inversion Omission
Hallucination
Predicate Substitution
Figure 40: The structural error distributions of typical LLMs (GPT-5, GPT-5 mini, Gemini2.5Pro and Intern VL3.5-241B-A222 (referred as Intern VL3.5... in figure)) and Human-level prediction in both forward and inverse tasks.
46

---

Published as a conference paper at ICLR 2026
Forward Structural Error Example (GPT-5)
Inverse Structural Error Example (GPT-5)
The robot r1 changes to be using the right gripper grasp
The dice transitions to be inside the bookcase.
robot r1 now becomes using the right gripper grasp the Swiss cheese.
board game changes Inside the bookcase.
Gg
Task: assembling gift baskets.
Task: collecting children's toys.
Error Type: Entity Substitution
Error Type:
Entity Substitution
The model substituted Swiss cheese butter cookie, while keeping the remaining terms accurate.
Reason:
Reason: model substituted game with
aice, while keeping the remaining terms accurate.
Figure 41: Example of structural error Entity Substitution by GPT-5 under forward and inverse tasks.
Forward Structural Error Example (GPT-5)
Inverse Structural Error Example (GPT-5)
The Tupperware transitions to be under the half chicken. The half chicken becomes on top of and touching the Tupperware.
The robot transitions using gripper to grasp the Swiss cheese.
The half chicken becomes inside the Tupperware.
dx
The robot becomes using right gripper to grasp the Swiss cheese.
Task: clearing food from table into fridge.
Task: assemblin gift baskets.
Error Type: Predicate Substitution
Error Type: Predicate Substitution
The chicken should on top Tusperware, but the model predicted being inside the Tupperwore.
Reason:
using left gripper, the model predicted as using the right gr pope..
The
Figure 42: Example of structural error Predicate Substitution by GPT-5 under forward and inverse tasks.
47

---

Published as a conference paper at ICLR 2026
Forward Structural Error Example (GPT-5)
Inverse Structural Error Example (GPT-5)
The baking sheet becomes covered by the diced Vidalia onion.
The cutting board is no longer on top of and touching the
The baking sheet is no longer containing diced Vidalia
The cutting board changes to be top of and touching the
making pizza.
Task: chopping onion.
Type: Polarity Inversion
Type:
Polarity Inversion
Reason: The baking sheet shou ii transit from state 'no onion" "has onion", but the model predicted inversely,
Reason: The cutting board should transit from state "on the bar" "no
of the bar", but model predicted inversely.
Figure 43: Example of structural error Polarity Inversion by GPT-5 under forward and inverse tasks.
Forward Structural Error Example (GPT-5)
Inverse Structural Error Example (GPT-5)
The robot r1 changes to be using the left gripper grasp the microwave.
dg
The bottom cobinet without top becomes open.
The microwave now becomes turned on.
The Tupperware langer inside the battom cabinet na top.
Task: cooking hot dogs.
Task: freezing pies.
Type:
Type:
The model omitted the action "The robot changes to using the left gripper to the microwave. which supposed to be predicted.
Reason:
Reason: The model om omitted the action "The bottom cabinet without top becomes open." which supposed be predicted.
Figure 44: Example of structural error Omission by GPT-5 under forward and inverse tasks.
48

---

Published as a conference paper at ICLR 2026
Forward Structural Error Example (GPT-5)
Inverse Structural Error Example (GPT-5)
robot changes using left gripper grasp the microwave,
The bottom cabinet without top becomes open,
The micrawave now becomes turned on.
The Tupperware is no longeer inside the bottom no top.
Task: cooking dogs.
Task: freezing pies.
Error Type:
Hallucination
Error Type: Hallucination
Reason: The model ucinated action The microwave now becomes turned which should not predicted current action.
Reason: The madel hallucinated the action "The Tupperware is longer inside bottom cabinet top. which should not be predicted in current action.
Figure 45: Example of structural error Hallucination by GPT-5 under forward and inverse tasks.
GPT-5 Forward
GPT-5 Inverse
GPT-5 mini Forward
GPT-5 mini Inverse
Gemini2.5Pro Forward
Gemini2.5Pro Inverse
IntemVL3.55.. Forward
IntemVL3.5.. Inverse
Human Forward
Human Inverse
0%
25%
50%
75%
100%
Spatial Relations
Functional States
Material States
Agent Interactions
Figure 46: The semantic error distributions of typical LLMs (GPT-5, GPT-5 mini, Gemini2.5Pro and InternVL3.5515 VL3.5-241B-A28B (referred as VL3.5. in figure)) and Human-level prediction both forward and inverse tasks.
49

---

Published as a conference paper at ICLR 2026
Forward Semantic Error Example (GPT-5)
Inverse Semantic Error Example (GPT-5)
The Tupperware transitions under half chicken. The half chicken now becomes on top of touching the Tupperware,
The cutting board is no longer on top of and touching the
Gn
The half chicken becomes inside the Tupperware.
The cutting board changes to be top of and touching the
Task: clearing food from table into fridge.
Task: chopping onion.
Error Type: Spatial Relations
Type: Spatial
the model made a mistake when predicting an action related to the spatial position of the half chicken (on top of).
Reason:
Reason: The model made mistake when aredicting an action related the spatial position of cutting board (on top of).
Figure 47: Example of semantic error Spatial Relations by GPT-5 under forward and inverse tasks.
Forward Semantic Error Example (GPT-5)
Inverse Semantic Error Example (GPT-5)
The robot r1 changes using the left gripper to grasp the microwave.
On
The robot r1 changes to be using the right gripper grasp tube of toothpaste.
now
The dispenser becomes turned
cooking dogs.
Task: sorting household items.
Error Type: Functional States
Type: Functional
predicting oction related to the microwave's 's functionality (turned on).
Reason:
The model made mistake related to the functionality of soop dispenser (turned on) when predicting on action.
Reason:
Figure 48: Example of semantic error Functional States by GPT-5 under forward and inverse tasks.
50

---

Published as a conference paper at ICLR 2026
Forward Semantic Error Example (GPT-5)
Inverse Semantic Error Example (GPT-5)
Fu
Vidalia onion no longer drop- in sink.
The Vidalia onion now becomes the diced onion.
The Vidalia onion stopped being inside the becomes
drop
Vidalia
The baking sheet transitions by the grated cheese.
the diced Vidalla onlon.
Task: chopping onion.
Task: making pizza.
Type:
Material States
Type: Material
The model made mistake when predicting the material state of the Vidalia onion (diced).
Reason: The model made a mistake when predicting an action related to the material state of the Vidalia onion (diced).
Figure 49: Example of semantic error Material States by GPT-5 under forward and inverse tasks.
Forward Semantic Error Example (GPT-5)
Inverse Semantic Error Example (GPT-5)
The robot changes to be using the right gripper to the butter cookie.
The robot r1 transitions be using the left gripper to grasp the Swiss cheese.
The robot r1 becomes using the right gripper to grasp the Swiss cheese.
The robot r1 now becomes using the right gripper to grasp the Swiss cheese.
Task: assembling gift baskets.
Task: assembling gift baskets.
Error Type: Agent Interactions
Error Type: Agent Interactions
The model made mistake when predicting action involving the agent's 's interaction (grasp using the gripper).
Reason: The model made a mistake when predicting action involving the agent's 's interaction (grasp using the gripper).
Figure 50: Example of semantic error Agent Interactions by GPT-5 under forward and inverse tasks.
51

---

Published as a conference paper at ICLR 2026
Algorithm 2: Action-level Parsing of Signatures Data
Input: Dataset of signatures Daig. each with ground-truth signatures agt and predicted signatures 0.50
Output: Data of signatures Diiig with missing, matched and hallucinated components
Signatures filtering: Daig
foreach (age) "agttapp .. E Daig do
lgg = =lagig then add (a sig a to Daig
else
discard sig 999
Action-pairwise Comparison: foreach (o . a.i)) D sig do
Gmi missing component.s, Cma matched components, Ch hallucinated components
Cmi 0, Cma + 0, Ch.
foreach (Cge, Cp) E (agh) sig
: a ag0)) do
if Cgt = Cp then
add Cgt to Cra
else
add Cyt Cmi add Cp to Ch
add (Cmi: Cmay Ch) to D' Digg discard (agi ag return D'
52

---

Published as a conference paper at ICLR 2026
Algorithm 3: Action-level Structural and Semantic Error Categorization
Input: Parsed signatures dataset D sigg predicates preds
Output: Categorized errors dataset Derr Structural errors categorization: PI 4 PS+0, 0. ES+ 0. OM - 0, HA
foreach ai E D do
(Cmi: Cr.) + (Cmi (Cmi (9* on (a)1)))
(Cmis Ch.) 6- F.FFFinPaarisse::eeee
(Cmi, Ch, polarity inversion);
(Cmss ch) * 0 then
add (Cmi, (1. to PI
remove (Cmss oh) from sig
(Cmis Ch) f- +=FindPairwiserrrre (Cmss Ch) * 0 then add (Cmi, (h) to PS remove (Cmss oh) from sig
(Omis Oho predicate substitution);
(Cmis Cn) f- *FFindPairiseerrrooee
(Omis Chy entity substitution);
(Cmis ch) * 0 then
add (Cmis Ch) to ES remove (Cmss Ch) from sig
foreach Cmá E Cun do L add mi OM
foreach C₁ C₁ do add to HA
PI: Polarity Inversion, PS: Predicate Substitution, ES:
Entity Substitution OM: Omission, НA: Hallucination Derr + (PI, ES, OM,
x/
Semantic errors labeling: foreach c in Derr do foreach pred in preds do
if pred then label e with Semant cError(pred)
return Derr
53

---

Published as a conference paper at ICLR 2026
Algorithm 4: Dataset-Level Detection of Left-Right Hand Confusion
Input:
Dataset of signature-leved differences Dais {Cmis Cmas Ch) Output: Confusion dataset Dhand = {D Dr21)
Left to right hand confusion: D/2r <- 0
foreach (Cmis Cma: Ch) E Daify do
if 3m Cmi that involves left hand then 3h E Cs that involves left hand then continue,
else if 3h cn that involves right hand then foreach m E Cmi do if involves left hand then add m Di2r
Right to left hand confusion: D,21 +
foreach (Emis Cmay Ch) E Dasfs do
3m E Cys that involves right hand then 3h Ch that involves right hand then L continue,
else if 2h on that involves left hand then
foreach E Cyu do
m involves right hand then add m to Dr21
Drand (Dier: Dr21)
return Dhand
54

---

Published as a conference paper at ICLR 2026
D ADDITIONAL RELATED WORK
Additional World Modeling Discussion. Recent methods explore transformer-basee dynamics (Chen al.] 2022 Wang et al. 2025c|al a Wan al.
2025) and propose diagnostics for temporal abstraction and causality (e.g.. CATER and CLEVRER) (Girdhar et al. 2020, Yi et al., 2019). Several benchmarks extend this direction. Dang et al. (2025) introduces ECBench under the theme of embodied cogmition, but their evaluation focuses on multimodal perception QA without grounding in MDP theory or assessment interactive dynamics, leaving the role of action and consequence largely unexplored. By contrast. Gao et al. (2025) examines whether vision-language models acquire internal models of the world, studying both spatial reasoning and physical interactions in contrived static setups. While non-egocentric, this work provides useful insight into how large models capture specific dimensions of reasoning. Aurora-Bench (Qiu et al. 2025) focuses on short-horizon and general-purposs, video forward and inverse world modeling. More recently, Chen et al. (2025) evaluates sequence-level coherence by having models predict the order of intermediate clips given only an initial and final state. While this assesses high-level planning, our benchmark, ENACT, differs in several fundamental ways to capture a more fine-grained understanding of interaction dynamics. First, their work lacks a clear action space, defining actions as video clips, which can lead to inconsistent semantic granularity. Second, their prediction of the entire intermediate sequence tests one-shot planning rather than a step-by-step, causal understanding of how actions lead to state changes. Furthermore, their evaluation is limited to forward prediction, whereas ENACT also evaluates inverse modeling. Finally, ENACT is built with a scalable data generation pipeline specifically designed to serve controlled proxy for characterizing the properties of VLM-based embodied agents. In doing so, it complements prior benchmarks by grounding embodied world modeling in norghooizon,, fine-grained dynamies.
Vision-Language Models for Embodied Agents. Scaled foundation models advance vi- sual-linguistic reasoning, yet their primarily disembodied training raises a natural question of whether embodied cognition emerges without interaction (OpenAl 2025 DeepMind 2025 Anthropic (025).
Robotics integrates
VLMs as planners and policies grounding language in affordances and control
(Ahn et al. 2022
Huang al. 20233b 2022b; Liang al. 2022a Driess al. 2023 Zitkovich et al.,
2023 Kim et al.
2024] Team et al.] (024). Parallel embodied agents in simulation assess navigation
and interaction (Anderson et al. 2018 Das al.) 2018 Shridhar et al. 2020 Padmakumar et al.
2022 Mees et al. [022] Fan et al. (022). while egocentric corpora broaden sensorimotor coverage (Damen et al. | Grauman | 2002). Complementary benchmarks examine isolated facets: spatial perception in static scenes (Ramakrishnan et al.) (2024) Yin et al., 2025), contrived physical interactions (Yi et al. (2019] Bakhtin al.][2019] or purely linguistic reasoning (Li et al. 2024b), whereas ENACT unifies these elements via egocentric trajectories and evaluates consequence-awant
al.
world modeling over extended horizons. RoboEXP (Jiang al.
2024) utilizes action-conditioned
scene graphs for manipulation with VLMs, yet it operates primarily within fully observable, allocen- tric environments restricted to short-horizon tasks. GVL (Ma et al.. 2024) uses VLMs as in-context value functions by casting task progress estimation as a temporal ordering problem over shuffled frames from relatively short-horizon, fixed-camera trajectories. In contrast, ENACT evaluates world modeling: agents must autoregressively predict the next egocentric state conditioned on previous state and action in long-horizon horizon home-scale tasks with moving cameras, effectively acting as transition models rather than value models.
55

---

Published as a conference paper at ICLR 2026
E THE USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs), including Google's Gemini 2.5 Pro and OpenAl's GPT-5, as auxiliary tools assist with writing, editing, and conducting the literature review for this manuscript. All content was critically revised and fact-checked by the human authors to ensure its scientific validity and originality. The authors are fully responsible for all statements and conclusions presented in this paper. Specifically, we use LLMs for polishing our wording and writing, and we use LLMs to retrieve several related works.
F LIMITATIONS
Our work has limitations primarily related to its scope. First, while we introduce several diagmostic tasks that reveal key model biases, this set is not exhaustive. The experiments factors like camera configuration and agent appearance serve as foundational examples, but the ENACT framework is designed to be extensible tool. It can support future, more complex investigations into much broader spectrum different embodied--elated settings. Second, due to the significant computational cost of evaluation on VLMs, the in-depth ablation experiments were necessarily focused on representative subset of models and data. A broader evaluation across more architectures and larger data scales would be beneficial generalize our findings. Furthermore, we do explore finetuning VLM in this work, but we expect our automatic and scalable dataset can also bring benefits to improving VLM's embodied world modeling abilities. Additionally, due to the frequent physical inconsistency of generated rollouts and the difficulty of designing fair evaluation metrics, which often require costly human studies, we do not evaluate video generative models on ENACT (for unified VLMs, evaluate BAGEL (Deng al. 2025) and the result is shown in Table and 10p.
56


Reviews:
[{"review_id": "xyDl1YgZOI", "reviewer": "Reviewer_r94E", "summary": "To investigate whether embodied cognition emerges in VLMs, the authors introduce ENACT, a benchmark designed for two sequence reordering tasks: 1. Forward World Modeling: Predicting an ordered sequence of future states (egocentric RGB images) from an initial observation and a set of actions. 2. Inverse World Modeling: Inferring an ordered sequence of actions from an ordered sequence of states.\n\nUsing both proprietary and open-weight models, the authors find:\n(1) a significant performance gap across VLMs,\n(2) all models perform better on inverse than forward modeling, and\n(3) models exhibit embodied biases.", "questions": "- If inverse world modeling is consistently easier than forward modeling, what implications does this have? What directions do the authors foresee for future work? Does this asymmetry apply specifically to VLMs, or might it generalize to generative models or unified VLM architectures?\n\n\nI would consider raising my score accordingly if my questions are sufficiently answered.\nI also encourage the authors to consider open-sourcing the ENACT benchmark.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Compares the ability of VLMs to perform forward vs. inverse world modeling on the Behavior1k dataset with egocentric inputs.\n\n- Provides several novel discussions and insights.\n\n- Demonstrates consistent trends between open- and closed-source baselines.", "weaknesses": "1. Novelty:\n\nHow does this approach compare to GVL[1]? GVL also performs zero shot reordering task using closed-source VLMs and reports high success rates on existing embodied benchmarks. I wonder if ENACT is just evaluating different models using GVL's proposed methods? Also, a citation is missing. \n\n2. Limited Discussion of Model-Specific Differences:\n\nThe paper mentions that the tested VLMs are trained on “static” datasets, and I think the static here is unclear what this means (image-only/dis-embodied data?). For instance, Cosmos-Reason1, which is trained on embodied datasets, performs the worst. If ENACT is designed to measure embodied cognition, I would expect Cosmos-Predict1 to outperform other baselines trained not in the embodied domain.\n\n3. Real-World Comparisons:\n\nThe “real-world” comparisons appear to rely on GPT-image-1 performing style transfer, and comparing across ray tracing and other rendering effects disabled in the simulator, and the authors conclude certain baselines are indifferent to rendering changes, thus they are robust to the sim-to-real gap. I think there's still a gap between real-world captured data and style-transfered data, and degrading rendering effects do not account for real-world challenges such as lighting / physics simulation difference etc.\n\n[1] Ma, Y. J., Hejna, J., Wahid, A., et al., Vision Language Models are In-Context Value Learners, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QmZzcfH01i", "reviewer": "Reviewer_hbkZ", "summary": "The paper proposes ENACT, a simulator-derived benchmark that tests whether VLMs exhibit embodied cognition through sequence reordering tasks grounded in a POMDP. The benchmark includes two tasks: forward world modeling (ordering future images given the action sequence) and inverse world modeling (ordering actions given the image sequence).\n\nThe authors derive their data from BEHAVIOR demos, which they convert to key-frame trajectories by detecting abstract scene graph deltas. They form QAs by shuffling future states or actions and employ a parser-friendly output along with an online verifier.\nTesting across 29 long-horizon household activities (8,972 QAs, a passing number for benchmark of this kind), the results show that SOTA VLMs underperform humans, particularly as horizons lengthen. The models consistently find the inverse task easier than forward modeling. The probing experiments reveal interesting patterns: the models show little sensitivity to rendering realism but experience sizable performance drops with non-human-like camera intrinsics and heights. They also exhibit a clear right-handed bias.\n\nOverall, ENACT provides a scalable and reasonably clean probe of consequence-aware reasoning in egocentric interaction. While the benchmark offers meaningful signals about embodiment, these insights remain simulator-bound in nature.", "questions": "**[Q1]**: The paper asks “to what extent does embodied cognition emerge from such training,” but the benchmark measures sequence ordering accuracy. How exactly does success on this proxy reflect embodied cognition rather than just temporal correlation learning?\n- Action: Include an operational mapping, e.g., a table or diagram linking each task (forward/inverse) to specific cognitive constructs (for example, causal modeling, temporal abstraction, affordance understanding, ....).\n\n**[Q2]**: The inverse task may benefit from language priors, aka models are better at mapping visuals to familiar verbs than to unseen symbolic actions. How robust is this advantage when action descriptions are out of vocabulary?\n- Action: Re-run a subset of evaluations with non-linguistic action encodings (e.g., icon sequences or structured tokens) and a held-out predicate set. Report whether the inverse > forward gap remains when language grounding is removed.\n\n**[Q3]**: Because multiple QAs come from the same trajectory, the reported averages might conflate model ability with activity-specific bias. Are results consistent across different activities and noise levels in the abstract deltas?\n- Action: Provide a per-activity breakdown of Pairwise Accuracy, plus variance across random seeds where $∆Vis$ predicates are randomly perturbed or dropped. This would clarify whether conclusions hold beyond a few well-behaved trajectories.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The benchmark design is solid and principled. It uses POMDP framing with two tasks formulated as permutation prediction to avoid hand-crafted distractors. The forward task requires reordering images to match a given action sequence, while the inverse task requires reordering actions to match a given image sequence. An online verifier allows multiple valid permutations, and the metrics include Task Accuracy (exact match) and Pairwise Accuracy (adjacent consistency). The benchmark achieves solid human IAA ($α = 0.83$).\n\n- The probe studies are focused and actionable, examining camera priors and handedness rather than just presenting leaderboard tables. The findings are particularly interesting. For image realism, there's no significant effect across path tracing, style-transfer, or ray-tracing variants. However, camera intrinsics and height matter considerably: performance drops with large apertures, fisheye lenses, and high camera positions, with best results using human-like intrinsics. The models show clear embodied biases, including a right-handed advantage with higher precision/recall and lower mixing. The error taxonomy reveals that failures are dominated by Omissions and Hallucinations of state changes.\n\n- The main evaluation is mostly solid, testing 7 proprietary and 22 open-weight VLMs against human performance as reference. Results consistently show that models perform better on inverse tasks than forward tasks, accuracy falls monotonically with L, and there's a large gap between human and model performance at longer horizons.", "weaknesses": "- The central research question, in my view, would benefit from clarification. The phrasing “to what extent does embodied cognition emerge from such training?” seems to suggest an over-time evaluation across training stages, yet the current evaluation (on end models; based solely on benchmark performance) cannot adequately address that. VLMs typically undergo multiple distinct training phases (e.g., large-scale language pretraining, multimodal alignment, instruction tuning, reinforcement learning for human preference / reasoning). Therefore, the results can at best reveal how well these models are capable of solving embodied tasks, rather than whether or how embodied cognition itself emerges during training.\n- The benchmark relies heavily on simulator semantics. Scene-graph deltas and visibility constraints are simulator-privileged, which simplifies supervision and evaluation but may bake in ontology choices that models can exploit while limiting transfer to real-world scenarios. **The authors should consider adding a real-video subset (such as curated egocentric clips with weak labels) to stress test ontology mismatch.** The scene graph itself also represents a strong choice by assuming graph structure, which may not fully capture the complexity of real-world visual reasoning.\n- (I am less sure about this, authors can correct me if I'm wrong.) Key-frame sampling by predicate deltas seems to underweight fine motor dynamics and overemphasize symbolic toggles (Open/Inside/OnTop). Authors should consider an ablation with low-level pixel change gates or dense optical-flow checkpoints to verify that results aren't an artifact of symbolic sparsity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RW62O45Ft3", "reviewer": "Reviewer_NXiX", "summary": "### ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction\n\n#### Main Research Problem\n\n* **Embodied Cognition (EC)**: The theory posits that intelligence arises from continuous sensorimotor interaction with the world.\n* **VLM Limitation**: Modern Vision-Language Models (VLMs) are trained **disembodiedly** on vast static datasets.\n* **Core Question**: **To what extent does embodied cognition emerge** in these VLMs despite their non-interactive training, specifically their ability to understand the **consequences** of actions and the dynamics of the environment?\n\n#### Research Methodology (The ENACT Benchmark)\n\n* **Framework**: Introduced the **ENACT** benchmark, grounded in a Partially Observable Markov Decision Process (POMDP) framework, to evaluate EC through **egocentric** (first-person) interaction world modeling.\n* **Task Types**: Two complementary **sequence reordering** tasks test a model's understanding of how the world evolves with its actions:\n    1.  **Forward World Modeling**: Given a sequence of abstract actions, the model must predict and correctly order the sequence of **future states**.\n    2.  **Inverse World Modeling**: Given a sequence of state changes (observations), the model must infer and correctly order the corresponding **actions**.\n* **Data**: A scalable dataset of 8,972 QA pairs derived from diverse, long-horizon household activities in the BEHAVIOR simulator.\n\n#### Key Experimental Findings\n\n1.  **Significant Gap**: State-of-the-art VLMs (including GPT-5) show a **substantial performance gap** compared to human performance.\n2.  **Horizon Failure**: Model accuracy **drops sharply** as the interaction time horizon (number of steps) increases, indicating a lack of robust, **long-term** understanding of action consequences.\n3.  **Embodied Biases**: VLMs exhibit distinct embodied biases, such as a preference for recognizing **right-side actions** and a significant performance drop when camera views deviate from typical human perspectives.\n4.  **Error Analysis**: The primary error modes for top models are **Omission** (failing to recognize a state change that occurred) and **Hallucination** (recognizing a state change that did not occur).\n\n#### Conclusion\n\nThe ENACT benchmark demonstrates that while powerful, current VLMs lack the necessary robust, long-term understanding required for embodied world modeling, indicating that **key elements of embodied intelligence have not fully emerged** from their static training paradigm.", "questions": "1. I still have some doubts about the selection of keyframes. The author emphasized during the construction of the dataset to avoid having repetitive frames and overly similar frames as adjacent keyframes, and proposed a series of methods for this purpose. However, what readers are concerned about is whether it is no longer similar to the previous frame. This is particularly crucial throughout the entire trajectory. In the readers' understanding, a key frame contains sufficient event information. Can the author propose a series of criteria to measure this?\n\n2. In my opinion, this is actually a piece of work that assesses the model's ability to understand the embodied world. Since the author attempts to propose a benchmark or dataset, it seems that more measurement criteria are necessary in the eyes of the readers. The current embodied tasks do not seem to be satisfied with merely evaluating the \"ordering tasks\" of forward and backward modeling. And if such a benchmark only includes these two evaluation tasks, then a series of video generation models such as Sora would be excluded (we also observed this in the experimental part). However, we know that many of the current world models are actually constructed based on image/video generation models. This would significantly limit the widespread use of benchmarks.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "### Main Strengths of the ENACT Paper\n\n* **1. Novelty and Comprehensiveness in Evaluation (Consequence-Aware World Modeling)**\n    * **Summary**: ENACT elevates the evaluation of embodied AI beyond simple perception or isolated interactions to **consequence-aware world modeling** over **extended time horizons**.\n    * **Detail**: It introduces complementary **Forward** (Action $\\rightarrow$ State) and **Inverse** (State $\\rightarrow$ Action) sequence reordering tasks, providing a holistic test of a VLM's ability to reason about the **causal effects** of its actions, unlike prior static or limited benchmarks.\n\n* **2. Focus on Egocentric Interaction and Fidelity (Robust POMDP Framework)**\n    * **Summary**: The benchmark is explicitly designed around **egocentric (first-person)** interaction trajectories within a **Partially Observable Markov Decision Process (POMDP)** framework.\n    * **Detail**: By utilizing data from the **high-fidelity BEHAVIOR simulator**, ENACT ensures the evaluation tasks are realistic and relevant to the challenges faced by embodied agents operating in complex, dynamic household environments.\n\n* **3. Effective Diagnosis of SOTA VLM Failures (Long-Term Reasoning and Biases)**\n    * **Summary**: ENACT provides a crucial diagnostic tool by successfully revealing fundamental weaknesses in the **state-of-the-art VLMs** (e.g., GPT-5) concerning long-term embodied reasoning.\n    * **Detail**: Experimental findings pinpoint specific failure modes—such as a **sharp performance drop** with increasing horizon, and prevalent **Omission** and **Hallucination** errors—which are essential for directing future research efforts in embodied AI development.", "weaknesses": "### Main Limitations of the ENACT Benchmark (Weaknesses)\n\n* **1. Reliance on Simulation and the Sim-to-Real Gap**\n    * **Summary**: The entire ENACT dataset and evaluation are restricted to high-fidelity trajectories within the **BEHAVIOR simulator**.\n    * **Detail**: This introduces the inherent **sim-to-real gap** limitation. The model's strong performance (or failures) in the clean, deterministic physics and visuals of the simulation may not directly translate to the stochastic, noisy, and complex dynamics of a real-world robotic agent.\n\n* **2. Abstraction of States and Actions (Missing Low-Level Embodiment)**\n    * **Summary**: The benchmark evaluates understanding based on high-level **symbolic scene graphs** and **abstract action primitives** (e.g., `Open(object)`), rather than raw sensorimotor data.\n    * **Detail**: By removing the necessity to process raw pixels, continuous motor commands, and solve low-level control problems, ENACT fails to test critical aspects of true embodied cognition related to continuous planning and fine-grained sensor processing.\n\n* **3. Limitation to Sequence Reordering (Absence of Generative Modeling)**\n    * **Summary**: The core evaluation is a **sequence reordering task**, which tests recognition and temporal alignment against a limited set of choices.\n    * **Detail**: This task is less demanding than **free-form generative world modeling**, where the VLM would have to *generate* novel, unconstrained future states or action sequences from scratch, which is the ultimate goal of a robust world model.\n\n* **4. Focus on Functional Dynamics (Neglects Fine-Grained Physics)**\n    * **Summary**: The benchmark primarily assesses object-level, functional state changes (e.g., object is open/closed, object is grasped/placed).\n    * **Detail**: It may insufficiently test complex, fine-grained **physical reasoning** such as geometric constraints, stability, momentum, or soft-body dynamics, which are integral to sophisticated manipulation in unstructured environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8RGHSFxmGH", "reviewer": "Reviewer_6SYx", "summary": "The authors propose a framework called ENACT to evaluate if vision language models (VLMs) can exhibit embodied cognition. The framework uses 2 main sequence reordering tasks (forward and inverse world modelling) and data collected from the egocentric robot manipulation simulator BEHAVIOR from a diverse set of household scenarios. Authors conduct an extensive analysis on 29 VLMs (both open and closed source) on this framework and compare it with human evaluation from 3 annotators, with strong agreement reported. They also present and discuss failure modes of VLMs.", "questions": "- Have you considered discussing how more recent pretrained world models can act in embodied settings, as opposed to world modelling via VLMs?\n- In Fig 2, should N, the number of trajectories, be replaced with R, as described on line 132?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors conduct an extensive evaluation on 29 VLMs (both open and closed source), on increasing horizons lengths. They then use the most promising models (one open InternVL3.5, one closed GPT5-mini) and conduct a further set of experiments to explore the VLMs sensitivity to image realism, camera configurations and embodied biases.\n- The paper is clearly presented and easy to follow, with descriptive and detailed figures, comprehensive discussion on each dimension evaluated and a good overview of the data collection process.\n- The work offers a good overview of the limitations of current VLMs models and future directions for research.\n- The failure modes analysis in section 3.5 is really informative, highlighting the challenges of hallucinations and omissions in extending VLMs to embodied tasks.\n- The authors are happy to opensource the code and the QA dataset to benefit the research community.", "weaknesses": "- I'm unsure about the claim the authors make on VLMs being trained on vast static datasets. Recent VLMs, including those used as benchmarks in this work make extensive use of video data during training which include temporal dynamics (motion, causality, progression), as well as context across frames.\n- I would've liked to see more details about the online verifier, which is key in computing the 2 main metrics: task and pairwise accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RDEitlGrn7", "reviewer": "Reviewer_Wf5o", "summary": "The paper introduces ENACT, a benchmark for testing embodied cognition in Vision-Language Models via world modeling from egocentric interaction. Using BEHAVIOR simulator trajectories, it builds scalable sequence-reordering tasks: Forward (order future images given actions) and Inverse (order actions given images), with a semantic verifier over symbolic scene-graph deltas. Across 8,972 QAs (29 activities; horizons L=3–10), state-of-the-art VLMs lag far behind humans, degrade sharply with longer horizons, and perform better on inverse than forward. Probing shows robustness to rendering realism but sensitivity to non-human camera intrinsics and a notable right-handed bias. Error analysis reveals failures dominated by omissions and hallucinations of state changes, positioning ENACT as a scalable, diagnostic stress test for consequence-aware, long-horizon reasoning.", "questions": "The semantic verifier uses subset inclusion rules. How often do accepted predictions omit/hallucinate parts of transitions? \n\nLimiting to 11 predicates and “visible” deltas excludes many dynamics (forces, latent preconditions). Do you have a plan to include non-visible or continuous variables, and how would the verifier adapt?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. 8,972 QAs over 29 household activities, balanced lengths (L=3…10) and 11 predicate classes, with visibility checks and de-duplication—good breadth while keeping semantics precise. Also, evaluation spans 7 proprietary and 22 open-weight VLMs with unified prompts/decoding, plus a robust human baseline – this breadth makes results meaningful to the community.\n\nS2. Converting predictions to atomic state-change signatures and breaking errors into omission, hallucination, polarity inversion, predicate/entity substitution gives rare, fine-grained insight into failure modes.\n\nS3. Scalable, well-engineered data pipeline:  The DAG+DP path-counting and weighted backtracking for key-frame synthesis is technically neat and makes long-horizon QA generation tractable without manual curation.", "weaknesses": "W1 . ENACT reduces forward/inverse dynamics to sequence reordering under a symbolic action space and an online subset-based verifier. A model can succeed by learning robust pairwise visual diffs (relying on dection of local difference and adjacent changes) and text matching without learning an internal, action-conditioned dynamics model (e.g., no counterfactual rollouts, no latent transition consistency beyond local adjacency). That undermines the claim that scores reflect “world modeling” rather than discriminative sorting.\n\nW2. Several benchmarks already test temporal ordering, causal clip inference, or egocentric state changes. ENACT’s main novelty is pairing reordering with symbolic scene-graph deltas in a simulator. But it still evaluates static post-hoc reasoning over pre-rendered frames—not interactive rollouts, planning, or closed-loop control. For ICLR standards, this may be too incremental without a stronger theoretical or algorithmic contribution. Meanwhile, the “image realism” study shows no significant differences, but all variants are style transforms of the same rendered content with the same camera geometry. That cannot test the core sim-to-real generalization problem (novel textures, clutter, domain shift in object distributions, sensor noise, real lens artifacts). Thus in terms of data quality, there’s no significant advantage over previous works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)