RoboScape: Physics-informed Embodied World Model
Authors: Yu Shang
Xin Zhang
Yinzhou Tang
Lei Jin
Chen Gao
Wei Wu
Yong Li
Venue: NeurIPS 2025 spotlight ()
Keywords: Embodied World model
Physics Priors
Video Generation
Primary area: deep_learning
TL;DR: We propose RoboScape, a unified physics-informed embodied world model that jointly learns video generation and physics knowledge within an integrated framework.

Abstract: World models have become indispensable tools for embodied intelligence, serving as powerful simulators capable of generating realistic robotic videos while addressing critical data scarcity challenges. However, current embodied world models exhibit limited physical awareness, particularly in modeling 3D geometry and motion dynamics, resulting in unrealistic video generation for contact-rich robotic scenarios. In this paper, we present RoboScape, a unified physics-informed world model that jointly learns RGB video generation and physics knowledge within an integrated framework. We introduce two key physics-informed joint training tasks: temporal depth prediction that enhances 3D geometric consistency in video rendering, and keypoint dynamics learning that implicitly encodes physical properties (e.g., object shape and material characteristics) while improving complex motion modeling. Extensive experiments demonstrate that RoboScape generates videos with superior visual fidelity and physical plausibility across diverse robotic scenarios. We further validate its practical utility through downstream applications including robotic policy training with generated data and policy evaluation. Our work provides new insights for building efficient physics-informed world models to advance embodied intelligence research. Our code and demos are available at: https://github.com/tsinghua-fib-lab/RoboScape.

RoboScape: Physics-informeet Embodied World Model
Yu Shang', Xin Zhang?, Yinzhou Tang', Lei Jin', Chen Gao', Wei :u²; Yong Li
Tsinghua University Manifold AI
Abstract
World models have become indispensable tools for embodied intelligence, serving as powerful simulators capable of generating realistic robotic videos while address- ing critical data scarcity challenges. However, current embodied world models exhibit limited physical awareness, particularly modeling 3D geometry and motion dynamics, resulting in unrealistic video generation for contact-rich robotic scenarios. In this paper, we present RoboScape, a unified physics-informed world model that jointly learns RGB video generation and physics knowledge within an integrated framework. We introduce two key physics-informed joint training tasks: temporal depth prediction that enhances 3D geometric consistency in video rendering, and keypoint dynamics learning that implicitly encodes physical prop- erties (e.g., object shape and material characteristics) while improving complex motion modeling. Extensive experiments demonstrate that RoboScape generates videos with superior visual fidelity and physical plausibility across diverse robotic scenarios. We further validate its practical utility through downstream applications including robotic policy training with generated data and policy evaluation. Our work provides new insights for building efficient physics-inforie advance embodied intelligence research. Our code and demos are available tattps://gitenn wwwccomm//iaa.eeeeeeeecc tsinghua-nnnnnne fib-fillllle weeb--aa//Ree Lab/Roooocapeee
world models
Introduction
The udvancement of large language and vision models [ has demonstrated the critical role of high-quality, large-scale training data for their superior performance. However, the robotic learning is significantly hindered by the prohibitive cost of collecting real-world data 3 relies human teleoperation to acquire high-quality demonstrations. This limitation poses great challenge for scaling robotic learning and deploying agents in complex, real-world environments.
4 5. which often
World models 77, 8 9 1O 11,, which simulate environmental dynamics by predicting future states based on current observations and given actions, offer promising solution to this data scarcity problem. Such models hold significant promise for advancing embodied intelligence by generating realistic robotic data 12] and enabling scalable simulation environments [33. However, current embodied world models [13 14 [5] predominantly focus video generation, with training ob- jectives centered on optimizing the RGB pixels. While capable of producing visually plausible 2D images, they often fail to maintain crucial physical properties, such motion plausibility and spatial consistency [16]. Particularly, in robotic manipulation tasks involving deformable objects (e.g., cloth), the generated videos frequently contain artifacts such unrealistic object morphing or discontinuous motion. These limitations become particularly detrimental interaction-ric robotic scenarios, where even minor physical inconsistencies can dramatically compromise the effectiveness of learned policies.
Cooresponniig author, correspondence to liyong07 @ssinghua.cdu.cn.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The root cause lies in existing models' overreliance on visual token fitting without awareness of physical knowledge [T7 [9. To address this, we propose a physsis--innieee informed world model that jointly learns depth information and temporal keypoint consistency to implicitly encode physical constraints. Existing efforts of integrating physical knowledge into video generation fall into three categories: physics-prior regularization, physics simulator- -aase knowledge distillation, and material field modeling. Current regularization-batt based methods enforce constraints such as local rigidity [20] or rotational similarity (21 on Gaussian splatting (GS) features or 3D point clouds. However, these methods are limited to narrow domains like human motion [22] or rigid-body dynamics [20), hindering generalization to diverse robotic scenarios. Another line of work employs physics simulators to extract motion signals or semantic maps as conditions to guide video generation models [23 24 25| 26]. Although this approach yields reliable physical priors, the resulting cascaded pipeline introduces excessive computational complexity, hindering their practical deployment. There have been some recent works trying to enhance the physical simulation via material field modeling 27 28 However, such methods are confined to object-level modeling and are hard to apply to scene-level generation.
To overcome these limitations, we propose RoboScape, physics-informet world model based on a multi-task learning auto-regressive framework to generate visually realistic and physics-adherent robotic videos. Specifically, our approach incorporates physics knowledge through two auxiliary physics-informet supervision tasks within the world model itself to alleviate heavy external model cascading. First, to empower the model with 3D spatial physical understanding, we augment the RGB prediction backbone with a temporal depth prediction branch and inject the learned depth features into the RGB prediction to enhance spatial awareness. Such synergistic learning of temporal depth maps enables the model to implicitly acquire 3D) scene reconstruction priors rather than merely fitting 21) RGB images. Second, we introduce adaptive keypoint dynamics learning task to address unrealistic object deformation and implausible motion issues. To achieve this, we first perform dynamic keypoint sampling automatically identify regions with significant motion (typically involving robots and interacting objects), then encourage temporal token consistency for these keypoints across frames. Through this, the model effectively captures the deformation properties and motion behaviors of objects, implicitly encoding material properties (e.g., rigidity and softness) through self supervupe keypoint consistency, eliminating the need for explicit material modeling. Although some recent world models (29) (30 also explore joint RGB-depth prediction, their learning remains constrained at the image level, failing capture the fine-graine greeggraied motion dynamics and object deformation details that are crucial for robotic manipulation scenarios. Furthermore, these approaches exhibit a performance trade-off, where gains in 3D perception come at the cost of reduced RGB prediction fidelity. Differently, our model captures global spatial knowledge through learning temporal depth dynamics, while modeling local object deformation and motion characteristics via learning temporal keypoint tracking.
regg.ty
We conduct comprehensive experiments to evaluate our world model from three aspects: video gener- ation quality, robotic policy learning using synthetic data, and robotic policy evaluation. RoboScape achieves state-of-the the-art performance in both RGB and depth prediction accuracy, achieving supe- rior balance between these metrics compared to existing world model baselines. Additionally, we validated that synthetic data from our world model consistently improves the performance of robotic policy models within a simulated robotic environment including Diffusion Policy [31) and piO (2). confirming the model's practical utility for robotic learning. Finally, our model can also serve as reliable policy evaluator, with assessment results showing strong correlation with ground-truth simulator outcomes, confirming our model's capability to accurately model the physical world.
In summary, the main contributions of the paper are as follows:
We propose RoboScape, a physics-inforee informed embodied world model that unifies RGB video generation, temporal depth prediction, and adaptive keypoint tracking in a joint learning framework, achieving both high visual fidelity and physical plausibility.
We design an automated robotic data processing pipeline with physical prior information labels. Trained on the carefully curated large-scale, high-quality dataset, our model achieves SOTA performance on visual quality, geometric accuracy, and action controllability.
We demonstrate the practical utility of RoboScape on downstream applications including robotic policy training and evaluation. Extensive experimental results demonstrate its effectiveness in accurately modeling embodied environments, validating its potential for advancing real-world robotic deployment.

---

Levell:
fel.
BOll Sequence
Depth Gentration
Camera Houndory Detection
Key From Labeling
Action Difficuly Ranking
Kchar
stane
noc
Action Sequence
"pict battee
"Jlse Astion Clipping
office
indarity
Keypoint Tracking
Clip Quality Fillering
Scctes Grouping
Data Collecting
Physical Property Amnutaling
Video Slicing
Clip Filtering
Clip Categortzacion
Figure I: Illustration of the proposed robotic data processing pipeline with physical priors annotation.
2 Methodology
2.1 Problem Formulation
In this work, we focus on robot manipulation scenarios and learn an embodied world model fa as a dynamics function that predicts the next visual observation 0,4-1 given past observations Oiz and robotic actions a
O1+1 ~ Jo(O 01st, a13),
where Hx is video frame and E R is k-degree continuous action control vector.
2.2 Robotic Data Processing Pipeline with Physical Priors Annotation
Learning a physics-informed embodied world model requires high-quality dataset covering high- resolution RGB and depth sequences, action sequences that control the robot, and state sequences that the robot executes. In this section, present our data processing pipeline to construct multi-modal embodied dataset with physical priors based on AGIBOT-World dataset [6. as shown Fig.
Physical Property Annotating based on Depth Generation and Keypoint Tracking. Integrating explicit physics constraints remains significant challenge for current video generation baased world models. To address this, our approach concentrates on two crucial, expreslleepeessible physics constraints highly relevant to robotic manipulation: temporal depth consistency and keypoint mo- tion trajectories. These features can be efficiently extracted using off-the-shelf pretrained models, enabling enhanced generalization while maintaining practical feasibility. Specifically, we utilize Video Depth Anything [33 to generate the depth map sequence of the video. Furthermore, apply SpatialTracker (34] as the keypoint tracking model to sample the keypoint and track their trajectories.
Video Slicing based on Camera Boundary Detection and Action Semantic. The original videos have different attributes, such as lengths and resolution, with camera jumps or editing traces, and video may contain multiple action semantics. Thus, we slice the video into clips with normalized attributes, consistent motion, no camera jumps, and single action semantics. Specifically, we use T2a (35 to perform camera boundary detection and use Intern-VI. 36 to generate the action semantic of a specific clip.
Clip Filtering based on Key Frame and Clip Quality. The generated clips are highly heterogeneous terms of quality, semanties, and presentation form. To ensure the validity and adaptability of the training data, we introduce a clip filtering mechanism including: (1) using FlowNet (37] to filter out clips with indistinct motion and disordered movement patterns, and (2) using Ln (36) to label the key frame of the clip and filter out the frames without explicit relationship to the key frame.
Clip Categorization based Action Difficulty and Scenes. In this stage, categorize and reorganize the dataset based on action difficulty and clip scenes to support the curriculum learning strategy 38 which trains the world model from easier harder tasks.
2.3 RoboScape: A Physics-informed Embodied World Model
RoboScape is designed to achieve frame-level action-controllable robot video generation, enabling interactive future frame prediction. At its core, we adopt an auto-regressive Transformer-basel framework that iteratively predicts the next frame based on historical frames and the current robot

---

Adaptive Sampler
Keypoint Dynamics Learning
P
Minimize [ps 3p
Keypoint Trajectory
RGB Prediction
Regularization
Tokenizer
Visual
STBLOR
STBLock
8₂
St
Alignment
Frame Sequence
Current-time RGB Frame
Depth Feedback
Action Sequence
Action Embedding
Tokenizer
Visual
STBLock
STBLock
Alignment
Zt
Temporal Depth Prediction
Current-time Depth Frame
Depth Frame Sequence
Overview of the chysics-informen world model, where physical knowledge is integrated Figure 2:
through joint learning of temporal depth estimation and adaptively sampled keypoint dynamics.
action. To enhance the physical plausibility of generated videos, we introduce two physics-informed auxiliary training tasks in addition to the normal RGB image prediction: (1) temporal depth prediction. which encourages global geometric consistency across frames, and (2) adaptively sampled keypoint dynamics learning, which captures the motion and deformation details of local dynamic objects. The whole pipeline is illustrated Figure 2 Joint training with these physics- awwre regularizers provides an efficient approach to embed physical priors into world models, significantly reducing the reality gap between generated videos and real-world dynamics.
Video Tokenization. To enable efficient video generation, we leverage MAGVIT-2 39 to compress raw RGB frames O E x into discrete latent tokens $1.7 IR XW xD where H' = H//a and W" W/a denote the reduced spatial dimensions (a being the downsampling factor), and D represents the latent channel dimension. Similarly, we tokenize temporal depth maps XD
d1.r Xx into latent depth tokens 21.T
_R T'
Geometry Consistency Enhancement via Temporal Depth Prediction. While RGB-based video generation has achieved remarkable progress, it often suffers from inconsistent 3D geometry due to the lack of explicit spatial constraints. Considering that inter-frame depth variations encode crucial 3D structure information, we propose to jointly learn temporal RGB and depth information, leveraging depth features as geometric constraints to ensure spatially coherent video generation. For uniform modeling, we also utilize MAGVIT-2 as the visual encoder for depth maps. We convert depth maps to a threc-channel RGB format to ensure compatibility with MAGVIT-2. For joint prediction of both RGB and depth images, propose dual- oo-autoregressive Transformer (DCT). Each branch consists stack of 32 spatial-temporal Transformer Transfssfoomm blocks, which implement causal attention mechanism in the temporal attention layers for generation causality, and bidirectional attention in the spatial attention layers to enable full context modeling.
At timestep f, the model processes historical latent tokens through parallel branches FROB and Depphh, conditioned on learned action embeddings C1a-11 E [[t-1)xxxxxx) E, (ai:-1)) and position
embeddings C1-1-1 E (t= xW where &, denotes the robot action encoder. The auto- regressive prediction of each branch is formulated as:
FRGB FRGB(SI::-1 Cii-1 el.--1)
(2)
i = Dephh (Zi:t1 @ C1a-1 2 @1a-11),
where @ denotes element- wise addition with broadcasting. Empirically, we find that simple additive fusion provides effective action control while maintaining model efficiency.

---

To inject depth predictions as physical priors into the RGB branch and enhance spatial structure fidelity of rendered videos, we introduce cross-branch interaction pathways. Specifically, at each ST-Transformer block 1, we project the depth branch's intermediate features dopph and fuse them additively with the corresponding RGB features:
BRGB = hRGB + W (hilep)))
(3)
where W is learnable linear projection layer. This hierarchical feature fusion enables the RGB branch to maintain precise geometric structure while generating photorealistic video frames. Both RGB and depth branches are optimized using the cross-entropy loss of tokens:
CRGB Σ
St log p(:)),
DDpph
Σ
Ee log p(2)).
Implicit Material Understanding via Keypoint Dynamics Learning, Modeling physically plausible object deformations and motions in robot manipulation scenarios remains challenging for RGB-base world models, as material properties (c.g., rigidity, clasticity) cannot be effectively learned through RGB pixel fitting alone. While physics engines provide accurate simulations, their computational expense and scene-specific constraints limit practical applicability. To tackle this, propose keypoint-induced material learning approach, with the insight that physical material understanding can emerge from self-supervised tracking of contact-driven keypoint dynamics. For example, when robot places an apple into a plastic bag, accurately capturing the motion of keypoints on the deforming bag implicitly captures the material properties. This method be integrated naturally with video generation frameworks while maintaining strong generalization capabilities.
Specifically, for each video V. we utilize SpatialTracker (34) densely sample No keypoints in the initial frame and track their temporal coordinate trajectories across T frames, yiclding Tdense = {(p..... p..' )) 1 where the element P E R represents its coordinates in the tokenized feature map of frame f. Rather than relying on costly segmentation masks to identify contact regions and guide keypoint sampling, we observe that the most informative keypoints are empirically characterized by large motion magnitudes. Thus, we adaptively select the top-K most active keypoints based on their motion magnitudes M, ET-1 ||p pillz, Vi L...,NN producing the sampled trajectory
set Teample {pp....... .... p. /-
To enhance the keypoint dynamic learning, we encourage temporal consistency between the visual tokens of sampled keypoints by aligning all frames the initial frame (1 1) through the following loss:
K
T
KKeypoint
(ps)
$1 (p
(T 1)K
where (pt)) E I. denotes the i-th keypoint-locatee predicted token at frame t.
Furthermore, we observe that these dynamically active keypoint regions often exhibit higher token emoor due their complex motion patterns. To address this, we propose keypoint-guidee attention mechanism that adaptively enhances token learning in regions intersected by keypoint trajectories. Specifically, we compute spatiotemporal attention map A E x''xH'''W' with each element defined as:
(t, E Teamples
A1
(6)
otherwise,
where 7 is hyperparameter controlling the importance weight. The attention-augmentte training objective is formulated as:
Σ A sp 1og.(S:).
CAttention
Physics-informed Joint Training Objectives. By integrating the above designs, train unified physics-aware world model through multi-task learning, with the final objective formulated as:
LRGB + X DDephh + >2 + AgL.nmetion L
(8)
where A2, Ag E IR+ are tunable coefficients balancing the loss terms.
5

---

Time
1-0 (condition)
1-1
t=2
1-3
t-5
1-6
t=7
1-8
Prediction
GT
Prediction
Prediction
GT
Figure 3: Qualitative results visualization of our model (only the subsequent frames are shown). More results can be found in the appendix.
3 Experiments
In this section, we begin by detailing our experimental protocol (Section 3.1), including the dataset statistics, baseline information, and the implementation of our model. We then evaluate our model from three aspects: video quality evaluation (Section 3.2), robot policy learning with synthetic data (Section 3.3), and robotic policy evaluation (Section 3.4).
3.1 Experimental Settings
Dataset Statistics. In experiment,
50,000 videos extracted from the AgiBotWorldd-ert dataset [40], covering 147 tasks and 72 skills. We concatenate the end position, end orientation, and effector position of the embodiment the action sequence. Our dataset comprises approximately 6.5M training clips and 1 1.2K test clips.
Baselines. We compare model with four advanced baselines, including both embodied world mod- els (IRASim |H and iVideoGPT [3]) and general world models (Genie [4] and Cog VideoX (42)).
Due to unavailable training codes in some recent works (29 30] these methods excluded from direct comparison. Details of baselines are presented the appendix.
Implementation Details. We preprocess videos by extracting 16-frame clips sampled at 2Hz, yielding approximately 6.5 million training clips. The model trained for epochs using the following hyperparameters: A 1, A2 0.01, A 1, and 5. Training completes in approximately 24 hours on a cluster of 32 NVIDIA A800-SXM4-80M GPUs. During inference, we use the first frame as conditional input to autoregressively predict the subsequent 15 frames.
3.2 Video Quality Evaluation
We evaluate video generation quality through three key dimensions: appearance fidelity, geometric consistency and action controllability. The details the six used metrics are as follows:
PSNR: It measures pixel-level reconstruction accuracy between generated and ground-truth frames.
LPIPS: It assesses perceptual quality using visual feature similarity.
AbsRel: It computes relative depth estimation errors.
81 02: They evaluate depth prediction accuracy at different precision levels.

---

0.22
0.50
0.23
0.45
LPWN
0.18
Abrotel
0.16
0.35
0.32
18
0.30
5400
5400
240
Size
Model Size
Model Size
0.700
0.06
0.01
PSWR
0.025
33 0.82
0.78
0.559
0.76
13
12:4
1204 Model Size
544
Model Size
Modil Size
Sohericapes
RencSc-pe-M
EogVideoX
Figure 4: Model scaling law of RoboScape.
Table 1: Quantitative comparison of our model and baselines with independent runs.
Appearance Fidelity
Geometric Consistency
Action Controllability
Method
LPIPS (U)
PSNR (1)
AbsRcl
8 (?)
82 (1)
APSNR (f)
IRASim
0.66330.0055
11.382+0.122
0.641#0.030
0.508=0.014
0.701#0.026
0. 160+0. 122
iVideoGPT
0.501
16.23420.038
0.74920.018
0.36420.012
0,586#0.010
0, 0.148+0.06)
Genie
0.165+0.002
19.875+0.064
0.44640.014
0.545#0.011
0.778+0.016
1.86840 090
VidexX
0,202=0.008
17.957=0.153
0.508+0.026
0.594#0.017
0.739+0.016
RoboScape
0.12820.002
21.730=0.120
0.37820.026
0.60820.011
0.81420.013
3.442:0.139
APSNR: It quantifies output sensitivity to action condition, with higher values indicating better action control ability.
We present some generation results in Figure 3 where we predict future frames conditioned on an initial frame and robot action commands (we visualize frames while the model supports long- horizon rollouts). The visualizations demonstrate that our model effectively simulates realistic robot manipulation scenarios, with generated sequences showing strong similarity to ground truth observations. Notably, our approach successfully handles deformable object interactions, evidenced by the cloth-dragging sequence where the generated deformations accurately follow physical laws and capture material properties.
As shown in Table we conduct comprehensive comparisons with four advanced baselines: two embodied world models (IRASim and iVideoGPT) and two general world models (Genic and CogVideoX( Our model consistently outperforms all baselines across six evaluation metrics, demon- strating its superior capability in video prediction for robotic scenarios. Detailed analysis reveals that while Cog VideoX can generate high-quality videos, its inability to follow action commands leads to substantial deviations in future frames. The two embodied world models are not good at motion learning when conducting long-term generation, thus receiving poor metrics. Our model's novel integration of keypoint dynamics learning effectively addresses these limitations, simultaneously achieving high-fidelity visual generation and superior action controllability.
We further conduct ablation studies to demonstrate the complementary benefits of our two core components: temporal depth learning and keypoint dynamics learning. The results are shown in Table 2 The quantitative results reveal that both components contribute significantly to overall performance; removing either one leads measurable degradation across different metrics. The depth learning primarily preserves geometric consistency of moving objects, and the keypoint learning proves essential for maintaining both visual fidelity and action controllability. We provide a case study in Figure 6 It can be seen that the missing of temporal depth learning will lead to geometric distortions moving objects, while the absence key-point dynamics learning results in unreal motion patterns. These findings collectively validate the necessity of our key designs.
7

---

Table 2: Ablation study of our key designs of physics prior injection with 5 independent runs.
Method
LPIPS (J)
PSNR (1)
AbsRel (1)
61 (T)
02 (7)
APSNR (
whole model
0.12820.002
21.730=0. 120
0.378#0.026
0.608=0.011
0.81420.013
3,442#0. 139
w/o depth
0. 0.12640.00
21.885=0.046
0.408+0.010
0.560=0.012
0.789=0.022
3.514=0.023
w/o kcypoint
0.128=0.001
21.63400044
0,346+0.012
0.637=0.012
0.848=0.012
2.953=0.036
w/o depth keypoint
0.130=0.001
21.47740.029
0.371#0.012
0.598=0.018
0.800#0.009
1.945=0.054
0.20
LSP5
0.30
Absitel
0.36
0.40
0.37
0005
3000
6005
1200
3032
6200
1000
3030
6000
Data Scale
Data Scale
Data Stale (K)
3.3
0.60
0.66
0.64
0.62
0.59
0.5
(K)
RoboScape-5
RoboSccae--M
RoboScape-L
Genie
OngVideoX
Figure 5: Data scaling law of RoboScape-S.
We also investigate the scaling behavior of RoboScape in terms of both model and data scales, As we evaluate three model variants-Roooocappe
shown in Figure 4 and RoboScape-L (544M) and observe clear scaling law: all six evaluation metrics improve significantly as model capacity increases. In addition, we study the impact of data scale by training RoboScape-S on 1,000K, 3, 3,000K, and 6,000K clips (Figure 55. While increasing data size consistently enhances visual quality and action controllability, geometric accuracy exhibits marginal improvement even slight degradation. We find that this is because smaller datasets encourage overfitting to the final frame of conditional inputs, artificially inflating geometric metrics without generating meaningful temporal dynamics. Despite this, the overall trend confirms that more training data leads to better model performance.
(34M), RoboScape-M (13IM),
3.3 Robotic Policy Learning with Synthetic Data
We validate our world model's utility by generating synthetic robotic video data for downstream policy learning based on Diffusion Policy (DP) 31 and m0 (32]. The actions and initial observations for our synthetic data are directly drawn from the raw RoboMimic and LIBERO datasets. Through controlled experiments with progressively adding synthetic data, we systematically measure the impact of generated data on policy learning performance. The results are shown in Table 3 "Real data" mentioned consists of original videos with their corresponding action annotations from the raw dataset. For "Synthetic Data", we use our world model to generate the videos based on actions and initial observations.
In the experiments on the Robomimic Lift task [43] DP trained for 10k steps with only generated data achieved nearly the same performance as DP trained with real data. Notably, the policy success rate exhibited consistent improvement with increasing synthetic training data, highlighting the effectiveness of our model. We further validated our approach using the mo [32] model on the challenging LIBERO 44] task suite. These tasks present three key challenges beyond the Robominic Lift environment: (1) complex multi-object manipulation requirements, (2) cluttered scene configurations. and (3) extended action sequence horizons. Therefore, we employ small amount of real data (200 traeectories) as a training warm-up. Remarkably, when training mo policies with increasing generated data, the model performance achieves gradual improvement. These results

---

Table Results of policy learning with DP on Robomimic task and x0 on LIBERO tasks.
DP on Robomimic lasks
T0 on LIBERO tasks
# Symthetic Data
Success Rale
#Synthetic Data
Spatial
Object
Goal
10
Average
50
40%
200
77.6%
81.8%
71.0%
36.0%
66.6%
100
77%
400
79.4%
85.2%
74.6%
46.2%
71.4%
150
84%
600
81.6%
86.0%
78.0%
51.8%
74.4%
200
91 %
800
84.6%
89.0%
82.8%
60.0%
79.1%
Real (200)
92%
Real (200)
77.2%
79.8%
68.8%
34.8%
65.2%
Time
too (condition)
t=
#=2
t=3
t=4
1=5
teB
1=7
te
Prediction
Prediction (wlo keypoint)
Prediction (wlo depth)
GT
Figure 6: Effect of the physics knowledge learning. Omission of temporal depth learning leads to geometric distortions moving objects, while the absence of key-point dynamics learning results in unreal motion patterns.
demonstrate our model's capability to generate physically plausible trajectories even for demanding, long-horizon manipulation scenarios.
3.4 Robotic Policy Evaluation
In this section, we investigate whether our world model can act as a policy evaluator for different robotic policies. In policy evaluation, the world model acts an environment that receives policy- generated action sequences and predicts subsequent observations in rollout manner. Policy quality is then assessed by checking success rates in the predicted videos. Here compare IRASim,, IVideoGPT, and our model as the policy evaluator and use Diffusion Policy [31] as the policy model. Specifically, train the policy on the Robomimic Lift task 43] using 200 trajectories and save the policy every 250 epochs until it is fully converged. Then we post-train the world model and evaluate the policy in both the ground-truth simulator and the world model by 100 runs. The success signal of each run can be directly given by the simulator, while requires manual judgment when the policy interacts with the world model. The generated videos from all models were presented to participants in a randomized, blind order, with no model identifiers displayed. Participants were instructed judge whether the task depicted in each video was successfully completed. This setup ensures objectivity and impartiality. Afterwards, we calculate the Pearson correlation and R²2 between different world models and the ground- trouhd-truth simulator. The results in Figure 7 show that the Pearson correlation of our model is 0.953, while the correlation of other models is rather low, indicating that our world model can be utilized better policy evaluator.
4
Related Work
4.1
World Model
World models learn representations of environmental states through neural networks, enabling the prediction of future states based on current observations and actions [45] Recent advances in world
models primarily leverage video generation techniques, with applications spanning three key domains
including autonomous driving [46] 47 48 49
50 51 52), embodied intelligence [53] 14 15). and

---

Bearson -0.134
18-
-0.195
0.953
R 0.018
0.900
Succids Flate Diher Vooe
Roto Modee
0.8
0.7
64
32
62
24
66
66
10
0.2 03
24
66
6.1
0.9
10
022
0.3
044
0.3
6.7 3.0
Success RatagGround
Success Ratu Samolaorr
Success RateegGrrundl thuth Simu
(a) IRASim
(b) iVideoGPT
(c) RoboScape
Figure Correlation between the success rate different world models and the ground-truth simulator. Each point represents policy, and the trained epochs are shown above the point.
gaming 14
The dominant modeling approaches fall into two main categories: diffusion models and autoregressive models. Diffusion models, such as DiT [57], generate sequences through gradual denoising process and are well-suited for producing diverse and short-term consistent visual content. Autoregressive models, such as Genie[4]], reconstruct sequences via masking mechanisms and demonstrate superior efficiency and controllability. Compared to diffusion-based approaches, autoregressive methods offer advantages inference speed and training stability. Our work utilizes masked autoregressive models with physical information injection, aiming to build efficient and interactive world models for embodied intelligence.
55
56 48
Physics-aware Generative Model
Recent advances in video generation have increasingly focused improving the modeling of physical properties [1]] Current methods in this area can roughly divided into explicit and implicit physical modeling. Explicit methods incorporate physical information by learning explicit textures and material representations 28 30 In contrast, implicit methods mainly embed physical knowledge
into models via training loss terms 20 by using generative models to jointly generate RGB videos and other physical representations
29 58 These approaches aim to enhance physical understanding
through data-driven approaches rather than predefined physical rules. Currently, there's 's much room for existing embodied world models to enhance the integration of physical knowledge into video generation. To advance this field, introduce physics-informed embodied world model that jointly learns RGB video generation, temporal depth prediction, and keypoint dynamics within a unified framework, achieving both high visual fidelity and physical plausibility.
5 Conclusion and Future Work
In this work, we propose RoboScape, a physics- informed embodied world model that efficiently integrates physical knowledge into video generation through thysics-inspired multi-task joint training framework, eliminating the need for cascaded external models such physics engines. By incorporating temporal depth prediction, our model learns the 3D geometric structure of scenes, while dynamic keypoint learning enables implicit modeling of object deformation and motion patterns. Extensive evaluations demonstrate that our approach outperforms baseline methods in video generation quality, synthetic data utility for downstream robotic manipulation policy training, and effectiveness as policy evaluator. In the future, we plan to combine the generative world model with real-world robots to test performance further.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key Research and Development Program of China (N0.2024YF333376603..
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[2] Gemini Team, Rohan Anil, Sehastian Borgeaud, Jean- SannBBppistt Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 11805, 2023.
[3] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng. Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning, pages 1723-1736. PMLR, 2023.
[4] Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, al. Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration O. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892-6903. IEEE, 2024.
[5] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: large-scale ni-th--wld robot manipulation dataset. arXiv preprint arXiv:2403. 12945, 2024.
[6] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding. Siyuan Feng, Shenyuan Gao, Xindong He, Xu Huang. Shu Jiang, al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint wwXiv:2503.0666) 2025.
[7] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Ad-
vances in neural information processing systems, 31, 2018
[8] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint 4rX1v:2310.06114 1(2):6 2023.
[9] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Erie Luhman, al. Video generation models world simulators. OpenA/ Blog, 1:8, 2024.
[10] Jingtao Ding, Yunke Zhang, Yu Shang, Yuheng Zhang, Zefang Zong, Jie Feng, Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, et al. Understanding world or predicting future? comprchensive survey of world models. ACM Computing Surveys, 58(3):1-38, 2025.
[11] Yu Shang, Yinzhou Tang, Xin Zhang, Shengyuan Wang, Yuwei Yan, Honglin Zhang, Zhiheng Zheng, Jie Zhao, Jie Feng, Chen Gao, et survey of embodied world models.
[12] Lirui Wang, Kevin Zhao, Chaoqi Liu, and Xinlei Chen. Learning re al-world action-video dynamics with heterogeneous masked autoregression. arXiv preprint NXXi::2502.04296 2025.
[13] Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. Advances in Neural Information Processing Systems, 37:68082-66 181199 2024.
[14] Fangqi Zhu, Hongtao Wu, Song Guo, Yuxiao Liu, Chilam Cheang, and Tao Kong. Irasim: Learning interactive real-robot action simulators. arXiv preprint arXiv:2406. 1136066.4540 14540, 2024.
[15] Niket Agarwal, Arslan Ali, Maciej Bala. Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03377. 2025.
[16] Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: physical law perspective. arXiv preprint arXiv:2411.02358 2024.
11

---

[17] Minghui Lin, Xiang Wang, Yishan Wang, Shu Wang. Fenggi Dai, Pengxiang Ding, Cunxiang Wang, Zhengrong Zuo, Nong Sang, Siteng Huang, et al. Exploring the evolution of physics cognition in video generation: A survey. arXiv preprint arXiv:2503.21765, 2025.
[18] Xuyang Guo, Jiayan Huo, Zhenmei Shi, Zhao Song, Jiahao Zhang, and Jiale Zhao. T2yphysbench: A first-principles benchmark for physical consistency in text-to-video gen- eration. arXiv preprint arXiv:2505.000 00337, 2025.
[19) Fanqing Meng. Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense- based benchmark for video generation. arXiv preprint ar arXiv:2410.05363 2024.
[20] Yunuo Chen, Junli Cao, Anil Kag, Vidit Goel, Sergei Korolev, Chenfanfu Jiang, Sergey Tulyakov, and Jian Ren. Towards physical understanding in video generation: A 3d point regularization approach. arXiv preprint 2.Xx:22502.03639, 2025.
[21] Gaurav Rai and Ojaswa Sharma. Enhancing sketch animation: Text-to-vider diffusion models with temporal consistency and rigidity constraints. arXiv preprint urXiv:2411.19385 2024.
[22] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan, Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800-809, IEEE, 2024.
[23] Tianyi Xie, Yiwei Zhao, Ying Jiang, and Chenfanfu Jiang. Physanimator: Physics-guided generative cartoon animation. arXiv preprint 000::2501.16500 2025.
[24] Jiaxi Lv, Yi Huang, Mingfu Yan, Jiancheng Huang, Jianzhuang Liu, Yifan Liu, Yafei Wen, Xiaoxin Chen, and Shifeng Chen. Gpt4motion: Scripting physical motions in text-to-video generation via dlender-oriented gpt planning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1430- 1440, 2024.
[25] Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics-grounder image-to- tonge-to-video generation. In European Conference on Computer Vision, pages 360-378. Springer, 2024.
[26] Xiyang Tan, Ying Jiang. Xuan Li, Zeshun Zong, Tianyi Xie. Yin Yang. and Chenfanfu Jiang. Physmotion: eyysics-grouned dynamics from a single image. arXiv preprint arXiv:2411.17189, 2024.
[27] Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. In European Conference on Computer Vision, pages 388-406. Springer, 2024.
[28] Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, and Yueqi Duan. Physics3d: Learning physical properties of 3d gaussians via video diffusion. arXiv preprins arXiv:2406.043388 2024.
[29] Aether Team, Haoyi Zhu, Yifan Wang. Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang, et al. Aether: Geometric-aware unified world modeling. arXiv preprint arXiv:2503. 18945, 2025.
[30] Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. Tesseract: Learning 4d embodied world models. arXiv preprint arXiv:2504.20995, 2025.
[31] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273888
2023.
[32] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. To: A sasiion-anguaae-acttinn flow model for general robot control. arXiv preprint ar
2024.
[33] Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang. Jiashi Feng, and Bingyi Kang. Video depth anything: Consistent depth estimation for super-long videos. arXiv preprint arXiv:2501.12375. 2025.
12

---

[34] Yuxi Xiao. Qianqian Wang. Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei
Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20406-20417, 2024.
[35] Tomás Soucek and Jakub Lokoc. Transnet v2: An effective deep network architecture for fast shot transition detection. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11218-12 11221, 2024.
[36] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic risual-linguise tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2-1988 2024.
[37] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages
2015.
[38] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555-4577 2021.
[39] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion-tokeenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023.
[40]
ggiBot-WorddCoontrbuooe Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng. Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan NIu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang. Wenhao Wang. Dafeng Wei, Chengen Xie. Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang. Shukai Yang. Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, and Jianchao Zhu. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. arXiv preprint arXiv:2503. 06669, 2025.
[41] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[42] Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint arXiv:2408. 260702 2024.
[43] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkamni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.00 03800328, 2021.
[44] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776-44791, 2023.
[45] David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv: 10122, 2018.
[46] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards
world models for autonomous driving. In European Confer- ence on Computer Vision, pages 55-72. Springer, 2024.
[47] Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Gen- erative end-to-end autonomous driving. In European Conference on Computer Vision, pages 87-104. Springer, 2024.
[48] Haoxuan Che, Xuanhua He, Quande Liu, Cheng Jin, and Hao Chen. Gamegen-x: Interactive open-world game video generation. arXiv preprint 241.022411.00769, 2024.
13

---

[49] Jingcheng Ni, Yuxin Guo, Yichen Liu, Rui Chen, Lewei Lu, and Zehuan Wu. Maskgwm: generalizable driving world model with video mask reconstruction. arXiv preprint arXiv:2502.1 1663, 2025.
[50] Yuqi Wang. Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14749-14759, 2024.
[51] Shenyuan Gao, Jiazhi Yang. Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprini arXiv:2405. /7398, 2024.
[52] Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523. 2025.
[53] Chaoqi Liu Xinlei Chen Lirui Wang, Kevin Zhao. Learning robotic video dynamics with heterogeneous masked autoregression. Arxiv, 2025.
[54] Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating games with generative interactive videos. arXiv preprint arXiv:2501.03322 2025.
[55] Haoran He, Yang Zhang, Liang Lin, Zhongwen Xu, and Ling Pan. Pre-trained video generative models as world simulators. arXiv preprint arXiv:2502.07825, 2025.
[56] Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, and Xin- gang Pan. Worldmem: Long-ter term consistent world simulation with memory. arXiv preprint a504v:2504. /2369, 2025.
[57] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[58) Dingkang Liang, Dingyuan Zhang, Xin Zhou, Sifan Tu, Tianrui Feng, Xiaofan Li, Yumeng Zhang, Mingyang Du, Xiao Tan, and Xiang Bai. Seeing the future, perceiving the future: A uni- fied driving world model for future generation and perception. arXiv preprint arXiv:2503. 135877 2025.
[59] Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models understand physical principles? arXiv preprint arXiv:2501.0900 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper proposes physics-informee embodied world model, which is reflected in the methodology and experiment validation part.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This work does not invlove theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details in the experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provide an anonymous code repository in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuiu wwwticccguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See the "implementation details" section in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide independent repeated experimental results in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See the "implementation details" section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our video generation experiment, in every respect, follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: No risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We have added citations and footnotes to note the source of used codes and datasets.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLM usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur/a: /0tpp////.////offeeeer/////////0 for what should or should not be described.
21

---

A Broader Impacts
Our physics-informed world model offers positive impacts by advancing robotic learning: it generates high-fidelity synthetic data with inherent physical plausibility, which drastically reduces reliance on costly real-world data collection and improves sim-to-real transfer for applications healthcare, disaster response, and industry. While this computational efficiency lowers research barriers, the high fidelity necessitates ethical consideration. Specifically, the generated content could misused to create deepfakes misrepresent safe procedures. To mitigate these risks, future work should focus on training fake detection models to identify synthetic content and incorporating anomaly detection mechanisms to flag prevent the generation of videos depicting unsafe or non-standard robot behaviors, ensuring responsible deployment in safety-critical domains.
B Limitations
While our physics-informe world model achieves significant gains in physical consistency, we ac- knowledge the following shortcomings and inherent assumptions: (I) Our current physical constraints focus primarily on geometric (depth) and kinematic (keypoint dynamics) properties. This leaves out broader critical information such as dynamic force interactions. Furthermore, our model relies on the empirical assumption that keypoint dynamics implicitly capture material properties, which lacks theoretical guarantee. (2) Our model's inference is currently limited to 48 48-frame rollout. This computational constraint makes it challenging to simulate and evaluate very long- aag-horizon tasks (e.g.. complete cloth folding), where we have observed that cumulative error can lead physically implau- sible outcomes. (3) Generalization to more diverse embodiments, such as quadrupeds or humanoids, is currently limited by the lack of readily available, high-quality, multi-domain datasets. Our current validation is primarily confined to single- and dual-arm manipulation settings. Future work will be dedicated overcoming these limitations by incorporating richer physical laws, exploring more efficient hybrid architectures, and expanding our training to more diverse embodiment data.
C Baseline Details
We provide details the compared baselines as follows:
DīT-based robotic video generation model, capable of generating videos conditioned on robot actions and trajectories.
IRASim:
iVideoGPT: An auto-regressive interactive world model that takes the current video frame observa- tion and action as input predict the next frame while simultaneously estimating the reward signal for robotic operations.
Genie: A foundation world model trained through unsupervised learning on massive video data. We implement it with reproduced open-source repository [
Cog VideoX: An advanced DiT-based text-to-video generation framework, with superior perfor- mance in prompt-driven video generation.
D Supplemented Evaluation toward Physics Correctness
In this section, we provide a comparison regarding the physical correctness of our model and baselines. While existing physics benchmarks for video generation primarily designed for general text-to- video models and aren't 't directly applicable to our embodied world model (which is driven by robotic actions), we've ve identified that key metrics from these benchmarks can be utilized for our evaluation. Specifically, we utilize four metrics from the Physics-IQ benchmark (59) that are particularly effective in assessing the physical realism of motion plausibility and object deformation, detailed follows:
Spatial IoU: Measures "Whether the location where an action happens is correct".
Spatiotemporal IoU: Measures "Where does action happen and whether it occurs at the right time".
22

---

Table 4: Comparison of world models on physics correctness.
Model
Spatial IoU (t)
Spatiotemporal IoU (1)
Weighted Spatial IoU (1)
MSE ())
IRASim
0.2431
0.1255
0.2081
0.0655
Genie
0.6429
0.3420
0.6082
0.0397
Cog CogVideoX
0.6523
0.2058
0.4675
0.0943
RoboScape 0.7573
0.4454
0.7023
0.0184
Table 5: Policy success rate using generated data from different world models on LIBERO benchmark.
Model
LIBERO-Spatial
LIBERR--Oject LIBERO-Goal LIBERO-10
Average
Only real data
77.2%
79.8%
68.8%
34.8%
65.2%
IRASim
72.8%
77.4%
75.2%
34.0%
64.8%
Cog VideoX
81.0%
79.6%
74.4%
44.2%
69.8%
RoboScape
84.6% %
89.0% %
82.8%
60.0%
79.1 %
Weighted Spatial IoU: Measures "Where and how much action happens".
MSE: Measures "How objects look and interact" at pixel-level.
The results shown in Table demonstrate our model's superior performance in terms of physical correctness.
Supplemented Results of Policy Learning
We compare training the piO model using data synthesized baseline world models and model. For fair comparisons, both Cog VideoX and IraSim were fine-tuned on the LIBERO dataset, with each model generating 800 synthetic data points. 5 the experimental results across various task subsets in the LIBERO environment, indicating the superiority of our model in generating synthetic data for VLA training compared baselines.
F
More Visualization Results
F1
Video Generation Results
We provide more visualization results of generated videos using our model, as illustrated in Figure
F.2
Robotic Policy Learning
We provide some visualization results of generated data on Robomimic and LIBERO using our model, which are shown in Figure 9 and Figure 10
F.3 Robotic Policy Evaluation (add visualization results of our model and baselines
In this part, we provide visualization results of RoboScape and other baselines in policy evaluation. The failure cases are presented in Figure 11 while the successful cases shown in Figure 12
23

---

Time (condition)
1-2
[-4
[+8
148
Prediction
GT
Prediction
Prediction
GT
Figure 8: Supplemented visualization results from our model (only the subsequent frames are shown).
Time
1+0 (condition)
1+5
t-10
[+15
1+20
1425
t+30
[+35 t+40
Prediction
GT
Figure 9: Supplemented visualization results Robomimic (displaying every 5th frame; 8 frames shown from t=0 to 1=4)).
Time
F-0 (condition)
1-0
t=20
t=30
t=4
1=50
t=60
t=70
t=8
Prediction
Prediction
Figure 10: Supplemented visualization results on LIBERO (displaying every 10th frame: 8 frames shown from t=0 to t=8))
24

---

Time
t=0 (condition)
t=1
t-2
1=3
t-4
1=5
t=8
tw7
1=
Ground Truth
IRASim
MideoGPT
Figure 11: Supplemented visualization results of failure cases in policy evaluation.
Time
1-0 (condition)
L-1
t-2
L-3
1-4
1-5
1-6
L-B
Ground Truth Simulator
IRASim
MideoGPT
RoboScape
Figure 12: Supplemented visualization results of successful cases in policy evaluation.
25


Reviews:
[{"review_id": "uEzXfOH2uL", "reviewer": "Reviewer_Z6ho", "summary": "The paper presents an action-conditioned video forward model for robotic manipulation environments. It uses autoregressive prediction with a spatio-temporal transformer architecture. The main proposed improvements comprise 1) training a forward model for depth images and using latent information from this part in the image prediction branch as well as 2) identifying highly-moving keypoints and encouraging visual consistency across time at their coordinates via an additional loss term. Also, a multi-step video data processing pipeline is proposed for annotating, structuring and filtering the training data. The method is quantitatively compared to several baselines in raw prediction performance as well in application-oriented settings like synthetic data generation for policy training and as simulator for policy evaluation for a simulated robot.", "questions": "Please address the questions in the above weaknesses list.", "limitations": "I don't think the limitations section serves its purpose. It discusses future work in the sense of enhancements, but does not discuss shortcomings of the method.\nWhat are inherent assumptions, how does the method generalize? What are the compute limitations, f.e., in terms of prediction horizon?\nWhen did the method fail in the experiments?\nAlso, the limitations should be part of the main paper and not the supplemental material.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The considered problem of robotic action-conditional video generation is relevant. Improvements on this topic could have impact on many downstream problems, such as planning and generalization. Physical plausibility is a major issue with image-based formulations and attacking this with methological advances is important.\n* The outlined data processing pipeline (sec 2.2) seems quite advanced but reasonable. I found the presentation, including Figure 1, very clear.\n* I appreciate the evaluation of the model as synthetic data generator, since it directly tackles an important downstream problem and strongly shows usefulness of the method. However, to make the experiments stronger, I would like to see a comparison that always adds similarly many amounts of real data. This would disentangle the total dataset size from the fact if it is synthetic or real.\n\nWeaknesses:\n* The paper tries to set the approach in the domain of physics-informed world models. However, physics constraints such as physics-based dynamics, elastic bounces, contacts, friction, inertia, non-penetration, energy-conservation etc. are not explicitly incorporated. Rather, the method imposes 3D static and dynamic consistency across predicted frames.\n* It is unclear how the model segments the image into static and dynamic parts, since depth consistency can only be imposed in static parts.\n* Eq. 4, are tokens discrete categories or continuous feature vectors? Why is cross entropy used?\n* The paper gives no details on how the comparison to the baseline is done exactly. Are the baselines re-trained on the proposed dataset?   What are the adjustments? For example, how is the latent action space from Genie adapted to the fixed action space in this setting?\n* What causes the large performance differences in Table 1?  Unfortunately, these differences and their potential cause are not discussed. The relatively small performance difference in the ablations demonstrates that the full ablated variant (w/o deph & keypoint), still achieves better metrics than, for example, iVideoGPT. The latter seems to use a related architecture. Have hyperparameters of the baselines methods been tuned and the methods been trained properly to enable fair comparison? The proposed method, including its ablations, should also be evaluated on the datasets from IRASim and iVideoGPT and compared to the results reported in the respective papers.\n* It is hard to interpret the results in Table 1 without more information and statistics. How big is the evaluation dataset? What is the standard deviation? Also, multiple seeds should be evaluated and statistics over these should be presented.\n*  The explanation of the quantitative metrics misses precise definitions or references. This hinders assessibility of the performance of the proposed method compared to the baselines. Besides the definition of the metrics, the section is missing details on how the action-conditioning is evaluated. In the dataset, how many different trajectories exist for the same initial configuration? The paper also misses a precise evaluation protocol. What are e.g. the training and test split sizes of the created dataset?\n* Occasionally, there are some wordings/claims which are not justified scientifically. Examples:\n  - 2.3 \"maintaining model efficiency\" how / what is meant by that?\n  - 2.3 \"maintain precise geometric structure\" ... \"photorealistic video frames\"  how do you quantify this?\n  -  In Sec 2.3, there is a claim that would need to be shown e.g. in experiments \"... cannot be effectively learned through RGB pixel fitting \"\n* Similarly to Table 1, Table 2 is missing standard deviations and additional information such as evaluation dataset size and training/evaluation protocol. In the section discussing Table 2, the paper claims that \"both components contribute significantly to overall performance\". On what results is this claim based on? E.g, how can significance be determined when not even the standard deviation of the results are provided?\n* The paper mentions a curriculum learning strategy in 2.2, but then never again. How is the model trained? Does it use a curriculum?\n* Sec 3.3, How are the actions generated for the synthetic data? How are the initial observations picked?\n* Sec 3.4,  What does post training the world model mean?  How is the success condition rated, what is the exact evaluation protocol? Is it made sure that the human rating sucess does not know which model is being evaluated in order to have no bias? \n* Please provide a discussion why the proposed model predicts the success rate better than the other ones. What is a likely cause? How does one count if a prediction is not interpretable?\n* l. 231, what constitutes an \"embodied\" and a \"general\" world model. What are the differences?\n* It should be stated early on in the paper, that the policies are evaluated in simulation and not with a real robot.\n* l. 137ff, which image encoder is used for the depth maps? MAGVIT-2 ? Does it work for depth maps? How is it adapted?\n* l. 152, what does the notation c_1:t-1 ... : \\epsilon_a(a_1:t-1) mean ?\n \nFurther minor comments:\n- 2.3 \"enforce\" temporal consistency is not adequate IMO, since only incorporated via loss. Maybe \"encourage\"?\n- Sec 3: naming of sections: 4 instead of 3", "was_revised": true, "final_justification": "The author response addressed most of my concerns. The comments of the reviews need to be addressed in the paper.", "rebuttal": "**Response to W1:**  Physics constraints like friction or inertia are hard to express and challenging to integrate into current video generation-based world models. Therefore, our approach focuses on two crucial, **visually expressible** physics constraints highly relevant to robotic manipulation: spatial geometric accuracy (via depth prediction) and plausible object deformation (via keypoint dynamics learning). These constraints can be seamlessly integrated into a **data-driven** video generation model. Furthermore, these **annotations are widely available**, ensuring the **generalizability**.\n\n\n\n**Response to W2:** Our model doesn't explicitly segment images into static and dynamic parts, as depth consistency isn't imposed on different regions of the depth map over time. Our objective is to enhance **consistency between the predicted RGB and depth maps**. The model is designed to predict both the next-time RGB and depth image. Then the depth features directly guide the RGB prediction to improve spatial geometric accuracy.\n\n\n\n**Response to W3:** The tokens are discrete. We use MAGVIT-2 to compress raw RGB frames into discrete latent tokens. Therefore, the video prediction task is fundamentally modeled as predicting the next frame's tokens, a process analogous to the learning objective of LLMs, where the cross-entropy loss is widely employed for next-token prediction.\n\n\n\n**Response to W4:**  To ensure a fair evaluation, **all baselines were re-trained on our used dataset** and hyperparameters were carefully tuned. The original Genie model was trained on Internet videos lacking action annotations, thus necessitating a latent action model. In our setup, we mapped the robot's raw action vectors to a latent action space via an MLP. The dynamic model component of Genie retained its original architectural settings.\n\n\n\n**Response to W5:**  Issues we've observed in baseline models are as follows:\n\n- CogVideoX: Lacks action controllability, failing to produce outputs consistent with ground-truth actions.\n\n- IRASim & iVideoGPT: Suffer from accumulated drift and distortion during long-range rollouts.\n\nBeyond the physical constraints we introduce, our spatial-temporal transformer block architecture contributes to better performance. This design is inherently better suited for video generation compared to iVideoGPT which reuses the language model architecture like LLaMA.\n\nFor a fair comparison, **all baselines were fine-tuned on the same dataset with carefully tuned hyperparameters.** Additionally, we also evaluate our model on the **BAIR dataset (from iVideoGPT)** under identical experimental settings, confirming our model's robust effectiveness.\n\n| Model     | LPIPS(↓)   | PSNR(↑)  |\n| --------- | ---------- | -------- |\n| iVideoGPT | 0.0500     | 24.5     |\n| RoboScape | **0.0321** | **26.4** |\n\n\n\n**Response to W6:**  Our evaluation dataset consists of 1.2K video clips for testing. Here we conducted 5 independent runs for each model and calculated the standard deviations of parts of metrics (due to page limit). The results demonstrate that **our method significantly outperforms existing approaches.** We will update these results in Table 1.\n\n| **Model** | PSNR             | AbsRel        | △PSNR         |\n| --------- | ---------------- | ------------- | ------------- |\n| IRASim    | 11.3820±0.1215   | 0.6407±0.0299 | 0.1600±0.1215 |\n| CogVideoX | 17.9570±0.1531   | 0.5083±0.0264 |        -     |\n| Genie     | 19.8750 ± 0.0643 | 0.4457±0.0139 | 1.8680±0.0899 |\n| RoboScape | 21.7304±0.1204   | 0.3783±0.0264 | 3.4420±0.1386 |\n\n\n\n**Response to W7:** \n\n**About quantitative metrics:** Appearance fidelity is evaluated using PSNR and LPIPS as in [1]. Geometric consistency is assessed via AbsRel, δ1, and δ2 for depth estimation consistency between generated and real videos [2]. Action Controllability is quantified by ΔPSNR, which measures the difference in generation quality when using ground-truth actions versus random actions [3].\n\n**About action-conditioning evaluation:** We provide the world model with the initial frame and an action sequence to predict the subsequent 15 frames for evaluation, similar to [1]. \n\n**About dataset details and evaluation protocol:** Each video clip in our dataset represents a unique, independent robot action execution trajectory. No multiple trajectories originate from the same initial configuration. Our dataset comprises approximately 6.5M training clips and 1.2K test clips.\n\n[1] Cosmos world foundation model platform for physical ai.\n\n[2] Aether: Geometric-aware unified world modeling.\n\n[3] Genie: Generative interactive environments.\n\n\n\n**Response to W8:**  \n\n- \"Maintaining model efficiency\" refers to our use of additive fusion for action control, a more computationally lightweight approach compared to more complex methods like feature modulation or adding external modules (e.g., ControlNet).\n- \"Maintain precise geometric structure\" is quantified by AbsRel, δ1, and δ2 and \"photorealistic video frames\" are quantified by PSNR and LPIPS.\n- To better demonstrate \"material properties can not effectively learned...\", we will provide a visualization case in the revised manuscript.\n\n\n\n**Response to W9:**  We perform 5 independent runs for Table 2 and the results with standard deviations of some metrics are as follows (with the same evaluation set as Table 1), demonstrating that our previous results are robust.\n\n| **Model**            | PSNR           | AbsRel        | △PSNR         |\n| -------------------- | -------------- | ------------- | ------------- |\n| whole model          | 21.7304±0.1204 | 0.3783±0.0264 | 3.4420±0.1386 |\n| w/o depth            | 21.8848±0.0462 | 0.4083±0.0098 | 3.5137±0.0234 |\n| w/o keypoint         | 21.6341±0.0426 | 0.3460±0.0117 | 2.9527±0.0362 |\n| w/o depth & keypoint | 21.4772±0.0288 | 0.3710±0.0119 | 1.9449±0.0543 |\n\n\n\n**Response to W10:**  Our approach employs a two-stage curriculum learning strategy based on task difficulty. We manually classify tasks into \"simple\" (e.g., single atomic operations like pick and place) and \"difficult\" (e.g., long-term tasks like folding/ironing clothes). The model is initially trained for 2 epochs on the simple tasks, then for 3 epochs on the difficult tasks. We will add this discussion to Section 3.1.\n\n\n\n**Response to W11:**  The actions and initial observations for our synthetic data are directly drawn from the raw RoboMimic and LIBERO datasets. To be specific, \"Real data\" mentioned in Table 3 consists of original videos with their corresponding action annotations from the raw dataset. For \"Synthetic Data\", we use our world model to generate the videos based on actions and initial observations.\n\n\n\n**Response to W12:** Post training the world model refers to fine-tuning the base model pre-trained on Agibot dataset using the robomimic dataset. For success condition rating, generated videos from all models were presented to participants in a **randomized, blind order**, with **no model identifiers displayed**. Participants were simply instructed to judge whether the task depicted in each video was successfully completed.  This setup ensures objectivity and impartiality.\n\n\n\n**Response to W13:**  Baseline models often produce distorted or deformed videos, making it challenging to definitively assess task completion, leading to discrepancies between the predicted and actual success rates. In contrast, our model achieves **better appearance fidelity and action controllability**, generating visually normal videos that reliably respond to given actions. This facilitates a more accurate evaluation of policy execution success.\n\n\n\n**Response to W14:**  The distinction between an \"embodied\" and a \"general\" world model lies in their training data. **General world models** are trained on a wide variety of video data (most from the Internet), whereas **embodied world models** are specifically trained on data from robotic videos. To ensure a fair comparison, we fine-tuned all models on the same robotic dataset.\n\n\n\n**Response to W15:** Thanks for the suggestion. We will explicitly state in the introduction (Line 79-81) that our policy evaluations are conducted within a simulated robotic environment, rather than with a real robot. \n\n\n\n**Response to W16:**  For uniform modeling, we utilize MAGVIT-2 as the image encoder for depth maps. We convert depth maps to a three-channel RGB format to ensure compatibility with MAGVIT-2. This approach aligns with existing works, such as [1], which also employ a unified encoder (VAE) for both RGB and depth images. \n\n[1] Aether: Geometric-aware unified world modeling.\n\n\n\n**Response to W17:**  \n\nThis signifies the encoding of robot actions. Here, $a_{1:t−1}$ represents the original 16-dimensional robot action vector (comprising dual-arm end position, orientation, and gripper range) for frames 1 to t−1. $E_a$ is the robot action encoder, which transforms these raw actions into $c_{1:t−1}$, a D-dimensional action latent embedding for each frame.\n\n\n\n**Response to W18:**  Thanks for this suggestion. We also agree that \"encourage\" is a more fitting term and will change it in the revised manuscript.\n\n\n\n**Response to W19:** Thanks for pointing that out. We'll correct it.\n\n\n\n**Response to Limitations:** We will revise our limitations section within the main manuscript to include the following:\n\n- **Assumptions & Generalization:** Our model empirically assumes keypoint dynamics implicitly capture material properties; this is not theoretically guaranteed. Generalization to other robot types (e.g., quadrupeds) is currently limited by data availability, which is a focus for future work.\n-  **Computational Limitation:** Our model's reliable inference is currently limited to a 48-frame rollout, which is a computational constraint.\n- **Failure Cases:** We observe failure cases in long-horizon tasks (e.g., cloth folding), sometimes exhibiting physically implausible outcomes."}, {"review_id": "epECBuPq31", "reviewer": "Reviewer_XShs", "summary": "This paper presents RoboScape, a world model aiming to mitigate failures of prior world models in 3D geometry and realistic object dynamics. The model unifies RGB video generation, temporal depth prediction, and adaptive keypoint tracking in one learning objective. The paper also presents a robotic data processing pipeline with physical prior information labels, as well as experimental results in simulation. \n\nThe core innovations are in the objective and data. For the objective, first are RGB and depth reconstruction losses. Second is temporal depth prediction loss that enhances geometric consistency. Third is keypoint dynamics loss and attention loss for upweighting pixels on the keypoint track. These four are added and optimized together with separate weights (hyperparameters). Though it isn't a novel element of the paper, a key element of the system is a dual-branch co-autoregressive transformer architecture that is novel for world models. \n\nExperiments are conducted on three tasks: video generation, policy learning with synthetic data, and policy evaluation with learned simulation. For video generation, RoboScape consistently outperforms baselines. For synthetic data for policy learning, the generated data helps agents reach far higher success. And for policy evaluation, it receives very strong correlation with ground-truth simulators, unlike baselines. Ablations show far worse performance particularly in geometry and physics awareness.", "questions": "- How might you expand evaluation suite in terms of domain? \n- How adaptable is this system to new domains?", "limitations": "Limitations: yes\nBroader impact: mostly about potential positive impact, which is insufficient", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Quality \n#### Strengths\n- Well-motivated. The concerns the paper highlights are some of the most important and prohibitive for using powerful world models for robotics \n- System design is thoughtful: loss components make sense, objective is well-designed and analyzed. Joint learning approach is difficult but the right thing if it can be done\n- Evaluation approach is very strong: downstream tasks are general and difficult, ablations provide insights, and baselines make sense, even if not the most recent. Though the most relevant and important baselines are withheld, I realize this is due to accessibility, and I don't think it takes away from the paper unless they use the same approach\n- Metrics are thorough \n- Results are strong and extensive and show clear promise \n#### Weaknesses\n- Lack of other geometry-aware models. If these are truly all inaccessible, it's fair not to have.\n- Holdout or otherwise out-of-distribution input not tested, which would help understand the true generality of the 3D/dynamics representation. However, the tasks are complex, which helps mitigate this issue though not entirely \n\n### Clarity \n#### Strengths\n- Well-written paper \n- Method is very thoroughly described\n- Claims-driven results section is useful\n #### Weaknesses\n- Method is thorough, but also dense and somewhat difficult to stay on top of, even though the writing is good \n- It would be good to have a more intuitive way of explaining the architecture and objective, e.g. a diagram\n- Figures are difficult to follow, both image and graph figures. More annotation, size, and salience would help\n\n### Originality and significance \nThis work seems highly significant and to my knowledge, original. If it can generalize and/or the data requirements are automatable or at least reproducible and extendable, it is very significant in making progress on one of the most notable problems in world modeling.", "was_revised": true, "final_justification": "The paper concerns I had have been addressed to the degree the authors can given the discussion limitations. Their rebuttal was verbose and I hope that doesn't make its way into an already opaque paper, but I do appreciate the info they gave. My concerns about broader impacts were unaddressed but I will maintain my score.", "rebuttal": "**Q1: Method is thorough, but also dense and somewhat difficult to stay on top of, even though the writing is good**\n\n**Response:** \nThanks for your valuable feedback. We appreciate your positive comments on the writing and will refine the revised manuscript with more intuitive figures and concise language to make the methodology easier to follow. If you have any specific questions or suggestions regarding a particular section, we would be more than happy to provide a detailed explanation and engage in further discussion. \n\n\n\n**Q2: It would be good to have a more intuitive way of explaining the architecture and objective, e.g. a diagram**\n\n**Response:** \nThank you for your valuable feedback. While rebuttal formatting restrictions prevent us from including a new figure, we would like to provide a more intuitive explanation here for better understanding.\n\n- **Further explanation of the architecture:** Similar to LLMs, our model is fundamentally an **autoregressive model** that predicts the next frame based on the current frame and an action command. We chose this design because it is well-suited for interactive world modeling and action control. Unlike standard autoregressive models that only predict RGB images, our architecture features **two parallel, action-driven branches**: one for **RGB prediction** and a second for **depth prediction**. The core idea is to leverage the depth information to improve the geometric plausibility of the generated RGB video frames. To achieve this, we introduce an interaction between the branches, where intermediate features from the depth branch are fused (via additive fusion) into the corresponding layers of the RGB branch.\n\n- **Further explanation of the training objective:** As mentioned above, the injected depth information directly helps improve the geometric fidelity of the RGB images. Further, to address the common issue of unrealistic object deformation in embodied videos, we also introduce a **keypoint temporal consistency task**. This objective encourages the model to **implicitly learn the deformation rules** of interacting objects. For instance, the relative positions of keypoints on a rigid object will remain consistent across frames, whereas a soft object will show variations. This approach has two key advantages: it can be seamlessly integrated as a constraint within the RGB prediction objective, making it \"**data-driven**\" and easy for a video generation model to understand, and its **annotations are widely available**, ensuring **good generalizability** across various scenes.\n\n\n\n**Q3: Figures are difficult to follow, both image and graph figures. More annotation, size, and salience would help**\n\n**Response:** \nThanks for your feedback. Considering the required rebuttal format, we would like to provide more intuitive explanations to help clarify our figures.\n\n**Fig 1: Data Preparation Pipeline**\n\nThis figure outlines our five-step data curation process: extracting raw video/action/text, annotating with depth/keypoints, segmenting into atomic action clips, filtering ambiguous data and annotating key-frames with Intern-VL, and finally organizing for training.\n\n**Fig 2: Multi-task Learning Framework**\n\nThis figure illustrates our multi-task training framework. At its core is a dual-branch, autoregressive model architecture. The top line shows our **keypoint dynamic regularization**, where we extract keypoint trajectories of dynamic regions to encourage temporal consistency between corresponding pixels in different frames (indicated by the red dashed lines with \"regularization\" text). Below this, our model has two parallel, action-driven branches: one predicts the next **RGB frame**, and the other predicts the next **depth map**. A crucial part of our design is that intermediate features from the depth branch are explicitly injected into the RGB prediction branch (indicated by the gray dashed lines), providing direct geometric guidance to improve the visual realism of the generated video frames.\n\n**Fig 3: Video Generation Results**\n\nThis figure visualizes our model's video generation results on three cases. The setup is to provide the model with the first frame (t=0) and ask it to predict the 16 subsequent frames (only 8 frames are shown). The \"Prediction\" row shows the output of our model, while the \"GT\" row represents the ground truth video. The high degree of consistency between our predictions and the ground truth demonstrates our model's ability to generate plausible and coherent video sequences.\n\n**Fig 4: Ablation Study**\n\nThis figure presents a key ablation study to validate the effectiveness of our physical constraints. The first row shows the results from our full model with both depth and keypoint constraints. The second and third rows show the results when the keypoint and depth constraints are removed, respectively. The red dashed circles highlight where these omissions lead to specific failures. The figure clearly demonstrates that the absence of depth learning leads to geometric distortions in moving objects, while the lack of keypoint dynamics learning results in unrealistic motion patterns. This confirms that both constraints are essential for generating high-fidelity, physically plausible videos.\n\n**Fig 5: Policy Evaluation**\n\nThis figure showcases our world model's use as a policy evaluator. We compare its performance against two baselines, IRASim and iVideoGPT. The x-axis shows the policy's performance in a physics simulator, while the y-axis shows its performance as evaluated by the world model. Each green dot represents a single policy model (Diffusion Policy), with the associated number indicating its training epoch. The R2 and Pearson coefficients quantify the correlation between the world model's evaluation and the ground-truth simulator. Our model’s significantly higher correlation values compared to the baselines prove that it can more accurately reflect real-world physics, making it a reliable tool for policy evaluation.\n\n\n\n**Q4: How might you expand evaluation suite in terms of domain?**\n\n**Response:** \nOur current evaluation framework is designed to be highly versatile and can be readily expanded to new domains. It consists of three key components:\n\n1. **Video Generation Metrics:** We evaluate the generated videos based on **visual quality**, **geometric fidelity**, and **action controllability**. The metrics we use, such as PSNR, LPIPS, AbsRel, and δ1/δ2, are general-purpose and can be directly applied to any new domain, such as different robots.\n2. **Downstream Policy Training:** We assess our model by training a downstream policy (e.g., using Diffusion Policy or VLA like Pi0) on our synthesized data. Since these policy models are broadly applicable, we can easily adapt this evaluation by simply training a new policy on data generated for a new domain.\n3. **World Model as a Policy Evaluator:** We can also use our world model to evaluate a policy's performance by having it interact with the policy model. The world model receives the policy's actions, generates observations in response, and allows us to assess the world model based on the simulated outcomes. This process is fully transferable, as long as a policy model is trained for the new domain.\n\nBy relying on these three generalizable evaluation methods, our framework is well-equipped to handle new domains without requiring significant changes to the evaluation methodology itself.\n\n\n\n**Q5: How adaptable is this system to new domains?**\n\n**Response:** \nOur central idea is to integrate depth maps and keypoint dynamics learning into the world model's next-state prediction task through a multi-task learning approach. This approach ensures strong adaptability from two key perspectives:\n\n- **Data Perspective:** The required data annotation for our model is highly generalizable. Both depth maps and keypoint tracking information are widely available from off-the-shelf models across various embodied scenarios. This includes, but is not limited to, humanoid robots, single- and dual-arm robots, human-operated scenarios, and locomotion tasks.\n- **Model Architecture Perspective:** The core model architecture remains largely unchanged when adapting to a new domain. The primary difference between domains is typically the dimensionality of the raw action vector. This can be easily accommodated by simply adjusting the input dimension of our action encoder, without requiring any modification to the main model architecture.\n\n\n\n**Q6: Limitations: yes Broader impact: mostly about potential positive impact, which is insufficient**\n\n**Response:** \nThanks for the valuable feedback. We would like to discuss more about the limitations as below, and will supplement them within the main manuscript:\n\n**1、About Assumptions and Generalization:** A core assumption of our model is that learning keypoint dynamics can implicitly teach the model about object deformation and material properties. This is a data-driven, empirical assumption rather than a theoretically guaranteed one. Regarding generalization, our current training data includes both single and dual-arm robot manipulation. Due to the unavailability of high-quality data for other robot types, such as quadrupeds, we are unable to test for generalization to these new domains. We plan to explore this as data becomes available.\n\n**2、About Computational Limitation:** Our current model can reliably support a maximum rollout inference of up to 48 frames. While this is sufficient for many tasks, it is a clear computational limitation.\n\n**3、About Failure Cases:** We observed that our method can suffer from error accumulation during long-horizon tasks, such as generating an entire sequence of folding clothes. In these cases, the model often fails to produce a coherent or physically plausible outcome. We will consider adding more long-horizon manipulation tasks in the training data to improve this."}, {"review_id": "iX80kAs1Bv", "reviewer": "Reviewer_NSMY", "summary": "This paper proposes integrating physics into embodied world models to generate physically accurate robotic videos. It utilizes two main components: temporal depth prediction to improve 3D consistency and keypoint tracking to learn dynamics. The resulting model demonstrates enhanced performance in downstream robotic policy training.", "questions": "- Can the authors elaborate on the scope of the dataset varieties in terms of environment and robot settings? For example, what types of robots, objects, and scene settings are included, and how are they distributed? What robotic tasks are modeled? It helps to assess the breadth of this contribution knowing how general the dataset is.\n\n- The authors have specifically chosen two features in this work (depth and keypoint trajectory), both of which have been explored in previous studies to improve subject consistency and motion smoothness in generated video. How does this approach differ from previous work, perhaps in terms of model architecture or training strategies? Additionally, why were these features chosen for this study? Further insight into these aspects may help clarify the unique contribution of this work.\n\n- Can the authors provide more details on the model architecture design?", "limitations": "See \"weaknesses\" and \"questions\" above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe proposed method outperforms various baselines in terms of appearance fidelity and geometry consistency.\n\nWeaknesses:\n\n- The paper claims that existing embodied/general world models often exhibit artifacts like morphing or discontinuous motion. However, it lacks visual illustrations of these failure cases. To better showcase the robustness of the proposed method, a more extensive qualitative comparison (e.g., generation results under the same prompts) is needed, in addition to the current quantitative comparisons.\n\n- Also, the absence of accompanying videos for visual illustration. This makes it difficult to evaluate the visual quality, particularly the smoothness of motion.\n\n- Additionally, there’s concern regarding the variety of the dataset and trained world model. The paper only covers certain robot arm settings, leaving out many other embodied models such as humanoid robots, UAVs, quadrupeds, and soft robots. This limitation could stem from the scalability of the data, which might be a potential barrier.", "was_revised": true, "final_justification": "The authors have adequately addressed most of my concerns. Given the unavailability of the visual content, I would ask them to include the additional results as discussed in their rebuttal to Q1 and Q2.", "rebuttal": "**Q1: The paper claims that existing embodied/general world models often exhibit artifacts like morphing or discontinuous motion. However, it lacks visual illustrations of these failure cases. To better showcase the robustness of the proposed method, a more extensive qualitative comparison (e.g., generation results under the same prompts) is needed, in addition to the current quantitative comparisons.**\n\n**Response:** \nWe appreciate this valuable feedback. We acknowledge that a visual demonstration would enhance our claims. While we cannot provide images or videos during this rebuttal phase due to formatting limitations, we assure that we will add it in the revised paper. \n\n\n\n**Q2: Also, the absence of accompanying videos for visual illustration. This makes it difficult to evaluate the visual quality, particularly the smoothness of motion.**\n\n**Response:** Thank you for the suggestion. Similar to the response to Q1, while we cannot provide videos during the rebuttal phase due to formatting limitations, we will establish a project page for the paper to include more videos to further demonstrate the visual quality.\n\n\n\n**Q3: Additionally, there’s concern regarding the variety of the dataset and trained world model. The paper only covers certain robot arm settings, leaving out many other embodied models such as humanoid robots, UAVs, quadrupeds, and soft robots. This limitation could stem from the scalability of the data, which might be a potential barrier.**\n\n**Response:** \nWe appreciate this point. Our current data already includes a range of manipulation scenarios with both single-arm and dual-arm gripper-based robots, which have varying degrees of freedom. By adapting the action encoder, our framework can be easily generalized to work with data from different robots. Just as you mentioned, currently high-quality data and testbeds for other types of robots like humanoid robots, UAVs, quadrupeds, and soft robots are limited. As these resources become more widely available, we plan to expand our training data and validate our model on these diverse robot morphologies.\n\n\n\n**Q4: Can the authors elaborate on the scope of the dataset varieties in terms of environment and robot settings? For example, what types of robots, objects, and scene settings are included, and how are they distributed? What robotic tasks are modeled? It helps to assess the breadth of this contribution knowing how general the dataset is.**\n\n**Response:** \nWe utilized the Agibot-World dataset [1], a recently released large-scale robotic learning dataset collected with humanoid robots. Specifically, we focused on data from dual-arm end-effector type robots, excluding those with dexterous hands. The data we used included 147 tasks (e.g. closing the refrigerator door, placing objects, etc.), and 72 skills (e.g. pick, push, etc.). The dataset covers 5 scenes including home (40%), dining (20%), industry (20%), supermarkets (10%) and office (10%) and each scene consists of multiple environments.\n\n[1] https://huggingface.co/agibot-world\n\n\n\n**Q5: The authors have specifically chosen two features in this work (depth and keypoint trajectory), both of which have been explored in previous studies to improve subject consistency and motion smoothness in generated video. How does this approach differ from previous work, perhaps in terms of model architecture or training strategies? Additionally, why were these features chosen for this study? Further insight into these aspects may help clarify the unique contribution of this work.**\n\n**Response:** \nExisting methods using extra depth or motion information for video generation typically fall into two categories:\n\n- **As extra channels:** Some approaches, like those in [1] and [2], treat depth or motion as additional channels alongside RGB, using a diffusion model to denoise them jointly. The key limitation here is the lack of explicit supervision, leading to a weak coupling between these channels.\n- **As external conditions:** Other methods, such as those in [3] and [4], use depth or motion as external conditions to guide video generation. This approach restricts flexibility during inference, as it requires pre-annotated depth or motion information.\n\n[1] Team, Aether, et al. \"Aether: Geometric-aware unified world modeling.\" *arXiv preprint arXiv:2503.18945* (2025).\n\n[2] Xi, Dianbing, et al. \"Omnivdiff: Omni controllable video diffusion for generation and understanding.\" *arXiv preprint arXiv:2504.10825* (2025).\n\n[3] Esser, Patrick, et al. \"Structure and content-guided video synthesis with diffusion models.\" ICCV 2023.\n\n[4] Wang, Xiang, et al. \"Videocomposer: Compositional video synthesis with motion controllability.\" NeurIPS 2023.\n\n**Our differences with prior works:**\n\n- **Model Architecture:** Unlike the prevalent use of diffusion models, we employ an autoregressive model, which is more suitable for interactive world modeling. Our architecture includes two parallel branches for RGB and depth prediction. Crucially, the intermediate features from the depth prediction branch are explicitly injected into the RGB prediction branch, providing direct guidance for video frame synthesis. Such dual branch-based world model architecture is a novel design.\n- **Training Strategy:** We employ a novel multi-task learning strategy that **jointly learns world model dynamics and physical constraints**. Our model is trained to predict both the next RGB frame and its corresponding depth map in an autoregressive manner, driven by robot actions. The predicted depth information then acts as an explicit constraint on the RGB prediction process. Additionally, we use the temporal consistency of keypoints as a physical prior to locally constrain the RGB frames, enhancing the plausibility of deformation and motion. This unified training paradigm is distinct from the conditional diffusion training commonly used in prior works.\n\n**Motivation for our feature choices:**\n\nWe chose to focus on depth prediction and keypoint dynamic learning for two primary reasons:\n\n- **Addressing key robotic manipulation challenges:** General video generation models often struggle with two critical issues: unrealistic object deformation and motion, and inaccurate scene geometry. Both of these are fatal flaws when creating synthetic data for robot tasks. By incorporating depth and keypoint dynamics, we directly impose physical constraints that correct these deficiencies, leading to more physically plausible robotic manipulation videos.\n\n- **Practicality and generalizability:** These features are highly practical because they can be clearly and directly expressed through visual information. This allows for seamless integration into our video generation-based world model. Moreover, annotations for depth and keypoints are widely available and can be extracted using existing tools, which ensures our method's scalability and generalizability to various datasets.\n\n  \n\n**Q6: Can the authors provide more details on the model architecture design?**\n\n**Response:**  \nOur framework is composed of four main modules: an action encoder, a visual tokenizer, and two parallel prediction branches for RGB and depth images, detailed as follows:\n\n- **Visual Tokenizer:** We uniformly use MAGVIT-2 to compress both RGB frames and depth maps into discrete tokens. All subsequent video frame prediction is conducted within this token space.\n- **Action Encoder:** A simple linear layer is used to map the 16-dimensional robot action vector into a latent embedding space (with a default dimension of 256).\n- **RGB and Depth Prediction Branches:** Both branches consist of a stack of 32 spatial-temporal attention blocks. Each block is a sequence of spatial attention, temporal attention, and an MLP layer with residual connections. Both spatial and temporal attention layers have 8 attention heads, and the latent embedding dimension is set to 256. The RGB branch takes the fused action and current RGB features as input to predict the next RGB features. The depth branch does the same for depth. A key architectural detail is that intermediate features from the depth branch are explicitly injected into the corresponding layers of the RGB branch to provide direct geometric guidance and enhance the RGB feature representation.\n\nWe are willing to provide further details on any specific part of the architecture that you are concerned about."}, {"review_id": "Mkumehn7Pl", "reviewer": "Reviewer_isFR", "summary": "This paper presents a world model for robot learning. The proposed method, RoboScape, learns to predict video and depth images that are physically consistent. Physical knowledge is transferred through a proxy task of keypoint tracking, where latent features of keypoints are enforced to be consistent across time. Experiments show that the proposed method can be used to generate synthetic data for robot training or evaluate robot policies.", "questions": "Please refer to the \"weaknesses\" section.", "limitations": "The authors have stated their limitations in the \"Conclusion\" section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n1. The writing is clear, and the paper is well-motivated. World models that are capable of generating physically consistent videos have large potential applications in robotics, and the paper suggests a solution to the problem.\n\n2. Performance-wise, the paper outperforms existing world models in terms of video prediction performance.\n\n3. I find the experiment in Section 3.4 to be particularly interesting, suggesting the possibility of RoboScape to be used as a policy evaluator.\n\n[Weaknesses]\n1. The paper currently only uses the world model for generating synthetic data. I am curious how the authors could formulate the world model directly for planning, similar to the recently proposed V-JEPA2 work [1].\n\n2. The performance against the baselines are all measured in terms of video prediction quality. Would it be possible to perform comparisons in the actual task domain, i.e., generate synthetic data using each model and compare $\\pi 0$ model's performance? This would better highlight the effectiveness of the proposed method for robot data generation.\n\n3. Have the authors also tried giving \"dense supervisions\" for world model training, namely via optical flow? I am curious how the performance would differ if the authors used the loss function from Equation 5 with a *shorter* timespan but with *denser* pixels.\n\n[Related Work]\n1. Assran et al., V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, arXiv 2025.", "was_revised": true, "final_justification": "The rebuttal has allayed all my concerns, and I especially appreciate the new sets of experiments conducted using $\\pi0$. Thus I am keeping my initial rating of accept.", "rebuttal": "**Q1: The paper currently only uses the world model for generating synthetic data. I am curious how the authors could formulate the world model directly for planning, similar to the recently proposed V-JEPA2 work.**\n\n**Response:** \n\nThank you for this insightful question. Our model can also be directly used for action planning, much like the approach in V-JEPA 2. In fact, both video-generative world models and JEPAs are fundamentally modeling the dynamics of state transitions, with the key difference being whether the world state is represented in the pixel space or a latent space.\n\nTherefore, our model can be formulated for action planning using a model predictive control (MPC) approach. Specifically, we can sample multiple potential action trajectories and use our world model to predict the future frames for each trajectory. By using the similarity between the generated final frame and a target goal image as an energy function, we can select the action sequence that minimizes this energy as the result. We will adopt this evaluation as another evaluation dimension of our world model. \n\n**Q2: The performance against the baselines are all measured in terms of video prediction quality. Would it be possible to perform comparisons in the actual task domain, i.e., generate synthetic data using each model and compare π0 model's performance? This would better highlight the effectiveness of the proposed method for robot data generation.**\n\n**Response:** We appreciate this valuable suggestion, which indeed offers a more direct assessment of our method's effectiveness in robot data generation. We've conducted additional experiments comparing the π0 model's performance when trained on synthetic data generated by different world models. Following the setup in Table 3, we compared training the π0 model using data synthesized by **CogVideoX** and **IraSim** (experiments for IraSim are ongoing and will be updated soon). For fair comparisons, both CogVideoX and IraSim were fine-tuned on the **LIBERO dataset**, with each model generating 800 synthetic data points. The table below presents the experimental results across various task subsets in the LIBERO environment, indicating the superiority of our model in generating synthetic data for VLA training compared to baselines.\n\n| **Model**      | LIBERO-Spatial | LIBERO-Object | LIBERO-Goal | LIBERO-10 | Average |\n| -------------- | -------------- | ------------- | ----------- | --------- | ------- |\n| Only real data | 77.2%          | 79.8%         | 68.8%       | 34.8%     | 65.2%   |\n| CogVideoX      | 81.0%          | 79.6%         | 74.4%       | 44.2%     | 69.8%   |\n| RoboScape      | 84.6%          | 89.0%         | 82.8%       | 60.0%     | 79.1%   |\n\n\n\n**Q3: Have the authors also tried giving \"dense supervisions\" for world model training, namely via optical flow? I am curious how the performance would differ if the authors used the loss function from Equation 5 with a *shorter* timespan but with *denser* pixels.**\n\n**Response:** \n\nThanks for this insightful question. Our motivation for learning keypoint consistency (Equation 5) is its **inherent inter-frame consistency for the same keypoint in the RGB space**. This provides a natural physical constraint that significantly enhances the model's **understanding of object deformation and motion dynamics**, implicitly aiding in capturing material properties like rigidity and softness. What's more, the number of keypoints is a tunable hyper-parameter, also allowing for denser supervision as you mentioned, though this might impact training speed. \n\nRegarding optical flow, as you correctly noted, it provides a dense signal representing pixel motion vectors between consecutive frames. However, **optical flow feature maps between different frames are not necessarily continuous or strictly inter-frame consistent**. This inherent lack of strict consistency makes it an unsuitable constraint for direct application using Equation 5. Our chosen key-point supervision provides the necessary inter-frame consistency to effectively guide the model's learning of object dynamics and properties."}]

Author rebuttal:

Meta-review: The paper presents RoboScape, a physics-informed embodied world model. The reviewers consistently acknowledged the paper's strengths: the work a incorporates physics priors (depth prediction and keypoint dynamics) into world models, the method is well-motivated and clearly presented, and the experimental evaluation is comprehensive across video generation, policy training with synthetic data, and policy evaluation tasks. All reviewers recognized the technical soundness and potential impact of the approach.

The main concerns raised included requests for more detailed architecture explanations, clearer figures, additional baselines (particularly geometry-aware models), missing statistical significance tests, and questions about generalizability to other robot types. Several reviewers also noted the absence of visual demonstrations and videos to showcase the method's improvements over baselines.

The authors provided thorough responses during the rebuttal process, addressing technical questions about model architecture, clarifying methodological choices, conducting additional experiments with statistical significance tests, and providing quantitative comparisons using physics-aware metrics from established benchmarks. Notably, they demonstrated superior performance on metrics measuring physical correctness and conducted downstream policy training experiments that directly validated the utility of their synthetic data generation.

The reviewers engaged constructively in the discussion, with concerns being addressed.
Decision: Accept (spotlight)