Title: RoboScape: Physics-informed Embodied World Model
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embodied World model

Physics Priors

Video Generation
Abstract: World models have become indispensable tools for embodied intelligence, serving as powerful simulators capable of generating realistic robotic videos while addressing critical data scarcity challenges. However, current embodied world models exhibit limited physical awareness, particularly in modeling 3D geometry and motion dynamics, resulting in unrealistic video generation for contact-rich robotic scenarios. In this paper, we present RoboScape, a unified physics-informed world model that jointly learns RGB video generation and physics knowledge within an integrated framework. We introduce two key physics-informed joint training tasks: temporal depth prediction that enhances 3D geometric consistency in video rendering, and keypoint dynamics learning that implicitly encodes physical properties (e.g., object shape and material characteristics) while improving complex motion modeling. Extensive experiments demonstrate that RoboScape generates videos with superior visual fidelity and physical plausibility across diverse robotic scenarios. We further validate its practical utility through downstream applications including robotic policy training with generated data and policy evaluation. Our work provides new insights for building efficient physics-informed world models to advance embodied intelligence research. Our code and demos are available at: https://github.com/tsinghua-fib-lab/RoboScape.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a world model for robot learning. The proposed method, RoboScape, learns to predict video and depth images that are physically consistent. Physical knowledge is transferred through a proxy task of keypoint tracking, where latent features of keypoints are enforced to be consistent across time. Experiments show that the proposed method can be used to generate synthetic data for robot training or evaluate robot policies.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1. The writing is clear, and the paper is well-motivated. World models that are capable of generating physically consistent videos have large potential applications in robotics, and the paper suggests a solution to the problem.\n\n2. Performance-wise, the paper outperforms existing world models in terms of video prediction performance.\n\n3. I find the experiment in Section 3.4 to be particularly interesting, suggesting the possibility of RoboScape to be used as a policy evaluator.\n\n[Weaknesses]\n1. The paper currently only uses the world model for generating synthetic data. I am curious how the authors could formulate the world model directly for planning, similar to the recently proposed V-JEPA2 work [1].\n\n2. The performance against the baselines are all measured in terms of video prediction quality. Would it be possible to perform comparisons in the actual task domain, i.e., generate synthetic data using each model and compare $\\pi 0$ model's performance? This would better highlight the effectiveness of the proposed method for robot data generation.\n\n3. Have the authors also tried giving \"dense supervisions\" for world model training, namely via optical flow? I am curious how the performance would differ if the authors used the loss function from Equation 5 with a *shorter* timespan but with *denser* pixels.\n\n[Related Work]\n1. Assran et al., V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning, arXiv 2025.\n\nQUESTIONS:\nPlease refer to the \"weaknesses\" section.\n\nLIMITATIONS:\nThe authors have stated their limitations in the \"Conclusion\" section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes integrating physics into embodied world models to generate physically accurate robotic videos. It utilizes two main components: temporal depth prediction to improve 3D consistency and keypoint tracking to learn dynamics. The resulting model demonstrates enhanced performance in downstream robotic policy training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe proposed method outperforms various baselines in terms of appearance fidelity and geometry consistency.\n\nWeaknesses:\n\n- The paper claims that existing embodied/general world models often exhibit artifacts like morphing or discontinuous motion. However, it lacks visual illustrations of these failure cases. To better showcase the robustness of the proposed method, a more extensive qualitative comparison (e.g., generation results under the same prompts) is needed, in addition to the current quantitative comparisons.\n\n- Also, the absence of accompanying videos for visual illustration. This makes it difficult to evaluate the visual quality, particularly the smoothness of motion.\n\n- Additionally, there’s concern regarding the variety of the dataset and trained world model. The paper only covers certain robot arm settings, leaving out many other embodied models such as humanoid robots, UAVs, quadrupeds, and soft robots. This limitation could stem from the scalability of the data, which might be a potential barrier.\n\nQUESTIONS:\n- Can the authors elaborate on the scope of the dataset varieties in terms of environment and robot settings? For example, what types of robots, objects, and scene settings are included, and how are they distributed? What robotic tasks are modeled? It helps to assess the breadth of this contribution knowing how general the dataset is.\n\n- The authors have specifically chosen two features in this work (depth and keypoint trajectory), both of which have been explored in previous studies to improve subject consistency and motion smoothness in generated video. How does this approach differ from previous work, perhaps in terms of model architecture or training strategies? Additionally, why were these features chosen for this study? Further insight into these aspects may help clarify the unique contribution of this work.\n\n- Can the authors provide more details on the model architecture design?\n\nLIMITATIONS:\nSee \"weaknesses\" and \"questions\" above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents RoboScape, a world model aiming to mitigate failures of prior world models in 3D geometry and realistic object dynamics. The model unifies RGB video generation, temporal depth prediction, and adaptive keypoint tracking in one learning objective. The paper also presents a robotic data processing pipeline with physical prior information labels, as well as experimental results in simulation. \n\nThe core innovations are in the objective and data. For the objective, first are RGB and depth reconstruction losses. Second is temporal depth prediction loss that enhances geometric consistency. Third is keypoint dynamics loss and attention loss for upweighting pixels on the keypoint track. These four are added and optimized together with separate weights (hyperparameters). Though it isn't a novel element of the paper, a key element of the system is a dual-branch co-autoregressive transformer architecture that is novel for world models. \n\nExperiments are conducted on three tasks: video generation, policy learning with synthetic data, and policy evaluation with learned simulation. For video generation, RoboScape consistently outperforms baselines. For synthetic data for policy learning, the generated data helps agents reach far higher success. And for policy evaluation, it receives very strong correlation with ground-truth simulators, unlike baselines. Ablations show far worse performance particularly in geometry and physics awareness.\n\nSTRENGTHS AND WEAKNESSES:\n### Quality \n#### Strengths\n- Well-motivated. The concerns the paper highlights are some of the most important and prohibitive for using powerful world models for robotics \n- System design is thoughtful: loss components make sense, objective is well-designed and analyzed. Joint learning approach is difficult but the right thing if it can be done\n- Evaluation approach is very strong: downstream tasks are general and difficult, ablations provide insights, and baselines make sense, even if not the most recent. Though the most relevant and important baselines are withheld, I realize this is due to accessibility, and I don't think it takes away from the paper unless they use the same approach\n- Metrics are thorough \n- Results are strong and extensive and show clear promise \n#### Weaknesses\n- Lack of other geometry-aware models. If these are truly all inaccessible, it's fair not to have.\n- Holdout or otherwise out-of-distribution input not tested, which would help understand the true generality of the 3D/dynamics representation. However, the tasks are complex, which helps mitigate this issue though not entirely \n\n### Clarity \n#### Strengths\n- Well-written paper \n- Method is very thoroughly described\n- Claims-driven results section is useful\n #### Weaknesses\n- Method is thorough, but also dense and somewhat difficult to stay on top of, even though the writing is good \n- It would be good to have a more intuitive way of explaining the architecture and objective, e.g. a diagram\n- Figures are difficult to follow, both image and graph figures. More annotation, size, and salience would help\n\n### Originality and significance \nThis work seems highly significant and to my knowledge, original. If it can generalize and/or the data requirements are automatable or at least reproducible and extendable, it is very significant in making progress on one of the most notable problems in world modeling.\n\nQUESTIONS:\n- How might you expand evaluation suite in terms of domain? \n- How adaptable is this system to new domains?\n\nLIMITATIONS:\nLimitations: yes\nBroader impact: mostly about potential positive impact, which is insufficient", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an action-conditioned video forward model for robotic manipulation environments. It uses autoregressive prediction with a spatio-temporal transformer architecture. The main proposed improvements comprise 1) training a forward model for depth images and using latent information from this part in the image prediction branch as well as 2) identifying highly-moving keypoints and encouraging visual consistency across time at their coordinates via an additional loss term. Also, a multi-step video data processing pipeline is proposed for annotating, structuring and filtering the training data. The method is quantitatively compared to several baselines in raw prediction performance as well in application-oriented settings like synthetic data generation for policy training and as simulator for policy evaluation for a simulated robot.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The considered problem of robotic action-conditional video generation is relevant. Improvements on this topic could have impact on many downstream problems, such as planning and generalization. Physical plausibility is a major issue with image-based formulations and attacking this with methological advances is important.\n* The outlined data processing pipeline (sec 2.2) seems quite advanced but reasonable. I found the presentation, including Figure 1, very clear.\n* I appreciate the evaluation of the model as synthetic data generator, since it directly tackles an important downstream problem and strongly shows usefulness of the method. However, to make the experiments stronger, I would like to see a comparison that always adds similarly many amounts of real data. This would disentangle the total dataset size from the fact if it is synthetic or real.\n\nWeaknesses:\n* The paper tries to set the approach in the domain of physics-informed world models. However, physics constraints such as physics-based dynamics, elastic bounces, contacts, friction, inertia, non-penetration, energy-conservation etc. are not explicitly incorporated. Rather, the method imposes 3D static and dynamic consistency across predicted frames.\n* It is unclear how the model segments the image into static and dynamic parts, since depth consistency can only be imposed in static parts.\n* Eq. 4, are tokens discrete categories or continuous feature vectors? Why is cross entropy used?\n* The paper gives no details on how the comparison to the baseline is done exactly. Are the baselines re-trained on the proposed dataset?   What are the adjustments? For example, how is the latent action space from Genie adapted to the fixed action space in this setting?\n* What causes the large performance differences in Table 1?  Unfortunately, these differences and their potential cause are not discussed. The relatively small performance difference in the ablations demonstrates that the full ablated variant (w/o deph & keypoint), still achieves better metrics than, for example, iVideoGPT. The latter seems to use a related architecture. Have hyperparameters of the baselines methods been tuned and the methods been trained properly to enable fair comparison? The proposed method, including its ablations, should also be evaluated on the datasets from IRASim and iVideoGPT and compared to the results reported in the respective papers.\n* It is hard to interpret the results in Table 1 without more information and statistics. How big is the evaluation dataset? What is the standard deviation? Also, multiple seeds should be evaluated and statistics over these should be presented.\n*  The explanation of the quantitative metrics misses precise definitions or references. This hinders assessibility of the performance of the proposed method compared to the baselines. Besides the definition of the metrics, the section is missing details on how the action-conditioning is evaluated. In the dataset, how many different trajectories exist for the same initial configuration? The paper also misses a precise evaluation protocol. What are e.g. the training and test split sizes of the created dataset?\n* Occasionally, there are some wordings/claims which are not justified scientifically. Examples:\n  - 2.3 \"maintaining model efficiency\" how / what is meant by that?\n  - 2.3 \"maintain precise geometric structure\" ... \"photorealistic video frames\"  how do you quantify this?\n  -  In Sec 2.3, there is a claim that would need to be shown e.g. in experiments \"... cannot be effectively learned through RGB pixel fitting \"\n* Similarly to Table 1, Table 2 is missing standard deviations and additional information such as evaluation dataset size and training/evaluation protocol. In the section discussing Table 2, the paper claims that \"both components contribute significantly to overall performance\". On what results is this claim based on? E.g, how can significance be determined when not even the standard deviation of the results are provided?\n* The paper mentions a curriculum learning strategy in 2.2, but then never again. How is the model trained? Does it use a curriculum?\n* Sec 3.3, How are the actions generated for the synthetic data? How are the initial observations picked?\n* Sec 3.4,  What does post training the world model mean?  How is the success condition rated, what is the exact evaluation protocol? Is it made sure that the human rating sucess does not know which model is being evaluated in order to have no bias? \n* Please provide a discussion why the proposed model predicts the success rate better than the other ones. What is a likely cause? How does one count if a prediction is not interpretable?\n* l. 231, what constitutes an \"embodied\" and a \"general\" world model. What are the differences?\n* It should be stated early on in the paper, that the policies are evaluated in simulation and not with a real robot.\n* l. 137ff, which image encoder is used for the depth maps? MAGVIT-2 ? Does it work for depth maps? How is it adapted?\n* l. 152, what does the notation c_1:t-1 ... : \\epsilon_a(a_1:t-1) mean ?\n \nFurther minor comments:\n- 2.3 \"enforce\" temporal consistency is not adequate IMO, since only incorporated via loss. Maybe \"encourage\"?\n- Sec 3: naming of sections: 4 instead of 3\n\nQUESTIONS:\nPlease address the questions in the above weaknesses list.\n\nLIMITATIONS:\nI don't think the limitations section serves its purpose. It discusses future work in the sense of enhancements, but does not discuss shortcomings of the method.\nWhat are inherent assumptions, how does the method generalize? What are the compute limitations, f.e., in terms of prediction horizon?\nWhen did the method fail in the experiments?\nAlso, the limitations should be part of the main paper and not the supplemental material.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}