DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning

Authors: Gaoyue Zhou, Hengkai Pan, Yann LeCun, Lerrel Pinto
Primary area: applications to robotics, autonomy, planning
Keywords: World Models, Planning, Representation Learning

Abstract:
The ability to predict future outcomes given control actions is fundamental for physical reasoning. However, such predictive models, often called world models, have proven challenging to learn and are typically developed for task-specific solutions with online policy learning. We argue that the true potential of world models lies in their ability to reason and plan across diverse problems using only passive data, without requiring online interactions with the environment. Concretely, we require world models to have the following three properties: 1) be trainable on offline, pre-collected trajectories, 2) support test-time behavior optimization, and 3) facilitate task-agnostic reasoning. To realize this, we present DINO World Model (DINO-WM), a new method to model visual dynamics without reconstructing the visual world. DINO-WM leverages spatial patch features pre-trained with DINOv2, enabling it to learn from offline behavioral trajectories by predicting future patch features. This design allows DINO-WM to achieve observational goals through action sequence optimization, facilitating task-agnostic behavior planning by treating desired goal patch features as prediction targets. We evaluate DINO-WM across various domains, including maze navigation, tabletop pushing, and particle manipulation. Our experiments demonstrate that DINO-WM can generate zero-shot behavioral solutions at test time without relying on expert demonstrations, reward modeling, or pre-learned inverse models. Notably, DINO-WM exhibits strong generalization capabilities compared to prior state-of-the-art work, adapting to diverse task families such as arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle scenarios.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730487521978

thread_type:
official_review

<thread_object>
creator: reviewer_fu9l

<content>
summary:
This work proposes a simple approach for learning a task-agnostic world model from offline data. They use frozen DINOv2 tokens and a Transformer transition model with a single-step forward prediction loss. This world model is then shown to enable model predictive control in toy environments as well as two simulated robotic environments involving manipulating a rope and many particles.

strengths:
1. The authors demonstrate that frozen visual tokens from DINOv2 are enough to learn a forward model that successfully predicts the dynamics of several different robot control environments.
2. It is further shown that the world model is able to plan in altered environments because, in part, it does not depend on a task-specific reward function.
3. The proposed world model compares favorably to task-specific world models, DreamerV3 and TD-MPC2, and its decoder generates more plausible trajectories than a prior video prediction method, AVDC.
4. The rope and granular manipulation environments demonstrate the simulation of difficult transition dynamics.

weaknesses:
1. The paper includes an insufficient set of benchmarks. I would recommend including more standard reinforcement learning and robotics benchmarks, such as DMControl, RoboMimic, MimicGen, RLBench, MetaWorld, FrankaKitchen, ManiSkill, etc. These benchmarks would better assess the world model’s ability to capture robot-object dynamics and sequential tasks. For example, the TD-MPC2 paper (used as a baseline) includes MetaWorld, DMControl, ManiSkill and MyoSuite as benchmarks. It would only be fair to compare the proposed method on some subset of the benchmarks used in TD-MPC2 or DreamerV3. I would recommend adding at least one benchmark from TD-MPC2 to strengthen the comparison. E.g., MetaWorld covers a diverse set of manipulation tasks.
2. The paper focuses on learning task-agnostic world models from offline data, but this entails a major assumption. It is assumed that the training dataset sufficiently covers the environment’s state-action space. This appears to be the case in Point Maze, Two Room, Rope Manipulation and Granular Manipulation, where the datasets are collected by executing random actions. This would not work in, e.g., RoboMimic or FrankaKitchen, where the agent has to execute a sequential task that is difficult to solve with random actions. Here, we have a chicken-and-egg problem. We need an expert dataset to train an accurate world model, but we need an accurate world model to plan expert behavior. This is why prior world modeling approaches focus on online improvement and joint learning of world models and policies.
3. I am confused by the first sentence of Section 3.1.2. It states that the ViT model is used as the transition model. But ViT is just a Transformer with tokenized images and the input to the transition model is already tokenized. Wouldn’t it be more accurate to say that a decoder-only Transformer is used as the transition model?
4. Section 3.1.2 further describes a causal masking approach of treating patch vectors of one observation as a whole unit. This design decision should be ablated in the experiments. In particular, comparing the planning success rate of DINO-WM with and without the per-image causal mask.
5. The comparison with baselines would be more fair if DreamerV3 and TD-MPC2 were modified to also use a DINOv2 image encoder, which I assume is not the case in the current experiments. Could you swap the convolutional encoder of DreamerV3 with DINOv2 and measure the impact on its planning success rate?
6. Embed to Control (https://arxiv.org/abs/1506.07365) is a major missing citation, which also investigates offline world model learning without rewards.

questions:
1. The decoder could propagate gradients into the transition model if you first predicted the next state, then decoded it and compared it to the original next state image? Have you considered this design decision?
2. Some world modeling approaches roll out trajectories N steps into the future to improve the consistency of predictions over time. Would that be compatible with your approach?
3. Do you fine-tune AVDC on your datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674588523

thread_type:
official_review

<thread_object>
creator: reviewer_cbfu

<content>
summary:
The paper proposes Dino-WM a method to model visual dynamics without reconstructing the visual world by leveraging patch features from pretrained visual encoder DinoV2, allowing it to learn the world model from offline datasets of trajectories. This formulation allows Dino-WM to realize observational goals by using action sequence optimizing enabling task-agnostic behavior planning. The paper demonstrates Dino-WM can generate solutions for tasks at test-time zero-shot without any expert demonstration, reward modeling or inverse dynamics models. In addition, Dino-WM outperforms existing methods like DreamerV3 and TD-MPC2 on complex control environments. Additionally, paper present additional analysis on using various pretrained visual encoder for world modelling to demonstrate effectiveness of using DinoV2 features and show generalization to novel environment configurations.

strengths:
1. The paper is well written and easy to follow
2. The idea proposed in the paper is quite simple, scalable and effective as demonstrated by the results. 
3. The experiments section is thorough and clearly demonstrates effectiveness of Dino-WM compared to existing methods.

weaknesses:
1. The authors argue in the intro and abstract of the paper that a world model should enable learning from **only passive data** but the experiments demonstrated in the paper always require action metadata during training of Dino-WM which contradicts the motivation. I’d suggest authors to make that argument a bit less constrained to convey the world model training should include both passive and action conditioned data.
2. The comparisons presented in experiments section 4.1 do not include comparisons to diffusion based methods. I understand it might not be feasible to get results for all environments but I’d recommend authors to add comparison of AVDC or another diffusion based methods for whichever task possible in table.1. If not possible, I’d recommend authors to add a discussion about why the comparison is not possible. Currently, the section 4.2 implies there’ll be quantitative comparison with diffusion based methods.
3. The generalization experiments in section 4.5 tests generalization to unseen configurations of environments seen during training but the description in section 4.5 (L 450-451) implies that experiments test generalization to new environments on which the model was never trained which is not true. I’d recommend authors to please fix the text description in this section. 
4. I couldn’t find any training details for world models used in experiments section. It is not clear to me what datasets were used for training Dino-WM, how were the trajectories collected, what was the scale of those datasets,etc.  The appendix has hyperparam details but no training details. It’d be great if authors can add description for those details in the paper. 
5. I would also be interested in seeing scaling laws analysis for the a subset of environments presented in the paper. It’d be interesting to see how evaluation performance improves as the size of training dataset used for Dino-WM pretraining was used. I believe it will make paper stronger and would be a nice analysis to have.

questions:
In addition to AVDC there are more recent WM style methods based on generative methods like Genie [1] that work quite well. I'd like authors to either present comparison with this method or add a discussion addressing why the comparison is not possible.

[1] Genie: Generative Interactive Environments
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676539796

thread_type:
official_review

<thread_object>
creator: reviewer_iuyc

<content>
summary:
The authors present a world model to facilitate solving control problems in the physical world. The world model learns the dynamic of the environments in the latent space based on pre-trained DINOv2 that extracts both spatial-aware and object-centric latent representation from 2D images. The approach enables gradient-based trajectory optimization at inference time to deal with various control problems (maze navigation, robot manipulation, etc.). Moreover, it achieves so by only using offline trajectory data during model learning.

strengths:
+ The proposed world model learns to generalize to different variants of several tasks (maze navigation, robot manipulation, etc.) by using only the offline trajectory data. This makes it a potential candidate for training robot foundation models with large quantities of offline data.
+ The trajectory optimization-based test-time control enabled by the proposed world model achieves good results in a range of navigation and manipulation tasks.
+ The comparison of DINOv2 over others (e.g., R3M) as the pre-trained backbone for the world model gives us insights into how good these visual foundation models are for facilitating solving control problems.

weaknesses:
- As the authors have pointed out, latent world models are not a new thing. While the proposed framework might be unique in terms of a reconstruction-free reward-free latent-space world model, I do feel the overall novelty is a little bit limited here.
- While the authors have shown results on a range of control tasks, IMO they are more or less 2D tasks (e.g., in the particular manipulation the particles mostly move on a plane). I am curious how will the approach perform for more complicated manipulation or control tasks (e.g., Humanoid Run from DMControl and Move Chair from ManiSkill2). These tasks might require a much more accurate level of physical reasoning capabilities (objects have momentum and move in a more complex way) which can help us understand better the limit of DINOv2.

questions:
- What is the inference speed of the framework? This show how well they can potentially deal with tasks that heavily require reactive controls.
- Missing references to some recent work that uses Transformers [1] or Diffusion Models [2] to learn to perform latent/visual planning for control problems from offline data.

[1] Chain-of-Thought Predictive Control

[2] Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning
</content>
</thread_object>

</thread>
<thread>
creation date:
1730780371246

thread_type:
official_review

<thread_object>
creator: reviewer_uasa

<content>
summary:
This paper introduces a visual dynamics learning approach called DINO-WM, leveraging pre-trained DINOv2 latent features for predictive modeling. DINO-WM incorporates a visual transformer that processes DINO features and action sequences to forecast future observations in an auto-regressive manner, trained through teacher forcing. The model demonstrates zero-shot planning capabilities when integrated with MPC using CEM. The study evaluates DINO-WM across a diverse set of tasks—PointMaze, PushT, Rope, and Granular Manipulation—and achieves 
better performance compared to state-of-the-art model-based reinforcement learning approaches.

---

post rebuttal: The authors addressed several concerns. Though I am still worried about the novelty but I will not argue for a rejection as the contribution of the paper is clear. So I increased my score.

strengths:
- Effectively leverages a pre-trained vision model to enhance world model learning.
- Conducts comprehensive evaluations across multiple tasks, validating the model's effectiveness.

weaknesses:
- The paper's writing needs to be improved. The citations should be clearly separated from the main text using brackets when needed.
- Although DINO-WM uses DINOv2's latent features, the approach of learning a visual dynamics model in the latent space is not groundbreaking. Pretraining latent features for world model learning or learning latent world models through rewards are both exploited in previous literature. Leveraging existing vision models is not novel to me.
- In one comparison, the authors evaluate TD-MPC2 without rewards, which might not provide a fair assessment. Since reward signals are central to TD-MPC2’s optimization, omitting them potentially undermines its performance.
- The work does not include results on established image-based RL benchmarks, such as DM-Control or Minecraft, which are widely used to validate model-based approaches like TD-MPC2 and Dreamer. Including these benchmarks would strengthen claims of general applicability​.
- The model’s lack of Q-learning hinders its capacity to handle tasks requiring long-term planning. In RL, Q-learning or similar approaches are often necessary for efficiently solving complex, high-horizon tasks. This limitation may affect DINO-WM’s scalability to challenging environments.​

questions:
- I wonder what causes the low performance of TDMPC, is the reward fully ignored or the reward depends on the latent representation that it just learned?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732203692955

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank the reviewers for their time and for finding that our work DINO-WM "effectively leverages a pre-trained vision model for world modeling," (**UASa**) has potential for "training robot foundation models," (**iuyC**), is "simple, scalable, and effective," (**Cbfu**) and "compares favorably to task-specific world models." (**fU9L**). There are, however, several significant concerns, especially around comparisons to task-specific RL settings. In this general comment, we would like to address clarifications of our setting and additional experiments that should assuage the reviewers’ concerns.
# Clarification: Offline and Reward-Free Task Solving
Several reviewers (**UASa**, **iuyC**, & **fU9L**) suggested evaluations on established RL benchmarks like DMControl. However, we want to clarify that **DINO-WM is not an task-specific online RL algorithm**. In fact, the entire motivation behind DINO-WM is that embodied agents need to be able to solve a variety of tasks instead of a single one. Most RL benchmarks such as DMControl typically assume: 
- Access to rewards throughout training
- Knowledge of the task or goals during training
- Access to simulators for online interactions

In contrast, DINO-WM operates in a purely offline and reward-free setting. It is trained without tasks, goals, or rewards, and the evaluation focuses on the generalization abilities of the learned world model to arbitrary goals and environment layouts at test time. Nevertheless, when trying to find appropriate DMControl tasks, we found that the Reacher task can be adapted to a multi-goal setting, and have added results to the paper.
# Clarification: Novelty of This Work
Reviewers **UASa** & **iuyC** highlighted that prior work also leverages “latent world models” or “use pre-trained vision models for world models.” We want to clarify that our work **does not claim these choices as novel contributions**. Instead, the novelty of this work lies in addressing whether it's possible to build general-purpose world models that exclusively capture environment dynamics and enable reasoning and planning to solve arbitrary states. This long-term objective of task-agnostic world models, capable of generalizing to unseen dynamics and goals, remains largely unexplored in prior work and represents the true potential of world models.
# Additional Experiments
## Evaluations on DMControl and LIBERO (**UASa**, **iuyC**, & **fU9L**)
To address reviewers’ suggestions on evaluations on established RL benchmarks, we include additional experiments of the Reacher task in DMControl in the manuscript in Appendix A.4. We observe a 43% performance increase than DreamerV3, our most competitive baseline.

We also evaluate on LIBERO[1], a task suite featuring offline imitation learning, and show that learned world models can be effectively used to distinguish expert vs. non-expert trajectories in Appendix A.4.
## Evaluations on Diffusion-based Baselines (AVDC) (**Cbfu**)
While the original AVDC can’t be applied for planning as they only provide a sequence of observations instead of executable actions, we further introduce an action-conditioned AVDC variant, and report its planning performance in Appendix A.7. 
## Scaling Laws of DINO-WM
Reviewer **Cbfu** suggests experiments scaling with the amount of training data. We agree that this could provide more insights on the scalability and data efficiency of DINO-WM, and we show results in Appendix A.5.1.
## DINO-WM with and without Causal Mask
Reviewer **fU9L** suggested ablation experiments on the predictor's added causal mask. We thank the reviewer for this constructive feedback, and included results in Appendix A.5.2.
## Backprop Decoder Loss to Predictor
We follow Reviewer **fU9L**’s suggestion on a variant of the model where reconstruction loss from the decoder is back propagated to the predictor, and updated the results in Appendix A.5.3.
# Additional Results
## Inference Speed
Reviewer **iuyC** raised this important point for understanding the applicability of DINO-WM. We include the inference speed and planning time for DINO-WM in Appendix A.8.
## Physical Reasoning on Complex Tasks
To show DINO-WM’s physical reasoning capabilities mentioned by Reviewer **iuyC**, we provide two new visualizations in our anon website: 
- Full rollouts of deformable tasks involving 7-DoF robotic arms to highlight tasks beyond simple 2D manipulation. 
- Unconditioned predictions on CLEVRER[2] featuring multiple 3D objects sliding and colliding. These demonstrate the model's ability to reason about physical interactions accurately.
## Training Details
We include additional training hyperparameters and model architectures in Appendix A.9, following Reviewer **Cbfu**’s feedback.

We welcome further discussions and are happy to provide additional results or clarifications as needed.
# References
[1] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

[2] CLEVRER: CoLlision Events for Video REpresentation and Reasoning
</content>
</thread_object>

</thread>
<thread>
creation date:
1732663220299

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for their thoughtful and constructive feedback. In response, we have made several updates and added experiments to address the points raised:
- Evaluations on the Reacher task of DMControl, showing a 43% performance improvement of DINO-WM compared to DreamerV3, our most competitive baseline. (**UASa**, **iuyC** & **fU9L**)
- Evaluations on the LIBERO[1] benchmark, demonstrating DINO-WM’s ability to identify expert vs. non-expert trajectories. (**UASa**, **iuyC** & **fU9L**)
- Quantitative planning evaluations of an action-conditioned AVDC variant as a diffusion-based baseline. (**Cbfu**)
- Ablation experiments with varying dataset sizes, showing DINO-WM's scaling laws. (**Cbfu**)
- Ablation experiments demonstrating the effectiveness of the causal mask introduced in DINO-WM. (**fU9L**)
- Ablation comparing DINO-WM with and without decoder loss, highlighting the benefits of decoupling world model training from reconstruction objectives. (**fU9L**)
- Inference and planning speed evaluations showing DINO-WM’s wall-clock time efficiency over simulators with complex dynamics. (**iuyC**)
- Evaluations on the CLEVRER[2] dataset, showing DINO-WM’s physical reasoning capabilities for objects with complex dynamics and behaviors like colliding, rotating, and sliding. (**iuyC**) 
- Included additional training details and model hyperparameters in the appendix of the updated manuscript. (**Cbfu**)
- Added missing references, improved citation formats, and refined writing per reviewer suggestions. (**UASa**, **iuyC**, **Cbfu** & **fU9L**)
- Addressed reviewers’ questions regarding baselines, clarifications of DINO-WM, and comparisons with recent world model style work like Genie[3]. (**UASa**, **iuyC**, **Cbfu** & **fU9L**)

We are very grateful for your time and your help in improving our work, and we believe that we have addressed all reviewers’ feedback and questions. As the rebuttal period closes to its end, we kindly ask reviewers to review our responses and the updated manuscript. We are happy to answer any additional questions you might have!

[1] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, Peter Stone. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

[2] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum. CLEVRER: CoLlision Events for Video REpresentation and Reasoning

[3] Genie: Generative Interactive Environments
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents DINO-WM, a method for learning world models that leverage pre-trained DINOv2 visual features to enable zero-shot planning. The core idea is to predict future states in the latent space of DINOv2's patch features, trained purely from offline trajectories without requiring rewards or online interaction.

Strengths: The paper demonstrates empirically that pre-trained visual features can serve as a useful representation space for world models. The evaluations are thorough, showing strong results on maze navigation, pushing, and particle manipulation tasks compared to baselines like DreamerV3 and TD-MPC2.

However, the paper has a fundamental disconnect between its claimed contribution and actual analysis. The authors identify a key challenge in offline world models: without sufficient data coverage, these methods typically require auxiliary learning objectives (like reconstruction or rewards) to learn a good latent planning space (lines 53-83). The paper proposes that using DINOv2's pre-trained features, which contain spatial and object-centric priors, can address this challenge by enabling effective planning without such auxiliary objectives.

Yet this core claim lacks rigorous justification in two critical ways. First, the paper does not establish that auxiliary objectives actually help compensate for limited data coverage - this baseline connection is necessary to motivate their solution. Second, while empirical results show DINOv2 features are effective, the paper fails to analyze why these features provide a good planning space without auxiliary learning. Without understanding what properties of DINOv2 (geometric, semantic, or physical) enable this capability, it's unclear if the method truly addresses the stated coverage challenge or simply works well for unrelated reasons. A rigorous comparison with other pre-trained representations, analyzing their relevant properties for dynamics modeling, would be necessary to properly justify the proposed solution.

Moreover, the paper falls short in benchmarking against established robotics tasks (like DMControl or RLBench) where random exploration is insufficient. While the authors added some comparisons during rebuttal, the results remain limited to simpler scenarios. The strong assumptions about data collection and the lack of a clear path to scaling to more complex tasks suggest this approach.

Given these limitations, particularly the lack of rigorous analysis explaining why DINOv2 features enable better world modeling, I recommend rejecting this paper.