DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning

## Abstract

Abstract The ability to predict future outcomes given control actions is fundamental for physical reasoning. However, such predictive models, often called world models, have proven challenging to learn and are typically developed for task-specific solutions with online policy learning. We argue that the true potential of world models lies in their ability to reason and plan across diverse problems using only passive data. Concretely, we require world models to have the following three properties: 1) be trainable on offline, pre-collected trajectories, 2) support test-time behavior optimization, and 3) facilitate task-agnostic reasoning. To realize this, we present DINO World Model (DINO-WM), a new method to model visual dynamics without reconstructing the visual world. DINO-WM leverages spatial patch features pre-trained with DINOv2, enabling it to learn from offline behavioral trajectories by predicting future patch features. This design allows DINO-WM to achieve observational goals through action sequence optimization, facilitating task-agnostic behavior planning by treating desired goal patch features as prediction targets. We evaluate DINO-WM across various domains, including maze navigation, tabletop pushing, and particle manipulation. Our experiments demonstrate that DINO-WM can generate zero-shot behavioral solutions at test time without relying on expert demonstrations, reward modeling, or pre-learned inverse models. Notably, DINO-WM exhibits strong generalization capabilities compared to prior state-of-the-art work, adapting to diverse task families such as arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle scenarios.

## 1 Introduction

Robotics and embodied AI has seen tremendous progress in recent years. Advances in imitation learning and reinforcement learning has enabled agents to learn complex behaviors across diverse tasks Lee et al. (); Zhao et al. (); Ma et al. (); Hafner et al. (); Hansen et al. (); Agarwal et al. (); Haldar et al. (). Despite this progress, generalization remains a major challenge Zhou et al. (). Existing approaches predominantly rely on policies that, once trained, operate in a feed-forward manner during deployment—mapping observations to actions without any further optimization or reasoning. Under this framework, successful generalization inherently requires agents to possess solutions to all possible tasks and scenarios once training is complete, which is only possible if the agent has seen similar scenarios during training Brohan et al. (; ); Reed et al. (); Etukuru et al. (). However, it is neither feasible nor efficient to learn solutions for all potential tasks and environments in advance.

Instead of learning the solutions to all possible tasks during training, an alternate is to fit a dynamics model on training data and optimize task-specific behavior during runtime. These dynamics models, also called world models Ha & Schmidhuber (), have a long history in robotics and control Sutton (); Todorov & Li (); Williams et al. (). More recently, several works have shown that world models can be trained on raw observational data Hafner et al. (; ); Micheli et al. (); Robine et al. (); Hansen et al. (). This enables flexible use of model-based optimization to obtain policies as it circumvents the need for explicit state-estimation. Despite this, significant challenges still remain in it use for solving general-purpose tasks.

To understand the challenges in world modeling, let us consider the two broad paradigms in learning world models: online and offline. In the online setting, access to the environment is often required so data can be continuously collected to improve the world model, which in turn improves the policy and the subsequent data collection. However, the online world model is only accurate in the cover of the policy that was being optimized. Hence, while it can be used to train powerful task-specific policies , it requires retraining for every new task even in the same environment . Instead in the offline setting, the world model is trained on an offline dataset of collected trajectories in the environment, which removes its dependence on the task specificity given sufficient coverage in the dataset. However, when required to solve a task, methods in this domain require strong auxiliary information to overcome the lack of dense coverage on the task-specific domain. This auxiliary information can take the form of expert demonstrations Pathak et al. (), structured keypoints Ko et al. (); Wen et al. (), access to pretrained inverse models Du et al. (); Ko et al. () or dense reward functions , all of which reduce the generality of using offline world models. The central question to building better offline world models is if there is alternate auxiliary information that does not compromise its generality?

Figure: Figure 1: We present DINO-WM, a method for training visual models by using pretrained DINOv2 embeddings of image frames (a). Once trained, given a target observation $o_{T}$, we can directly optimize agent behavior by planning through DINO-WM using model-predictive control (b). The use of pretrained embeddings significantly improves performance over prior state-of-the-art world models (c).
Refer to caption: extracted/5982932/figures/intro.png

In this work, we present DINO-WM, a new and simple method to build task-agnostic world models from an offline dataset of trajectories. DINO-WM models the dynamics of world on compact embeddings of the world, rather than the raw observations themselves. For the embedding, we use pretrained patch-features from the DINOv2 model, which provides both a spatial and object-centric representation prior. We conjecture that this pretrained representation enables robust and consistent world modeling, which relaxes the necessity for task-specific data coverage. Given these visual embeddings and actions, DINO-WM uses the ViT architecture to predict future embeddings. Once this model is trained, planning to solve tasks is constructed as visual goal reaching, i.e. to reach a future desired goal given the current observation. Since the predictions by DINO-WM are high quality (see Figure 4), we can simply use model-predictive control with inference-time optimization to reach desired goals without any without any extra information during testing.

DINO-WM is experimentally evaluated on four environment suites spanning maze navigation, sliding manipulation, and particle manipulation tasks. Our experiments reveal the following findings:

- •
DINO-WM produce high-quality future world modeling that can be measured by improved visual reconstruction from trained decoders. On LPIPS metrics for our hardest tasks, this improves upon prior state-of-the-art work by 56% (See Section 4.7).
- •
Given the latent world models trained using DINO-WM, we show high success for reaching arbitrary goals on our hardest tasks, improving upon prior work by 45% on average (See Section 4.3).
- •
DINO-WM can be trained across environment variations within a task family (e.g. different maze layouts for navigation or different object shapes for manipulation) and achieve higher rates of success compared to prior work (See Section 4.5).

Code and models for DINO-WM will be open-sourced to ensure reproducibility and videos of policies are made available on our project website: [https://dino-wm.github.io](https://dino-wm.github.io).

## 2 Related Work

We build on top of several works in building world models, optimizing them, and using compact visual representations. For conciseness, we only discuss the ones most relevant to DINO-WM.

Model-based Learning: Learning from models of dynamics has a rich literature spanning the fields of control, planning, and robotics Sutton (); Todorov & Li (); Astolfi et al. (); Holkar & Waghmare (); Williams et al. (). Recent works have shown that modeling dynamics and predicting future states can significantly enhance vision-based learning for embodied agents across various applications, including online reinforcement learning Hafner et al. (); Micheli et al. (); Hansen et al. (); Robine et al. (), exploration Mendonca et al. (; ); Sekar et al. (), planning Finn & Levine (); Ebert et al. (); Hafner et al. (), and imitation learning Pathak et al. (). Several of these approaches initially focused on state-space dynamics Deisenroth & Rasmussen (); Chua et al. (); Lenz et al. (); Nagabandi et al. (), and has since been extended to handle image-based inputs, which we address in this work. These world models can predict future states in either pixel space Finn & Levine (); Ebert et al. (); Ko et al. (); Du et al. () or latent representation space Yan et al. (). Predicting in pixel space, however, is computationally expensive due to the need for image reconstruction and the overhead of using diffusion models . On the other hand, latent-space prediction is typically tied to objectives of reconstructing images Hafner et al. (; ); Micheli et al. (), which raises concerns about whether the learned features contain sufficient information about the task. Moreover, many of these models incorporate reward prediction Hafner et al. (); Micheli et al. (); Robine et al. (), or use reward prediction as auxiliary objective to learn the latent representation Hansen et al. (; ), inherently making the world model task-specific. In this work, we aim to decouple task-dependent information from latent-space prediction, striving to develop a versatile and task-agnostic world model capable of generalizing across different scenarios.

Generative Models as World Models: With the recent excitement of large scale foundation models,
there have been initiatives on building large-scale video generation world models conditioned on agent’s actions in the domain of self-driving Hu et al. (), control Yang et al. (); Bruce et al. (), and general-purpose video generation Liu et al. (). These models aim to generate video predictions conditioned on text or high-level action sequences. While these models have demonstrated utility in downstream tasks like data augmentations, their reliance on language conditioning limits their application when precise visually indicative goals need to be reached. Additionally, the use of diffusion models for video generation makes them computationally expensive, further restricting their applicability for test-time optimization techniques such as MPC. In this work, we aim to build a world model in latent space rather than in the raw pixel space, which enables more precise planning and control.

Pretrained Visual Representations: Significant advancements have been made in the field of visual representation learning, where compact features that capture spatial and semantic information can be readily used for downstream tasks. Pre-trained models like ImageNet pre-trained ResNet He et al. (), I-JEPA Assran et al. (), and DINO Caron et al. (); Oquab et al. () for images, as well as V-JEPA Bardes et al. () for videos, and R3M Nair et al. (), MVP Xiao et al. () for robotics have allowed fast adaptation to downstream tasks as they contain rich spatial and semantic information. While many of these models represent images using a single global feature, the introduction of Vision Transformers (ViTs) Dosovitskiy et al. () has enabled the use of pre-trained patch features, as demonstrated by DINO Caron et al. (); Oquab et al. (). DINO employs a self-distillation loss that allows the model to learn representations effectively, capturing semantic layouts and improving spatial understanding within images. In this work, we leverage DINOv2’s patch embeddings to train our world model, and demonstrate that it serves as a versatile encoder capable of handling multiple precise tasks.

## 3 DINO World Models

Overview and Problem formulation: Our work follows the vision-based control task framework, which models the environment as a partially observable Markov decision process (POMDP). The POMDP is defined by the tuple $(\mathcal{O},\mathcal{A},p)$, where $\mathcal{O}$ represents the observation space, and $\mathcal{A}$ denotes the action space. The environment’s dynamics are modeled by the transition distribution $p(o_{t+1}\mid o_{\leq t},a_{\leq t})$, which predicts future observations based on past actions and observations.

In this work, we aim to learn task-agonstic world models from pre-collected offline datasets, and use these world models to perform visual reasoning and control at test time. At test time, our system starts from an arbitrary environment state and is provided with a goal observation in the form of an RGB image, in line with prior works Wu et al. (); Ebert et al. (); Mendonca et al. (), and is asked to perform a sequence of actions $a_{0},...,a_{T}$ such that the goal state can be achieved. This approach differs from world models used in online reinforcement learning (RL) where the objective is to optimize rewards for a fixed set of tasks at hand Hafner et al. (); Hansen et al. (), or from text-conditioned world models, where goals are specified through text prompts Du et al. (); Ko et al. ().

Figure: Figure 2: Architecture of DINO-WM. Given observations $o_{t-k:t}$, we optimize the sequence of actions $a_{t:T-1}$ to minimize the predicted loss to the desired goal $o_{g}$. All forward computation is done in the latent space $z$. Here $p_{\theta}$ indicates DINO-WM’s dynamics model, which is used for making future predictions.
Refer to caption: x1.png

### 3.1 DINO-based World Models ( DINO-WM )

We model the dynamics of the environment in the latent space. More specifically, at each time step $t$, our world model consists of the following components:

$$ Observation model: $\displaystyle\quad z_{t}\sim\text{enc}_{\theta}(z_{t}\mid o_{t})$ Transition model: $\displaystyle\quad z_{t+1}\sim p_{\theta}(z_{t+1}\mid z_{t-H:t},a_{t-H:t})$ Decoder model (optional for visualization): $\displaystyle\quad\hat{o}_{t}\sim q_{\theta}(o_{t}\mid z_{t})$ $$

where the observation model encodes image observations to latent states $z_{t}$, and the transition model takes in a history of past latent states of length $H$. The decoder model takes in a latent $z_{t}$, and reconstruct the image observation $o_{t}$. We use $\theta$ to denote the parameters of these models. Note that our decoder is entirely optional, as the training objectives for the decoder is independent for training the rest part of the world model. This eliminates the need to reconstructing images both during training and testing, which reduces computational costs compared to otherwise coupling together the training of the observational model and the decoder, as in Hafner et al. (); Micheli et al. ().

DINO-WM models only the information available from offline trajectory data in an environment, in contrast to recent online RL world models that also require task-relevant information, such as rewards Hansen et al. (; ); Hafner et al. (), discount factors Hafner et al. (); Robine et al. (), and termination conditions Hafner et al. (); Micheli et al. ().

#### 3.1.1 Observation Model

With the goal of learning a generic world model across many environments and the real world, we argue that the observation model should 1) be task and environment independent, and 2) contain rich spatial information which is crucial in navigation and manipulation tasks. Contrary to previous works where the observation model is always learned for the task at hand Hafner et al. (), we argue instead that it is not always possible for world models to learn an observation model from scratch when facing a new environment, as perception is a general task that can be learned from the large corpus of internet data. Therefore, we choose the out-of-the-box pre-trained DINOv2 model as our world model’s observation model, as it has been shown to excel at object detection, semantic segmentation, and depth estimation tasks which require substantial spatial understanding. The observation model is kept frozen throughout both training and testing time. At each time step $t$, it encodes an image $o_{t}$ to patch embeddings $z_{t}\in\mathbb{R}^{N\times E}$, where $N$ denotes the number of patches, and $E$ denotes the embedding dimension. This process is visualized in Figure 2.

#### 3.1.2 Transition Model

We adopt the ViT Dosovitskiy et al. () architecture for the transition model as it is a natural choice for processing patch features. However, a few modifications are required to the architecture to allow for additional conditioning on proprioception and controller actions.

Our transition model takes in a history of past latent states $z_{t-H:t-1}$ and actions $a_{t-H:t-1}$, where $H$ is a hyperparameter denoting the context length of the model, and predicts the latent state at next time step $z_{t}$. To properly capture the temporal dependencies, where the world state at time $t$ should only depend on previous observations and actions, we implement a causal attention mechanism in the ViT model, enabling the model to predict latents autoregressively at a frame level. Specifically, each patch vector $z^{i}_{t}$ for the latent state $z_{t}$ attends to $\{z^{i}_{t-H:t-1}\}_{i=1}^{N}$. This is different from past work IRIS Micheli et al. () which similarly represent each observation as a sequence of vectors, but autoregressively predict $z^{i}_{t}$ at a token level, attending to $\{z^{i}_{t-H:t-1}\}_{i=1}^{N}$ as well as $\{z^{i}_{t}\}_{i=1}^{<k}$. We argue that predicting at a frame level and treating patch vectors of one observation as a whole better captures global structure and temporal dynamics, modeling dependencies across the entire observation rather than isolated tokens, leading to improved temporal generalization.

To model the effect of the agent’s action to the environment, we condition the world model’s predictions on these actions. Specifically, we concatenate the $K$-dimensional action vector, mapped from the original action representation using a multi-layer perceptron (MLP), to each patch vector $z^{i}_{t}$ for $i=1,\ldots,N$. When proprioceptive information is available, we incorporate it similarly by concatenating it to the observation latents, thereby integrating it into the latent states.

We train the world model with teacher forcing. During training, we slice the trajectories in to segments of length $H+1$, and compute a latent consistency loss on each of the $H$ predicted frames. For each frame, we compute

$$ $\mathcal{L}_{pred}=\left\|p_{\theta}\left(\text{enc}_{\theta}(o_{t-H:t}),\phi( a_{t-H:t})\right)-\text{enc}_{\theta}\left(o_{t+1}\right)\right\|^{2}$ (1) $$

where $\phi$ is the action encoder model that can map actions to higher dimensions. Note that our world model training is entirely performed in latent space, without the need to reconstruct the original pixel images.

#### 3.1.3 Decoder for Interpretability

To aid in visualization and interpretability, we use a stack of transposed convolution layers to decode the patch representations back to image pixels, similar as in Razavi et al. (). Given a pre-collected dataset, we optimize the parameters $\theta$ of the decoder $q_{\theta}$ with a simple reconstruction loss defined as:

$$ $\mathcal{L}_{rec}=\left\|q_{\theta}(z_{t})-o_{t}\right\|^{2},\quad\text{where} \quad z_{t}=\text{enc}_{\theta}(o_{t})$ (2) $$

The training of the decoder is entirely independent of the transition model training, offering several advantages: 1) The quality of the decoder does not affect the world model’s reasoning and planning capabilities for solving downstream tasks, and 2) During planning, there is no need to reconstruct raw pixel images, thereby reducing computational costs. Nevertheless, the decoder remains valuable as it enhances the interpretability of the world model’s predictions.

### 3.2 Visual Planning with DINO-WM

Arguably, to evaluate the quality of the world model, it needs to be able to allow for downstream reasoning and planning. A standard evaluation metric is to perform trajectory optimization at test time with these world models and measure the performance. While the planning methods themselves are fairly standard, it serves as a means to emphasize the quality of the world models. For this purpose, our world model receives the current observation $o_{0}$ and a goal observation $o_{g}$, both represented as RGB images. We formulate planning as the process of searching for a sequence of actions that the agent would take to reach $o_{g}$. To achieve this, we employ model predictive control (MPC), which facilitates planning by considering the outcomes of future actions.

We utilize the cross-entropy method (CEM), a stochastic optimization algorithm, to optimize the sequence of actions at each iteration. The planning cost is defined as the mean squared error (MSE) between the current latent state and the goal’s latent state, given by

$$ $\mathcal{C}=\left\|\hat{z}_{T}-z_{g}\right\|^{2},\quad\text{where}\quad\hat{z} _{t}=p(\hat{z}_{t-1},a_{t-1}),\quad\hat{z}_{0}=\text{enc}(o_{0}),\quad z_{g}= \text{enc}(o_{g}).$ $$

The MPC framework and CEM optimization procedure is detailed in Appendix A.4.1.
Since our world model is differentiable, a possibly more efficient approach is to optimize this objective through gradient descent (GD), allowing the world model to directly guide the agent toward a specific goal. The details of GD are provided in Appendix A.4.2. However, we empirically observe that CEM outperforms GD in our experiments. We hypothesize this is due to our choice to not constrain the terrain smoothness of the world model during training, potentially leading to issues with the gradient. Full results for both planners can be found in Appendix A.4.3.

## 4 Experiments

Our experiments are designed to address the following key questions: 1) Can we effectively train DINO-WM using pre-collected offline datasets? 2) Once trained, can DINO-WM be used for visual planning? 3) To what extent does the quality of the world model depend on pre-trained visual representations? 4) Does DINO-WM generalize to new configurations, such as variations in spatial layouts and object arrangements? To answer these questions, we train and evaluate DINO-WM across five environment suites (full description in Appendix A.1) and compare it to a variety of state-of-the-art world models that model the world both in latent space and in raw pixel space.

### 4.1 Environments and Tasks

We consider five environment suites in our evaluations spanning simple navigation environments and manipulation environments with varying dynamics complexity. For all environments, the observation space is RGB images of size (224, 224).

Figure: Figure 3: We evaluate DINO-WM on 5 environment suites, from left to right: PointMaze, Push-T, Two Room, Rope Manipulation, and Granular Manipulation.
Refer to caption: extracted/5982932/figures/env.png

- a)
Point Maze: A simple 2D point maze navigation environment in the D4RL suite Fu et al. (). A point agent with 2-dimensional action space moves in a U-shape maze. The agent’s dynamics incorporate physical properties such as velocity, acceleration, and inertia, making the movement realistic. The objective of the task is to navigate the maze and reach arbitrary goal locations from arbitrary starting location.
- b)
Push-T: This manipulation environment was introduced in Chi et al. () to study precise manipulation. The environment features a pusher agent interacting with a T-shaped block. The goal is to guide both the agent and the T-block from a randomly initialized state to a known feasible target configuration within 25 steps. The task requires both the agent and the T to match the target locations. Unlike previous setups, the fixed green T no longer represents the target position for the T-block but serves purely as a visual anchor for reference. Success requires precise understanding of the contact-rich dynamics between the agent and the object, making it a challenging test for visuomotor control and object manipulation. We also introduce a variant of this where we have multiple object shapes.
- c)
Wall: This custom 2D navigation environment featuring two rooms separated by a wall with a door opening. The task requires the agent to navigate from a randomized starting location in one room to a goal in the other room, which requires the agent to pass through the door. We introduce a variant of this environment where the positions of the wall and door are randomized to assess the model’s ability to generalize to novel configurations of familiar environment dynamics.
- d)
Rope Manipulation: This task is simulated with Nvidia Flex Zhang et al. () and consists of an XArm interacting with a rope placed on a tabletop. The objective is to move the rope from an arbitrary start configuration to a specified goal configuration.
- e)
Granular Manipulation: Granular manipulation uses the same setting as Rope manipulation and manipulates about a hundred particles to form desired shapes.

### 4.2 Baselines

We compare DINO-WM with the following state-of-the-art models commonly used for control:

- a)
IRIS Micheli et al. (): IRIS employs a discrete autoencoder to translate visual inputs into tokens, and a GPT Transformer that predicts tokens of future observations. It combines these components to learn policies and value functions through imaginative procedures.
- b)
DreamerV3 Hafner et al. (): DreamerV3 learns a world model to interpret visual inputs into categorical representation. It predicts future representations and rewards based on given action and trains an actor-critic policy from its imagined trajectories.
- c)
TD-MPC2 Hansen et al. () : TD-MPC2 learns a decoder-free world model in latent space and uses reward signals to optimize the latents. It serves as a strong baseline for reconstruction-free world modeling.
- d)
AVDC Ko et al. (): AVDC leverages a diffusion model to generate an imagined video of task execution based on initial observation and a textual goal description. It then estimates optical flow between frames to capture object movements and generates robot arm commands.

Figure: Figure 4: Openloop rollout of world models trained with various pre-trained encoders on Push-T and Granular environment. For each trajectory, the model is given the first frame as well as sequence of actions. The world models performs openloop rollout with these actions, and the images are reconstructed by a pre-trained decoder. For each environment, the bottom row denotes the ground truth. DINO-WM (Ours) rollouts are bolded and are visually indistinguishable from the ground truth observations.
Refer to caption: extracted/5982932/figures/openloop4.png

### 4.3 Optimizing Behaviors with DINO-WM

With a trained world model, we study if DINO-WM be used for zero-shot planning directly in the latent space.

For the PointMaze, Push-T, and Wall environments, we sample 50 initial and goal states to measure the success rate across all instances. Due to the environment stepping time for the Rope and Granular environments, we evaluate the Chamfer Distance (CD) on 10 instances for them. In the Granular environment, we sample a random configuration from the validation set, with the goal of pushing the materials into a square shape at a randomly selected location and scale.

Table: Table 1: Planning results for offline world models on five control environments.

As seen in Table 1, on simpler environments such as Wall and PointMaze, DINO-WM is on par with state-of-art world models like DreamerV3. However, DINO-WM significantly outperforms prior work at manipulation environments where rich contact information and object dynamics need to be accurately inferred for task completion. We notice that for TD-MPC2, the lack of reward signal makes it difficult to learn good latent representations, which subsequently results in poor performance. Visualizations of some planning results can be found in Figure 5.

Figure: Figure 5: Planning visualizations for PointMaze, Push-T, and Granular, on randomly sampled initial and goal configurations. The task is defined by Start and Goal, denoting the initial and goal observations. Final shows the final state the system arrives at after planning with each world model. For comparison, we show the best performing world models DINO CLS and DreamerV3.
Refer to caption: extracted/5982932/figures/plan_results2.png

### 4.4 Does pre-trained visual representations matter?

We use different pre-trained general-purpose encoders as the observation model of the world model, and evaluate their downstream planning performance. Specifically, we use the following encoders commonly used in robotics control and general perception: R3M Nair et al. (), ImageNet pretrained ResNet-18 Russakovsky et al. (); He et al. () and DINO CLS Caron et al. (). Detailed descriptions of these encoders are in Appendix A.3.

Table: Table 2: Planning results for world models with various pre-trained encoders.

In the PointMaze task, which involves simple dynamics and control, we observe that world models with various observation encoders all achieve near-perfect success rates. However, as the environment’s complexity increases—requiring more precise control and spatial understanding—world models that encode observations as a single latent vector show a significant drop in performance. We posit that patch-based representations better capture spatial information, in contrast to models like R3M, ResNet, and DINO CLS, which reduce observations to a single global feature vector, losing crucial spatial details necessary for manipulation tasks.

### 4.5 Generalizing to Novel Environment Configurations

We would like to measure the generalization capability of our world models not just across different goals in an environment, but across different environments themselves. For this we construct three families of environments, where the world model will be deployed in an unseen environment for unseen goals. Our families of environments consist of WallRandom, PushObj, and GranularRandom with detailed descriptions in Appendix A.2. Visualizations of training and testing examples are shown in Figure 6.

Figure: Figure 6: Training and testing visualizations for WallRandom, PushObj and GranularRandom. Test setups are highlighted in blue boxes, showcasing unseen configurations for assessing the model’s generalization ability.
Refer to caption: extracted/5982932/figures/generalization_exps.png

Table: Table 3: Planning results for offline world models on three suites with unseen environment configurations.

From Table 3, we observe that DINO-WM demonstrates significantly better performance in the WallRandom environment, indicating that the world model has effectively learned the general concepts of walls and doors, even when they are positioned in locations unseen during training. In contrast, other methods struggle to accurately identify the door’s position and navigate through it. The PushObj task remains challenging for all methods, as the model was only trained on the four object shapes, which makes it difficult to infer physical parameters like the center of gravity and inertia precisely. In GranularRandom, the agent encounters fewer than half the particles present during training, resulting in out-of-distribution images compared to the training instances. Nevertheless, DINO-WM accurately encodes the scene and successfully gathers the particles into a designated square location with the lowest Chamfer Distance (CD) compared to the baselines, demonstrating better scene understanding. We hypothesize that this is due to DINO-WM’s observation model encoding the scene as patch features, making the variance in particle number still within the distribution for each image patch.

### 4.6 Qualitative comparisons with generative video models

Given the prominence of generative video models, it is reasonable to presume that they could readily serve as world models. To investigate the usefulness of DINO-WM over such video generative models, we compare it with imagined rollouts from AVDC Ko et al. (), a diffusion-based generative model. As seen in Figure 7, we find that the diffusion model trained on benchmarks produce future images that are mostly visually realistic, however they are not physically plausible as we can see that large changes can occur in a single timestep of prediction, and may have difficulties in reaching to the exact goal state. Potentially stronger generative models in the future could alleviate this issue.

We also compare DINO-WM with a variant of AVDC, where the diffusion model is trained to generate the next observation $o_{t+1}$ conditioned on the current observation $o_{t}$ and action $a_{t}$, rather than generating an entire sequence of observations at once conditioned on a text goal. As detailed in Appendix A.5, it can be seen that the action-conditioned diffusion model diverges from the ground truth observations over long-term predictions, making it insufficient for accurate task planning.

Figure: Figure 7: Comparison of plans generated by DINO-WM and AVDC, a diffusion-based generative model.
Refer to caption: extracted/5982932/figures/avdc_comp.png

### 4.7 Decoding and Interpreting the Latents

Although DINO-WM operates in latent space and the observation model is not trained with pixel reconstruction objectives, training a decoder is still valuable for interpreting the model’s predictions. We evaluate the image quality of predicted futures across all models and find that our approach outperforms others, even those whose encoders are trained with environment-specific reconstruction objectives. We show openloop rollout visualizations in Figure 4. This demonstrates the robustness of DINO-WM despite the lack of explicit pixel-level supervision. We report two key metrics: Structural Similarity Index (SSIM) Wang et al. () and Learned Perceptual Image Patch Similarity (LPIPS) Zhang et al. () on the reconstruction of world models’ predicted future states. SSIM measures the perceived quality of images by evaluating structural information and luminance consistency between predicted and ground-truth images, with higher values indicating greater similarity. LPIPS, on the other hand, assesses perceptual similarity by comparing deep representations of images, with lower scores reflecting closer visual similarity.

**Table 4: Comparison of world models across different environments on LPIPS and SSIM metrics.**
|  | LPIPS $\downarrow$ | SSIM $\uparrow$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | PushT | Wall | Rope | Granular | PushT | Wall | Rope | Granular |
| R3M | 0.045 | 0.0083 | 0.023 | 0.08 | 0.956 | 0.994 | 0.982 | 0.917 |
| ResNet | 0.063 | 0.0024 | 0.025 | 0.08 | 0.950 | 0.996 | 0.980 | 0.915 |
| DinoCLS | 0.039 | 0.004 | 0.029 | 0.086 | 0.973 | 0.996 | 0.980 | 0.912 |
| AVDC | 0.046 | 0.030 | 0.060 | 0.106 | 0.959 | 0.983 | 0.979 | 0.909 |
| Ours | 0.007 | 0.0016 | 0.009 | 0.035 | 0.985 | 0.997 | 0.985 | 0.940 |

## 5 Conclusion, Limitations & Future Work

In this work, we introduce DINO-WM, a simple yet effective technique for modeling visual dynamics in latent space without the need for pixel-space reconstruction. We have demonstrated that DINO-WM captures environmental dynamics and generalizes to unseen configurations, independent of task specifications, enabling visual reasoning at test time and generating zero-shot solutions for downstream tasks through planning. DINO-WM takes a step toward bridging the gap between task-agnostic world modeling and reasoning and control, offering promising prospects for generic world models in real-world applications. For limitations, DINO-WM still relies on the availability of ground truth actions from agents, which may not always be feasible when training with vast video data from the internet. Additionally, while we currently plan in action space for downstream task solving, an extension of this work could involve developing a hierarchical structure that integrates high-level planning with low-level control policies to enable solving more fine-grained control tasks.

## Ethics Statement

This work explores creation of latent world models that can be used for better downstream planning. While we do not anticipate a potential for current misuse as this particular work, we can imagine future work that builds on this can lead to impact in robotics. Such potential applications to robotics open up a potential to misuse, which we acknowledge.

## Reproducibility Statement

All code, models, and benchmarks produced from this project will be made open-source on our project website. We also provide thorough textual descriptions of all experimental procedures in the Appendix. Appendix A.1 describes our environments, data generation, and task definitions. In Appendix A.4, we outline all the planning optimization methods that we used in this paper. We provide further comparisons for DINO-WM and using generative models as world models in Appendix A.5, and additional planning visualizations with DINO-WM in Appendix A.7.
Finally, Appendix A.6 provides the hyperparameters we used for training the world model for reproducing our experiment results in Section 4.1.

## Appendix A Appendix

### A.1 Environments and Dataset Generation

- a)
Point Maze: In this environment entroduced by Fu et al. (), the task is for a force-actuated 2-DoF ball in the catesian directions x and y to reach a target goal. The agent’s dynamics incorporate physical properties such as velocity, acceleration, and inertia, making the movement realistic. We customize the environment by altering the maze configuration to test the model’s generalization ability in unseen situations. We generate 2000 fully random trajectories to train our world models.
- b)
Push-T: This environment introduced by Chi et al. () features a pusher agent interacting with a T-shaped block. The goal is to guide both the agent and the T-block from a randomly initialized state to a known feasible target configuration within 25 steps. The task requires both the agent and the T to match the target locations. Unlike previous setups, the fixed green T no longer represents the target position for the T-block but serves purely as a visual anchor for reference. Success requires precise understanding of the contact-rich dynamics between the agent and the object, making it a challenging test for visuomotor control and object manipulation. Additionally, we introduce variations by altering the shape and color of the object to assess the model’s capability to adapt to novel tasks. We generate a dataset of 18500 samples replayed the original released expert trajectories with various level of noise and evaluate the model’s performance across all different shapes to assess its adaptability.
- c)
Wall: This custom 2D navigation environment features two rooms separated by a wall with a door. The agent’s task is to navigate from a randomized starting location in one room to a goal in the other, passing through the door. We present a variant where wall and door positions are randomized, testing the model’s generalization to novel configurations. For the fixed wall setting, we train on a fully random dataset of 2000 trajectories each with 50 time steps. For the variant with multiple training environment configurations, we generate 10240 random trajectories.
- d)
Rope Manipulation: Introduced in Zhang et al. (), this task is simulated with Nvidia Flex Zhang et al. () and consists of an XArm interacting with a soft rope placed on a tabletop. The objective is to move the rope from an arbitrary starting configuration to a goal configuration specified at inference time. For training, we generate a random dataset of 1000 trajectories of 20 time steps of random actions from random starting positions, while testing involves goal configurations set from varied initial positions, incorporating random variations in orientation and spatial displacement.
- e)
Granular Manipulation: This environment uses the same simulation setup as Rope Manipulation and involves manipulating about a hundred particles to form desired shapes. The training data consists of 1000 trajectories of 20 time steps of random actions starting from the same initial configuration, while testing is performed on specific goal shapes from diverse starting positions, along with random variations in particle distribution, spacing, and orientation.

### A.2 Environment families for testing generalization

- 1.
WallRandom: Based on the Wall environment, but with randomized wall and door positions. At test time, the task requires navigating from a random starting position on one side of the wall to a random position on the other side, with non-overlapping wall and door positions seen during training.
- 2.
PushObj: Derived from the Push-T environment, where we introduce novel block shapes, including Tetris-like blocks and a ”+” shape. We train the model with four shapes and evaluate on two unseen shapes. The task involves both the agent and object reaching target locations.
- 3.
GranularRandom: Derived from the Granular environment, where we initialize the scene with a different amount of particles. The task requires the robot to gather all particles to a square shape at a randomly sampled location. For this task, we directly take the models that are trained with a fixed amount of materials used in Section 4.3.

Visualizations can be found in Figure 6.

### A.3 Pretraining features

- a)
R3M: A ResNet-18 model pre-trained on a wide range of real-world human manipulation videos Nair et al. ().
- b)
ImageNet: A ResNet-18 model pre-trained on the ImageNet-1K dataset Russakovsky et al. ().
- c)
DINO CLS: The pre-trained DINOv2 model provides two types of embeddings: Patch and CLS. The CLS embedding is a 1-dimensional vector that encapsulates the global information of an image.

### A.4 Planning Optimization

In this section, we detail the optimization procedures for planning in our experiments.

#### A.4.1 Model Predictive Control with Cross-Entropy Method

- a)
Given the current observation $o_{0}$ and the goal observation $o_{g}$, both represented as RGB images, the observations are first encoded into latent states:
$\hat{z}_{0}=\text{enc}(o_{0}),\quad z_{g}=\text{enc}(o_{g}).$
(3)
- b)
The planning objective is defined as the mean squared error (MSE) between the predicted latent state at the final timestep $T$ and the goal latent state:
$\mathcal{C}=\left\|\hat{z}_{T}-z_{g}\right\|^{2},\quad\text{where}\quad\hat{z}
_{t}=p(\hat{z}_{t-1},a_{t-1}),\quad\hat{z}_{0}=\text{enc}(o_{0}).$
(4)
- c)
At each planning iteration, CEM samples a population of $N$ action sequences, each of length $T$, from a distribution. The initial distribution is set to be Gaussian.
- d)
For each sampled action sequence $\{a_{0},a_{1},\ldots,a_{T-1}\}$, the world model is used to predict the resulting trajectory in the latent space:
$\hat{z}_{t}=p(\hat{z}_{t-1},a_{t-1}),\quad t=1,\ldots,T.$
(5)
And the cost $\mathcal{C}$ is calculated for each trajectory.
- e)
The top $K$ action sequences with the lowest cost are selected, and the mean and covariance of the distribution are updated accordingly.
- f)
A new set of $N$ action sequences is sampled from the updated distribution, and the process repeats until success is achieved or after a fixed number of iterations that we set as hyperparameter.
- g)
After the optimization process is done, the first $k$ actions $a_{0},...a_{k}$ is executed in the environment. The process then repeats at the next time step with the new observation.

#### A.4.2 Gradient Descent:

Since our world model is differentiable, we also consider an optimization approach using Gradient Descent (GD) which directly minimizes the cost by optimizing the actions through backpropagation.

- a)
We first encode the current observation $o_{0}$ and goal observation $o_{g}$ into latent spaces:
$\hat{z}_{0}=\text{enc}(o_{0}),\quad z_{g}=\text{enc}(o_{g}).$
(6)
- b)
The objective remains the same as for CEM:
$\mathcal{C}=\left\|\hat{z}_{T}-z_{g}\right\|^{2},\quad\text{where}\quad\hat{z}
_{t}=p(\hat{z}_{t-1},a_{t-1}),\quad\hat{z}_{0}=\text{enc}(o_{0}).$
(7)
- c)
Using the gradients of the cost with respect to the action sequence $\{a_{0},a_{1},\ldots,a_{T-1}\}$, the actions are updated iteratively:
$a_{t}\leftarrow a_{t}-\eta\frac{\partial\mathcal{C}}{\partial a_{t}},\quad t=0
,\ldots,T-1,$
(8)
where $\eta$ is the learning rate
- d)
The process repeats until a fixed number of iteractions is reached, and we execute the first $k$ actions $a_{0},...,a_{k}$ in the enviornment, where $k$ is a pre-determined hyperparameter.

#### A.4.3 Planning Results

Here we present the full planning performance using various planning optimization methods. CEM denotes the setting where we use CEM to optimize a sequence of actions, and execute those actions in the environment without any correction or replan. Similarly, GD denotes optimizing with gradient decent and execute all planned actions at once in an open-loop way. MPC denotes allowing replan and receding horizon with CEM for optimization.

**Table 5: Planning results of DINO-WM**
|  | PointMaze | Push-T | Wall | Rope | Granular |
| --- | --- | --- | --- | --- | --- |
| CEM | 0.8 | 0.86 | 0.74 | NA | NA |
| GD | 0.22 | 0.28 | NA | NA | NA |
| MPC | 0.98 | 0.90 | 0.96 | 0.41 | 0.26 |

### A.5 Comparison with Action-Conditioned Generative Models

We compare DINO-WM with a variant of AVDC, where the diffusion model is trained to generate the next observation $o_{t+1}$ conditioned on the current observation $o_{t}$ and action $a_{t}$, rather than generating an entire sequence of observations at once conditioned on a text goal. We then present openloop rollout results on validation trajectories using this action-conditioned diffusion model, with visualizations shown in Figure 8. It can be seen that the action-conditioned diffusion model diverges from the ground truth observations over long-term predictions, making it insufficient for accurate task planning.

Figure: Figure 8: Openloop rollout on PushT with DINO-WM and action-conditioned AVDC (AVDC-AC). For each trajectory, the model is given the first frame as well as sequence of actions. The world models performs openloop rollout with these actions.
Refer to caption: extracted/5982932/figures/ac_avdc.png

### A.6 Hyperparameters and implementation

We present the DINO-WM hyperparameters and relevant implementation repos below. We train the world models for all environments with the same hyperparameters.

**Table 6: Environment-dependent hyperparameters for DINO-WM training**
|  | $H$ |
| --- | --- |
| PointMaze | 3 |
| Push-T | 3 |
| Wall | 1 |
| Rope | 1 |
| Granular | 1 |

- •
DINOv2: [https://github.com/facebookresearch/dinov2](https://github.com/facebookresearch/dinov2)
- •
DreamerV3:[https://github.com/NM512/dreamerv3-torch](https://github.com/NM512/dreamerv3-torch)
- •
AVDC: [https://github.com/flow-diffusion/AVDC](https://github.com/flow-diffusion/AVDC)
- •
R3M: [https://github.com/facebookresearch/r3m/](https://github.com/facebookresearch/r3m/)

We base our predictor implementation on [https://github.com/lucidrains/vit-pytorch/](https://github.com/lucidrains/vit-pytorch/).

### A.7 Additional Planning Visualizations

We present additional visualizations for planning with DINO-WM. In this setting, all planning instances share the same initial observations but have different goal observations to demonstrate DINO-WM’s generalization capabilities in planning. We show trajectory pairs to compare the environment’s observations after executing a sequence of planned actions with DINO-WM’s imagined trajectories. The left-most column denotes the initial observations, and the right-most shaded column denotes the goal observations. Each pair of rows represents a planning instance: the top (shaded) row shows the environment’s observation after executing 25 planned actions, and the bottom row shows the world model’s imagined observations.

Figure: Figure 9: Trajectories planned with DINO-WM on PushT with the same initial states but different goal states.
Refer to caption: extracted/5982932/figures/plan_pusht_same_init.png

Figure: Figure 10: Trajectories planned with DINO-WM on PointMaze with the same initial states but different goal states.
Refer to caption: extracted/5982932/figures/plan_pointmaze_same_init.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - I wonder what causes the low performance of TDMPC, is the reward fully ignored or the reward depends on the latent representation that it just learned?
soundness: 3
strengths: - Effectively leverages a pre-trained vision model to enhance world model learning.
- Conducts comprehensive evaluations across multiple tasks, validating the model's effectiveness.
summary: This paper introduces a visual dynamics learning approach called DINO-WM, leveraging pre-trained DINOv2 latent features for predictive modeling. DINO-WM incorporates a visual transformer that processes DINO features and action sequences to forecast future observations in an auto-regressive manner, trained through teacher forcing. The model demonstrates zero-shot planning capabilities when integrated with MPC using CEM. The study evaluates DINO-WM across a diverse set of tasks—PointMaze, PushT, Rope, and Granular Manipulation—and achieves 
better performance compared to state-of-the-art model-based reinforcement learning approaches.

---

post rebuttal: The authors addressed several concerns. Though I am still worried about the novelty but I will not argue for a rejection as the contribution of the paper is clear. So I increased my score.
weaknesses: - The paper's writing needs to be improved. The citations should be clearly separated from the main text using brackets when needed.
- Although DINO-WM uses DINOv2's latent features, the approach of learning a visual dynamics model in the latent space is not groundbreaking. Pretraining latent features for world model learning or learning latent world models through rewards are both exploited in previous literature. Leveraging existing vision models is not novel to me.
- In one comparison, the authors evaluate TD-MPC2 without rewards, which might not provide a fair assessment. Since reward signals are central to TD-MPC2’s optimization, omitting them potentially undermines its performance.
- The work does not include results on established image-based RL benchmarks, such as DM-Control or Minecraft, which are widely used to validate model-based approaches like TD-MPC2 and Dreamer. Including these benchmarks would strengthen claims of general applicability​.
- The model’s lack of Q-learning hinders its capacity to handle tasks requiring long-term planning. In RL, Q-learning or similar approaches are often necessary for efficiently solving complex, high-horizon tasks. This limitation may affect DINO-WM’s scalability to challenging environments.​

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - What is the inference speed of the framework? This show how well they can potentially deal with tasks that heavily require reactive controls.
- Missing references to some recent work that uses Transformers [1] or Diffusion Models [2] to learn to perform latent/visual planning for control problems from offline data.

[1] Chain-of-Thought Predictive Control

[2] Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning
soundness: 3
strengths: + The proposed world model learns to generalize to different variants of several tasks (maze navigation, robot manipulation, etc.) by using only the offline trajectory data. This makes it a potential candidate for training robot foundation models with large quantities of offline data.
+ The trajectory optimization-based test-time control enabled by the proposed world model achieves good results in a range of navigation and manipulation tasks.
+ The comparison of DINOv2 over others (e.g., R3M) as the pre-trained backbone for the world model gives us insights into how good these visual foundation models are for facilitating solving control problems.
summary: The authors present a world model to facilitate solving control problems in the physical world. The world model learns the dynamic of the environments in the latent space based on pre-trained DINOv2 that extracts both spatial-aware and object-centric latent representation from 2D images. The approach enables gradient-based trajectory optimization at inference time to deal with various control problems (maze navigation, robot manipulation, etc.). Moreover, it achieves so by only using offline trajectory data during model learning.
weaknesses: - As the authors have pointed out, latent world models are not a new thing. While the proposed framework might be unique in terms of a reconstruction-free reward-free latent-space world model, I do feel the overall novelty is a little bit limited here.
- While the authors have shown results on a range of control tasks, IMO they are more or less 2D tasks (e.g., in the particular manipulation the particles mostly move on a plane). I am curious how will the approach perform for more complicated manipulation or control tasks (e.g., Humanoid Run from DMControl and Move Chair from ManiSkill2). These tasks might require a much more accurate level of physical reasoning capabilities (objects have momentum and move in a more complex way) which can help us understand better the limit of DINOv2.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: In addition to AVDC there are more recent WM style methods based on generative methods like Genie [1] that work quite well. I'd like authors to either present comparison with this method or add a discussion addressing why the comparison is not possible.

[1] Genie: Generative Interactive Environments
soundness: 3
strengths: 1. The paper is well written and easy to follow
2. The idea proposed in the paper is quite simple, scalable and effective as demonstrated by the results. 
3. The experiments section is thorough and clearly demonstrates effectiveness of Dino-WM compared to existing methods.
summary: The paper proposes Dino-WM a method to model visual dynamics without reconstructing the visual world by leveraging patch features from pretrained visual encoder DinoV2, allowing it to learn the world model from offline datasets of trajectories. This formulation allows Dino-WM to realize observational goals by using action sequence optimizing enabling task-agnostic behavior planning. The paper demonstrates Dino-WM can generate solutions for tasks at test-time zero-shot without any expert demonstration, reward modeling or inverse dynamics models. In addition, Dino-WM outperforms existing methods like DreamerV3 and TD-MPC2 on complex control environments. Additionally, paper present additional analysis on using various pretrained visual encoder for world modelling to demonstrate effectiveness of using DinoV2 features and show generalization to novel environment configurations.
weaknesses: 1. The authors argue in the intro and abstract of the paper that a world model should enable learning from **only passive data** but the experiments demonstrated in the paper always require action metadata during training of Dino-WM which contradicts the motivation. I’d suggest authors to make that argument a bit less constrained to convey the world model training should include both passive and action conditioned data.
2. The comparisons presented in experiments section 4.1 do not include comparisons to diffusion based methods. I understand it might not be feasible to get results for all environments but I’d recommend authors to add comparison of AVDC or another diffusion based methods for whichever task possible in table.1. If not possible, I’d recommend authors to add a discussion about why the comparison is not possible. Currently, the section 4.2 implies there’ll be quantitative comparison with diffusion based methods.
3. The generalization experiments in section 4.5 tests generalization to unseen configurations of environments seen during training but the description in section 4.5 (L 450-451) implies that experiments test generalization to new environments on which the model was never trained which is not true. I’d recommend authors to please fix the text description in this section. 
4. I couldn’t find any training details for world models used in experiments section. It is not clear to me what datasets were used for training Dino-WM, how were the trajectories collected, what was the scale of those datasets,etc.  The appendix has hyperparam details but no training details. It’d be great if authors can add description for those details in the paper. 
5. I would also be interested in seeing scaling laws analysis for the a subset of environments presented in the paper. It’d be interesting to see how evaluation performance improves as the size of training dataset used for Dino-WM pretraining was used. I believe it will make paper stronger and would be a nice analysis to have.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The decoder could propagate gradients into the transition model if you first predicted the next state, then decoded it and compared it to the original next state image? Have you considered this design decision?
2. Some world modeling approaches roll out trajectories N steps into the future to improve the consistency of predictions over time. Would that be compatible with your approach?
3. Do you fine-tune AVDC on your datasets?
soundness: 3
strengths: 1. The authors demonstrate that frozen visual tokens from DINOv2 are enough to learn a forward model that successfully predicts the dynamics of several different robot control environments.
2. It is further shown that the world model is able to plan in altered environments because, in part, it does not depend on a task-specific reward function.
3. The proposed world model compares favorably to task-specific world models, DreamerV3 and TD-MPC2, and its decoder generates more plausible trajectories than a prior video prediction method, AVDC.
4. The rope and granular manipulation environments demonstrate the simulation of difficult transition dynamics.
summary: This work proposes a simple approach for learning a task-agnostic world model from offline data. They use frozen DINOv2 tokens and a Transformer transition model with a single-step forward prediction loss. This world model is then shown to enable model predictive control in toy environments as well as two simulated robotic environments involving manipulating a rope and many particles.
weaknesses: 1. The paper includes an insufficient set of benchmarks. I would recommend including more standard reinforcement learning and robotics benchmarks, such as DMControl, RoboMimic, MimicGen, RLBench, MetaWorld, FrankaKitchen, ManiSkill, etc. These benchmarks would better assess the world model’s ability to capture robot-object dynamics and sequential tasks. For example, the TD-MPC2 paper (used as a baseline) includes MetaWorld, DMControl, ManiSkill and MyoSuite as benchmarks. It would only be fair to compare the proposed method on some subset of the benchmarks used in TD-MPC2 or DreamerV3. I would recommend adding at least one benchmark from TD-MPC2 to strengthen the comparison. E.g., MetaWorld covers a diverse set of manipulation tasks.
2. The paper focuses on learning task-agnostic world models from offline data, but this entails a major assumption. It is assumed that the training dataset sufficiently covers the environment’s state-action space. This appears to be the case in Point Maze, Two Room, Rope Manipulation and Granular Manipulation, where the datasets are collected by executing random actions. This would not work in, e.g., RoboMimic or FrankaKitchen, where the agent has to execute a sequential task that is difficult to solve with random actions. Here, we have a chicken-and-egg problem. We need an expert dataset to train an accurate world model, but we need an accurate world model to plan expert behavior. This is why prior world modeling approaches focus on online improvement and joint learning of world models and policies.
3. I am confused by the first sentence of Section 3.1.2. It states that the ViT model is used as the transition model. But ViT is just a Transformer with tokenized images and the input to the transition model is already tokenized. Wouldn’t it be more accurate to say that a decoder-only Transformer is used as the transition model?
4. Section 3.1.2 further describes a causal masking approach of treating patch vectors of one observation as a whole unit. This design decision should be ablated in the experiments. In particular, comparing the planning success rate of DINO-WM with and without the per-image causal mask.
5. The comparison with baselines would be more fair if DreamerV3 and TD-MPC2 were modified to also use a DINOv2 image encoder, which I assume is not the case in the current experiments. Could you swap the convolutional encoder of DreamerV3 with DINOv2 and measure the impact on its planning success rate?
6. Embed to Control (https://arxiv.org/abs/1506.07365) is a major missing citation, which also investigates offline world model learning without rewards.

Meta-review: The paper presents DINO-WM, a method for learning world models that leverage pre-trained DINOv2 visual features to enable zero-shot planning. The core idea is to predict future states in the latent space of DINOv2's patch features, trained purely from offline trajectories without requiring rewards or online interaction.

Strengths: The paper demonstrates empirically that pre-trained visual features can serve as a useful representation space for world models. The evaluations are thorough, showing strong results on maze navigation, pushing, and particle manipulation tasks compared to baselines like DreamerV3 and TD-MPC2.

However, the paper has a fundamental disconnect between its claimed contribution and actual analysis. The authors identify a key challenge in offline world models: without sufficient data coverage, these methods typically require auxiliary learning objectives (like reconstruction or rewards) to learn a good latent planning space (lines 53-83). The paper proposes that using DINOv2's pre-trained features, which contain spatial and object-centric priors, can address this challenge by enabling effective planning without such auxiliary objectives.

Yet this core claim lacks rigorous justification in two critical ways. First, the paper does not establish that auxiliary objectives actually help compensate for limited data coverage - this baseline connection is necessary to motivate their solution. Second, while empirical results show DINOv2 features are effective, the paper fails to analyze why these features provide a good planning space without auxiliary learning. Without understanding what properties of DINOv2 (geometric, semantic, or physical) enable this capability, it's unclear if the method truly addresses the stated coverage challenge or simply works well for unrelated reasons. A rigorous comparison with other pre-trained representations, analyzing their relevant properties for dynamics modeling, would be necessary to properly justify the proposed solution.

Moreover, the paper falls short in benchmarking against established robotics tasks (like DMControl or RLBench) where random exploration is insufficient. While the authors added some comparisons during rebuttal, the results remain limited to simpler scenarios. The strong assumptions about data collection and the lack of a clear path to scaling to more complex tasks suggest this approach.

Given these limitations, particularly the lack of rigorous analysis explaining why DINOv2 features enable better world modeling, I recommend rejecting this paper.

Decision: Reject