DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Authors: Gaoyue Zhou
Hengkai Pan
Yann LeCun
Lerrel Pinto
Venue: ICML 2025 poster (2025)
Keywords: World Models
Planning
Representation Learning
Primary area: applications->robotics
TL;DR: We introduce a simple yet effective method for constructing versatile world models with pre-trained DINOv2 that generalize to complex environment dynamics, which enables zero-shot solutions at test time for arbitrary goals.

Abstract: The ability to predict future outcomes given control actions is fundamental for physical reasoning. However, such predictive models, often called world models, remain challenging to learn and are typically developed for task-specific solutions with online policy learning. To unlock world models' true potential, we argue that they should 1) be trainable on offline, pre-collected trajectories, 2) support test-time behavior optimization, and 3) facilitate task-agnostic reasoning. To this end, we present DINO World Model (DINO-WM), a new method to model visual dynamics without reconstructing the visual world. DINO-WM leverages spatial patch features pre-trained with DINOv2, enabling it to learn from offline behavioral trajectories by predicting future patch features. This allows DINO-WM to achieve observational goals through action sequence optimization, facilitating task-agnostic planning by treating goal features as prediction targets. We demonstrate that DINO-WM achieves zero-shot behavioral solutions at test time on six environments without expert demonstrations, reward modeling, or pre-learned inverse models, outperforming prior state-of-the-art work across diverse task families such as arbitrarily configured mazes, push manipulation with varied object shapes, and multi-particle scenarios.

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Gaoyue Zhou'
Hengkai Pan
Yann LeCun 12
Lerrel Pinto"
Abstract
2024; Hansen al., 2024; Haldar et al., 2024; Jia et al., 2024). Despite this progress, generalization remains ma- jor challenge (Zhou et al., 2023). Existing approaches pre- dominantly rely on policies that, once trained, operate in feed-forward manner during deployment- servations actions without any further optimization or reasoning. Under this framework, successful generalization inherently requires agents to possess solutions to all possi- ble tasks and scenarios once training is complete, which is only possible if the agent has seen similar scenarios during training (Reed et al., 2022; Brohan et al., 2023b;a; Etukuru et al., 2024). However, it is neither feasible nor efficient to learn solutions for all potential tasks and environments in advance.
The ability to predict future outcomes given con- trol actions is fundamental for physical reasoning. However, such predictive models, often called world models, remain challenging to learn and are typically developed for task- specific solutions with online policy learning. To unlock world mod- els' true potential, we argue that they should I) trainable on offline, pre-collected trajectories. 2) support test-time behavior optimization, and 3) facilitate task-agnostic reasoning. To this end, we present DINO World Model (DINO-WMI, a method model visual dynamics without reconstructing the visual world. DINO-W HMM lever- ages spatial patch features pre- pre-trained with DI- NOv2, enabling it to learn from offline behavioral trajectories by predicting future patch features. This allows DINO-WM to achieve observational goals through action sequence optimization, fa- cilitating task-agnostic planning by treating goal features as prediction targets. We demonstrate that DINO-W MWM achieves zero-shot behavioral so- lutions at test time on six environments without expert demonstrations, reward modeling, or pre- learned inverse models, outperforming prior state- of-the-art art work across diverse task families such arbitrarily configured mazes, push manipula- tion with varied object shapes, and multi-particle scenarios.
ob-
Instead of learning the solutions to all possible tasks dur- ing training, an alternative is to fit dynamics model on training data and optimize task-specific behavior at runtime. These dynamics models, also called world models (Ha & Schmidhuber, 2018), have a long history in robotics and control (Sutton, 1991; Todorov & Li, 2005; Williams et al., 2017). More recently, several works have shown that world models be trained 2019; Micheli et al., 2023; Robine et al., 2023; Hansen et al., 2024; Hafner et al., 2024). This enables flexible use of model-based optimization to obtain policies as circum- vents the need for explicit state-cstimation. Despite this, significant challenges remain its use for solving general- purpose tasks.
sensory data (Hafner et al.,
To understand the challenges in world modeling, let us con- sider the two broad paradigms in learning world models: online and offline. In the online setting, access to the envi- ronment is often required so data can be continuously col- lected to improve the world model, which turn improves the policy and the subsequent data collection. However, the online world model is only accurate in the cover of the policy that was being optimized. Hence, while it can be used to train powerful task-specific policies, it requires re- training for every new task even in the same environment. Instead, the offline setting, the world model is trained on an offline dataset of collected trajectories the environ- ment, which removes its dependence on the task specificity given sufficient coverage in the dataset. However, when
1. Introduction
Robotics and embodied AI have seen tremendous progress in recent years. Advances in imitation learning and rein- forcement learning have enabled agents to leam complex behaviors across diverse tasks (Agarwal et al., 2022: Zhao et al., 2023; Lee al., 2024; Ma et al., 2024; Hafner al.,
'Courant Institute, New York University "Meta AL. Correspon- dence to: Gaoyue Zhou <gz2123 (0r2213@@mmm nyu.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
required to solve a task, methods in this domain require strong auxiliary information which can take the form of ex- pert demonstrations (Pathak et al., 2018; Wang et al., 2023), structured keypoints (Ko et al., 2023: Wen et al., 2024), ac- cess to pretrained inverse models (Du et al., 2023; Ko et al., 2023) or dense reward functions (Ding et al., 2024), all of which reduce the generality of using offline world models. The central question in building better offline world models is if there is alternate auxiliary information that does not compromise its generality?
available on our project website: https://dino github.io/.
2. Related Work
We build on top of several works in developing world mod- els, optimizing behaviors from them, and leveraging com- pact visual representations. For conciseness, we only dis- cuss the ones most relevant to DINO-WM.
Model-based Learning: Learning from models of dynam- ics has a rich literature spanning the fields of control, plan- ning, and robotics (Sutton, 1991; Todorov & Li, 2005; As- tolfi et al., 2008; Holkar & Waghmare, 2010; Williams et al., 2017). Recent works have shown that modeling dynam- ies and predicting future states can significantly enhance vision-based learning for embodied agents across various ap- plications, including online reinforcement learning (Micheli et al., 2023; Robine et al., 2023; Hansen et al., 2024; Hafner et al., 2024), exploration (Sekar et al., 2020; Mendonca et al., 2021; 2023a), planning ( (Watter et al., 2015) (Finn & Levine, 2017; Ebert et al., 2018; Hafner et al., 2019), and imitation learning (Pathak et al.. 2018). Several of these approaches initially focused on state-space dynamics (Deisenroth & Rasmussen, 2011; Lenz et al., 2015; Chua et al., 2018; Nagabandi et al., 2019), and have since been extended to handle image- bassee inputs, which we address in this work. These world models can predict future states in either pixel space (Finn & Levine, 2017; Ebert et al., 2018, 88; Ko et al., 2023; Du et al., 2023) or latent representation space (Yan et al., 2021). Predicting in pixel space, however, is computationally expensive due to the need for image re- construction and the overhead of using diffusion models (Ko et al., 2023). On the other hand, latent-space prediction is typically tied to image reconstruction objectives (Hafner et al., 2019; Micheli et al., 2023; Hafner et al., 2024), which raises concerns about whether the learned features contain sufficient information about the task. Moreover, many of these models incorporate reward prediction (Micheli et al., 2023; Robine et al., 2023; Hafner et al., 2024), or use reward prediction as an auxiliary objective to learn the latent repre- sentation (Hansen et al., 2022; 2024), inherently making the world model -ssk-spec...c. In this work, we aim to decou- ple task pask-dependen information from latent-space prediction, striving develop versatile and task-agnostic world model capable of generalizing across different scenarios.
In this work, we present DINO- WM, a new and simple method to build task-agnostic world models from an offlin dataset of trajectories (Figure 1). DINO- WM models the world dynamics on compact embeddings of the world, rather than the raw observations themselves. For the embedding, use pretrained patch-features from the DINOv2 model, which provides both a spatial and object-centric representa- tion prior. We conjecture that this pretrained representation enables robust and consistent world modeling, which re- laxes the necessity for task-specific data coverage. Given these visual embeddings and actions, DINO- WM uses the ViT architecture to predict future embeddings. Once this model is trained on the offline dataset, planning to solve tasks is constructed as visual goal reaching, i.c. to reach future desired goal given the current observation. Since the predictions by DINO-WM M are high quality (see Figure 4), we can simply use model predictive control with inference- time optimization to reach desired goals without any extra information during testing.
DINO- WM is experimentally evaluated six environ- ment suites spanning maze navigation, sliding manipulation, robotic arm control, and deformable object manipulation tasks. Our experiments reveal the following findings:
DINO-WN produces high-quality future world modeling that can be measured by improved visual reconstruction from trained decoders. On LPIPS metries for our hardest tasks, this improves upon prior ttat--of-the-ar work by 56% (See Section 4.7).
Given the latent world models trained using DINO-WM, we show high success for reaching arbitrary goals our hardest tasks, improving upon prior work by 45% average (See Section 4.3).
DINO-WM can be trained across environment variations within a task family (e.g. different maze layouts for nav- igation or different object shapes for manipulation) and achieve higher rates of success compared to prior work (See Section 4.5).
With the recent excitement of large scale foundation models, there have been initiatives on building large-scale video generation world models conditioned on agent's actions in the do- main of self-driving (Hu et al., 2023), control (Yang et al., 2023; Bruce et al., 2024), and general-purposs video gen- eration (Liu et al., 2024). These models aim to generate video predictions conditioned on text or high-level action
Generative Models as World Models:
Code and models for DINO-WM are oounesourced to ensure reproducibility and videos of planning are made
2

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
(a) Training DINO-WM
(b) Test-time Inference
(c) Planning Performance
2T
IRIS
Zt
2++1
1
Ze
g
TD-MPC2
22
DINOv2
DINOv2
DINOY2
DINOv2
Dreamerv3
..04
920
DINO-WM
ag
1+1
01
O g
Chamfer distance
Figure present DINO-WM, WM, method for training visual models by using pretrained DINOv2 embeddings of image frames (a). Once trained. given target observation or, can directly optimize agent behavior by planning through DINO- DINO-WM using model predictive control (b). The use pretrained embeddings significantly improves performance over prior state-of-the-ar world models (c).
sequences. While these models have demonstrated utility in downstream tasks like data augmentations, their reliance on language conditioning limits their application when precise visually indicative goals need to be reached. Additionally, the use of diffusion models for video generation makes them computationally expensive, further restricting their applica- bility for test-time optimization techniques such MPC. In this work, we aim to build a world model in latent space instead of raw pixel space, enabling more precise planning and control.
3. DINO World Models
Overview and Problem Formulation: Our work follows the vision-based control task framework, which models the environment as a partially observable Markov deci- sion process (POMDP). The POMDP is defined by the tu- ple (O, p). where represents the observation space, and A denotes the action space. The dynamies of the environment are modeled by the transition distribution p(o++1 0gt: Age), which predicts future observations based past actions and observations.
Pretrained Visual Representations: Significant advance- ments have been made in the field of visual representa- tion learning, where compact features that capture spatial and semantic information can be readily used for down- stream tasks. Pre-trained models like ImageNet pre-trained ResNet (He et al., 2016), I-JEPA (Assran et al., 2023), and DINO (Caron et al., 2021; Oquab et al., 2024) for images, as well as V-JEPA (Bardes al., 2024) for videos, and R3M (Nair et al., 2022), MVP (Xiao al., 2022) for robotics have allowed fast adaptation to downstream tasks they contain rich spatial and semantic information. While many of these models represent images using single global feature, the introduction of Vision Transformers (ViT') (Dosovitskiy et al., 2021) has enabled the use of pre-trained patch fea- tures, as demonstrated by DINO (Caron et al., 2021; Oquab al., 2024). DINO employs a celf-distillation loss that allows the model to learn representations effectively, captur- ing semantic layouts and improving spatial understanding within images. In this work, we leverage DINOv2's patch embeddings to train our world model, and demonstrate that serves as a versatile encoder capable of handling various precise tasks.
In this work, we aim to learn task-agnostic world models from precollected offline datasets, and use these world mod- els to perform visual reasoning and control at test time. At test time, our system starts from an arbitrary environment state and is provided with goal observation in the form of RGB image, in line with prior works (Ebert et al., 2018; Wu et al., 2020; Mendonca et al., 2023b), and is asked to perform a sequence of actions ao .... ar to reach the goal state. This approach differs from the world models used in online reinforcement learning (RL) where the objective is to optimize the rewards for fixed set of tasks at hand (Hafner et al., 2024; Hansen et al., 2024), or from text-conditioned world models, where the goals are specified through text prompts (Du et al., 2023; Ko et al., 2023).
3.1. DINO-based World Models (DINO-WM)
We model the dynamics of the environment in the latent space. More specifically, at each time step t, our world model consists of the following components:
Observation model:
~ enco(21 0z)
Transition model:
2t41 ~ po | 2t H:t. Hc)Ha)
Decoder model:
Ô, ~ qo 22)
(optional for visualization)
3

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Actions optimized test-time
Or-k
0++1
DINOV2
DINOV2
Planning loss
1
2/42
Figure 2.
Architecture of DINO-WM. Given observations Ortect, we optimize the sequence of actions d:T- minimize the predicted loss to the desired goal og. All forward computation is done in the latent space 2. Here pa indicates DINO-WM' dynamics model, which is used for making future predictions.
where the observation model encodes image observations to latent states 2t, and the transition model takes in history of past latent states of length H. The decoder model takes in a latent 22. and reconstructs the image observation oe. We use 0 denote the parameters of these models. Note that our decoder is entirely optional, as the training objectives for the decoder are independent for training the rest part of the world model. This eliminates the need to reconstruct images both during training and testing, which reduces com- putational costs compared to otherwise coupling together the training of the observational model and the decoder, as in (Micheli et al., 2023; Hafner et al., 2024). We ablate and show the effectiveness of this choice in Appendix A.4.2.
each time step f. it encodes an image Or to patch embed- dings 24 E N where N denotes the number of patches, and E denotes the embedding dimension. This process visualized in Figure 2.
1.1.2. TRANSITION MODEL
We adopt the VIT architecture (Dosovitskiy et al., 2021) for the transition model due to its suitability for processing patch features. We remove the tokenization layer, as it operates on patch embeddings, effectively transforming it into a decoder-only transformer. We further make a few modifications to the architecture to allow for additional conditioning proprioception and controller actions.
DINO-WAY models only the information available from offline trajectory data in an environment, in contrast to re- cent online RL world models that also require task-relevant information, such as rewards (Hafner et al., 2020; Hansen al., 2022: 2024), discount factors (Hafner al., 2022: Robine et al., 2023), and termination conditions (Micheli et al., 2023; Hafner et al., 2024).
Our transition model takes in history of past latent states 2t-Htt- and actions H--H:t1, where H is a hyperparam- eter denoting the context length of the model, and predicts the latent state at next time step žt. To properly capture the temporal dependencies, where the world state at time t should only depend on previous observations and actions, implement a causal attention mechanism in the ViT model, enabling the model to predict latents autoregres- sively at frame level. Specifically, each patch vector z for the latent state 2, attends to 1 This is different from past work IRIS (Micheli et al., 2023) which similarly represents each observation as a sequence of vec- tors, but autoregressively predict of at a token level, attend- ing { 1 as well as {=])<k We argue that predicting at frame level and treating patch vectors of one observation as whole better captures global structure and temporal dynamics, modeling dependencies across the entire observation rather than isolated tokens, leading to improved temporal generalization. The effectiveness of this attention mask has been shown in our ablation experiments in Appendix A.4.1
3. OBSERVATION MODEL
To learn generic world model across many environments and the real world, we argue that the observation model should 1) be task and environment independent, and 2) capture rich spatial information for navigation and manip- ulation. Contrary previous work where the observation model is always learned for the task at hand (Hafner et al., 2024), we argue instead that it can be inefficient and often not possible to learn good observation model from scratch when facing a new environment, as perception is a general task that benefits from large-scale internet data. Therefore, we use the pre-trained DINOv2 model as our world model's observation model, leveraging its strong spatial understand- ing for tasks like object detection, semantic segmentation, and depth estimation (Oquab et al., 2024). The observa- tion model remains frozen during training and testing. At
To model the effect of the agent's 's action to the environ- ment, we condition the world model's 's predictions on these
4

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
actions. Specifically, concatenate the K-dimensional ac- tion vector, mapped from the original action representation using a multi-layer perceptron (MLP), to each patch vector 2 for = ... When proprioceptive information is available, we incorporate it similarly by concatenating it to the observation latents, thereby integrating it into the latent states.
We utilize the cross-entropy method (CEM) optimize the sequence of actions each iteration. The planning cost is defined as the mean squared error (MSE) between the current latent state and the goal's s latent state, given by
& p( ac-1);
C = 112 2gl1", where
Eo = enc(ou).
We train the world model with teacher forcing, During training. slice the trajectories into segments of length H + 1, and compute a latent consistency loss on each of the H predicted frames. For each frame, we compute
Zg = enc(og)-
The MPC framework and CEM optimization procedure are detailed in Appendix 1. Since our world model is differ- entiable, a possibly more efficient approach is to optimize this objective through gradient descent (GD), allowwng the world model to directly guide the agent toward a specific goal. The details of GD are provided in Appendix A.5.2. However, we empirically observe that CEM outperforms GD in our experiments with full results in Appendix A.5.3, We hypothesize that incorporating regularizations during training and in the planning objectives could further improve performance, and leave this for future work.
Cpred = lipe (enco(ot-
- ence (lI+1)ll? (1)
where o is the action encoder model that can map actions to higher dimensions. Note that our world model training is entirely performed in latent space, without the need to
reconstruct the original pixel images.
3.1.3. DECODER FOR INTERPRETABILITY
To aid in visualization and interpretability, we use a stack of transposed convolution layers to decode the patch repre- sentations back to image pixels, similar as in (Razavi et al., 2019). Given a pre-collected dataset, we optimize the pa- rameters of the decoder qo with a simple reconstruction loss defined as:
4. Experiments
Our experiments are designed to address the following key questions: 1) Can we effectively train DINO- WW using precollected offline datasets? 2) Once trained, can DINO- WM be used for visual planning? 3) To what extent does the quality of the world model depend on pre-trained vi- sual representations? 4) Does DINO-WM generalize to new configurations, such variations in spatial layouts and object arrangements? 5) How does DINO-WM' perfor- mance scale with offline dataset size? We train and evaluate DINO- WM across six environment suites (full description in Appendix A. comparing it to state-of-the-arr world models that predict in either latent space or raw pixel space.
Free (z,) oell? where 2 (o.)
(2)
The training of the decoder is entirely independent of the transition model training, offering several advantages: 1)
The decoder does not affect the world model's reasoning and planning capabilities for solving downstream tasks, and 2) There is no need to reconstruct raw pixel images during plan- ning, thereby reducing computational costs. Nevertheless, the decoder remains valuable as it enhances the interpretabil- ity of the world model's predictions. While backpropagating this decoder loss to the predictor is possible, we ablate this choice and find that it negatively impacts performance com- pared to omitting the decoder loss. Full details are provided in Appendix A.4.2.
4.1. Environments and Tasks
We evaluate six environment suites with varying dynam- ies complexity, some of which are drawn from standard robotics benchmarks, such as D4RL (Fu et al., 2021) and DeepMind Control Suite (Tassa et al., 2018), as shown in Figure 3. These environments include maze navigation (Maze, Wall), (PushT) and robotic arm control (Reach), and deformable object manipulation with an XArm (Rope, Granular).
3.2. Visual Planning with DINO- -M
control for tabletop pushing
To evaluate the quality of the world model, we perform tra- jectory optimization at test time and measure performance. While the planning methods themselves are fairly standard, they serve as means to emphasize the quality of the world models. For this purpose, our world model receives the current observation ou and goal observation og. both rep- resented as RGB images. We formulate planning as the process of searching for a sequence of actions that the agent would take to reach o,- We employ model predictive con- trol (MPC), which facilitates planning by considering the outcomes of future actions
In all environments, the task is reach a randomly sampled goal state specified by a target observation, starting from arbitrary initial states. For PushTT, target configurations are sampled to ensure feasibility within 25 steps. For Granu- lar, targets require gathering all particles into a square with randomized locations and sizes. Observations in all environ- ments are RGB images of size (224, 224). A full description of the environments provided in Appendix A.
5

---

DINO-WIY WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Distance (CD) 10 instances for them. In Granular, we sample random configuration from the validation set, with the goal of pushing the materials into square shape at randomly selected location and scale.
Table I. Planning results for offline world models on six control
environments.
Model
Maze
Wall
Reach
PushT
Rope
Granular
SR f
SR f
SR t
SR t
CD +
CD |
IRIS
0.74
0.04
0.18
0.32
1.11
0.37
DreamerV3
1.00
1.00
0.64
0.30
2.49
1.05
TD-MPC2
0.00
0.00
0.00
0.00
2.52
1.21
Ours
0.98
0.96
0.92
0.90
0.41
0.26
Figure evaluate DINO WM on six environment suites, from left to right, top to bottom: Maze, Reach, Wall, Push-T, Rope Manipulation, and Granular Manipulation.
As seen in Table 1, on simpler environments such as Wall and PointMaze, DINO-WM is on par with state-of-art world models like DreamerV3. However, DINO-W WM sig- nificantly outperforms prior work at manipulation environ- ments where rich contact information and object dynamics need to be accurately inferred for task completion. We no- tice that for TD-MPC2, the lack of reward signal makes it difficult to learn good latent representations, which sub- sequently results in poor performance. Visualizations of planning on all environments can be found in Appendix A.8.
4.2. Baselines
We compare DINO-WI with the following state-of-the-art models commonly used for control. For IRIS, Dreamer V3, and TD-MPC2, we train the models with our offline datasets without any reward task information, and perform MPC on the learned world model for solving downstream tasks.
Does DINO- WM learn better environment dynamics as more data become available? We conduct set of ablation experiments in Section 4.8, showing that the planning per- formance scales positively with the amount of training data. We also present the full inference and planning times for DINO-WM, in Appendix A.6, showing significant speedup over traditional simulation, particularly in the computation- ally intensive deformable environments.
a) IRIS (Micheli et al., 2023): IRIS encodes visual in- puts into tokens via a discrete autoencoder and predicts future tokens using GPT Transformer, enabling policy and value learning through imagination.
b) DreamerV3 (Hafner et al., 2024): DreamerV3 en- codes visual inputs into categorical representations, pre- dicts future states and rewards, and trains an actor-critic policy from imagined trajectories.
4.4. Does pre-trained visual representations matter?
TD-MPC2 (Hansen et al., 2024) : TD-MPC2 learns decoder-free world model in latent space and uses reward signals to optimize the latents.
We use different pre-trained general-purpose encoders as the observation model the world model, and evaluate their downstream planning performance. Specifically, we use the following encoders commonly used in roboties control and general perception: R3M (Nair et al., 2022), ImageNet pretrained ResNet-18 18 (Russakovsky et al., 2015; He et al., 2016) and DINO CLS (Caron et al., 2021). Detailed descrip- tions of these encoders are in Appendix A.3.
d) AVDC (Ko et al., 2023): AVDC uses diffusion model to generate task execution videos from an initial observa- tion and textual goal. We provide qualitative evaluations and MPC planning results for an action-conditioned variant Section 4.6.
Table 2. Planning results for world models with various pre-trained encoders. Model
4.3. Optimizing Behaviors with DINO- MM
Maze SR
Wall
Reach
PushT SR
Rope
Granular
With a trained world model, study if DINO- WM can be used for zero-shot planning directly in the latent space.
SR f
1
CD |
CD
R3M
0.94
0.34
0.40
0.42
1.13
0.95
ResNet
0.98
0.12
0.06
0.20
1.08
0.90
For Maze, Reach, PushT, and Wall environments, we sample 50 initial and goal states and measure the success rate across all instances. Due to the environment stepping time for the Rope and Granular environments, we evaluate the Chamfer
DINO CLS
0.96
0.58
0.60
0.44
0.84
0.79
DINOPatch (Ours)
0.98
0.96
0.92
0.90
0.41
0.26
We report the planning performance in Table 2. the Point-
6

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
DreamerVS
X
H
H
IRIS
F
Ours
X
F
T
GT
T
DreamerV3
IRIS
Ours
Figure Open-loop rollouts of world models Push-Ti T and Granular. Given the first frame and action sequence, each model predicts future frames, reconstructed by its decoder. For each environment, the bottom row denotes the ground truth. DINO- WW (Ours) rollouts bolded and are visually indistinguishable from the ground truth observations.
Maze task, which involves simple dynamics and control, observe that world models with various observation coders all achieve near-perfect success rates. However, more
we compare the linear probe results on three enviromments across both patch-based features (DINOv2 of various ViT sizes, pre-trained MAE (He et al., 2021) and global features (DINO CLS, R3M). The validation loss for these linear probes is reported in Table 3, where DINO-S Patch and DINO-B Patch achieve the lowest validation loss, indicating their superior task representation capabilities. While the pre-trained MAE also has patch-based features, has much higher validation loss. We hypothesize this is because MAE prioritizes reconstruction less preferable choice for control tasks.
the environment's complexity precise control and spatial meristaaninngworle models that encode observations as a single latent vector show a significant drop performance. We posit that patch-based representations better capture spatial information, con- trast to models like R3M, ResNet, and DINO CLS, which reduce observations to single global feature vector, losing crucial spatial details necessary for manipulation tasks.
task relevance, making it
To better understand the strong planning performance en- abled by DINOy2 features, we analyze the features directly. well-established method for evaluating feature quality in downstream control tasks is linear probing from the fea- tures to environment states, which assesses how well the features encode task-relevan state information. To this end,
4.5. Generalizing to Novel Environment Configurations
We evaluate the generalization of world models only across different goals but also across various en- vironment configurations. We construct three environ- ment families- WallRandom, PushObj. and GranularRan-

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
encounters fewer than half the particles present during train- ing, resulting in out-of-distribation images compared the training instances. Nevertheless, DINO-WM accurately encodes the scene and successfully gathers the particles into designated square location with the lowest Chamfer Distance (CD) compared to the baselines, demonstrating better scene understanding. We hypothesize that this is due to DINO COW''ss observation model encoding the scene as patch features, making the variance in particle number still within the distribution for each image patch.
Table Linear Probe Validation Loss for Pre-trained Encoders. We evaluate the linear probe performance by mapping the embed- dings from each encoder the state vector of each environment. DINO-S and DINO-B denote DINOv2 models with ViT-Small ViT-Base architectures, respectively. For patch-based features (DINO-S Patch, DINO-B Patch, and Pre-traincd MAB), we first flatten the patch embeddings, project them to a 1536-dimensional vector, and then feed them into linear probe model. Our results show that DINO-S Patch and DINO-B Patch achieve the lowest validation loss.
Method
PointMaze
PushT
Wall
DINO-S Patch
0,017
0.434
0.184
4.6. Qualitative Comparisons with Generative Video Models
DINO-B Patch
0.014
0.504
0.163
DINO-S CLS
0.475
0.833
0.519
Given the prominence of generative video models, it's natu- ral to assume they could serve as world models. We compare DINO-WM with AVDC (Ko et al., 2023), diffusion- generative model. As shown in Figure 6, while AVDC can generate visually realistic future images, these images lack physical plausibility. Large, unrealistic changes can occur within single timestep, and the model struggles to reach the exact goal state. Future advancements in generative models may help address these issues.
Pre-trained MAE
0.856
0.804
0.711
R3M
0.192
0.902
0.539
Test
Test
Test
Ours
Figure 5. Training and testing setups for WallRandom, PushObj and GranularRandom. Test setups are highlighted in blue.
AYDO
dom-where the model is tested on unseen configurations with random goals. Visualizations of training and testing examples are shown in Figure and detailed descriptions of the environments can be found Appendix A.2.
Start
Goal
Figure 6. Plans generated by DINO-WM and AVDC.
We further compare DINO- WM with a variant of AVDC, where the diffusion model is trained to generate the next observation 0:+1 conditioned on the current observation o and action d,, rather than generating entire sequence of observations at once conditioned on a text goal. We present open-loop rollout results on validation trajectories using this action-conditioned AVDC, with visualizations shown in Figure 7. It can be seen that the action-conditio conditioned AVDC diverges from the ground truth observations over long-term predictions, making insufficient for accurate task planning.
Table 4. Planning results for offline world models three suites with unseen environment configurations.
Model
WallRandom
PushObj GranularRandom
SR 1
SR f
CD +
IRIS
0.06
0.14
0.86
D3eamerV3
0.76
0.18
1.53
R3M
0.40
0.16
1.12
ResNet
0.40
0.14
0.98
DINO CLS
0.64
0.18
1.36
Ours
0.82
0.34
0.63
4.7. Decoding and Interpreting the Latents
From Table 4, we observe that DINO- WM demonstrates significantly better performance in WallRandom, indicat- ing that model has effectively learned the general concepts of walls and doors, even when they are positioned lo- cations unseen during training. In contrast, other methods struggle to accurately identify the door's position and nav- igate through it. The PushObj task remains challenging for all methods, as the model was only trained on the four object shapes, which makes difficult to precisely infer rel- evant physical parameters. Granular the agent
Although DINO- WM operates in latent space and the ob- servation model is not trained with pixel reconstruction ob- jectives, training a decoder aids in interpreting predictions. We evaluate the image quality of predicted futures across all models and find that our approach outperforms others, even those whose encoders are trained with cavironment-speciinn reconstruction objectives. Open-loop rollouts in Figure demonstrate DINO- WM's robustness despite the lack of explicit pixel supervision. We report the Learned Perceptual
8

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
4.8. Scaling Laws of DINO-WM
GT
To analyze the scaling behavior of DINO-WM, we trained world models and performed planning using datasets of varying sizes, ranging from 200 to 18500 trajectories the PushT environment. Our results in Table demonstrate a clear trend: as the dataset size increases, both the quality of the world model's predictions and the performance of the planned behavior improve significantly. Larger datasets enable the world model to capture more diverse dynamies and nuances of the environment, leading to more accurate predictions and better-informed planning.
AVDC-AC
Gr
AV
E
Table 7. Planning performance and prediction quality on PushT with DINO WM trained on datasets of various sizes. SSIM and LPIPS are measured on the predicted future latents after decoding. We observe consistent improvement performance as we increase
A
AVOC-AC
the datasct size.
Dataset Size
SR f
SSIM t
LPIPS 4
n=200
0.08
0.949
0.056
Figure 7. Open-loop rollout on PushT with DINO-W WM and action- conditioned AVDC (AVDC-AC). For each trajectory, the model is given the first frame well sequence of actions. The world models perform ooen-loop rollout with these actions.
n= 1000
0.48
0.973
0.013
n=5000
0.72
0.981
0.007
n=10000
0.88
0.984
0.006
n= 18500
0.92
0.987
0.005
Image Patch Similarity (LPIPS) (Zhang et al., 2018) and Structural Similarity Index (SSIM) (Wang et al., 2004) on the world models' predicted future frames in Table and Table 6. SSIM measures the perceived quality of images by evaluating structural information and luminance consistency between predicted and ground-trutt images, with higher values indicating greater similarity. LPIPS assesses percep- tual similarity by comparing deep representations of images, with lower scores reflecting closer visual similarity.
5. Conclusion
We introduce DINO-WNY a simple yet effective technique for modeling visual dynamics in latent space without the need for pixel-space reconstruction. We have demonstrated that DINO- WM captures environmental dynamics and gen- eralizes unseen configurations, independent of task spec- ifications, enabling visual reasoning at test time and gen- erating zero-shot solutions for downstream tasks through planning. DINO-WM takes step toward bridging the gap between task-agnostic world modeling and reasoning and control, offering promising prospects for generic world models in real-world applications.
Table 5. Comparison of world model precitions LPIPS (L).
Method
P
Wall
Rope
Granular
R3M
0.045
0.008
0.023
0.080
ResNet
0.063
0.002
0.025
0.080
DINO CLS
0.039
0.004
0.029
0.086
Limitations and Future Work: First, DINO- WWM assumes access to offline datasets with sufficient state-action cover- age, which can be challenging to obtain for highly complex environments. This can potentially be addressed by combin- ing DINO- WM with exploration strategies and updating the model as new experiences are available. Second, DINO- WM still relies on the availability of ground truth actions from agents, which may not always be feasible when train- ing with vast video data from the internet. Lastly, while we currently plan in action space for downstream task solv- ing, an extension of this work could involve developing a hierarchical structure that integrates high-level planning with low-level control policies to enable solving more fine- grained control tasks.
AVDC
0.046
0.030
0.060
0.106
Ours
0.007
0.0016
0.009
0.035
Table Comparison of world model precitions SSIM (t)-
Method
PushT
Wall
Rope
Granular
R3M
0.956
0.994
0.982
0.917
ResNet
0.950
0.996
0.980
0.915
DINO CLS
0.973
0.996
0.980
0.912
AVDC
0.959
0.983
0.979
0.909
Ours
0.985
0.997
0.985
0.940
9

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Acknowledgements
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis,
J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog,
We would like to thank Ademi Adeniji, Alfredo Canziani, Amir Bar, Kevin Zhang. Mido Assran, Vlad Sobal, Zichen Jeff Cui for their valuable discussion and feedback. This work supported by grants from Honda, Hyundai, NSF award 2339096 and ONR awards N00014-21-1-2777 and N00014-22-1-2773. LP is supported by the Packard Fellow- ship.
A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T., Jesmonth, S., Joshi, N. J.. Julian, R., Kalashnikov, D., Kuang, Y. Leal, I., Lee, K.H Levine, S., Lu, Y. Malla, U., Manjunath, D., Mordatch, I., Nachum, O., Parada, C., Peralta, J., Perez, E., Pertsch, K. Quiambao, J., Rao, K., Ryoo, M., Salazar, G., Sanketi, P., Sayed. K., Singh, J.. Sontakke, S., Stone, A., Tan, C., Tran, H., Vanhoucke, V. Vega, S., Vuong, Q. Xia, F,, Xiao, T., Xu, P., Xu, S., Yu, and Zitkovich, B. Rt-1: Robotics transformer for real-world control at scale, 2023b. URL https: / //arxiv. owg. abs 221.222222..0022 06817.
Impact Statement
This paper presents work whose goal is to facilitate the learning and applications of task-agnostic world models. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Bruce, J., Dennis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C., Aytar, Y., Bechtle, S., Behbahani, F. Chan, S., Heess, N., Gonzalez, L., Osindero, S., Ozair, S., Reed, S., Zhang, J., Zolna, K., Clune, J., de Freitas, N., Singh, S., and Rocktäschel, T. Ge- nie: Generative interactive environments, 2024. URL https: http//axxivvi...... httpgg/aa///442// wttp://www..r.././2//2222222..2.
References
Agarwal, A., Kumar, A., Malik, J., and Pathak, D. Legged locomotion in challenging terrains using egocentric vi- sion, 2022. URL https: : /tt////axivvv///a//////eeee.aaa/ www.absg/////oooooooo/.o..o....e 07638.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers, 2021. URL https: / //arxiv..
Assran, M., Duval, Q., Misra, I., Bojanowski, P., Vincent, P., Rabbat, M., LeCun, Y., and Ballas, N. Self-supervised learning from images with joint-embedding predictive architecture. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619- 15629, 2023.
Chi, C., Xu, Z.. Feng, S., Cousineau, E., Du. Y., Burchfiel, B., Tedrake, R., and Song. Diffusion policy: Visuo- motor policy learning via action diffusion, 2024. URL https///wwwww....aaaeeeaeee
Astolfi, A.. Karagiannis, D., and Ortega, R. Nonlinear and adaptive control with applications, volume 187, Springer, 2008.
04137.
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials us- ing probabilistic dynamics models, 2018. URL https: / wwrxxiv.org/aee 121111114......
Bardes, A., Garrido, Q., Ponce, J., Chen, X., Rabbat, M., LeCun, Y., Assran, M., and Ballas, N. V-JEPA: La- tent video prediction for visual representation learning, 2024. URL httpttpp///ppeeeeeie..eeeeeeeeee Yd=EEYbBOOO.v.
Deisenroth, M. P. and Rasmussen, C. E. Pilco: A model-based and data-efficient approach policy search. In International Conference Machine Learning, 2011. URL https: http://appimimmmme wer..sssaaaniccholl...o org org/ stg//oorpuIDD:147730
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X.. Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., Florence, P., Fu, C., Arenas, M. G., Gopalakr- ishnan, K., Han, K., Hausman, K., Herzog, A., Hsu, J., Ichter, B., Irpan, A., Joshi, N., Julian, R., Kalashnikov, D., Kuang, Y.. Leal, I., Lee, L., Lee, T.-W. E., Levine, S., Lu, Y., Michalewski, H.. Mordatch, I.. Pertsch, K., Rao, K., Reymann, K., Ryoo, M., Salazar, G., Sanketi, P. Sermanet, P., Singh, J.. Singh, A., Soricut, R., Tran, H., Vanhoucke, V., Vuong, Q., Wahid, A., Welker, S., Wohlhart, P., Wu, J., Xia, F. Xiao, T., Xu, P., Xu, S., Yu, T., and Zitkovich, B. Rt-2: Vasion-language-actin models transfer web knowledge to robotic control, 2023a. URL https::/ //a //t.xvvvvvv///////////./////wwvv org/ abs/ 15818.
Ding, Z.,, Zhang, A., Tian, Y., and Zheng, Q. Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning, 2024. URL https: a arxiv. abss////////// 2402.045 03570.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image worth 16x16 words: Transformers for image recognition at scale, 2021. URL https: //arxiv. war.iv.org/abs/20000 1999..
10

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Du, Y., Yang, M., Dai, B., Dai, H., Nachum, O., Tenenbaum, B., Schuurmans, D., and Abbeel, P. Learning universal policies via text-guided video generation, 2023. URL https: arxiv.a....... org/ abs / 2202. 00111.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A., and Levine, S. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control, 2018. URL https: /twpaaaxvvv///////////..///aaav. org/ abs 1812. 00568.
He, K., Chen, X., Xie, S., Li. Y., Dollär, P., and Girshick, Masked autoencoders are scalable vision learners, 2021. URL https: wrwaa.awiv..oo org/ / aabs /2///11111 06377.
Holkar, K. and Waghmare, I.. M. An overview of model predictive control. International Journal of control and automation, 3(4):47-63, 2010.
Etukuru, H., Naka, N., Hu, Z., Lee, S., Mehu, J., Edsinger, A., Paxton, C., Chintala, S., Pinto, L., and Shafiullah, N. M. M. Robot utility models: General policies for zero- shot deployment in new environments. arXiv preprint arXiv:2409.05885. 2024.
Hu, A., Russell, L., Yeo, H.,, Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., and Corrado, G. Gaia-1: generative world model for autonomous driving, 2023.
Finn, C. and Levine, S. Deep visual foresight for planning robot motion, 2017. URL https: http://aaxvvvv///e org/ abs/1610.0063
Jia, Z., Thumuluri, V., Liu, F. Chen, L., Huang, Z., and
Su, H. Chain-of-thought predictive control, 2024. URL https:/ wwtp /acbss//3344 230..00000777
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning, 2021. URL /tttps://axxiv.oot / 2004. 07219.
Ko, P.-.., Mao, Du, Y., Sun, S.-H,, and Tenenbaum, J. B. Learning to act from actionless videos through dense correspondences, 2023. URL https:/ /ttp::/aaxvvvooooee abs/2310.088 08576
Ha, D. and Schmidhuber, J. World models. 2018. doi: 10.581/7ZENOD0.122661 URL https: zenodo. www.er//eo00e...
Lee, S., Wang, Y., Etukuru, H., Kim, H. J., Shafiullah, N. M. M., and Pinto, L. Behavior generation with latent actions, 2024. URL https:// http:://axxvv.oo//// /abs/ 2403.0318 03181
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lec, H., and Davidson, J. Learning latent dynamics for planning from pixels, 2019. URL https: https://arxiv. /arxiv. org/abs/ /1lbs/1811.. 04551
Lenz, I., Knepper, R. A., and Saxena, A. Deepmpe: Learning deep latent features for model predic- tive control. In Robotics: Science and Systems, 2015. URL https: htttt:::/api.ssmmatt.. ieemanticschoo org/Corpus 101301144.
Hafner, D., Lillicrap, T., Ba, and Norouzi, M. Dream to control: Learning behaviors by latent imagination, 2020.
URL httpt: http//aaxvvvvvv. org/ abs / 01603.
Liu, Y., Zhang, K., Li, Y., Yan, 7., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H.. Gao, J., He, L., and Sun, L. Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024. URL mttps://arxiv. http/vrivvv.//// org/ htsp//bb//222//0
Hafner, D., Lillicrap, Norouzi, M., and Ba, J. Mastering atari with discrete world models, 2022. URL https: //aaxiv. org/ abs 2000.000000000000000000000000000 02193.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Master- ing diverse domains through world models, 2024. URL https: //ta:///axvv... org/ abs 2201 04104.
Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar,
A. Eureka: Human- reward design via coding large language models, 2024. URL https://// /aaaaaivvvv.o/
Haldar, S., Peng. 7., and Pinto, Baku: An efficient transformer for multi-task policy learning, 2024. URL https::/arxxt org/ abs/ 6.06. 07539.
abs
Mendonca, R., Rybkin, O.,, Daniilidis, K., Hafner, D., and Pathak, D. Discovering and achieving goals via world models, 2021. URL httt:s://arxiv.or//eeee 2110.
Hansen, N., Wang, X., and Su, H. Temporal differ- ence learning for model predictive control, 2022. URL https: aaaii.iii...//.. org/ org/abs// 2203..222222 04955.
Mendonca, R., Bahl, S., and Pathak, D. Alan: Au- tonomously exploring robotic agents in the real world, 2023a. URL https: /arxivv. org/ wwtg/bs///////oooooo...ooeeeeeee 2302. 06604.
Hansen, N., Su, H., and Wang, X. Td-mpe2: Scalable, robust world models for continuous control, 2024. URL https://arxx www. abs 22300 16888.

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Mendonca, R., Bahl, S., and Pathak, D. Structured world models from human videos, 2023b. URL https:// arxiv.org org/ abs abs/2308 .00901.
Sutton, R. Dyna, integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 164):160- 163, 1991.
Micheli, V., Alonso, E., and Fleuret, B. Transformers are sample-efficient world models, 2023, URL https:// arxiv. org/ abs 2209 00588.
Tassa, Y., Doron, Y., Muldal, A.. Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., Lillicrap, T., and Riedmiller, M. Deepmind control suite, 2018. URL. https: http:://arxxvvorg// abs/1801. 00690.
Nagabandi, A., Konoglie, K., Levine, S., and Kumar, V. Deep dynamics models for learning dexterous manipula- tion, 2019. URL https: /ttp/viiivv.////////////////aava wwt.:o/g///.........o....o...saa /1999. 11 652.
Todorov, E. and Li, W. A generalized iterative lqg method for locally-optimal feedback control of constrained non- linear stochastic systems. In Proceedings of the 2005, American Control Conference, 2005., pp. 300-306. IEEE, 2005.
Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A. R3m: A universal visual representation for robot ma- nipulation, 2022. URL
abs/
12601.
Wang, J., Dasari, S., Srirama, M. K., Tulsiani, S., and Gupta, A. Manipulate by seeing: Creating manipulation con- trollers from pre-trained representations, 2023. URL 08135.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Mísra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P. Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without supervision. 2024. URL https:/ / www.iivvorroooo.ooo abs/ 0709.
hbtrg/abb//3000 http:s///aaiivoorr/aa///////3/// https: /ttp:///axiiv../
Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13 (4):600-612, 2004. doi: 10.1 10.1109/TIP.0033. 101..8.......
Watter, M., Springenberg, T., Boedecker, J., and Ried- miller, M. Embed to control: A locally linear latent dynamics model for control from raw images, 2015. URL https://arxins wwwpsiivvorrggooooooo abs 1506. 073
Pathak, D., Mahmoudich, P., Luo, G., Agrawal, P., Chen, D., Shentu, Y., Shelhamer, E., Malik, J., Efros, A. A., and Darrell, Zero-shot visual imitation, 2018. URL https: //wxivaxvvvv/vv////v//eeee.a.av. org/ wtwg/bs///ireeoeeeeee /bs.///////000 08606.
Wen, C., Lin, X., So, J., Chen, K., Dou, Q., Gao, Y., and Abbeel, P. Any-point trajectory modeling for policy learn- ing, 2024. URL https: /arxiv...vvv org/ abs/ 24 2401. 00025.
Razavi, A., den Oord, A., and Vinyals, O. Generating diverse high-fidelity images with vq-vae-2, 2019. URL https: / /ttp://aavvvv/aa///aa/eeeeaaae.a wtwp/rgg/////oooo.eet....t..eese aaspa/9.../.... 00446.
Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. gener- alist agent, 2022. URL https: arxii. org/oagggoo abs/ 2205. 06175.
Williams, G., Wagener, N., Goldfain, B., Drews, P., Rehg, J. M., Boots, B., and Theodorou, E. A. Information theoretic mpc for model-based reinforcement learning. 2017 IEEE international conference on robotics and automation (ICRA), pp. 1714 1714-1721. IEEE, 2017.
Wu, Y., Yan, W., Kurutach, T., Pinto, L., and Abbeel, P. Learning to manipulate deformable objects without demonstrations, 2020. URL https:://aaxii..ett abs 1111111111113939
Robine, J., Höftmann, M., Uelwer, T., and Harmeling, S. Transformer-basee world models are happy with 100k in- teractions, 2023. URL /tatps://arxiv.... http:/aawvvv..aaaaaa././/. abs/ 2303. 07109.
Xiao, T., Radosavovie, I., Darrell, T., and Malik, J. Masked visual pre-training for motor control, 2022. URL https: / /aaxii.. org abs /ttpa//...2.211. 06173.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge, 2015. URL https:// arxiv.
Yan, W., Vangipuram, A., Abbeel, P., and Pinto, I.. Learning predictive representations for deformable objects using contrastive estimation. In Conference on Robot Learning, pp. 564-574. PMLR, 2021.
Sekar. R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via self-supervised world models, 2020. URL https:/ /arxiv.org/ abs/2005. 05960.
Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuur- mans, D., and Abbeel, Learning interactive real-world simulators, 2023.
12

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Zhang, K., Li, B., Hauser, K., and Li, Y. Adapti- graph: Material-adaptive graph-based neural dynamies for robotic manipulation, 2024. URL https: /arxiv. htg/abs/2407 07889.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features perceptual metric. CoRR, abs/1800.03990 2018. URL hittp://arxte org/abs/////ooooooo..o..../....r /ta.b//110000011
Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learn- ing fine-grained bimanual manipulation with low-cost hardware, 2023. URL https: / htttp::/arxxvv.rrg abs/ 2304. 13705.
Zhou, G., Dean, V., Srirama, M. K., Rajeswaran, A., Pari, J., Hatch, K., Jain, A., Yu, T., Abbeel, Pinto, L., Finn, C., and Gupta, A. Train offline, test online: A real robot learning benchmark, 2023. URL https: //tr///aiiii arxiv. org/abs/2306. 00942.
13

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
A. Appendix
A.1. Environments and Dataset Generation
a) PointMaze: In this environment introduced by (Fu ct al., 2021), the task is for force-actuated 2-DoF ball in the Cartesian directions x and y to reach a target goal. The agent's dynamics incorporate physical properties such as velocity, acceleration, and inertia, making the movement realistie. We generate 2000 fully random trajectories train our world models. We refer this task Maze for brevity in our tables.
b) Wall:
This custom 2D navigation environment features two rooms separated by a wall with a door. The agent's 's task to navigate from a randomized starting location in one room goal in the other, passing through the door. We present variant where wall and door positions are randomized, testing the model's generalization to novel configurations. For the fixed wall setting, we train on fully random dataset of 1920 trajectories each with 50 time steps. For the variant with multiple training environment configurations, we generate 10240 random trajectories.
c) Reacher: A continuous control task from DeepMind Control Suite (Tassa et al., 2018), where 2-joint robotic arm reaches a target in 2D space. We increase difficulty by requiring the entire arm, not just the end-effector, to match arbitrary target poses. To train the world model, we generate 3000 trajectories with 100 steps. We refer to this task as Reach for brevity in our tables.
d) Push-T:
This environment introduced by (Chi et al., 2024) features a pusher agent interacting with a T-shaped block. The goal is to guide both the agent and the T-block from a randomly initialized state to a known feasible target configuration within 25 steps. The task requires both the agent and the T to match the target locations. Unlike previous setups, the fixed green T no longer represents the target position for the T-block but serves purely for reference. Success requires precise understanding of the contact-rich dynamics between the agent and the object, making it challenging test for visuomotor control and object manipulation. We generate dataset of 18500 samples replayed the original released expert trajectories with various level of noise. Additionally, we introduce variations by altering the shape and color of the object to assess the model's capability to adapt to novel tasks. For this variant, we generate 20000 randomly sampled trajectories with 100 steps.
visual anchor
c)
Rope Manipulation: Introduced in (Zhang et al., 2024), this task is simulated with Nvidia Flex (Zhang et al., 2024) and consists of an XArm interacting with a soft rope placed on tabletop. The objective is to move the rope from an arbitrary starting configuration
goal configuration specified at test time. For training, we generate a random dataset of 1000 trajectories of 20 time steps of random actions from random starting positions, while testing involves goal configurations set from varied initial positions, incorporating random variations in orientation and spatial displacement.
f) Granular Manipulation: This environment uses the same simulation setup Rope Manipulation and involves manipulating about a hundred particles to form desired shapes. The training data consists of 1000 trajectories of 20 time steps of random actions starting from the same initial configuration, while testing is performed on specific goal shapes from diverse starting positions, along with random variations in partiele distribution, spacing, and orientation.
A.2. Environment Families for Testing Generalization
1. WallRandom:
Based on the Wall environment, but with randomized wall and door positions. At test time, the task requires navigating from a random starting position on one side of the wall to random position on the other side, with non-overlapping wall and door positions seen during training.
2. PushObj: Derived from the Push-T environment, where we introduce novel block shapes, including Tetris-like blocks and a "+" shape. We train the model with four shapes and evaluate on two unseen shapes. The task involves both the agent and object reaching target locations.
3. GranularRandomi Derived from the Granular environment, where we initialize the scene with a different amount of particles. The task requires the robot to gather all particles to square shape at randomly sampled location. For this task, we directly take the models that are trained with fixed amount of materials used in Section 4.3.
Visualizations can be found in Figure
14

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
A.3. Pretraining Features
a) R3M: ResNet-18 model pre-trained on wide range of real-world human manipulation videos (Nair et al., 2022).
b) ImageNet: A ResNet-18 model pre-trained on the ImageNet-IK IK dataset (Russakovsky al., 2015).
<) DINO CLS: The pre-trained DINOv2 model provides two types of embeddings: Patch and CLS. The CLS embedding is a I-dimensional vector that encapsulates the global information of an image.
d) Pre-trained MAE:
VIT model trained using Masked Autoencoding (He et al., 2021), where a large portion of input image patches are masked and the model learns to reconstruct them. We use ViT-Base checkpoint, which has feature dimension of 768 and approximately 86 million parameters.
A.4. Ablations
1. DINO-WM WITH WITHOUT CAUSAL ATTENTION MASK
We introduce causal attention mask in Section 3.1.2. We ablate this choice on PushT by training DINO-WM with and without this causal attention mask with varying history length h, such that the model takes in input 0t-h+1; 11.22....000 and output 0r-h+2; ...0.++- For models with mask, the model can only attend to past observations for predicting each o,, whereas in the w/o mask case, predicting any observation in the output sequence can attend to the entire input sequence of observations. We show planning success rate on our PushT settings in Table 8. When / = 1 where the model with and without this causal mask is equivalent, both models get decent and equivalent success rate. However, as we increase the history length, we see a rapid drop in the w/o mask case, since the model can cheat during training by attending to future frames, which is not available at test time. Adding the causal mask solves this issue, and we observe improvement in performance longer history could better capture dynamics information like velocity, acceleration, and object momentum.
Table Comparison of DINO-WM with and without causal attention mask on PushT. We train models with varying history h, representing the number of past observations the model takes as input.
= =
2 h = 3
w/o mask
0.76
0.36
0.08
with mask
0.76
0.88
0.92
A.4.2. DINO-WI WM WITH RECONSTRUCTION Loss
While DINO DINO-WM eliminates the need to train world models with a pixel reconstruction loss avoidiing the risk of learning features irrelevant to downstream tasks-we conduct an ablation study where the predictor is trained using a reconstruction loss propagated from the decoder. As shown in Table 9, this approach performs reasonably well on the PushT task but falls slightly short of our method, where the predictor is trained entirely independently of the decoder. This underscores the advantage of disentangling feature learning from reconstruction objectives.
Table Comparison of DINO-WY trained with and without loss from the decoder on PushT, highlighting the advantage disentangling feature lcarning from reconstruction objectives.
Success Rate
w/o decoder loss with decoder loss
0.92
0.80
A.5. Planning Optimization
In this section, we detail the optimization procedures for planning in our experiments.
15

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
A.5.1. MODEL PREDICTIVE CONTROL WITH
METHOD
a) Given the current observation og and the goal observation Og, both represented as RGB images, the observations are first encoded into latent states:
Eg enc(or), Zg enc(o,).
(3)
b) The planning objective is defined the squared (MSE) between the predicted latent state the final timestep T and the goal latent state:
C || =g
where i ((2)_1, 2g enc(oo)-
(4)
c) At each plamning iteration, CEM samples a population of N action sequences, each of length T, from a distribution. The initial distribution set to be Gaussian.
d) For each sampled action sequence (ao, 01,-... aT-1)) the world model is used to predict the resulting trajectory in the latent space:
2 = 1,, 1 = I=....,
And the cost calculated for each trajectory.
e) The top K action sequences with the lowest cost are selected, and the mean and covariance of the distribution are updated accordingly.
f) A new set of N action sequences is sampled from the updated distribution, and the process repeats until success is achieved after a fixed number of iterations that we hyperparameter.
g) After the optimization process done, the first & actions du....ay is executed in the environment. The process then repeats at the next time step with the new observation.
A.5.2. GRADIENT DESCENT:
Since our world model is differentiable, also consider optimization approach using Gradient Descent (GD) which directly minimizes the cost by optimizing the actions through backpropagation.
a) We first encode the current observation o and goal observation 0g into latent spaces:
cg = enc(on), = =nc(og))
(6)
b) The objective remains the same for CEM:
- 2r =y]1² where = 1() 2o enc(oog)
(7)
c) Using the gradients of the cost with respect to the action sequence {ao, di,...,, ar-1) the actions are updated iteratively:
ac Dai" t 0, 0
0₂ f-
(8)
where 77 is the learning rate
d) The process repeats until a fixed number of iteractions reached, and we execute the first actions ag, ag in the enviornment, where le is pre-determined hyperparameter.
A.5.3. PLANNING RESULTS
Here we present the full planning performance using various planning optimization methods in Table 10. CEM denotes the setting where we use CEM to optimize a sequence of actions, and execute those actions in the environment without any correction or replan. Similarly, GD denotes optimizing with gradient decent and execute all planned actions at once in an open-loop way. MPC denotes allowing replan and receding horizon with CEM for optimization.
16

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Table 10. Planning results of DINO-W WM M with various planning optimization methods.
PointMaze
Push-T
Wall
Rope
Granular
CEM
0.8
0.86
0.74
NA
NA
GD
0.22
0.28
NA
NA
NA
МPC
0.98
0.90
0.96
0.41
0.26
A.6. Inference Time
Inference time is critical factor when deploying model for real-time decision-making. Table 11 reports the time required on an NVIDIA A6000 GPU for a single inference step, the environment rollout time for advancing one step in the simulator, and the overall planning time for generating an optimal action sequence using the Cross Cross-Entropy Method (CEM). The inference time of DINO-WM remains constant across environments due to the fixed model size and input image resolution, resulting in significant speedup over traditional simulation rollouts. Notably, in environments with high computational demands, such deformable object manipulation, simulation rollouts require several seconds per step while DINO- W enables rapid inference and efficient planning. Planning time is measured with CEM using 100 samples per iteration and 10 optimization steps, demonstrating that DINO-WA can achieve feasible planning times while maintaining accuracy and adaptability across tasks.
Table 11. Inference time and planning time for DINO DINO-WM. Inference time represents the time required for a single forward pass for one step. while environment rollout time the simulator's speed for advancing step. Planning time corresponds to Cross-Entropy Method (CEM) with 100 samples per iteration and 10 optimization steps.
Metric
Time (s)
Inference (Batch 32)
0.014
Simulation Rollout (Batch 1)
3,0
Planning (CEM, 100x10)
15.89
A.7. Hyperparameters and Implementation
We present the DINO-W MMNO-WMPR hyperparameters and relevant implementation repos below. We train the world models for all environments with the same hyperparameters shown in Table 13.
The world model architecture consistent across all environments. We use an encoder based on DINOv2, which extracts features with shape of (14 x 14, 384) from input images resized to 196 196 pixels. The ViT backhone has depth of 16 attention heads, and an MLP dimension of 2048, amounting approximately 19M parameters.
To ensure the prediction task is meaningful, as nearby observations can be highly similar, introduce frameskip parameter during data processing. This parameter specifies how far into the future the model is predicting. The frameskip values for each environment are provided in Table 12.
AVDC: ANT..tttte///sttaaoocc/////r//rt
R3M: RBM.attte//attuooocccc//ccttttts
We base our predictor implementation on httppp//gghhaa..co/////ceaaaassp
A.8. Additional Planning Visualizations
We show visualizations of planning instances for DINO- WM and our baselines in Figure For comparison, we show the best performing world models DINO CLSS and DreamerV3. We also show visualizations of DINO- W on all tasks in
17

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Table 12, Envirommen--dppendent hyperparameters for DINO-
WM training. We report the number of trajectories in the dataset under Dataset Size, and length of trajectories under Traj. Len.
Table 13. Shared hyperparameters for DINO DINO-WM training
Name
Value
H
Frameskip
Dataset Size
Traj. Len.
Image size
224
Optimizer
Adam W
PointMaze
3
5
2000
100
Decoder
3e-4
Reacher
3
5
3000
100
Predictor
5e-5
Push-T
3
5
18500
100-300
Action encoder Ir
5e-4
PushObj
3
5
20000
100
Action emb dim
10
Wall
1
1920
50
Epochs 100
WallRandom
10240
50
Batch size
32
Rope
1
1000
5
Granular
1
1
1000
5
Start
Final
Goal
Start
Final
Goal
Start
Final
Goal
DINO DINO CLS DreamerV3
Ours
PointMaze
Push-T
Granular
Figure Planning visualizations for PointMaze, Push-T, and Granular, on randomly sampled initial and goal configurations. The defined by Start and Goal, denoting the initial and goal observations. Final shows the final state the system arrives at after planning with each world model. For comparison, we show the best performing world models DINO CLS and DreamerV3.
For each environment, the top (shaded) row shows the environment's observation after executing the planned actions, and the bottom row shows the world model's imagined observations.
Figure 9.
To demonstrate DINO- WM'ss ability to generalize to different goals at test time, we show additional visualizations for DINO-WM when provided with the same initial observation but different goal observations Figure 10 and Figure 11, Similarly, show trajectory pairs compare the environment's observations (top shaded rows) after executing sequence of planned actions with DINO-WM's imagined trajectories (bottom rows). The left-most column denotes the initial observations, and the mosh shaded column denotes the goal observations.
18

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Start
Goal
PointMaze
Wall
Reacher
K
+
PushT
Rope
Granular
Figure 9. Trajectories planned with DINO- WWM all six environments. For each environment, the top (shaded) row shows the
environment's 's observation after executing the planned actions, and the bottom row shows the world model's imagined observations.
19

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Start
Goal
PushT
4
Figure 10. Trajectories planned with DINO- WM on PushT with the same initial states but different goal states.
20

---

DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
Start
Goal
C
PointMaze
C
C
Figure 11. Trajectories planned with DINO-WM. on PointMaze with the initial states but different goal states.
21


Reviews:
[{"review_id": "jc40zqjoF7", "reviewer": "Reviewer_57WD", "summary": "This paper proposes DINO-WM, a task-agnostic world model that predicts future visual features using DINOv2 embeddings instead of reconstructing raw observations. Trained on offline trajectories with a Vision Transformer, it enables zero-shot test-time optimization via model predictive control. DINO-WM outperforms prior methods in goal-reaching success (45% improvement) and world modeling quality (56% improvement) across diverse tasks like maze navigation and robotic manipulation. By leveraging high-level feature prediction, it achieves flexible planning without task-specific retraining or auxiliary supervision.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback, for identifying that DINO-WM “significantly improves world modeling quality” and shows “a good embedding space makes world modeling + MPC a winning recipe”. We address the issues raised in the review below. \n\n>**“it is uncertain whether frozen DINOv2 feature patches provide sufficient resolution for contact-rich tasks”**\n\nWe show that frozen DINOv2 patch features capture state accurately, even in contact-rich tasks like PushT and deformable manipulation. Figure 4 in the manuscript illustrates how they precisely represent particle positions—challenging for global features like ImageNet-pretrained ResNet or R3M.\n\nWe further trained a DINO-WM on LIBERO [1], a tabletop environment manipulating diverse objects, with third-person image observations and a 7DoF action space. [Figure 1](https://tinyurl.com/5eckpz75) compares open-loop rollouts of WMs trained with DINO patch features vs. DINO CLS features. Reconstruction scores for the predicted frames can be seen in [Table 7](https://tinyurl.com/y4thskrv). This shows that DINO patch features can accurately represent the object within the gripper, whereas global representations struggle with dynamic elements—both the object in the gripper and the gripper itself. This further reinforces the suitability of patch features for contact-rich interactions.\n\nIn our response to **Reviewer 4KWu**, we provide linear probe results on the environment state using DINO patch features, along with PCA visualizations, demonstrating its ability to capture task-relevant information.\n\n>**“it is unclear whether planning will be fast enough”**\n\nWe address planning efficiency in 3 ways. \n1. We improved our inference code for DINO-WM since submission. For the same hyperparameters, planning now takes 15.89s, compared to 53s reported in the manuscript.\n2.  While we report results for CEM with 100 samples per iteration, this can be adjusted based on task complexity. [Table 1](https://tinyurl.com/2mn6aben) presents the tradeoff between sample size, planning time, and performance on PushT. This flexibility allows us to balance computational efficiency with performance depending on task requirements.\n3. Training and planning with a DINO-WM using a larger frameskip can speed up planning. This effectively increases the planning horizon as each prediction covers a longer time span. We train a DINO-WM with frameskip 25 on PushT and report the performance in [Table 6](https://tinyurl.com/525kjsza). While modeling long-term dependencies is more challenging as frameskip increases, it presents an opportunity to improve efficiency, making hierarchical planning promising directions for future research.\n\n>**“In cluttered environments with complex backgrounds, the model's performance remains uncertain”**\n\nWe ran experiments on ClutteredPushT, where the background of the PushT environment is a cluttered real-world tabletop. Open-loop rollouts of a trained DINO-WM is in [Figure 3](https://tinyurl.com/49j2c5av), and we report the task performance in [Table 5](https://tinyurl.com/4wvxtw84) compared to the original PushT.\n\nIn ClutteredPushT, DINO-WM can still identify the effect of agent actions and predict the motion of relevant objects. The final planning performance is only marginally degraded. This shows DINO-WM’s capability of modeling environments with complex backgrounds.\n\nAdditionally, we have trained an unconditional world model on the CLEVRER [2] dataset where multiple objects may collide. Videos of open-loop rollouts are provided [here](https://tinyurl.com/cx59vz8p).\n\n**Q1. Environment with SE(3) action space.** \n\nWe further provide results of DINO-WM on LIBERO which has a 7DoF action space. Due to space limit, we refer to our response to **Reviewer dmt6** for planning videos and performance. \n\n**Q2.** Addressed in the ClutteredPushT experiment.\n\n**Q3. Does increasing the latent-space planning horizon degrade performance?**\n\nNot necessarily. Longer planning horizon enables discovering states that are further away, with the tradeoff that the WM’s long-term prediction would be less accurate. We balance this via planning with receding horizons as in our deformable environments.\n\n**Q4.** We conduct ablations with different image sizes (yielding different patch sizes after DINO encoder) on PointMaze. We report the planning success rate (SR) for CEM, MPC, and the predicted frame’s image scores in  [Table 3](https://tinyurl.com/ycyua34p).\nWith MPC, all models eventually obtain a decent SR, but models with larger patch size are able to achieve better SR with CEM. This shows that bigger patch sizes can contain more precise state information, making the world model more accurate for zero-shot open-loop planning.\n\n**Q5.** Yes, our decoder is trained on the same data with all baselines including DreamerV3 and IRIS.\n\n[1] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning\n\n[2] CLEVRER: CoLlision Events for Video REpresentation and Reasoning", "claims_and_evidence": "DINO-WM claims to produce high-quality world modeling, supported by a 56% improvement in LPIPS over prior methods. It claims to achieve high success in reaching arbitrary goals, with a 45% improvement over previous approaches. Additionally, it claims to generalize across task variations, such as different maze layouts and object shapes, outperforming prior work in diverse environments.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes. The experiments are well-structured to evaluate DINO-WM’s ability to learn from offline datasets, optimize behavior at test time, and generalize across tasks. Comparisons against state-of-the-art baselines show that DINO-WM significantly improves world modeling quality. Results confirm that using DINOv2 patch embeddings enhances planning performance.", "relation_to_broader_scientific_literature": "Prior world modeling work learns the dynamics model together with the embedding function. This work leverages frozen DINO features and learns the dynamics model in isolation, showing that a good embedding space makes world modeling + MPC a winning recipe.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Most of the experiments focus on variations of reaching tasks (e.g., non-prehensile manipulation), making it unclear whether the same performance will hold for contact-rich manipulation. Specifically, it is uncertain whether frozen DINOv2 feature patches provide sufficient resolution for contact-rich tasks (even just for pick-and-place tasks). Additionally, for dexterous manipulation with high-DOF action spaces, it is unclear whether planning will be fast enough to find a solution, given that planar pushing already takes ~50 seconds planning time. Lastly, in cluttered environments with complex backgrounds, the model's performance remains uncertain.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Can you show examples of pick-and-place tasks, preferably with a non-trivial SE(3) action space beyond simple table-top pick-and-place?\n2. Can you demonstrate your existing experiments in environments with cluttered backgrounds?\n3. Does increasing the latent-space planning horizon degrade performance?\n4. How does changing the patch size impact performance?\n5. Figure 4 is very compelling, but was your decoder trained on the same data as DreamerV3 and IRIS?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kj0tqI36It", "reviewer": "Reviewer_4KWu", "summary": "The paper introduces DINO-WM, a world model that operates within the DINOv2 representation space without the need for reconstruction. The model is trained using teacher forcing with a frame-level causal mask. Due to its task-agnostic nature, DINO-WM can be used for zero-shot model predictive control without demonstration collection, reward modeling, or learning inverse dynamics models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback, and for acknowledging that DINO-WM “demonstrates superior performance and could serve as a solid baseline for future research” and “could inspire future research a lot.” We address the issues raised in the review below.  \n\n>**“why DINOv2 performs better are not clearly analyzed”“they do not provide evidence that the frozen DINOv2 representations are better in task representation”**\n\nWhile our manuscript presents results on feature reconstruction and task planning, demonstrating that DINOv2 patch features enable more accurate world modeling, we now provide further analysis of the features themselves. A well-established method for evaluating feature quality in downstream control tasks is linear probing, which assesses how well the features encode task-relevant state information. We conducted linear probe experiments on PointMaze, PushT, and Wall, comparing DINO-S Patch, DINO-S CLS, DINO-B Patch, IR3M, and Pre-trained MAE (DINO-S and DINO-B denotes DINO model with ViT Small and Base architecture, respectively). The validation loss for these linear probes is reported in [Table 2](https://tinyurl.com/mpurtfs), where DINO-S Patch and DINO-B Patch achieve the lowest validation loss, indicating their superior task representation capabilities.\n\nTo further analyze the DINOv2 features, we visualize the principal components after performing PCA on the patch embeddings from DINO-S on our deformable environments which has the most complex state space, following a procedure similar to that in the original DINOv2 paper ([Figure 2](https://tinyurl.com/r39zsb4r)). The visualizations show that DINOv2 features effectively identify objects of interest, distinguish the agent, and separate the foreground from the background, reinforcing that frozen DINOv2 representations encode task-relevant information effectively.\n\n>**“the authors only compare backbones with global representations in Tables 2-4, which are inherently unsuitable for capturing spatial relationships”“It would be more convincing if other semantic-rich alternatives, such as MAE [1], are included in the study”**\n\nWe totally agree that having more baselines with semantic-rich baselines would provide further insights. We first note that for the IRIS baseline which we compare with in Table 1 and Table 3 of the manuscript, it encodes an image with 16 tokens instead of a global representation, which is also capable of capturing spatial relationships explicitly. \n\nWe ran a new experiment on PushT using a pre-trained MAE suggested by the reviewer. The performance is presented in the Table below. We observe that the WM trained with MAE features has lower final MPC performance than the original DINO-WM. We hypothesize this is because MAE prioritizes reconstruction over task relevance, as indicated by linear probe results. Additionally, even the smallest MAE encoder, which we used in the experiments, is still significantly larger than the DINO-S model, with feature dimensionality twice than that of DINO-S. This highlights that a larger and more expensive feature extractor does not necessarily translate to better task performance, and the higher computational cost of MAE makes it a less efficient choice.\n\n| Encoder Model | Encoder Param Count | Feature Size | MPC  |\n|--------------|---------------------|--------------|------|\n| DINOv2      | 22,056,576           | 384          | 0.90 |\n| MAE         | 85,798,656           | 768          | 0.86 |\n\n>**“Q1 DINOv2 input resolution.”**\n\nAlthough DINOv2 is pre-trained on 224×224 images, its ViT-based architecture processes images as fixed-size patches, allowing it to handle arbitrary image sizes as long as the height and width are divisible by the patch size (e.g., 14 for DINO-S). In fact, the original DINOv2 paper [1] explicitly demonstrates its ability to work with non-224×224 images, including rectangular and high-resolution images (Section 7.5).\n\nIn DINO-WM, our choice of 196×196 input resolution is purely an engineering decision. Our decoder assumes a 16x spatial upscaling, and we aimed to match the environment’s 224×224 observation shape after decoding. Since DINOv2’s fixed patch size is 14, an input of 196×196 results in a 14×14 patch grid, which decodes to 224×224 after the 16× upscaling. However, using 224×224 directly with an additional interpolation layer at the decoder output would be equally valid.\n\nTo verify this, we conducted an ablation on PointMaze with input sizes ranging from 28×28 to 280×280. The results in [Table 3](https://tinyurl.com/ycyua34p) show that image size 224×224 performs on par with 196×196. This shows that our choice does not limit DINOv2’s capabilities. Moreover, the ability of DINO-WM to process images of various sizes enhances its flexibility in modeling environments of varying complexity, allowing for adaptive trade-offs between granularity and computational efficiency.\n\n [1] DINOv2: Learning Robust Visual Features without Supervision", "claims_and_evidence": "While leveraging DINOv2 representations as world states achieves superior results compared to image reconstruction, its working mechanism is unclear. The authors claim that reconstruction-based methods contain insufficient task information, but they do not provide evidence that the frozen DINOv2 representations are better in task representation.", "methods_and_evaluation_criteria": "The proposed methods are aligned with the motivation and evaluated using appropriate criteria.", "theoretical_claims": "I have checked the theoretical claims in this paper.", "experimental_designs_or_analyses": "The paper provides extensive experiments with sufficient details.", "relation_to_broader_scientific_literature": "The paper introduces DINOv2 as the representation space of the world model, thereby alleviating the need for image reconstruction both during training and testing. While this concept of latent world model is not new, the proposed world model demonstrates superior performance and could serve as a solid baseline for future research.", "essential_references_not_discussed": "As far as I know, all closely related works are cited appropriately.", "other_strengths_and_weaknesses": "W1) **Latent world model is not new.** Many related works have adopted the latent world model design, and switching to DINOv2 representation space is somewhat limited in technical innovation. However, the extensive experiments still make this work an insightful contribution to the community.\n\nW2) **The reasons why DINOv2 performs better are not clearly analyzed.** As I mentioned earlier, the authors fail to provide evidence that the frozen DINOv2 representations are better in task representation. In addition, the authors only compare backbones with global representations in Tables 2-4, which are inherently unsuitable for capturing spatial relationships. It would be more convincing if other semantic-rich alternatives, such as MAE [1], are included in the study.\n\n[1] Masked Autoencoders Are Scalable Vision Learners. Kaiming He, et al.", "other_comments_or_suggestions": "I believe this paper presents a solid practice in world model design and model predictive control. The generality of the proposed method could inspire future research a lot.", "questions_for_authors": "Q1) **DINOv2 input resolution.** If I remember it correctly, the pretrained DINOv2 backbone accepts images of 224x224 pixels as input. Therefore, I am curious why the authors resize the input images to 196x196 pixels (Line 927). Does it still work well without finetuning?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BX0SR22wEA", "reviewer": "Reviewer_dmt6", "summary": "This work aims to train a world model using large vision pre-trained features on offline trajectories in a task-agnostic fashion. They use the resulting trained world model to plan out Push-T, Maze, Reach, Rope, and Granalur control tasks in a zero-shot manner. Specifically, the authors train a world model using DINOv2 pre-trained features in the latent space. With extensive experimentation, the authors clearly show the benefit of using such a world model based on leveraging pre-trained large vision model features as opposed to the norm in the field -- to train the encoder from scratch.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful and constructive feedback, and for acknowledging that DINO-WM “expands the current MBRL setting by incorporating the learned visual features from large vision models”, and the idea of using pretrained features to be “neat and simple”. We now address the questions raised in the review below. \n\n\n**DINO-WM results on more robotic tasks**\n\nIn addition to the robotic deformable manipulation environments and the robotic arm reaching task presented in the manuscript, we further provide results of DINO-WM on LIBERO [1], a tabletop environment manipulating diverse objects, with third-person image observations and a 7DoF action space. We note that CEM in this raw action space would be inefficient starting from completely random action samples (which would result in the robot mostly jittering in place). Therefore, instead of sampling randomly, we sample from a pre-trained diffusion policy [2]. DINO-WM is capable of long-horizon predictions and assists in selecting the best action trajectories. We provide visualizations for open-loop rollouts of DINO-WM in [Figure 1](https://tinyurl.com/5eckpz75) and videos for task planning [here](https://tinyurl.com/yptacs64). Without planning with DINO-WM, the diffusion policy model achieves a 35% success rate (across 20 trajectories). With planning using DINO-WM, the task success rate is improved to 55%. This demonstrates the effectiveness of DINO-WM, even with a high-dimensional continuous action space and a long task horizon.\n\n\n**TD-MPC2 as a baseline**\n\nWe completely agree with the reviewer’s insight on why TD-MPC2 underperforms compared to other reconstruction-based baselines, as we have also discussed in the original manuscript (L290-293). Our motivation for including TD-MPC2 as a baseline is that it represents state-of-the-art work in world modeling while also incorporating planning to sample actions during training. We believe that comparing against TD-MPC2 provides valuable insight into the extent to which the performance of world models in this line of work depends on informative reward functions.\n\n**Architecture and size for decoders**\n\nWe thank the reviewer for raising this constructive question. For the (optional) DINO-WM decoder, the architecture is based on VQ-VAE and consists of two stacked Decoder modules. Each Decoder is a transposed CNN with residual blocks. For DreamerV3 and IRIS, we use the decoders provided within the respective algorithms, both of which are CNN-based decoders.\n\nWe have included the number of parameters for the decoder in each baseline in the Table below. For DreamerV3 and IRIS, we follow the default parameters from the original implementations. In response to the reviewer’s suggestion, we have also matched the decoder sizes for DreamerV3 and IRIS (denoted as DreamerV3 Large and IRIS Large). The decoded open-loop rollout images are shown in [Figure 4](https://tinyurl.com/mrku8uza). We observe that DreamerV3 Large shows improved prediction quality compared to the original DreamerV3. However, IRIS Large demonstrates slightly worse performance than IRIS, which could be due to the fact that IRIS requires identical parameters for both its encoder and decoder, making the model more difficult to optimize.\n\n| Model              | Decoder Size  |\n|--------------------|--------------|\n| Ours              | 10,140,163    |\n| IRIS              | 1,821,827     |\n| DreamerV3         | 6,985,667     |\n| IRIS Large        | 11,126,979    |\n| DreamerV3 Large   | 11,256,577    |\n\n\n\n[1] LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning\n\n[2] Diffusion Policy: Visuomotor Policy Learning via Action Diffusion", "claims_and_evidence": "I find all the claims made in the paper to be clear and supported by concrete experimental evidence.", "methods_and_evaluation_criteria": "Yes, the authors evaluate both plannings using the learned world model on (a) tasks on whose data they were trained on, (b) on novel unseen environment configurations -- testing the generalization capabilities as well as reporting visual similarity metrics such as LPIPS and SSIM when trained on a separate (and optional) decoder.", "theoretical_claims": "No theory is involved in this paper.", "experimental_designs_or_analyses": "Yes, and I don't have any particular concern with any experimental designs/analyses.", "relation_to_broader_scientific_literature": "I believe this work expands the current MBRL setting by incorporating the learned visual features from large vision models. Specifically training a world model on a task-agnostic dataset and being able to zero-shot plan on control tasks is a fundamental ability that a world model should possess. Although this paper does not evaluate on more complicated robotic tasks -- I believe this is a step in the right direction.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "**Strengths**: I find the idea of using a pre-trained representation such as DINOv2's patch embeddings to be a really neat and simple idea to enable zero-shot planning. Leveraging the strengths of the large pre-trained visual models is an appealing research direction in contrast to encoders in MBRL/world model literature that have been trained from scratch.\n\n**Weaknesses**: The authors can consider showing DINO-WM results on more robotic tasks (something like MetaWorld suite). I understanding that CEM planning over say picking or placing an object might be slightly challenging -- but it would be nice to see how DINO-WM does on those tasks. Even if it ends up failing, a discussion section describing the failures and potential reasons of that failure would be really helpful for the broader scientific community.", "other_comments_or_suggestions": "1. I find the choice of TDMPC2 as a baseline to be slightly strange. I believe TDMPC2 is learning a trivial (zero) representation because the only source for the TDMPC2-like world model to learn something meaningful is the reward. In the absence of reward, the \"consistency\" loss should immediately learn all zeros. Only reconstruction-based works like IRIS and DreamerV3 make sense as a baseline for this work. However, I haven't penalized the authors for this at all -- this was just a curious thought when I encountered the baseline.", "questions_for_authors": "1. What is the architecture that is used for the (optional) decoder to generate the results in Figure 4? Is the decode restricted to be the same across all the baselines (IRIS and DreamerV3) -- in which case I'd like the authors to report the number of parameters each decoder has. It would be ideal to showcase the results with the same decoder structure or with similar parameters to ensure that it is indeed the representation that is being evaluated.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ROF3WVfGen", "reviewer": "Reviewer_Bqcm", "summary": "The paper introduces DINO‐WM, building task‐agnostic world models for control and planning. Instead of operating directly in pixel space, the method leverages pre‐trained patch features from DINOv2 to encode observations into a rich, spatially-aware latent representation. The world model is trained offline using trajectories and employs a ViT-based transition model to predict future latent states. At test time, planning is carried out using model predictive control (MPC) with the cross-entropy method (CEM), allowing for zero-shot goal-reaching without additional reward signals or task-specific tuning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful review, especially in highlighting that DINO-WM demonstrates “robust generalization” and “is a significant step toward more general-purpose models.” We now address the issues raised by the reviewer.\n\n>**“I am a little worried about the real-world planning and control performance since the robot is always moving when the proposed model is planning the actions”**\n\nOur planning framework follows the principles of Model Predictive Control (MPC)—a well-established approach in real-world robotics. The planner optimizes future actions but executes only a subset before re-planning. The robot’s continuous movement is not an issue, as each planning step integrates updated proprioceptive and visual observations, ensuring robust closed-loop control.\n\nFor planning efficiency, we improved our inference code for DINO-WM since submission and now it takes 15.89s for CEM compared to 53s reported in the manuscript. We provide further ways to speed up planning and we refer to our response to **Reviewer 57WD** due to space limits.\n\n>**“More experiments on tasks with complex textured backgrounds need to be evaluated.”**\n\nWe evaluated ClutteredPushT, which adds a complex textured background to PushT. Please see our response to **Reviewer 57WD** for open-loop rollouts and final performance due to space limits.\n\n>**“DINO-WM requires a comprehensive offline dataset with sufficient state-action coverage” “validation on real-world robotic platforms”**\n\nWe agree with the reviewer that DINO-WM requires a diverse dataset for effective planning. While it avoids online interactions, reward signals, and expert demonstrations, it still follows the general trade-off that broader coverage improves generalization. However, this limitation can be mitigated by continuously updating the world model with new experiences, enabling progressive improvement without requiring extensive real-world interactions upfront.\n\nFor real-world validation, independent researchers have already successfully deployed DINO-WM on real robots. We are consulting with the AC on how to share this while maintaining anonymity.\n\n>**“incorporating hierarchical planning or multi-level control strategies could enhance performance on more complex or fine-grained tasks.”**\n\nWe totally agree that hierarchical planning can unlock further capabilities of DINO-WM. Our current work serves as a proof of concept that even single-level planning can be effective with pre-trained WMs. This establishes a strong foundation on which more advanced planning strategies can be built. We see this as an exciting and promising direction for future work.\n\n>**“hyperparameter sensitivity and computational trade-offs” “method’s robustness and scalability”**\n\nWe conducted a hyperparameter sweep for CEM on PushT. We refer to our response to **Reviewer 57WD** for the computation time and performance tradeoff due to space limits. \n\nFor WM training, we ablated image sizes on PointMaze (see response to **Reviewer 4KWu**). The training hyperparameters for DINO-WM is quite robust, as the same hyperparameters work across all six reported environments.\n\n>**“Appendix A.4.2 is total a waste.”**\n\nAppendix A.4.2 demonstrates the necessity of the frame-level attention mask in DINO-WM’s predictor. The experiments (L749-753) supports our claim that the causal mask effectively ensures attention to past frames only, enabling the model to capture essential temporal dynamics such as velocity and acceleration. Would the reviewer prefer that we remove this section?\n\n>**“More foundation models could be evaluated”**\n\nWe agree that evaluating more foundation models can provide further insights on the pros and cons of DINO-WM compared to existing models. To this end, we evaluated Genie, a foundation model for generating interactive environments. We train the Genie model on our PushT dataset (open-loop rollouts in [Figure 5](https://tinyurl.com/3659ydkn)). It is evident that Genie performs worse in prediction quality and future state estimation compared to DINO-WM, even with ground truth action conditioning.\n\n| Model         | LPIPS  |\n|--------------|--------|\n| Genie  | 0.043  |\n| Ours         | 0.007  |\n\nFor OpenVLA with SigLIP, we note that it is a language-conditioned feed-forward policy trained with expert trajectories, fundamentally differing from DINO-WM, which learns environment dynamics from any interaction dataset and performs goal-conditioned planning. Since our setting doesn’t assume expert data or language conditioning, OpenVLA is unsuitable for fine-tuning. Additionally, its released model assumes a fixed action space, making it infeasible to finetune the model on our datasets.\n\nWe also evaluate other foundational vision models such as pre-trained MAE as the encoder for DINO-WM. Please refer to our response to **Reviewer 4KWu** for the experiment details and final performance of this experiment due to the 5000 char response limit.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "This proposed principle holds when facing multiple experimental settings. However, I am a little worried about the real-world planning and control performance since the robot is always moving when the proposed model is planning the actions. If the planning is too slow (53s), the control system could collapse.", "theoretical_claims": "Theoretical claims are good.", "experimental_designs_or_analyses": "The experimental setups are a bit easy. More experiments on tasks with complex textured backgrounds need to be evaluated.", "relation_to_broader_scientific_literature": "The model demonstrates robust generalization, successfully handling novel environment configurations and object variations. This suggests that the learned latent representations capture essential underlying dynamics beyond the specific scenarios seen during training.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "## Strengths :\n\nThe approach allows for planning at test time without task-specific fine-tuning or the need for expert demonstrations. The experiments show that DINO-WM can generate effective control policies purely through offline training, which is a significant step toward more general-purpose models.\n\nThe authors validate their method on six simulation environments (e.g., maze navigation, push manipulation, robotic arm control, and deformable object manipulation). The extensive comparisons with several state-of-the-art baselines (IRIS, DreamerV3, TD-MPC2, among others) highlight the method’s superior performance in both success rates and reconstruction quality.\n\nThe model demonstrates robust generalization, successfully handling novel environment configurations and object variations.\n\n## Weakness\nDINO-WM requires a comprehensive offline dataset with sufficient state-action coverage. In real-world settings, gathering such data might be challenging, potentially limiting the method’s applicability outside of controlled environments.\n\nAll experiments are conducted in simulated tasks. Although the results are promising, further validation on real-world robotic platforms would be needed to assess practical deployment, as mentioned before.\n\nThe current planning framework operates solely in the action space. While effective for the tasks presented, incorporating hierarchical planning or multi-level control strategies could enhance performance on more complex or fine-grained tasks.\n\nAlthough the paper includes ablation studies, additional analysis on hyperparameter sensitivity and computational trade-offs would provide deeper insights into the method’s robustness and scalability.\n\n(Appendix A.4.2 is total a waste. Failed to understand why the authors want to add this paragraph.)", "other_comments_or_suggestions": "No", "questions_for_authors": "More foundation models could be evaluated, like OpenVLA using Dino+Siglip. It would add some value if authors could investigate such setups.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Discrimination / Bias / Fairness Concerns"]}}]

Author rebuttal:

Meta-review: ## Summary

This paper introduces DINO-WM, a novel approach to world modeling that leverages pre-trained DINOv2 visual features instead of reconstructing raw observations. The key innovation is using patch-level features from DINOv2 and learning a dynamics model in this rich representation space, which enables high-quality predictions and effective test-time planning. DINO-WM is trained on offline trajectories and can perform zero-shot planning for arbitrary goals without needing expert demonstrations, reward functions, or inverse dynamics models.

## Review Process

The paper received mixed initial reviews, with scores ranging from 2 to 4. Given this divergence in reviewer opinions, I initiated a discussion to better understand the strengths and limitations of the work. The authors provided thorough responses during rebuttal, addressing concerns about planning efficiency, performance on complex environments, and additional experiments with different foundation models. Following the discussion and rebuttal, one reviewer increased their score, resulting in three "Accept" ratings and one "Weak accept" rating.

## Strengths

- **Representation Innovation**: DINO-WM demonstrates that frozen pre-trained visual features can serve as an effective state space for world modeling, outperforming models that learn representations from scratch.
- **Strong Performance**: The approach shows substantial improvements over prior methods, with a 56% boost in world modeling quality (LPIPS) and 45% higher goal-reaching success rates across diverse environments.
- **Generalizability**: DINO-WM generalizes well to novel environment configurations, such as different maze layouts and object shapes.
- **Zero-shot Planning**: The method enables planning for arbitrary goals without requiring task-specific training, expert demonstrations, or reward signals.

## Limitations

- **Planning Efficiency**: While the authors improved planning speed from 53s to 15.89s in their rebuttal, real-world deployment may still require further optimization.
- **Dataset Requirements**: DINO-WM needs comprehensive offline datasets with sufficient state-action coverage, which could be challenging to collect in some real-world settings.
- **Complexity of Evaluated Tasks**: Most experiments focus on relatively simple simulation environments, though the authors provided additional results on more complex environments (LIBERO, ClutteredPushT) in their rebuttal.
- **Foundation Model Analysis**: While the authors evaluated different models in response to reviewer requests (MAE, SigLIP+DINO), a more comprehensive analysis of why DINOv2 features work particularly well would strengthen the paper.

## Recommendation

After carefully considering the reviews, author responses, and subsequent discussion, I recommend acceptance of this paper. DINO-WM represents a significant advance in world modeling by effectively leveraging pre-trained visual features to enable zero-shot planning. The authors have addressed key concerns raised during review, demonstrating the approach's effectiveness on more complex tasks and providing a thorough analysis of design choices.

The discussion among reviewers highlighted that despite some limitations, the core contribution of using pre-trained patch-level features for world modeling is valuable and opens promising directions for future research. I encourage the authors to incorporate their rebuttal material, especially the additional experiments and analyses, into the final version of the paper.
Decision: Accept (poster)