DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot
  Planning
Authors: name: Gaoyue Zhou; name: Hengkai Pan; name: Yann LeCun; name: Lerrel Pinto
Published: 2024-11-07 18:54:37

Abstract: The ability to predict future outcomes given control actions is fundamental
for physical reasoning. However, such predictive models, often called world
models, have proven challenging to learn and are typically developed for
task-specific solutions with online policy learning. We argue that the true
potential of world models lies in their ability to reason and plan across
diverse problems using only passive data. Concretely, we require world models
to have the following three properties: 1) be trainable on offline,
pre-collected trajectories, 2) support test-time behavior optimization, and 3)
facilitate task-agnostic reasoning. To realize this, we present DINO World
Model (DINO-WM), a new method to model visual dynamics without reconstructing
the visual world. DINO-WM leverages spatial patch features pre-trained with
DINOv2, enabling it to learn from offline behavioral trajectories by predicting
future patch features. This design allows DINO-WM to achieve observational
goals through action sequence optimization, facilitating task-agnostic behavior
planning by treating desired goal patch features as prediction targets. We
evaluate DINO-WM across various domains, including maze navigation, tabletop
pushing, and particle manipulation. Our experiments demonstrate that DINO-WM
can generate zero-shot behavioral solutions at test time without relying on
expert demonstrations, reward modeling, or pre-learned inverse models. Notably,
DINO-WM exhibits strong generalization capabilities compared to prior
state-of-the-art work, adapting to diverse task families such as arbitrarily
configured mazes, push manipulation with varied object shapes, and
multi-particle scenarios.