MaskGWM: A Generalizable Driving World Model with Video Mask
  Reconstruction
Authors: name: Jingcheng Ni, name: Yuxin Guo, name: Yichen Liu, name: Rui Chen, name: Lewei Lu, name: Zehuan Wu

World models that forecast environmental changes from actions are vital for
autonomous driving models with strong generalization. The prevailing driving
world model mainly build on video prediction model. Although these models can
produce high-fidelity video sequences with advanced diffusion-based generator,
they are constrained by their predictive duration and overall generalization
capabilities. In this paper, we explore to solve this problem by combining
generation loss with MAE-style feature-level context learning. In particular,
we instantiate this target with three key design: (1) A more scalable Diffusion
Transformer (DiT) structure trained with extra mask construction task. (2) we
devise diffusion-related mask tokens to deal with the fuzzy relations between
mask reconstruction and generative diffusion process. (3) we extend mask
construction task to spatial-temporal domain by utilizing row-wise mask for
shifted self-attention rather than masked self-attention in MAE. Then, we adopt
a row-wise cross-view module to align with this mask design. Based on above
improvement, we propose MaskGWM: a Generalizable driving World Model embodied
with Video Mask reconstruction. Our model contains two variants: MaskGWM-long,
focusing on long-horizon prediction, and MaskGWM-mview, dedicated to multi-view
generation. Comprehensive experiments on standard benchmarks validate the
effectiveness of the proposed method, which contain normal validation of
Nuscene dataset, long-horizon rollout of OpenDV-2K dataset and zero-shot
validation of Waymo dataset. Quantitative metrics on these datasets show our
method notably improving state-of-the-art driving world model.