MagicDriveDiT: High-Resolution Long Video Generation for Autonomous
  Driving with Adaptive Control
Authors: name: Ruiyuan Gao; name: Kai Chen; name: Bo Xiao; name: Lanqing Hong; name: Zhenguo Li; name: Qiang Xu
Published: 2024-11-21 03:13:30

Abstract: The rapid advancement of diffusion models has greatly improved video
synthesis, especially in controllable video generation, which is essential for
applications like autonomous driving. However, existing methods are limited by
scalability and how control conditions are integrated, failing to meet the
needs for high-resolution and long videos for autonomous driving applications.
In this paper, we introduce MagicDriveDiT, a novel approach based on the DiT
architecture, and tackle these challenges. Our method enhances scalability
through flow matching and employs a progressive training strategy to manage
complex scenarios. By incorporating spatial-temporal conditional encoding,
MagicDriveDiT achieves precise control over spatial-temporal latents.
Comprehensive experiments show its superior performance in generating realistic
street scene videos with higher resolution and more frames. MagicDriveDiT
significantly improves video generation quality and spatial-temporal controls,
expanding its potential applications across various tasks in autonomous
driving.