MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence
Authors: name: Canyu Zhao; name: Mingyu Liu; name: Wen Wang; name: Jianlong Yuan; name: Hao Chen; name: Bo Zhang; name: Chunhua Shen
Published: 2024-07-23 17:17:05

Abstract: Recent advancements in video generation have primarily leveraged diffusion
models for short-duration content. However, these approaches often fall short
in modeling complex narratives and maintaining character consistency over
extended periods, which is essential for long-form video production like
movies. We propose MovieDreamer, a novel hierarchical framework that integrates
the strengths of autoregressive models with diffusion-based rendering to
pioneer long-duration video generation with intricate plot progressions and
high visual fidelity. Our approach utilizes autoregressive models for global
narrative coherence, predicting sequences of visual tokens that are
subsequently transformed into high-quality video frames through diffusion
rendering. This method is akin to traditional movie production processes, where
complex stories are factorized down into manageable scene capturing. Further,
we employ a multimodal script that enriches scene descriptions with detailed
character information and visual style, enhancing continuity and character
identity across scenes. We present extensive experiments across various movie
genres, demonstrating that our approach not only achieves superior visual and
narrative quality but also effectively extends the duration of generated
content significantly beyond current capabilities. Homepage:
https://aim-uofa.github.io/MovieDreamer/.