VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video
  Generation
Authors: name: Mingzhe Zheng; name: Yongqi Xu; name: Haojian Huang; name: Xuran Ma; name: Yexin Liu; name: Wenjie Shu; name: Yatian Pang; name: Feilong Tang; name: Qifeng Chen; name: Harry Yang; name: Ser-Nam Lim
Published: 2024-12-03 08:33:50

Abstract: Current video generation models excel at generating short clips but still
struggle with creating multi-shot, movie-like videos. Existing models trained
on large-scale data on the back of rich computational resources are
unsurprisingly inadequate for maintaining a logical storyline and visual
consistency across multiple shots of a cohesive script since they are often
trained with a single-shot objective. To this end, we propose
VideoGen-of-Thought (VGoT), a collaborative and training-free architecture
designed specifically for multi-shot video generation. VGoT is designed with
three goals in mind as follows. Multi-Shot Video Generation: We divide the
video generation process into a structured, modular sequence, including (1)
Script Generation, which translates a curt story into detailed prompts for each
shot; (2) Keyframe Generation, responsible for creating visually consistent
keyframes faithful to character portrayals; and (3) Shot-Level Video
Generation, which transforms information from scripts and keyframes into shots;
(4) Smoothing Mechanism that ensures a consistent multi-shot output. Reasonable
Narrative Design: Inspired by cinematic scriptwriting, our prompt generation
approach spans five key domains, ensuring logical consistency, character
development, and narrative flow across the entire video. Cross-Shot
Consistency: We ensure temporal and identity consistency by leveraging
identity-preserving (IP) embeddings across shots, which are automatically
created from the narrative. Additionally, we incorporate a cross-shot smoothing
mechanism, which integrates a reset boundary that effectively combines latent
features from adjacent shots, resulting in smooth transitions and maintaining
visual coherence throughout the video. Our experiments demonstrate that VGoT
surpasses existing video generation methods in producing high-quality,
coherent, multi-shot videos.