Training-free Guidance in Text-to-Video Generation via Multimodal
  Planning and Structured Noise Initialization
Authors: name: Jialu Li, name: Shoubin Yu, name: Han Lin, name: Jaemin Cho, name: Jaehong Yoon, name: Mohit Bansal

Recent advancements in text-to-video (T2V) diffusion models have
significantly enhanced the visual quality of the generated videos. However,
even recent T2V models find it challenging to follow text descriptions
accurately, especially when the prompt requires accurate control of spatial
layouts or object trajectories. A recent line of research uses layout guidance
for T2V models that require fine-tuning or iterative manipulation of the
attention map during inference time. This significantly increases the memory
requirement, making it difficult to adopt a large T2V model as a backbone. To
address this, we introduce Video-MSG, a training-free Guidance method for T2V
generation based on Multimodal planning and Structured noise initialization.
Video-MSG consists of three steps, where in the first two steps, Video-MSG
creates Video Sketch, a fine-grained spatio-temporal plan for the final video,
specifying background, foreground, and object trajectories, in the form of
draft video frames. In the last step, Video-MSG guides a downstream T2V
diffusion model with Video Sketch through noise inversion and denoising.
Notably, Video-MSG does not need fine-tuning or attention manipulation with
additional memory during inference time, making it easier to adopt large T2V
models. Video-MSG demonstrates its effectiveness in enhancing text alignment
with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V
generation benchmarks (T2VCompBench and VBench). We provide comprehensive
ablation studies about noise inversion ratio, different background generators,
background object detection, and foreground object segmentation.