TrailBlazer: Trajectory Control for Diffusion-Based Video Generation
Authors: name: Wan-Duo Kurt Ma; name: J. P. Lewis; name: W. Bastiaan Kleijn
Published: 2023-12-31 10:51:52

Abstract: Within recent approaches to text-to-video (T2V) generation, achieving
controllability in the synthesized video is often a challenge. Typically, this
issue is addressed by providing low-level per-frame guidance in the form of
edge maps, depth maps, or an existing video to be altered. However, the process
of obtaining such guidance can be labor-intensive. This paper focuses on
enhancing controllability in video synthesis by employing straightforward
bounding boxes to guide the subject in various ways, all without the need for
neural network training, finetuning, optimization at inference time, or the use
of pre-existing videos. Our algorithm, TrailBlazer, is constructed upon a
pre-trained (T2V) model, and easy to implement. The subject is directed by a
bounding box through the proposed spatial and temporal attention map editing.
Moreover, we introduce the concept of keyframing, allowing the subject
trajectory and overall appearance to be guided by both a moving bounding box
and corresponding prompts, without the need to provide a detailed mask. The
method is efficient, with negligible additional computation relative to the
underlying pre-trained model. Despite the simplicity of the bounding box
guidance, the resulting motion is surprisingly natural, with emergent effects
including perspective and movement toward the virtual camera as the box size
increases.