SketchVideo: Sketch-based Video Generation and Editing
Authors: name: Feng-Lin Liu, name: Hongbo Fu, name: Xintao Wang, name: Weicai Ye, name: Pengfei Wan, name: Di Zhang, name: Lin Gao

Video generation and editing conditioned on text prompts or images have
undergone significant advancements. However, challenges remain in accurately
controlling global layout and geometry details solely by texts, and supporting
motion control and local modification through images. In this paper, we aim to
achieve sketch-based spatial and motion control for video generation and
support fine-grained editing of real or synthetic videos. Based on the DiT
video generation model, we propose a memory-efficient control structure with
sketch control blocks that predict residual features of skipped DiT blocks.
Sketches are drawn on one or two keyframes (at arbitrary time points) for easy
interaction. To propagate such temporally sparse sketch conditions across all
frames, we propose an inter-frame attention mechanism to analyze the
relationship between the keyframes and each video frame. For sketch-based video
editing, we design an additional video insertion module that maintains
consistency between the newly edited content and the original video's spatial
feature and dynamic motion. During inference, we use latent fusion for the
accurate preservation of unedited regions. Extensive experiments demonstrate
that our SketchVideo achieves superior performance in controllable video
generation and editing.