Enabling Versatile Controls for Video Diffusion Models
Authors: name: Xu Zhang; name: Hao Zhou; name: Haoming Qin; name: Xiaobin Lu; name: Jiaxing Yan; name: Guanzhong Wang; name: Zeyu Chen; name: Yi Liu
Published: 2025-03-21 09:48:00

Abstract: Despite substantial progress in text-to-video generation, achieving precise
and flexible control over fine-grained spatiotemporal attributes remains a
significant unresolved challenge in video generation research. To address these
limitations, we introduce VCtrl (also termed PP-VCtrl), a novel framework
designed to enable fine-grained control over pre-trained video diffusion models
in a unified manner. VCtrl integrates diverse user-specified control
signals-such as Canny edges, segmentation masks, and human keypoints-into
pretrained video diffusion models via a generalizable conditional module
capable of uniformly encoding multiple types of auxiliary signals without
modifying the underlying generator. Additionally, we design a unified control
signal encoding pipeline and a sparse residual connection mechanism to
efficiently incorporate control representations. Comprehensive experiments and
human evaluations demonstrate that VCtrl effectively enhances controllability
and generation quality. The source code and pre-trained models are publicly
available and implemented using the PaddlePaddle framework at
http://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl.