Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse
  Controls to Any Diffusion Model
Authors: name: Han Lin; name: Jaemin Cho; name: Abhay Zala; name: Mohit Bansal
Published: 2024-04-15 17:45:36

Abstract: ControlNets are widely used for adding spatial control in image generation
with different conditions, such as depth maps, canny edges, and human poses.
However, there are several challenges when leveraging the pretrained image
ControlNets for controlled video generation. First, pretrained ControlNet
cannot be directly plugged into new backbone models due to the mismatch of
feature spaces, and the cost of training ControlNets for new backbones is a big
burden. Second, ControlNet features for different frames might not effectively
handle the temporal consistency. To address these challenges, we introduce
Ctrl-Adapter, an efficient and versatile framework that adds diverse controls
to any image/video diffusion models, by adapting pretrained ControlNets (and
improving temporal alignment for videos). Ctrl-Adapter provides diverse
capabilities including image control, video control, video control with sparse
frames, multi-condition control, compatibility with different backbones,
adaptation to unseen control conditions, and video editing. In Ctrl-Adapter, we
train adapter layers that fuse pretrained ControlNet features to different
image/video diffusion models, while keeping the parameters of the ControlNets
and the diffusion models frozen. Ctrl-Adapter consists of temporal and spatial
modules so that it can effectively handle the temporal consistency of videos.
We also propose latent skipping and inverse timestep sampling for robust
adaptation and sparse control. Moreover, Ctrl-Adapter enables control from
multiple conditions by simply taking the (weighted) average of ControlNet
outputs. With diverse image/video diffusion backbones (SDXL, Hotshot-XL,
I2VGen-XL, and SVD), Ctrl-Adapter matches ControlNet for image control and
outperforms all baselines for video control (achieving the SOTA accuracy on the
DAVIS 2017 dataset) with significantly lower computational costs (less than 10
GPU hours).