Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model
Authors: Han Lin
Jaemin Cho
Abhay Zala
Mohit Bansal
Venue: ICLR 2025 Oral (2025)
Keywords: Adapter
Diffusion
ControlNet
Text-to-video Generation
Image-to-video Generation
Text-to-image Generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: ControlNets are widely used for adding spatial control to text-to-image diffusion models. However, when it comes to controllable video generation, ControlNets cannot be directly integrated into new backbones due to feature space mismatches, and training ControlNets for new backbones can be a significant burden for many users. Furthermore, applying ControlNets independently to different frames can not effectively maintain object temporal consistency. To address these challenges, we introduce Ctrl-Adapter, an efficient and versatile framework that adds diverse controls to any image/video diffusion models through the adaptation of pretrained ControlNets. Ctrl-Adapter offers strong and diverse capabilities, including image and video control, sparse-frame video control, fine-grained patch-level multi-condition control, zero-shot adaptation to unseen conditions, and supports a variety of downstream tasks beyond spatial control, including video editing, video style transfer, and text-guided motion control. With six diverse U-Net/DiT-based image/video diffusion models (SDXL, PixArt-α, I2VGen-XL, SVD, Latte, Hotshot-XL), Ctrl-Adapter matches the performance of pretrained ControlNets on COCO and achieves the state-of-the-art on DAVIS 2017 with significantly lower computation (< 10 GPU hours).

Published as a conference paper at ICLR 2025
CTRL-ADAPTER: AN EFFICIENT AND VERSATILE FRAMEWORK FOR ADAPTING DIVERSE CONTROLS TO ANY DIFFUSION MODEL
Han Lin" Jaemin Cho Abhay Zala
Mohit Bansal
UNC Chapel Hill
(hanlincs, jmincho, aszala, mbansal 1 |cs.. unc. edu https://ctr/ adapter github.
Video Control
CTRL-Adapter
Depth
Multi-oondtton Video Control
Imputt Conditon
Small image
Sparse Video Control
Large Image or Video Diffusion Model
No Condition
Conditon-Costnn Control-
Scroblc
Control
Frozen
Depth
Canny Edge
Normal May
Trained
Figure We propose CTRL-Adapter, an efficient and versatile framework for adding diverse controls to any diffusion model. CTRL-Adapter supports a variety of useful applications.
ABSTRACT
ControlNets are widely used for adding spatial control to text-to-image diffusion models with different conditions, such depth maps, scribbles/sketches, and human poses. However, when it comes to controllable video generation, Con- trolNets cannot be directly integrated into new backbones due to feature space mismatches, and training ControlNets for new backbones can be significant bur- den for many users. Furthermore, applying ControlNets independently to different frames cannot effectively maintain object temporal consistency. To address these challenges, we introduce CTRI-Adapter, an efficient and versatile framework that adds diverse controls to any image/video diffussin model through the adaptation of pretrained ControlNets. including image and video control, sparse-frame video control, fine- fine-grained patch- level multi-condition control (via an MoE router), zero-shot adaptation to unseen conditions, and supports variety of downstream tasks beyond spatial control, including video editing, video style transfer, and text-guided motion control. With six diverse U-Net/DiT-based image/video diffusion models (SDXL, PixArt-a: 12 12VGen-XL, SVD, Latte, Hotshot-XL), CTRL-Adapter matches the performance pretrained ControlNets on COCO and achieves the of-ee----h------ on DAVIS 2017 with significantly lower computation (< 10 GPU hours).
-ddater offers strong and diverse capabilities,
"Equal Contribution

---

Published as a conference paper at ICLR 2025
INTRODUCTION
images (Rom- bach et al., 2022; Podell et al., 2024; Saharia et al., 2022; Ramesh et al., 2022) and videos (Blattmann et al., 2023; Girdhar et al., 2023; Chen et al., 2024a; Lin et al., 2023; Long et al., 2024) from text descriptions. As it is often hard to describe every image/video detail only with text, there have been many works to control diffusion models in more fine-grained manner by providing additional condi- tion inputs such as bounding boxes (Li et al., 2023e; Yang et al., 2023), reference object images (Ruiz et al., 2023; Gal et al., 2023; Li et al., 2023a), and segmentation maps (Gafni et al., 2022; Avrahami ct al., 2023; Zhang et al., 2023c). Among them, Zhang et al. (Zhang et al., 2023c) have released a variety of ControlNet checkpoints based on Stable Diffusion (Rombach al., 2022) v1.5 (SDv1.5), and the user community has shared many ControlNets trained with different input conditions. Until now, ControlNet has become one of the most popular methods for controllable image generation.
Recent diffusion models have achieved significant progress in generating
However, there are challenges when using the existing pretrained image ControlNets for controllable video generation. First, pretrained ControlNet cannot be directly plugged into new backbone models, and the cost for training ControlNets for new backbone models is a big burden for many users due to high computational costs. For example, training& ControlNet for SDv1.5 takes 500-600 A100 GPU hours (Zhang et al., 2023b;a). Second, ControlNet was originally designed for controllable image generation; hence, applying pretrained image ControlNets directly to each video frame independently does not take the temporal consistency across frames into account.
To address this challenge, we design CTRt-Adapter, a novel, flexible framework that enables the efficient reuse of pretrained ControlNets for diverse controls with any new imuge/video diffusion models, by adapting pretruined ControlNets (and improving temporul alignment for videos). We illustrate the overall capabilities of CTRL-Adapter framework in Fig. 1. As shown in Fig. 3 left (in Sec. 2). CTRL-Adapter trains adapter layers (Houlsby et al., 2019; Yi-Lin Sung, 2022) to map the features of pretrained image ControlNet parameters of the ControlNet and the backbone diffusion model frozen. As shown in Fig. 3 right, each CTRL-Adapter consists of four modules: spatial convolution, temporal convolution, spatial attention, and temporal attention. The temporal convolution/attention modules effectively fuse the ControlNet features into image/video diffusion models for better temporal consistency. Additionally, to ensure robust adaptation of ControlNets to backbone models of different noise scales and sparse frame control conditions, we propose skipping the visual latent variable from the ControlNet inputs. We also introduce inverse timestep sampling to effectively adapt ControlNets to new backhones equipped with continuous diffusion timestep samplers. For more accurate control beyond single condition, we designed a novel and powerful Mfxture-of-Experr (MoE) router, which allows fine-grained, patch-level composition of spatial feature maps from multiple control conditions via CTRL-Adapters.
target image/video diffusion model, while keeping the
As shown in Table 5, CTRL-Adapter allows many useful capabilities, including image control, video control, video control with sparse frames, multi-condition control, and compatibility with different backbone models, while previous methods only support small subset of them (see details in Sec. 5). We demonstrate the effectiveness of CTRL-Adapter through extensive experiments and analyses. It exhibits strong performance when adapting ControlNets (pretrained with SDv1.5) various video and image diffusion backbones, including image-to-video generation - I2VGen-XL. (Zhang et al., 2023d) and Stable Video Diffusion (SVD) (Blattmann et al., 2023), text-to-video generation Latte (Ma et al., 2024b) and XL (Mullan et al., 2023), and text-to-image generation - SDXL (Podell et al., 2024) and PixArt-a (Chen al., 2024c). The ability of CTRL-Adapter to seamlessly adapt to based models such as Latte and PixArt-a, which are structurally different from U-Net based ControlNets, demonstrates the flexibility of our framework design.
In Sec. 4.1 and Sec. 4.2, we first show that CTRL- Addpttr matches the performance of a pretrained im- age ControlNet on COCO dataset (Lin et al., 2014) and outperforms previous methods in controllable video generation (achieving state-of-ther art performance on the DAVIS 2017 dataset (Pont-Tused et al., 2017)) with significantly lower training costs (less than 10 GPU hours, see Fig. 2). Next, we demonstrate that CTRL-Adapter enables more accurate video generation with multiple conditions compared to a single condition. Our fine-grained patch-level MoE router consistently outperforms both the equal weights baseline and the global weights MoE router (Sec. 4.3). In addition, we show that skipping the visual latent variable from ControlNet inputs allows video control only with a few frames of (i.e., sparse) conditions, eliminating the need for dense conditions across all frames
2

---

Published as a conference paper at ICLR 2025
Video Control w/ Depth Map
Image Control w/ Depth Map
Optical Flow Error (L)
12VGen-XL Ctri-Adapter (Ours)
0.84
Video Composer
Text2Video Zero
ControlVideo
0.82
SSIM
Controt-A-video
Ctri-Adapter (Ours)
3.5
0.80
ControlNet
SDXL X-Adapter
Training GPU hours
Training GPU hours
Figure 2: Training speed of CTRL-Adapter for video (left) and image (right) control with depth maps, measured on A100 80GB GPUs. For both video and image controls, CTRL-Adapter trained for 10 GPU hours outperforms strong baselines, including SDXL., which is trained for 700 GPU hours.
(Sec. 4.4). We also highlight zero-shot adaption CTRL-Adapter trained with one condition can easily adapt to another ControlNet trained with differrnt condition (Sec. 4.5). In Sec. 4.6 and Sec. 4.7, we provide comparison of the training efficiency with ControlNet, and comparison between a unified CTRL-Adapter with individual CTRL-Adapters, Moreover, CTRL-Adapter can be flexibly applied to variety of downstream tasks beyond spatial control, including video editing, video style transfer, and text-guided object motion control (Appendix H.3). Lastly, provide comprehensive ablations for CTRL-Adaptoe design choices (Appendix E), quantitative analysis including trade-off between visual quality and spatial control (Appendix F), and qualitative examples (Appendix G).
We propose an efficient and versatile framework (daptt) that adds diverse controls to any image/video diffusion model through the adaptation of pretrained ControlNets, which matches the performance of training ControlNet from scratch with significantly lower training costs.
We propose fine-grained patch-level MoE routing effectively compose ControlNet features, while previous works fuse the features of different control conditions only at the image level.
CTRL-Adapter can be seamlessly adapted to both UNet-based and DiT-based image/video back- bones (e.g., PixArt-a, Latte), diffusion models with continuous timestep samplers (via inverse noise sampling), different noise scales and sparse input frames (via latent-skipping).
Through extensive experiments, we show that CTRL-Adapter matches the performance of pretrained ControlNets both image and video generation backbones. In addition, Ctrl-Adapter can be flexibly applied to a variety of downstream tasks beyond spatial control, including video editing, video style transfer, and text-guided object motion control.
METHOD
2.1 PRELIMINARIES: LATENT DIFFUSION MODELS AND CONTROLNETS
Latent Diffusion Models. Many recent video generation works utilize latent diffusion models (LDMs) (Rombach al., 2022) to learn the compact representations videos. First, given a F-frame RGB video E IR Fx3xxxx video encoder (of pretrained autoencoder) provides C-dimensional E(x) E IR FxCxH' xw" where height and width are spatially downsampled (H' < and W' < W). Next, in the forward process, a noise scheduler (e.g.. DDPM (Ho et al., 2020)) adds noise to the latents Then, in the backward pass, diffusion model Fo(ze, I, Crestiig) learns to gradually denoise the latents, given a diffusion timestep f, and a text prompt Cuea (i.e., T2V) and/or initial frame Gmng (i.e., 12V) if provided. The diffusion model is trained with objective:
latent representation (i.e., latents):
LDMM Eg....N
c0 t, Caexting) 12
where and represent the added noise to latents and the predicted noise by Fo respectively. We apply the same objective for CTRL-Adapter training.
ControlNets. ControlNet (Zhang et al., 2023c) is designed to add spatial controls (e.g., depth, sketch, segmentation maps, etc.) to image diffusion models. Specifically, given a pretrained backbone image diffusion model Fo that consists of input/midddl/ouuppt blocks, ControlNet has similar

---

Published as a conference paper at ICLR 2025
Pretrained Image ControlNet
Pretrained Image/Video Diffusion Model
Input istents x H x
Clext
Cing
Of
Timestop Encoder
Toxt Encoder
Image
convolution
Up: Sample
x CXIIIX
Spenall Conv
2D Conv
Input Block A W1
X3
Inpul Block A x
X3
Add Time Norm
Inpat Block H2 W2
Imput
Temporal
Conv
Input Block HB WD
X3
Inpur Rinck HS x W9
Add Time Norm
3D Conv
We
CTRL-Adapter
x
Spalial Aln
FFN
Cjext
Middle Black
Middllo Acapter
Middle Block x
Cross Altendion
H5 WS
Cing
Zero convilution
Spatial
Output Blook Hi We
x1
Ourpot Adapter D
convolltion
Cutput
X3
Ourput UXX Adapter
zaro convalution
Tamporal Adn
Gross Attention
Owet
Temporal Adt
Cutput Block 12 W2
Ouguut Adapter B
zere convolution
Output
H1 W1
Block
Adaptar
SON
convolution
FXCXHXW
Vi
Forward
Backward
Timestep/Tex//
FX EXXXH'XX
Trainable
Frozen
Image Conditions
Figure 3: Left: CTRI-Adapter (colored orange) enables to pretrained image ControlNets (colored blue) for new image/video diffusion models (colored green). Right: Architecture details of CTRL-Adapter. Temporal convolution and attention layers are skipped for image diffusion backbones. architecture For where the input/middle blocks parameters of 6' are initialized from 0, and the output blocks consist of 1 1 convolution layers initialized with zeros. ControlNet takes the diffusion timestep f, text prompt Cext control image (e.g., depth map), and the noisy latents 2, as inputs, and the output features are merged into the backbone model F o for final image generation.
2.2 CTRL-ADAPTER
We introduce CTRt-Adapter, a novel framework that enables the efficient reuse of existing image ControlNets (SDv1.5) for spatial control with new diffusion models. We mainly describe our method details in the video generation settings, since CTRL-Adapter can be flexibly adapted to image diffusion models by regarding images single-frame videos.
Efficient adaptation of pretrained ControlNets. As shown Fig. 3 (left), we train an adapter module (colored orange) map the middle/output blocks of pretrained ControlNet (colored blue) to the corresponding middlc/output blocks of the target video diffusion model (colored green). If the target backbone does not have the same number of output blocks, CTRI--daapee maps the ControlNet features to the output block that handles the closest height and width of the latents. We keep all parameters both the ControlNet and the target video diffusion model frozen. Therefore, training CTRL-Adapter can be significantly more efficient than training a new video ControlNet.
CTRt-Adapter architecture. As shown in Fig. (right), each block of CTRL-Adapter consists of four modules: spatial convolution, temporal convolution, spatial attention, and temporal attention. We set the values for N1, .... N₂ and N by default. The temporal convolution and attention modules effectively fuse the ControlNet features to the video backbone models for better temporal consistency, Moreover, the spatial/temporal convolution modules incorporate the current denoising timestep by adding timestep embeddings into the adapter features and spatial/temporal attention modules incorporate the conditions (i.e., text prompt/initial frame) Creatiigg through cross-attention between image/text condition embeddings and the adapter features.. This design allows CTRL-Adapter to dynamically adjust its features according to different denoising stages and the objects generated. In addition, we skip the temporal comvolution/attention modules when adapting to image diffusion models. See Appendix I.I for architecture details of the four modules, and Appendix E for detailed ablation studies on the design choices of CTRI--Adapte.
4

---

Published as a conference paper at ICLR 2025
ControlNet Patch Feature(s)
Learnable Query Feature
ControlNees
W
W W
Video UNet
SM
Linear & SM
W
Learnable Emb.
CHRL Adapter
(4) Unconditional Global Weights
FFN
w2
W;
4 W.
1 1
SM
Cress Attn vt K
Q+
MLP
MLP N
Patch-Level MLP Weights
(c) Patch-Level O-Former Weights
Figure 4: Left: Framework for multi-condition video generation by combining multiple ControlNets. W1, W2, WN are the weights allocated to each ControlNet. Right: Three MoE router variants. (a) operates globally, while (b) and (c) operate the fine-grained patch-level. C' and N represent feature dimensions and number of ControlNet experts respectively. w/ represents the router weights position (i.j) of the hth ControlNet 2D feature map. SM stands for Softmax.
Adaptation to DiT-based image/video backhones. Our CTRI--Adapter can also adapt U-Net based ControlNets to DiT-based image/video generation backbones. One important observation we made that the spatial features encoded in the U-Net of ControlNets and the DiT blocks are structurally different (see Fig. 22). Specifically, the representation from U-Net blocks exhibits coarse to- fine, hierarchical patterns (e.g., earlier blocks output smaller size feature maps and control high-level information such as object presence, while later blocks output larger feature maps and control lower- level details like textures), while all DiT blocks handle the feature maps of same sizes. This indicates that mapping all middlc/output blocks of ControlNet to DiT blocks might not be the optimal solution. Therefore, we choose to map the feature maps of the largest size in ControlNet (i.e., block A) to the DiT blocks via CTRL-Adapters, which are followed by vero-convolutions for channel dimension matching. To improve computational efficiency for DiT-based video generation models (i.e., Latte Ma et al. (2024b)), see (a) in Fig. 16). See Appendix E.2 for more discussion on CTRL-Adapter designs for DIT.
only insert CTRL-Adapters into every other DiT block (i.e., blocks 2, 4, 6..., 28,
Skipping the latent from ControlNet inputs:
robust adaption to different noise scales & sparse frame conditions. Although the original ControlNets take the latent z, as part of their inputs, we find that skipping ze from ControlNet inputs is effective for CTRL-Adapter in certain settings, as illustrated in Fig. 12. (1) Different noise scales: while SDv1.5 samples noise e from (0, I). some recent diffusion models Hoogeboom et al. (2023); Esser et al. (2024); Blattmann al. (2023) sample noise e of much bigger scale (e.g. SVD Blattmann et al. (2023) sample noise from a N(, T), where Looooormaa 1.6): [0, +oo and B[o] = 7.24). We find that adding larger-scale 2, from the new backbone models to image conditions Cj dilutes the ej and makes the ControlNet outputs less informative, whereas skipping z, enables the adaptation of such new backbone models. (2) Sparse frame conditions: when the image conditions are provided only for the subset of video frames (i.e.. Cj = 9 for most frames f). ControlNet could rely on the information from ž₁ and ignore Cf during training. Skipping z, from ControlNet inputs also helps the CTRte-dapter to more effectively handle such sparse frame conditions (see Table 10). With latent-skipping, the input to ControlNet a becomes cy instead of ey + zt (i.e., ControlNet outputs Fe(c)) instead of F (c, + 2:)).
Inverse timestep sampling: robust adaptation to continuous diffusion timestep samplers. While SDv1.5 samples discrete timesteps uniformly from {0, 1,..10000),, some recent diffusion mod- els Esser et al. (2024); Ma et al. (2024a); Rombach et al. (2021) sample timesteps from continuous distributions, e.g., SVD Blattmann et al. (2023) samples timesteps from a LogNormal distribution. This gap between discrete and continuous distributions means that we cannot assign the same timestep t to both the video diffusion model and the ControlNet. Therefore, we propose inverse timestep sampling, an algorithm that creates timestep mapping between the continuous and discrete time dis- tributions (see Algorithm for PyTorch Ansel et al. (2024) code). The high-level idea of this algorithm is inspired by inverse transform sampling Estimation lemma (2010). Given the cumulative distribution functions (CDFs) of the continuous timestep distribution Fooms and the ControlNet timestep distribu- tion FONe, first uniformly sample value between [0, 1]. and then returns the smallest timesteps teom.. E [0, oc] C R. tCNes E {0, 1, 1,... 1000) C N, such that F procedure naturally creates mapping between two distributions. See Appendix B.2 for details.
2 u, Feser (feNe) 2 u. This
5

---

Published as a conference paper at ICLR 2025
Table 1:
Evaluation of video generation with single control condition DAVIS 2017 dataset. The
best number in each column is bolded, and the second best is underscored.
Depth Map
Canny Edge
Method
FID (U)
Optical Flow Eror (4)
FID ())
Optical Flow Error (U)
Texe2 (Khachatryan al,, 2023)
4.09
17.80
3.77
ControlVideo (Zhang et al., 2024)
27.84
4.03
25.58
3.73
Control-A Video (Chen et al., 2023)
22.16
3.61
22.82
3.44
VideoComposer (Wang et al., 2024)
22.09
4.55
Hotshot-XI. XI. backhone
SDXL ControlNet (von Platen et al., 2022)
45.35
4.21
25.40
4.43
SDel.5 ContmlNet CTRttAdapter (Ours)
20.83
4.15
Latte backbone (DiT-Based)
SDv1.5 ControNNet SoottiCContt +
CTRI-Adupler (Ours)
16.92
3.98
17.87
2.73
12V XL. backbone
SDel.5 ControlNet CTRI-Adupter (Ours)
3,20
6.42
3.37
SVD backhone
SVD Temporal ControlNet (Rowles, 2023)
4.91
4.84
SDvL.5 ControlNet + CTRL- Adaater (Ours)
3.82
2.96
3.96
2.39
Table 2: Evaluation of image generation with single control condition on COCO va12017 split. The best number in each column bolded, and the second best underscored.
Depth Map
Canny Edge
Soft Edge / HED
Method
FID (4)
MSE (4)
SSIM (t)
FID (1)
SSIM (*)
FID (
SSIM (1)
SDvL backbone
SDv1.5 ControlNet (Zhang et al., al., 2023c)
21.25
87.57
18,90
0.4828
26.59
0.4719
T21-Adapter (Mou ol., 2023)
21.35
89.82
18.98
0.4422
GLIGEN (1. al., 2023c)
21.46
88. 22
24.74
0.4226
28.57
0.4015
Uni ControlNet (Zhao 2024)
21.20
91.05
17.79
0.4911
17.86
0.5197
SDXL backbone
SDXL ControlNet
et 2022)
17.91
0.4458
SDv1.5 ControlNet + X-Adapter (Ran ot al,, 2024)
20.71
90.08
0,7885
19.71
0.3002
SDvL.5 ControlNe CrRI CrRI-Adapter (Ours)
19:26
87.54
0.8534
21.04
0.5806
18.08
0.6454
PixAAt-- backbonn (DiT-Based)
PixArt-5 ControlNet (Chen et 2024b)
0.6938
SDv1.5 ControlNet CTRL-Adapter (Ours)
22.54
84.78
0.8496
18.75
0.6359
17.52
0.6812
2.3 MULTI-CONDITION GENERATION VIA CTRL-ADAPTER COMPOSITION
Multi-Controlle (Zhang et al., 2023c) proposed for spatial control beyond single condition. However, this method naively combines different conditions with equal weights during inference time without training. For more effective control composition, we experiment with the following. For each variant, randomly select K E 2,, 4} control conditions to train CTRL-Adapter and the router in each training step. Comparisons of different variants are discussed Sec. 4.3 and Appendix E.4.
This variant replaces these fixed weights with unconditional global learnable weights via lightweight MoE (Shazeer et al., 2017) router. Specifically, this router is a simple linear layer with an input dimension of I and an output dimension equal to the number of conditions. Each output dimension represents the weight allocated certain condition, with the constraint that the sum of all weights equals 1.
(a) Unconditional Global Weights:
(b) Patch-Level MLP Weights: This variant processes the patch-level features of each ControlNet into scalar value independently, then uses softmax to assign ControlNet weights for each patch. Specifically, the router computes the weighted average of the feature maps from different control conditions associated with the same patch, with the weights learned by 3-layer MLP. The input dimension is cqual to the feature map embedding dimension, and the output dimension is equal to 1, producing scalar value that represents the weight of each patch's 's feature map.
(c) Patch-Level Q-Former Weights: This variant takes in all N ControlNet features associated with patch, using an architecture design inspired by Q- D-Former (Li et al., 2023b) to output expert weights Compared with variant (b), where the weights allocated to each patch are independent, variant (c) operates in more holistic way that allows the router and then determine the weights allocated to each patch.
all patches of feature map

---

Published as a conference paper at ICLR 2025
3 EXPERIMENTAL SETUP
ControlNets and Target Difflusion Models. We use ControlNets trained with SD v1.5. For target diffusion models, we experiment with two 12V models I2VGen-XL (Zhang et al., 2023d) and SVD (Blattmann et al., 2023), two T2V models - Latte (Ma et al., 2024b) and Hotshot-XI. (Mullan et al., 2023), and two T2I models SDXL (Podell et al., 2024) and PixArt-a (Chen et al., 2024c).
Training and Evaluation Datasets. We use 200K videos sampled from Panda-70M training set (Chen et al., 2024d) and 300K images from the LAION POP (Schuhmann & Bevan, 2023) dataset for video and image CTRL--Adapeess training respectively. During training, we extract various control conditions (e.g., depth map) on-the-fly to simplify the data-preparation process. Following previous works (Hu & Xu, 2023; Zhang et al., 2024), we evaluate video CTRL-Adaptert on DAVIS 2017 (Pont- Tuset et al., 2017), and image CTRL-Adapters on COCO va mal2017 7 split (Lin et al., 2014). Detailed training and inference setups for the experiments are provided in Appendix C and Appendix D.
Evaluation Metries. We perform evaluation on two folds: visual quality and spatial control. Fol- lowing previous works (Qin et al., 2023; Hu & Xu, 2023), we use FID (Heusel et al., 2017) to measure the visual quality of generated images/videos. For video datasets, following previous works (Hu & Xu, 2023; Li et al., 2024), report the 1.2 distance between the optical flow error (Ranjan & Black, 2017) between the conditions extracted from input and generated videos. For image datasets, following Uni-ControlNet (Zhao et al., 2024), we report the Structural Similarity (SSIM) (Wang et al., 2004) and mean squared error (MSE) between generated images and ground truth images.
4 RESULTS AND ANALYSIS
4.1 VIDEO GENERATION WITH SINGLE CONDITION
We compare SDv1.5 ControlNet + CTRL-Adapter built on Hotshot-XL, 12VGen-XL, SVD, and Latte with video control methods including Text2Video-Zert (Khachatryan al., 2023), Control- A- Video (Chen et al., 2023), ControlVideo (Zhang et al., 2024), and VideoComposer (Wang et al., 2024). As the spatial layers of Hotshot-XL XL are initialized with SDXL and remain frozen, the SDXL ControlNets are directly compatible with Hotshot-XLL XL. so we include Hotshot-XL. SDXL ControlNet as baseline. We also experiment with a temporal ControlNet (Rowles, 2023) trained with SVD.
Table 1 shows that in both depth map and canny edge input conditions, CTRL-Adapters on I2VGen- XL and SVD outperforms all previous strong video control methods in visual quality (FID) and spatial control (optical flow error) metrics. Note that it takes < 10 GPU hours for CTRI-Adapter to outperform the baselines (see Fig. 2). In Appendix H.1, we visualize the comparison between CTRL- CdaptAd and other baselines. We study visual quality-spatial control tradc-off in Appendix F.1.
4.2 IMAGE GENERATION WITH SINGLE CONDITION
We compare SDv ControlNet + CTRL- Adapter with controllable image generation methods that use SDv1.4, .4, SDv1.5, SDXL,
Input Prompt
Input Control Conditions
SOXL
SDXL
SDXL
PixArt-a
ControlNNet
X-Adapter
Atri-Adapter
Ctri-Adapter
metol status two wemen
and PixArt-a as backbones, including pre- trained 1.5SDLL ControlNets (Zhang
berch
al., 2023c; von Platen al., 2022), T2I-
Adapter (Mou et al., 2023), GLIGEN (Li et al., 2023c), Oni-ControlNet (Zhao al., 2024), X-Adapter (Ran et al., 2024), and PixArt-8 ControlNet (Chen et al., 2024b).
purtte citing a bathraom aink
Figure 5: Images generated with single condition.
As shown in Table 2, CTRI-Adapter outperforms baselines with 3Dv1.4/v1: backbones in almost all metrics. When compared to the baselines with SDXL backbones, CTRL-Adapter outperforms X-Adapter in most metrics, and matches (in FID/MSE with depth map inputs) or outperforms SDXL ControlNet (in SSIM with depth map and canny edge inputs). Note that SDXL ControlNet was trained for much longer than CTRL-Adapter (700 vs. 44 A100 GPU hours) and it takes less than 10 GPU hours for CTRL-Adapter to outperform the SDXL depth ControlNet in SSIM (see Fig. 2). In addition, when applied to Dise- backbone (i.e., PixArt-o), CTRI--Aapter achieves good improvement
7

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of different weighting methods (see Fig. 4 right part for details) for multi- condition video generation. The control sources are abbreviated as D (depth map), C (canny edge), N (surface normal), S (softedge), Seg (semantic segmentation map). (line art), and (human pose).
D+C
D+P
D-C+N+S
D+C-N-SHSqq+LAP
(1)
(1)
(})
Flow Error (.)
Bascline: Equal Weights
8.50
2.84
11.32
3.48
8.75
2,40
9.48
2.93
(a) Insonditional Global Weights
9.14
2.89
10.98
3.32
8.39
2.36
8.18
2,48
(b) Patch-Level Weights
840
2.34
9.37
3.17
7.87
2.11
8.26
2,00
(6) Patch-l evel Q-Former Weights
7.54
2.39
9.22
3.22
7.72
2.31
8.00
2,08
in FID (17.52 ours vs. 20.41 PixArt-ò ControlNet on soft edge) and competitive SSIM score. In Fig. 5, we visualize the comparison between CTRI-Adaater and other image control baselines. See Appendix H.2 for more visualizations.
4.3 VIDEO GENERATION WITH MULTIPLE CONTROL CONDITIONS
As described Sec. 2.3, users can achieve multi-soures control by simply combining the control features of multiple ControlNets via our CTRL-Adapter. Table 3 shows the result in two folds; firstly, patch-level MoE routers (i.e., variants b and с Fig. 4) consistently outperforms the equal weights baseline as well as the unconditional global weights (i.e., variant a in Fig. 4), which proves the effectiveness of patch-level fine-graing ane-grained control composition. Secondly, as shown in (b) and (c). control with more conditions almost always yields better spatial control and visual quality than control with a single condition. Fig. 28 and Fig. 29 show that multi-condition composition provides more accurate control compared to single condition. Table 9 extends (a) by conditioning image/tex//timestp embeddings.
Input Image/Prompt
Input Sparse Conditions (User Scribble, Depth Mapp
12VGen-XL CTRL-Adapter
Fly thrugg of museum with many aculprures beauhitu worka of ort atyles
CO
Condition
rrovelin through the Takeyn suburbe
Condition
Condition
Figure 6: Video generation from sparse frame conditions with CTRL-Adapter on 12VGen-XL. (which generates 16 frames in total). We only provide controls for the 1st, 6th, 11th, and 16th frames.
4.4 VIDEO GENERATION WITH SPARSE FRAMES AS CONTROL CONDITION
We experiment CTRIt-Adater with providing sparse frame conditions using 12VGen-XL. as backbone. During each training step, we first randomly select an integer E {1,..., N). where is equal to the total number of output frames (e.g., N 16 for 12VGen-XI.). Next, we randomly select & key frames from N total frames. We then extract these key frames' depth maps and user scribbles as control conditions. we do give the latents and only give the k frames ControlNet. In Fig. we can see that 12VGen-XX XL with our CTRL-Adapter can correctly generate videos that follow the control conditions for the given sparse key frames and make reasonable interpolations on the frames without conditions. In Appendix F.3, we show that skipping the latent from ControlNet inputs is important in improving the sparse control capability.
4.5 ZERO-SHIE SHOT GENERALIZATION ON UNSEEN CONDITIONS
ControlNet can be understood to the unified representation space of backbone generation models. This begs an interesting ques- tion: "Does CTRL-Adapter learn general feature mapping from one (smaller) backbone to another (larger) backbone?" " To answer this question, we experiment by directly plugging CTRL-Adapter to ControlNets that not seen during training. In Fig. 7, we observe the CTRL-Adapter trained on depth maps can adapt to normal map and soft edge ControlNets in a zero-shot manner. Quantitative analysis of different training strategies based such observation is illustrated in Sec. 4.7.
image feature extractor that maps different types of controls
8

---

Published as a conference paper at ICLR 2025
Input Image
Depth Map Video
Normal Map = Video
Soft Edge Video
Training condition
Zero-shat inforence with other conditions
Figure 7: zero-shot transfer of CTRL-Adapter trained only depth maps to unseen conditions.
Depth
OpenPose
Segmentation
0.80
0.83
Adagger LOuns
0.3
0.75
3000
10030 15000 20003 25003 3000 35000
5002
10300 15000
21000
25000
30000
32200
2002
1000
15000
22000
25100 33000 25030
SoftEdge
OpenPose
Segmentation
0.225
0.5
0.202
SSW
Tiskr Cirl Adapte Toural Halkn Cortiilee:
0.5
0.172
0.151
0.5
0.3
0.122
5020
10000 12002 20002 25300 20000 32003
5002
10320
15000
20000 25000 30000
35200
3000
20020 15000 20303 25002 20005 25030
Training Stops
Training Staps
Trairing Staps
Figure 9: CTRI-Adapters are trained much faster than training ControlNets from scratch.
4.6 TRAINING EFFICIENCY
Ctri-Adapter (Ours)
ControlNet
stops
stops
stops
stops 2000 stops
As a complement to Fig. 2, we present training efficiency comparison of our CTRI-Adapter and ControlNet trained from scratch under different con- trol conditions, using the same hyperparameter set- tings for a fair comparison. Fig. 9 shows that our CTRL-Adapter achieves significantly faster training and higher final performance in different backbones and tasks. Additionally, the clock time per train- ing step for our method is faster than that of Con- trolNet (CTRt-Adapter: 0.48s/step v.s. Control- Net: 0.60s/step for depth; CTRL-Adapter: 0.57s/step v.s. ControlNet: 0.68s/step for segmentation; CTRL-
(-1 min) (-3 mina)
min)
(-3 mina)
(-20 mins]
Figure 8: CTRL-Adapter achieves faster con-
vergence than training ControlNet from scratch
Adapter: 0.82s/step v.s. ControlNet: 0.95s/step for openpose). Lastly, Fig. 8 shows that it takes only 3 mins for CTRI--dapter to provide model that accurately follows the edge condition, while
takes 20 mins for ControlNet.
4.7 UNIFIED MULTI-TASK ADAPTER V.S. INDIVIDUAL TASK-SPECIFIC ADAPTERS
In main results from the above sections, train CTRL-Adapter for each control conditions. An interesting question to ask is: can we have single unified CTRL-Adapter that works for all control conditions? We conduct an experiment comparing (1) training single unified CTRL-Adapter (single adapter for all control conditions) v.s. (2) training an individual adapter for each control condition. Specifically, we use the same training settings with the only difference being that for individual AdRl-Adapter, we only extract one control condition (e.g. depth) from the input images, while for each training step of unified CTRI--dapter, we randomly select one control condition from depth, canny, softedge, normal, openpose, lineart, segmentation with equal probability. For evaluation, we compare the generated image quality and control quality on 1000 randomly selected images from the COCO val2017 dataset. As shown in Table 4, unified CTRL-Adapter achieves comparable FID and SSIM scores. Such a result is consistent with the strong zero-shot transferability as observed in Sec. 4.5. Therefore, when user has limited computational resource but still needs to work on multiple control conditions, we suggest training a unified CTRL-Adapter. Extended version of Table shown in Appendix F.4.

---

Published as a conference paper at ICLR 2025
Table 4: Training a unified CTRI-Adapter with SDXL backhone achieves comparable FID/SSIM training individual CTRL-Adapters evaluated on IK samples from COCO va12017.
Depth
Canyy
Sofadge
Segmercation
Nomnal
())
SSIM (1>
FID
(1)
(5)M(1)
FTD
S1)
(1)
SSIM
(T)
SSIM(1)
Individual CTRL- Adapters
14,77
14.00
05123
11.26
0.516
1603
0.632
1494
Unitled CrRt--ddapeer
15.33
03358
1397
0.5454
0.994
12.99
05117
15.88
0.0622
1499
0.1433
5 RELATED WORKS: ADDING CONTROL TO DIFFUSION MODELS
There have been many works using different types of additional inputs to control the image/video diffusion models, such as bounding boxes (Li et al., 2023c; Yang et al., 2023), reference object image (Ruiz et al., 2023; Gal et al., 2023; Li et al., 2023a), segmentation map (Gafni et al., 2022; Avrahami et al., 2023; Zhang et al., 2023c), sketch (Zhang et al., 2023c), etc.. and combinations of multiple conditions (Kim et al., 2023; Qin al., 2023; Zhao et al., 2024; Wang et al., 2024). As finetuning all the parameters of such image/video diffusion models is computationally expensive, several methods, such as ControlNet (Zhang al., 2023c), have been proposed to add conditional control capability via sarameter--fffciiee training (Zhang et al., 2023c; Ryu, 2022; Mou et al., 2023).
X-Adapter (Ran et al., 2024) learns an adapter mod- ule to reuse ControlNets pretrained with a smaller image diffusion model (e.g., SDv1.5) for bigger image diffusion model (e.g.. SDXL). While they fo- cus solely on learning an adapter for image control, CTRL-Adapter features architectural designs (e.g.. temporal convolution/aatention layers) for video gen- eration well. In addition, shown in Fig. 10 (b), X-Adapter needs to be used with the source image diffusion model (SDv1.55 during both training and inference, whereas CTRL-Adapter does not require the smaller diffusion model for image or video gen- eration, making it more memory and computation- ally efficient. SparseCtrl (Guo al., 2023) guides a video diffusion model with conditional inputs of few frames (instead of full frames), to alleviate the cost of collecting video conditions. Since SparseCtrl involves augmenting ControlNet with an additional channel for frame masks, it requires training new variant ControlNet from scratch. contrast, we leverage existing image ControlNets more efficiently by propagating information through temporal layers in adapters and enabling sparse frame control via skipping the latents from ControlNet inputs.
CTRL-Adapter
X-Adupter
Image Video UiNee
Image Controlletes
500L UNei
60v1. 5 UNNe
8Dv1.5
ControlNet
CTRL-Adapter (Durs)
(b) X-Adapter
Figure 10: Comparison of giving differ- ent inputs to ControlNet, where Zt, Cr and I represent latents, input control features, and timesteps respectively. (a): Default CTRL-Adaptes design. (b): X-Adapter (Ran et al., 2024) needs SDv1.5 U-Net as well as SDv1.5 ControlNet during training and infer- ence, whereas CTRI-Adapter doesn't 't need SDv1.5 U-Net at all.
Furthermore, compared with previous works that specially designed for specific condition controls on a single modality (image (Zhang et al., 2023c; Qin et al., 2023) or video (Hu & Xu, 2023; Zhang et al., 2024)), including image control, video control, sparse frame control, with significantly lower computational costs by reusing pretrained ControlNets (outperforms strong baselines in less than 10 GPU hours, see Fig. 2). To the best of our knowledge, we are also the first work that extends video control into fine- grained petch-level composition. Table 5 compares CTRL-Adapter with other relevant methods. See Appendix A. for extended related works.
work presents unified and versatile framework that supports diverse controls,
CONCLUSION
We propose CTRL-Adapter, an efficient, powerful, and versatile framework that adds diverse controls to any image/video diffusion model. Training an CTRL-Adapter is significantly more efficient than training a ControlNet for a new backbone, and it can outperform or match strong baselines in visual quality and spatial control. CTRL-Adapter not only provides many useful capabilities including image/video control, sparse frame control, multi multi-condition control, and zero-shot adaption to unseen conditions, but also can be easily and flexibly integrated into variety of downstream tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the reviewers for the thoughtful discussion and feedback. This work was supported by ARO W91 INIIN121102 I0220, DARPA MCS 166001-19-2-4033 DARPA KATROS Grant 197750-99-2-000 NSF-AI Engage Institute DRL211263, ONR N00014-23- 144-33-1-2356, and Accelerate Foundation Models Research program. The views, opinions, and/or findings contained in this article are those of authors and not of the funding agency.
REFERENCES
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky,
Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary De Vetito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang. Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yumjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang. Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24). ACM, April 2024. doi: 10.1 /tt.33266666666644330 URL https: /ttpttcctcccccccccoo..ooccc.htct org/ assets hetpsessspyttoe httppyytorh22222 pdf. 5, 21, 45
Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. SpaText: Spatio-Textur Representation for Controllable Image Generation. In CVPR, nov 2023, doi: 10.488550arxxv.2211... arxiv. org/ /tabs/22111 012211113305 14305v1. 2, 10, 19
URL https://
Andreas Blattmann, Tim Dockhor,, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets. arXiv preprint arXiv:2311.15122 2023. 2, 5, 7, 19, 21, 23, 28, 34, 45
G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000. 23
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7291-7299, 2017. 23
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Vidcocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024a. 2
Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li. Pixart-d: Fast and controllable image generation with latent consistency models. arXiv preprint arXiv:2401.05252 2024b. 6,
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference Learning Representations, 2024c. URL hctps://openrevieee net/ fore/formm?e femimmid==eAte de=eekk0Feeeeeeeee 3m1. 2, 7, 7,23, 26, 45
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-war Chao, Byung Eun Jeon, Yuwei Fang. Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In CVPR 2024, 2024d. 7, 22, 23,45 45
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305. 13840, 2023. 6,
11

---

Published as a conference paper at ICLR 2025
Xiaoliang Dai, Ji Hou, Yao Ma, Sam Tsai, Jialiang Wang. Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint (rxXfv2309.11800 2023. 23
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis, 2024. 5,
Estimation lemma. Estimation lemma -
Wikipedia, the free encyclopedia, 2010. URL https: //en. wixipedia wtt/wwki// frora wtw.smmmlllii.. 5,
Gunnar Farnebäck. Two-frame motion estimation based on polynomial expansion. In Scandina- vian Conference on Image Analysis, 2003a. URL pestps://ap:
org
CorpusID:156014: 23
Gunnar Farnebäck, Two-frame motion estimation based on polynomial expansion. Image Analysis: 13th Scandinavian Conference, SCIA 2003 Halmstad, Sweden, June 29-July 2, 2003 Proceedings
13, pp. 363-370. Springer, 2003b. 23
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- A-Scene: Scene-Based Text-to- Image Generation with Human Priors. In ECCV, 2022. URL
http:/ / a wxvii.i.. orgg abs 2203. 13131. 2, 10, 19
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. In ICLR, 2023. URL http: http//axvvvvo//// org / abs 28. 01 61 8. 2, 10, 19
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.1077 2023. 2
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020. 19
Qin Guo and Dongxu Yue. Dit-visualization.
petpss//githhe com/ hwap///uqqinccocc DiT-Vis silisualaaainn 2024. Exploring the differences between DiT-based and Unct-based diffusion models in feature aspects using code from diffusers, Plug-and-Play and PixArt. 32
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint 1 16933,, 2023. 10, 19, 20
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In International Conference on Learning Representations, 2024. 19
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei. Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models, 2023, 19
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi: 10. 10.1109/CVP..2016.999 25
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation, 2022. 19
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 7,
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 3, 19, 20
12

---

Published as a conference paper at ICLR 2025
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXtv:2210.02330: 2022. 19
Emiel Hoogeboom. Jonathan Heek, and Tim Salimans. simple diffusion: mad-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232.
PMLR, 2023. 5
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In ICMI,, volume abs/1902.00751, 2019. URL https: /api semant i cccchol
Otg/CorpusIDD IDp 595999 5959981 6.
Zhihao Hu and Dong Xu. Vidcocontrolnet: A motion-guider video-to-video translation framework by using diffusion model with controlnet. arXiv preprint 2XXi:2330..107733 2023. 7, 10, 19, 24
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In ICCV 2023, 2023. 6,
Sungnyun Kim, Junsoo Lee, Kibeom Hong, Daesik Kim, and Namhyuk Ahn. Diffblender: Scalable and composable multimodal text-to-image diffusion models. arXiv preprint 23 2023. 10, 19
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014. 19
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Danicl Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable transformer modelling library. https: //tp:://ggtttti htt.ttttthhhgggggg/ com wwtae///aeecooeeeeeeeeaeeea oformeror 2022. 22
Dongxu Li, Junnan Li, and Steven C. H. Hoi. BLIP-Diffusion: Pre-trained Subject Representation for Controllable fext-to-Image Generation and Editing. In NeurlPS, 2023a. URL http://arxiv. org/ htgg/ab//ae 2305 5. 1 4720. 2, 10, 19
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19742. PMLR, 2023b. 6
Ming Li, Taojiannan Yang, Huafeng Kuang. Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback. In European Conference on Computer Vision (ECCV), 2024.7 7
Yitong Li, Martin Renqiang Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In AAAI, 2017. 19
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. Storygan: sequential conditional gan for story visualization. In CVPR, 2019. 19
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded ext-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511 225212 2023c. 2, 6, 7, 10, 19
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via Ilm- Ilm-guided planning. arXiv preprint
2023. 2
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014. 2, 7, 24, 45
13

---

Published as a conference paper at ICLR 2025
Fuchen Long, Zhaofan Qiu, Ting Yao, and Tao Mei. Videodrafter: Content-consistent multi-scene video generation with Ilm. arXiv preprint arXiv:2401.01256, 2024. 2
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2018. 22
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden- Handen-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion- based generative models with scalable interpolant transformers. arXiv preprint 0al.::400..874444 2024a. 5, 21
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.0000 030448. 2024b. 2, 5, 7, 23, 26, 27, 34, 45
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, and Sergey Tulyakov. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis, 2024. 19
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. In AAAI 2024, 2023. 6, 7, 10, 19, 20
John Mullan, Duncan Crawbuck, and Aakash Sastry. Hotshot-XL. October 2023. URL https: //github. www.coom/oooooo shotcoo hwwshotttthhhoo....... x1. 2, 7, 19, 23, 34, 45
OpenAI. Video generation models as world simulators, 2024. URL https: / com/ research/vides generat interatiiooomdeeoo aa---sswwos- assworld-- simulators. 19
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022. 24
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In International Conference on Learning Representations, 2024. 2, 7, 20, 23, 45
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- -ooiinng and Luc Van Gool. The 2017 davis challenge video object segmentation. arXiv preprint 174.00075 2017. 2, 7, 24, 45
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: unified diffusion model for controllable visual generation in the wild. In NeurIPS 2023, 2023. 7, 10, 19,24
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents, 2022. URL attp:://arxiz org/ abs/ 2204. 06125.
Lingmin Ran, Xiaodong Cun, Jia- Wei Liu, Rui Zhao, Song Zijie, Xintao Wang, Jussi Keppo, and Mike Zheng Shou. X-adapter: Adding universal compatibility of plugins for upgraded diffusion model. In CVPR, 2024. 6, 7, 10, 19, 20, 23, 25
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Iransactions on patlern analysis and machine intelligence, 44(3):1623-1633 2020. 23
Anurag Ranjan and Michael J Black. Optical flow estimation using spatial pyramid network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4161-4170, 2017. 7. 24
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. Proceedings the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
URL https::////......... //tp://ppii femanticschon www.cchool...ooooooooooooooooeee Corpus 2211 191119999..1 193. 22
14

---

Published as a conference paper at ICLR 2025
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image symthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. URL https: //api. //apiisemante
semantt nt cachhlcccaeooooccccccc www.r.org/oooree
2,
Robin Rombach, Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022. 2, 20
Ciara Rowles. Stable Video Diffusion Temporal Controlnet. https:/ http:://gghhhubchoco
-aaaallcootoooooooooooo 2023. 6,
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine Tuning Imxa- Diffusion Models for Subject-Drived Generation. In CVPR, 2023. doi: 10.1 21109//cp55222..2222.022262 URL https: http://awwww.oooa/////////// 12242v1. 2, 10, 19
Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning, 2022. URL https: 10, 19
//github. com/co/cc//ooooo
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic Text-to- Image Diffusion Models with Deep Language Understanding. In NeurlPS, 2022. URL http:
//arxiv. owg/ abs 2205. 11487.
Christoph Schuhmann and Peter Bevan. Laion pop: 600,000 high-resolution images with detailed pop, 2023, 7, 22,
descriptions. https: 45
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- -25294,, 2022. 23
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: the Sparsely- Gated Mixture-o of of-Experts Layer In ICLR, 2017. URL pttttp::///penrrriii
net forum? d=B1ckMDqlg 6
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In /CLR, 2023. 19
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: continuous video generator with the price, image quality and perks of stylegan2. In CVPR, 2022. 19
Jascha Sohl-Dickstein,, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. ISBN 9781510810587 19
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1921-1930, 2023. 32
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-ar diffusion models. https: //gi //github.r www.com////ooo www..//./........cc. ng face di ffusers, 2022. 6, 4,45
Fu- Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305. /8264, 2023a. 19
Jiuniu Wang. Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope rext-to-video technical report, 2023b. 19
15

---

Published as a conference paper at ICLR 2025
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0307,, 2023c. 23
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024. 6, 7. 10, 19, 20, 25
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 1344):600-612. 2004. 7, 24
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-an art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https: www. aclweb. org/ www.anngooooo/oooooaoooooo 2020. amnlp-demos 6. 45
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding, In Proceedings of the European conference on computer vision (ECCV), pp. 418-434, 2018. 23
Enze Xie. Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 12117-122000 2021. 23
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310. 12190, 2023. 19
Zhengyuan Yang, Jianfeng Wang. Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
2, 10, 19
Mohit Bansal Yi-Lin Sung, Jaemin Cho. Vl-adapter: Parameter-efficient transfer learning for vision- and-language tasks. In CVPR, 2022.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang. Minheng Ni, Zhengyuan Yang. Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Ming Gong, Lijuan Wang. Zicheng Liu, Houqiang Li, and Nan Duan. NUWA-XL: Diffusion over diffusion for eXtremely long video generation. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 1309-1320, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. N0.18653/v1/222.acll/
URL https:
// //aclanthology org/2023. mong-ong..... 73. 19
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image dif- fusion models. https: htw.gi/iaaaae...o//llaaaare//aaa 2023a. 2
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image dif- fusion models. https:/ htt.ghhgggggccceeeeeeeeeoo 2023b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffuuion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023c. 2, 3. 6. 7, 10, 19, 20, 23, 45
Shiwei Zhang, Jiayu Wang. Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin. Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: xl: High-quality image-to-vide video synthesis via cascaded diffusion models. arXiv preprint 2rXXi:2311.04444 2023d. 2, 7, 19, 20, 23, 28, 34, 45
16

---

Published as a conference paper at ICLR 2025
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free frre controllable text-to-video generation. In ICLR, 2024. 6, 7, 10, 19, 20, 24
Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris Metaxas. Learning forecast and refine residual motion for image-to-video generation. In Proceedings of the European Conference
on Computer Vision (ECCV), September 2018. 19
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 7, 10, 19, 20, 24
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint ar 1101, 2022.
19
17

---

Published as a conference paper at ICLR 2025
Appendix
A Background
19
A.I Extended Related Works
19
A.2 Extended Preliminaries: LDM and ControlNet
20
B CTRL-Adapter Method and Architecture Details
21
B.1 CTRL-Adapter Architecture Details
B.2 Inverse timestep sampling: robust adaptation continuous diffusion timestep samplers
21
C Training and Inference Details
22
D Experimental Setup
23
ControlNets and Target Diffusion Models
D.2 Training Datasets for CTRI-Adapter
23
D.3 Input Conditions
23
D.4 Evaluation Datasets
24
D.5 Evaluation Metrics
24
Variants of CTRL-Adapter Architecture Design
24
E.1
CTRL-Adapter Design Ablations
24
E.1.1
Different combinations of CTRL-Adapter components
24
E.1.2
Where to fuse CTRL-Adapter outputs in backbone diffussin
25
E.1.3
Number of CTRL-Adapters in each output block position
26
E.2
Adaptation to DiT- Based Backbones
26
E.3
Skipping Latent from ControlNet Inputs
27
E.4
Different weighing modules for multi-condition generation
27
F
Additional Quantitative Analysis
29
F1
Trade-off between Visual Quality and Spatial Control
29
F2
CLIPScore as Evaluation Metric
30
F3
Comparison with Control-LoRA
30
F.4 A Unified Multi-Task Adapter V.S. Individual Task-Specific Adapters
31
G Additional Qualitative Analysis
32
G.1 Visualization of Spatial Feature Maps
32
G.2 Fast Training Convergence
33
H Additional Visualization Examples
34
H.1 Video Generation Visualization Examples
H.2 Image Generation Visualization Examples
39
18

---

Published as a conference paper at ICLR 2025
H.3 Visualization Examples for Additional Downstream Tasks
42
I Ethics Statement
45
Reproducibility Statement
45
K License
45
A BACKGROUND
A.1 EXTENDED RELATED WORKS
Text-to-video and image-to-video generation models. Generating videos from text descriptions or images (e.g., initial video frames) based on deep learning and has increasingly gained much attention. Early works for this task (Li et al., 2017: 2019; Zhao et al., 2018; Skorokhodov et al., 2022) have commonly used variational autoencoders (VAEs) (Kingma & Welling, 2014) and generative adversarial networks (GANs) (Goodfellow et al., 2020), while most of recent video generation works based on denoising diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015). Powered by large-scale training, recent video diffusion models demonstrate impressive performance in generating highly realistic videos from text descriptions (He et al., 2022; Ho al., 2022: Singer et al., 2023; Zhou et al., 2022; Khachatryan et al., 2023; Wang et al., 2023a; Yin et al., 2023; Wang et al., 2023b; Mullan et al., 2023; OpenAI, 2024; Gupta et al., 2023; Menapace al., 2024) initial video frames (i.e., images) (Blattmann et al., 2023; Zhang et al., 2023d; Guo et al., 2024; Xing et al., 2023).
Adding control to image/video diffusion models. While recent image/video diffusion models demonstrate impressive performance in generating highly realistic images/videos from text descrip- tions, it is hard to describe every detail of images/videos only with text or first frame image. Instead, there have been many works using different types of additional inputs to control the image/video diffusion models, such as bounding boxes (Li et al., 2023c; Yang et al., 2023), reference object image (Ruiz et al., 2023; Gal et al., 2023; Li et al., 2023a), segmentation map (Gafni et al., 2022; Avrahami et al., 2023; Zhang et al., 2023c), sketch (Zhang et al., 2023c), etc., and combinations of multiple conditions (Kim et al., 2023; Qin al., 2023; Zhao et al., 2024; Wang et al., 2024). As finetuning all the parameters of such image/video diffusion models is computationally expensive, several methods, such as ControlNet (Zhang al., 2023c), have been proposed to add conditional control capability via parameter-effiieent training (Zhang et al., 2023c; Ryu, 2022; Mou et al., 2023). X-Adapter (Ran et al., 2024) learns an adapter module to reuse ControlNets pretrained with a smaller image diffusion model (e.g., SDv1.5) for bigger image diffusion model (e.g.. SDXL.). While they fo- cus solely on learning an adapter for image control, CTRL-Adapter features architectural designs (e.g., temporal nanvolution/attentioo layers) for video generation well. In addition, X-Adapter needs the smaller image diffusion model (SDv1.5) during training and inference, whereas CdRte-dapter doesn't need the smaller diffusion model at all (for image/video generation), hence being more memory and computationally efficient (see Fig. 10 for details). SparseCtrl (Guo et al., 2023) guides video diffusion model with conditional inputs of few frames (instead of full frames), alleviate the cost of collecting video conditions. Since SparseCtrl involves augmenting ControlNet with an additional channel for frame masks. requires training new variant of ControlNet from scratch. In contrast, we leverage existing image ControlNets more efficiently by propagating information through temporal layers in adapters and enabling sparse frame control via skipping the latents from ControlNet inputs (see Sec. 2.2 for details). Furthermore, compared with previous works that are specially designed for specific condition controls on a single modality (image (Zhang et al., 2023e; Qin et al., 2023) video (Hu & Xu, 2023; Zhang et al., 2024)), our work presents a unified and versatile framework that supports diverse controls, including image control, video control, sparse frame control, and multi-source control, with significantly lower computational costs by reusing pretrained ControlNets (e.g., CTRL-Adapter outperforms baselines in less than 10 GPU hours). Table 5 summarizes the comparison of CTRI-Adapter with related works.
19

---

Published as a conference paper at ICLR 2025
Table 5: Overview of the capabilitiess supported by controllable image/video generation methods.
Method
Image Control
Control
Video
Video Control
Spanse l'rames
Mulni-Conditio
Control
Compatible w/ Different Backbomes
Image Control Methods
NontrolNet (Zhang al, 2023c)
Malti ControlNet (Zhang et al., 2023c)
121-Adapter (Mou ot al,, 20233
x
Cet
al., 2024)
X-Adapter (Ran ot ol,, 2024)
X
Control Methods
VoderolVieea (Zhang al,, 2024)
X
VideoComposer (Wang al., 2024)
X
SparseCrrl (Guo et al., 2023)
CTRI-Adapter (Ours)
Cpext/ /imtt//m
Coext//igg
Up/ Sample
Add lime & Norm
Add lime & Norm
Norm
Spatial Self Artn
Add Norm
Cross Attention
Add Norn
Add Norm
Temporal Sell- Sell-Attn
Add Norm
Attention
Add & Norn
Norm
Conv
Norm
FFN
FFN
FFN
FFN
Spatial Conv
Temporal Conv
SpatialAAtenttion Attention
Temporal Attention
Figure 11: Detailed architecture of CTRL-Adapter blocks.
A.2 EXTENDED PRELIMINARIES: LDM AND CONTROLNET
Latent Diffusion Models. Many recent video generation works are based latent diffusion models (LDMs) (Rombach et al., 2022), where a diffusion model learns the temporal dynamics of compact latent representations of videos. First, given F-frame RGB video z E R FX3xxxxW a video encoder (of a pretrained autoencoder) provides C-dimensional latent representation (i.e., latents): = = E(x) E R R'XCC xw' where height and width are spatially downsampled (H' < H and W" < W'. Next, the forward process, noise scheduler such as DDPM (Ho et al., 2020) gradually adds noise to the latents z: {(2, z2-1) (2;; /1 z,
1, 3,1). where Bg E (0, is the variance schedule with (1,.... T}. Then, in the backward pass. diffusion model (usually a U-Net architecture) F Fo t, Gexuingg) learns to gradually denoise the latents, given a diffusion timestep 4, and a text prompt Cext (i.e., T2V) or an initial frame Cany (i.e., 12V) provided. The Eo (Z Cuexting 11.
diffusion model trained with following objective: LLDM EE,...No. where e and represent the added noise to latents and the predicted noise by Fo respectively. We apply the same objective for CTRI-Adapter training.
ControlNets. ControlNet (Zhang et al., 2023c) is designed to add spatial controls to image diffusion models in the form of different guidance images (e.g., depth, sketch, segmentation maps, etc.). Specif- ically, given a pretrained backbone image diffusion model Fo that consists of input/middll//output blocks, ControlNet has similar architecture F o' where the input/middle blocks parameters of 0' are initialized from 0, and the output blocks consist of Ixl convolution layers initialized with zeros. ControlNet takes the diffusion timestep text prompt Crex,, control image Cr (e.g.. depth image), and the noisy latents as inputs, and provides the features that are merged into middle/output blocks of backbone image model Fo to generate the final image. The authors of ControlNet have released a variety of ControlNet checkpoints based on Stable Diffusion (Rombach et al., 2022) v1.5 (SDv1.5) and the user community have also shared many ControlNets trained with different input conditions based on SDv1.5. However, these ControlNets cannot be used with more recently released bigger and stronger image/video diffusion models, such as SDXL (Podell et al., 2024) and 12VGen-XL (Zhang et al., 2023d). Moreover, the input/middle blocks of the ControlNet are in the same size with those of the diffusion backbones (i.e., if the backbone model gets bigger, ControlNet also gets bigger). Due to this, it becomes increasingly difficult to train new ControlNets for each bigger and newer model that is released over time. To address this, we introduce CTRL-Adapter for efficient adaption of existing ControlNets for new diffusion models.
20

---

Published as a conference paper at ICLR 2025
2t
6
Cris-hdspter
Viseo UNie
Nentra
Vidao UNat
ControlNets
Latents given ControlNeet
Latents ControlNet
Figure 12: Left (default); latent 2, is given to ControlNet. Right: latent 2; not given to ControlNet.
B CTRL-ADAPTER METHOD AND ARCHITECTURE DETAILS
B.I CTRL-ADAPTER ARCHITECTURE DETAILS
In Fig. 11, we illustrate the detailed architecture of CTRL-Adapter blocks. See Fig. for how the CTRI-Adapter blocks are used to adapt ControlNets to image/video diffusion models. Fig. 11 is an extended version of Fig. 3 (right) with more detailed visualizations, including skip connections, nor- malization layers in each module, and the linear projection layers (i.e., FFN) in each spatial/temporal attention modules.
B.2 INVERSE TIMESTEP SAMPLING: ROBUST ADASTADAAA ADAPTATION TO CONTINUOUS DIFFUSION TIMESTEP SAMPLERS
While SDv1.5 55 samples discrete timesteps & uniformly from {0, 1, 10000. some recent diffusion models (Esser et al., 2024; Ma et al., 2024a; Rombach al., 2021) sample timesteps from continuous distributions, e.g., SVD (Blattmann al., 2023) samples timesteps from LogNormal distribution. This gap between discrete and continuous distributions means that we cannot assign the same timestep t to both the video diffusion model and the ControlNet. Therefore, sampling, an algorithm that creates a timestep mapping between the continuous and discrete time dis- tribations The high-level idea of this algorithm is inspired by inverse transform sampling (Estimation lemma, 2010). Given the cumulative distribution functions (CDFs) of the continuous timestep distribu- tion F ooot and the ControlNet timestep distribution FCNer. we first uniformly sample value between [0, 1]. and then returns the smallest timesteps feomt E [0, oo] C R. LCNet E {0, 1, ..., 1000} C N. such that Feom (feom.) u. FeNea (fcNa) 2 u. This procedure naturally creates mapping between two distributions. In Algorithm 1, we provide the Py PyTorch (Ansel et al., 2024) implementation of in- verse timestep sampling, described in Sec. 2.2. SVD (Blattmann et al., 2023) backbone.
propose inverse timestep
example, inverse time stamping adapts to
During each training step, the procedure for this algorithm can be summarized follows:
Sample a variable simeerss_timestap_ssaa
from Uniform[0. 1].
See
line
19
in function
Sample noise scale comt. via inverse transform sampling (Estimation lemma, 2010); Le., we derive the inverse cumulative density function of Toont. and sample coett by sampling u: ocont F (u). See function sample_sis sima and line 21 in function inverse stt.ttmmtttmmmmppllllll
Given a preconditioning function Jcont. that maps noise scale to timestep (typically associated with the continuous-time noise sampler), s sigma_to. timestep and line 23 in inverse_
can compute foott Gcont. Icont. (Dcont)). See function
Set the timesteps and noise scales for both ControlNet and our CTRL-Adapter as lcNer cound(1000w) and cCNa = u respectively, where 1000 represents the denoising timestep range over which the ControlNet is trained. See line 25 in inverse immsstmp_ssmaaaa
During inference, we follow the similar sampling strategy, with the only change in the first step. Instead of uniformly sample single value for u, uniformly sample equidistant values for
21

---

Published as a conference paper at ICLR 2025
within J0, 1] and derive corresponding Feont/NNet and Cont inputs for denoising steps, where here is the number of denoising steps during inference.
Algorithm PyTorch Implementation for Inverse Timestep Sampling import torch
def
sample_aige fi, 1oo=0., scale-1.1
**aDraw noise scale (sigma] following Razzas et al, (2022)
sigme_min, sigma_max
0.002, 700, values in the papez
min_inv_the, max_iny_rbo aigma_min ** cho), signa_max (1 xho)
7
sigme imv__inv_rke (1-ut inin_jav_rho max inv_roo)) sho
return sigma
9
sigma_to iimettep
***Map noise scale to timestep. leeee we use the Function used in SVD. 11
timestep
sigma.lcg()
return Limestep
def
15
senSomple noise aca lea and timeatape for Controllet and di diffusion models the setting used for SVD,
trained cont noise sampler,
18
1) sample a from Uniform[D, terch.rand
2) esiculate sigma_avd from pre- defined log normel distribution
sigme_svd e sample_signa lu, 1oc=0.7, scale=l.6)
3) colcuiate timestep_ava from aigma_avd via pre-defined mapping function timestep_svd
23
slgma_to_timeeteee (sigmapssiga_vd))
4J csiculate timestep sigma for controinet
25
sigma_cnet, timestep_cnes 2, round u)
return aigma_avd, timastep_avd, sigme_enctt, timestep_enet
C TRAINING AND INFERENCE DETAILS
Model architectures. Detailed illustration of our CTRL-Adapter architecture has been provided across several parts of our paper, including Sec. 2, Appendix B.I, Appendix E, and Appendix F. In addition, for all the backbone models used in this paper, we kept all their parameters frozen and made no modifications.
Training details. We use a learning rate of 5 x e-;; Adam W (Loshchilov & Hutter, 2018) optimizer with values for 31, 82, c, and weight decay as 0.9, 0.999, 1 e-8 and e respectively. We set trained on 4 A100 80GB GPUs with batch size the max gradient norm 1. All our experiments training with data type bfloat 6, we didn't use of Please note that other than mixeddppreiisie any additional methods to speed up the training/inference clock time, to save GPU memory. To be more specific, we didn't 't use any of the following methods: xformers (Lefaudeux et al., 2022), gradient checkpointing, 8bit Adam optimizer, and DeepSpeed (Rasley et al., 2020). In addition, to make our framework easy to use directly from raw input images/videos, we extract all control condition images/frames on-the-fly during training. We train the image and video CTRL-Adapters for 80k and 40k steps respectively, which can be finished in 24 hours measured by training clock time. The fast convergence of our method is shown in Fig.
Inference details. All inference can be done single A6000 GPU with 48GB memory. During inference, we use the default hyper-parametee for each backbone model, including the number of frames to generate, the number of denoising steps, and classifier-free guidance scale, etc..
Safeguards. When we generate images during inference, we also activate the NSFW filter of the backbone models. This ensures that users are protected from unnecessary exposure to explicit or objectionable materials. For training, the datasets we used (Chen et al., 2024d; Schuhmann & Bevan, 2023) both filter out the image/video samples with harmful contents. For example, as stated in the "Risk mitigation" section of Panda70M paper, they used the internal automatic pipeline to filter out the video samples with harmful or violent language and texts that include drugs or hateful speech. They also the NLTK framework replace all people's 's names with "person". ". LAION- -OPP dataset also created by filtering out samples based on the safety tags (using a customized trained NSFW classifier that they built).
22

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL SETUP
D.1 CONTROLNETS AND TARGET DIFFUSION MODELS
ControlNets. We use ControlNets trained with SDv1.5. SDv1.5 has the most number of publicly released ControlNets and has much smaller training cost compared to recent image/video diffusion models. Note that unlike X-Adapter (Ran et al., 2024), CTRL-Adapter does not need to load the source diffusion model (SDv1.5) during training or inference (see (a) and (b) in Fig. 10 for model architecture comparison).
Target diffusion models (where ControlNets are to be adapted). For video generation models, we experiment with two text-to-video generation models Latte (Ma al., 2024b) and Hotshot- XL (Mullan et al., 2023), and two image-to-vides generation models I2VGen XL (Zhang et al., 2023d) and Stable Video Diffusion (SVD) (Blattmann al., 2023). For image generation model, we experiment with Art-r (Chen et al., 2024c) and the hase model in SDXL (Podell et al., 2024). For all models, we use their default settings during training and inference (e.g., number of output frames, resolution, number of denoising steps, classifier-free guidance scale, etc.).
D.2 TRAINING DATASETS FOR CTAL-ADAPTER
Video datasets. For training CTRL-Adapter for video diffusion models, download around 1.5M videos randomly sampled from the Panda-70M training set (Chen et al., 2024d). Following recent works (Blattmann et al., 2023; Dai et al., 2023), filter videos of static scenes by removing videos whose average optical flow (Farnebäck, 2003b; Bradski, 2000) magnitude is below a certain threshold. Concretely, we use the Gunnar Farneback's algorithm" (Farnebäck, 2003a) 2FPS, calculate the averaged the optical flow for each video and re-scale it between and 1, and filter out videos whose average optical flow error is below a threshold of 0.25. This process gives us total of 200K remaining videos.
Image datasets. For training CTRtt-ddapeer for image diffusion models, we use 300K images randomly sampled from LAIÓN POP, which subset LAION (Schuhmann et al., 2022) dataset and contains 600K images in total with aesthetic values of least 0.5 and minimum resolution of 768 pixels on the shortest side. As suggested by the authors, we use the image captions generated with CogV VLM (Wang et al., 2023c).
D.3 INPUT CONDITIONS
We extract various input conditions from the video and image datasets described above.
Depth map: As recommended in Midas (Ranftl et al., 2020), we employ www.swiin_aarre__ee for the best preeddpprffrmmnce trade-off.
Canny edge, surface normal, line art, softedge, and sketching/seribbles:: Follow- ing ControlNet (Zhang et al., 2023c), we utilize the same annotator implemented the controlnet_aux" library.
Semantic segmentation map: To obtain higher-quality segmentation maps than UPer- Net (Xiao et al., 2018) used in ControlNet, we employ SegFormer (Xie et al., 2021) segformer-bS- foss-finetueed -eieeedddded-eeee aode-60--2 640-640 finetuned on ADE20k dataset at 640x640 resolution.
Human pose: We employ ViTPose (Xie al., 2021) ViTPose huge_ siire_ssmpee__eeeeee coco to improve both processing speed and estimation quality, compared to OpenPose (Cao et al., 2017) used in ControlNet.
httteerr//aggiineeeeco///srrrrrp httrps://aaaaaaaa/hho///lloppppp htttps://githob.coo
flowiialll...w..ooooooooooooo klow.html
23

---

Published as a conference paper at ICLR 2025
Table 6: Comparison of different architecture of CTRL-Adapter for image and video control, measured with visual quality (FID) and spatial control (MSE/optical flow error) metrics. The metrics are calculated from 1000 randomly selected COCO val2017 images and 150 vidcos from DAVIS 2017 dataset respectively. Top: image control on SDXL backbone. Bottom: video control on 12VGen-XL. backbone. SC, TC, SA, and TA: Spatial Convolution, Temporal Convolution, Spatial Attention, and Temporal Attention. "N: the number of blocks in CTRL-Adapter. We denote the default configurations (SC+SA for image SC+TC+SA+TA for video) with bold font.
SDXL
SC
SC+SA
SA*2
(SC+SA)*2
SC*3
SA33
(SC+SA)"3
FID (
12.15
11.52
11.57
11.74
12.36
10.91
11.66
12.11
12.07
MSE (L)
88.37
89.60
87.97
88.59
88.14
88.22
90.26
12VGen-XL
SC4SA
SA+TA
SC+TA
SC+TC
TC+TA
SA+TA"2
SC+TA"2
(TC+TA)'2
SC+TC+SA+TA
FID (L)
13.15
13.91
13.52
12,87
13.58
13.53
13.73
13,88
13:49
Optical Flow Err.
3.27
3.33
3.30
3.29
3.26
3.37
3,25
3.23
3,22
D.4 EVALUATION DATASETS
Video datasets. Following previous works (Hu & Xu, 2023; Zhang et al., 2024), we evaluate our video ControlNet adapters on DAVIS 2017 (Pont-Tuset et al., 2017). a public benchmark dataset also used in other controllable video generation works (Hu & Xu, 2023). We first combine all video sequences from TrainVal, Test-Dev 2017 and mest-Challenge each vidco into smaller clips, with the number of frames in each clip being the same as the default number of frames generated by each video backbone (e.g.. 8 frames for Hotshot-XL., 16 frames for 12VGen-XL -XL, and 14 frames for SVD). This process results in a total of 1281 video clips of 8 frames, 697 clips of 14 frames, and 608 video clips of 16 frames.
2017. Then, we chunk
Image datasets. We evaluate our image ControlNet adapters on COCO va 1 2017 split (Lin et al., 2014), which contains 5k images that cover diverse range of daily objects. We resize and center crop the images to 1024 by 1024 for SDXL evaluation.
D.5 EVALUATION METRICS
Visual quality. Following previous works (Qin et al., 2023; Hu & Xu, 2023), we use Frechet Inception Distance (FID) (Heusel al., 2017) to measure the distribution distance between our generated images/videos and input images/videos."
Spatial control. For video datasets, following VideoControlNer (Hu & Xu, 2023), we report the 1.2 distance between the optical flow (Ranjan & Black, 2017) of the input video and the generated video (Optical Flow Error). For image datasets, following Uni-ControlNet (Zhao et al., 2024), we report the Structural Similarity (SSIM) ( (Wang et al., 2004) and mean squared error (MSE)* between generated images and ground truth images.
E VARIANTS OF CTAI--DDAPTER ARCHITECTURE DESIGN
E.1 CTRL-ADAPTER DESIGN ABLATIONS
E.I.I DIFFERENT COMBINATIONS CTRL-ADAPTER COMPONENTS
As described See. 2.2, each CTRI-Adapter module consists of four components: spatial convolution (SC), temporal convolution (TC), spatial attention (SA), and temporal attention (TA). We experiment
'To be consistent with the numbers reported in Uni-ControlNet (Zhao et al., 2024), we pytorch-fid wttgghh...oo//aaaaae (attps: : / //githu.....////o///oo/.t http://sseiteeetttttrteeeee
in Table 2. For other results, we use clean-fid fid (Par- Gat.a/aaaaaaaaaa
mar et al., httets///aaattirrrggg///////s/as wwwpeammpes/tttteeeeeeeee
fid) which is more robust to aliasing
artifacts.
ots_sim
html
es/cl assess html #modu lesskeern. learn.
metrics
24

---

Published as a conference paper at ICLR 2025
20k
2.-
12k
ZCk
204
30k
5.0
4.5-
Optical
1.--
3.5
11
06.AA
(belaut)
(Delouk)
Adapter Insert Positions
Adapter Insert Positions
Figure 13: Comparison of inserting CIRL-Adapter to different U-Net blocks. 'Mid' represents the middle block, whereas 'Out ABCD' represents output blocks A, B, C, and D. The metries are calculated from 150 videos from DAVIS 2017 dataset.
Output Block D H4 x Vel
Outpui Adappeer
Output Block D Hét X Veil
Output Adagter
Outpul Blook D Het X V94
Outpui Adapler
connatttioon
convttttion
canvistiion
Output Btrcck
Outpul Adaptor
zem convolution
Ditpul Block
Outpul Flook
Output Block Hé W44
Output Adaator
Block H4 V44
Outpot
Block
convilttion
Adaptor
convelution
H4 194
3 Blocks (default)
2 Blocks
(c) Block
Figure 14: Comparison of inserting different numbers of CTRI-Adapters to the backbone diffusion U-Net's output blocks. We use output block D here for illustration. We insert three CTRI-Adapterss to the output blocks of the same feature map size by default.
with different architecture combinations of the adapter components for image and video control, and show the results in Table 6. Compared to X-Adapter (Ran et al., 2024), which uses stack of three spatial convolution modules (i.e., ResNet (He et al., 2016) blocks) for adapters, and VideoCom- poser (Wang al., 2024), which employs spatial convolution temporal attention for spatiotemporal condition encoder, we explore a richer combination that enhances global understanding of spatial information through spatial attention and improves temporal ability via combination of temporal convolution and temporal attention. For image control (Table 6 top), we find that the combining of SC+SA is more effective than stacking SC SA layers only. Stacking SC+SA twice further improves the visual quality (FID) slightly but hurts the spatial control (MSE) as tradcoff. Stacking SC+SA three times hurts the performance due insufficient training. We use the single SC+SA layer for image CTRL-Adapter by default. For video control (Table 6 bottom), we find that SC+TC+SA+TA shows the best balance of visual quality (FID) and spatial control (optical flow error). Notably, we find that the combinations with both temporal layers, SC+TC+SA+TA and (TC+TA)*2, achieve the lowest optical flow error. We use SC+TC+SA+TA for video CTRL-Adapter by default.
E. 1.2 WHERE TO FUSE CTRI-ADAPTER OUTPUTS IN BACKBONE DIFFUSION
We compare the integration of CTRI-Adapted outputs at different positions of video diffusion backbone model. As illustrated in Fig. 3, we experiment with integrating CTRL-Adapter outputs to different positions of 12VGen-XL's U-Net: middle block, output block A, output block B. output block C, and output block D. Specifically, we compared our default design (Mid+ + Out ABCD) with four other variants (Out ABCD, Out ABC, Out AB, and Out A) that gradually remove CTRL-Adapters from the middle block and output blocks at positions from B to D. As shown in Fig. 13, removing the CTRL-Adapters from the middle block and the output block D does not lead to noticeable increase in FID or optical flow error (i.e.. the performances similar in both left and right plots). However, Fig. 13 (right) shows that removing CTRL-Adapters from block C causes a significant increase optical flow error. Therefore, we recommend users retain CTRIe-Adpters in the mid and output blocks A/B/C to ensure good performance.
'Mid+Out ABCD', 'Out ABCD', and 'Out ABC'
25

---

Published as a conference paper at ICLR 2025
2.7
Training Stops 10k
Optica Flow Eror
20k
3.5
30k
1.4
22-
21
3.0
Mock
2blocks Number of Adapters per Black
blocks
Figure 15: Comparison of inserting different numbers of CTRL-Adapters to each U-Net output block. The metrics are calculated from 150 videos from DAVIS 2017 dataset. We insert -TRp-Adapters to each output block by default.
E.1.3 NUMBER OF CTRL-ADAPTERS IN BACH OUTPUT BLOCK POSITION
As illustrated in Fig. 3, there are three output blocks for each feature map dimension in the video diffusion model (represented by x3 in each output block). Here, we conduct ablation study by adding CTRL- CTRL-Adapters to only one or two of the three output blocks of the same feature size. The motivation is that using fewer CTRL-Adapters can almost linearly decrease the number of trainable parameters, thereby reducing GPU memory usage during training. We visualize the architectural changes with output block D as an example in Fig. 14. We insert CTRL-Adapters for three blocks as our default setting. As observed in Fig. 15, reducing the number of CTRL-Adapters increases the optical flow error. Therefore, we recommend adding CTRI-Adapters to each output block to maintain optimal performance.
E.2 ADAPTATION TO DIT-BASED BACKBONES
As illustrated in Sec. 2. we have observed that the spatial features encoded in the U-Net of ControlNets and the DiT blocks are structurally different (see Fig. 22 for visualization of such observation). Therefore, mapping
middle/output blocks ControlNet to DIT blocks might not be the optimal solution. In Fig. 16, we implement three different strategies to insert CTRL-Adapter to the DiT blocks. Specifically, variant (a) inserts CTRL-Adapter
interleavingly into the DiT blocks, while variant (b) and (c) insert CTRL-Adapters to the first 14 and the last 14 DiT blocks respectively. In Table 7, we perform quantitative analysis of these three variants on the DiT-based video generation model, Latte (Ma al., 2024b), with soft edge as control condition. As we can see, inserting Adaters interleavingly into the DiT blocks gives the best performance. This is consistent with finding: since all DiT blocks encode global information of the generated objects, it is optimal to treat these blocks equally, rather than inserting CTRL-Adapted only at the beginning or end. Between locations A and B. we use location A our default setting because its feature map size (64 64) directly matches the features of the DiT blocks (also 64 x 64) without resizing.
After finalizing where insert CTRL-Adapters, the next question is which block(s) of the Not we should create AdateAdapters to map from. In Table 8, we implement several variants on the DiT- based image generation model, PixArt-a (Chen et al., 2024c), including mapping from the block(s) at location A, location B, location C, and location D, respectively (see Fig. for the definitions of these locations). As we can see, mapping from location this is consistent with our findings in Fig. 22, since feature maps at locations C and D are too coarse to be informative. Moreover, we implemented two additional variants: (1) combining the ControlNet features from locations A and B (i.e., Output Blocks A+B), and (2) mapping more blocks from the same location (i.e., the second and third columns in Table 8). However, neither of these approaches provides sufficient gain compared to mapping single block from location A or B. Therefore, use mapping one block from location A as our default setting in our main paper.
location B gives the best performance. Again,
[Han: add rebuttal table]
26

---

Published as a conference paper at ICLR 2025
Pretrained DiT-Based Backbone
Pretrained DiT-Based Video Backbone
Pretrained DiT-Based Video Backbone 2t
FXG.
CTRL-Adapter
Transfores
CTRL-Adapter
Tunsiorns
Sluck
Turalerss
port
awe
Adazerr
Transforee
CTRteAdapter
Taneerrerileez it @ pr
Adazter Black
Toneturww Slock
ony
Booo
Adacter
Transtrer Block f
Sleck 20 p con
Rioos
Transforme Slock
Trnntorme Slock
ony
Broos
FXOXM W
FX X.
50
FX X.V
50
(a) Insert Adapters Interleaved (Default)
(b) Insert into First 14 DIT Blocks
(c) Insert into Last 14 DIT Blocks
Figure 16: Visualization of different routing methods for combining multiple ControlNet outputs. We use (a) as our default setting, and show the settings (b) and (c) as ablations.
Table 7: Ablation of inserting CTRL-Adapters to different DiT blocks in Latte (Ma et al., 2024b).
Visualization of the three architecture variants (interleaved, first half, and second half) are shown in Fig. 16. We use soft edge as control condition here for evaluation.
(a) Interleave (default)
(b) First Half
(c) Second Half
FID
Optical Flow Errrr (1)
FID (L)
Optical Flow Error (1) FID (L)
Optical Flow Error (L)
18.32
2.98
19.66
3.09
23.18
3.31
E.3 SKIPPING LATENT FROM CONTROLNET INPUTS
We find skipping the latent from ControlNet inputs can help CTRL-Adapter to more robustly handle (I) adaption the backbone with noise scales different from SDv1.5, such as SVD and (2) video control with sparse frame conditions. For the first scenario, latents in SVD leads to better visual quality (lower FID), but slightly worse spatial control (higher optical flow error). This is reasonable since skipping the noisy latents can avoid introducing large noise into the ControlNet, but also risks losing information encoded the latents. For the second scenario, skipping latents results in both better visual quality and better spatial control, adding dense noisy latents can make the sparse control conditions less informative.
from Table 10 that skipping
E.4 DIFFERENT WEIGHING MODULES FOR MULTI-CONDITION GENERATION
For multi multi-condition generation described Sec. 2.3, in addition the simple unconditional global weights, we also experimented with learning router module that takes additional inputs such as diffusion time steps and image/text embeddings and outputs weights for different ControlNets. Specifically, introduce three variants based on (a.1) unconditional global weights, which are (a.2) MLP router taking timestep as inputs; (a.3) MLP router - taking image/text embedding as inputs; and (a.4) MLP router taking timestep and image/text embedding as inputs. The MoE router in these variants are constructed as a 3-layer MI.. We illustrate the five methods in Fig. 17.
Table 9 show that all four global weighting schemes for fusing different ControlNet outputs perform effectively, and no specific method outperforms other methods with significant margins in all settings. With no surprise, patch-level MoE router performs consistently better than global MoE router in all control settings. Testing the effectiveness of incorporating text/image/immestep embeddings to patch-level MoE routers are left for future work.
27

---

Published as a conference paper at ICLR 2025
Table 8:
Ablation study of mapping ControlNet features from different locations, and mapping different number of blocks from the same location to the DiT blocks. The best numbers in each row are bolded, and the best numbers each column are underscored.
Block (default)
2 Blocks
Blocks
Insert Locations
FID (4)
SSIM (U)
FID (L)
SSIM (1)
FID (J)
SSIM (U)
Oulput Block A (default)
17.90
0.6802
19.08
0.6971
19.28
0.6855
Output Block
18.23
0.6712
18.61
0.6720
21.47
0.6549
Output Blocks A+B
17.52
0.6812
Oulput Block
22.22
0.5273
Output Block
34.16
0.3506
Table 9: Comparison of global weighting methods for multi-condition video generation (see Fig. 17 for visualization of the additional weighting methods (a.2, a.3, and a.4) developed based on (a. 1) unconditional global weights). The control sources are abbreviated as D (depth map), C (canny edge), (surface normal), S (softedge). Seg (semantic segmentation map). (line art), and P (human pose).
D4C
DdP
DoCoNaS
D-C-N.S4Seg-1.44
FID
(
Emoo
Equal Weights
8.50
2,84
11.32
3:48
8.75
2,40
9:48
2.93
Global Meli Router
(0.1) Unconditional Global Weights
9.14
2,89
10.98
3.32
8.39
2:36
8.18
2.48
Timestop Emnl Weights
9.41
3.51
11.13
3.35
2.78
8.17
2.45
Text'lmage
(a.4) Timestop + Tex/Emage Emn, Weights
8.64
3.31
10.69
3.43
8.09
2.69
8.51
2.43
Parch-l evel MoF. Router
(b) MLP Weights
3.40
2,34
9.37
3.17
7.87
2.11
8.26
2,00
(c) O-Fomee Weights
3.22
7.72
2.31
8.00
Table 10: Skipping latent from ControlNet inputs helps CTRL-Adapter for (1) adaptation backbone models with different noise scales and (2) video control with sparse frame conditions. We evaluate SVD and 12VGen-XX XL on depth maps and scribbles as control conditions respectively.
Method
Latent is given ControlNet FID Optical
(1)
Adaptation to different noise scales
SVD (Blattmann 2023) + CTRL-Adapter
4.48
2.77
SVD (Blattmann al., 2023) CYRL-Adapter
3.82
2.96
Spurse frame conditions
12VGen-XL (Zhang et al. 2023d) CTRI-Adaater
V
7.20
5.13
12VGen-XI, (Zhang et al, 2023d) Adatted
5.98
4.88
Ctext Text Encoder
Cing Image Encoder
Timestep Encoder
Learnable Emb. Softmax
W, W
MLP
MLP
MLP
W
Softmax
Softmax
Softmax
1/N
W, W W W
W,
W
W W W.
W, W....,,WWW W
Equal Weight
(a.1) Unconditional Global Weights
(a.2) Timestep Emb. Weights
(a.3) Text/img Emb. Weights
(a.4) Timestep Textilig Emb. Weights
Figure 17: Visualization of different global MoE routing methods.
28

---

Published as a conference paper at ICLR 2025
F ADDITIONAL QUANTITATIVE ANALYSIS
F.I TRADE-OFF BETWEEN VISUAL QUALITY AND SPATIAL CONTROL
In Fig. 18, Fig. 19, and Fig. 20, we show the visual quality (FID) and spatial control (SSIM/optical flow error) metrics with different numbers of denoising steps with spatial control (with the fusion of CTRI-Adapter outputs) on SDXL, SVD, and I2VGen-XL. backbones respectively. Specifically, suppose we use N denoising steps during inference. a control guidance level of æ% means that we fuse CTRL-Adapter features the video diffusion U-Net during the first x% N denoising steps, followed by (100- x%% x N regular denoising steps. In all experiments, we find that increasing the number of denoising steps with spatial control improves the spatial control accuracies (SSIM/optical flow error) but hurts visual quality (FID).
Depth
Canny
0.78
30
0.50
21
28
SSIM (73
0.822
26
0.52
0.84
19
0.86
22
20
40
60
100
40
60
80
100
Control Guidance Level (%)
Control Guiddacee Level (56
Figure 18: Trade-off between generated visual quality (FID) and spatial control accuracy (SSIM) on SDXL. Control guidance level of x represents that we apply CTRI-Adapter in the first x% of the denoising steps during inference. A control guidance level between 30% and 60% usually achieves the best balance between image quality and spatial control accuracy.
Depth
Canny
8.5
4.5 (47
8.0
7.6
Optical Flow Error
4.4
7.5
Flow
RD
3.9
4.1
6.8
6.5
3.6
3.8
20
60
100
20
60
80
100
Contral Guidance Level (16)
Control Guidance Level (9%)
Figure 19: Trade-off between generated visual quality (FID) and spatial control accuracy (Optical Flow Error) on SVD. Control guidance level of represents that apply CTRL-Adapter in the first %% of the denoising steps during inference. A control guidance level between 40% and 60% usually achieves the best balance between image quality and spatial control accuracy
Depth
Canny
Opticall Flow (4)
3.6
Opt Flow Error (4)
3.3
-3.5
8-
7
20
40
60
80
100
Control Guidance Level (%)
Control Guldance Level (96)
Figure 20: Trade-off between generated visual quality (FID) and spatial control accuracy (Optical Flow Error) on 12VGen-XL. Control guidance level of represents that apply CTRL-Adapter in the first a% the denoising steps during inference. control guidance level between 40% and 60% usually achieves the best balance between image quality and spatial control accuracy.
29

---

Published as a conference paper at ICLR 2025
F.2 CLIPSCORE AS AN EVALUATION METRIC
We also experiment with using CLIPScore on a total 13 different image control training settings, since it is common metric for visual-text alignment. However Fig. 21 shows that CLIPScore does not change much as the training proceeds, even though there are clear improvements visual quality and control accuracy. This indicates that CLIPScore is not meaningful metric for measuring accuracy of controlled image/video generation.
CLI Score Over training Steps
0.50
lopenpcse)
SDXL Cont olleet (depth)
8.300
50XL contet (segmentat
et (operpese)
Pixrt Controlk Isegmentat on)
0205
Controlket (softedge)
SDXL Adaptor (oponpose)
SDXL Ctrl Aei-Aeeeteer Idepthl
SDXL Cbri-Adepter bsegmentation
0400
Adopter (openpasc)
PicArt Chri-Adepter (depth)
Adapter (seamentte entation)
Adspter (softedge)
3000
1900
20006
20664
x000
4300
training Steps
Figure 21: CLIPScore-based evaluation on 13 different image control training settings. CLIPScore does not change much the training proceeds, while there clear improvements in visual quality and control accuracy.
F.3 COMPARISON WITH CONTROI-LLORA
We use the latest Control-LoRA (e.g., version 3), and evaluated all their available official checkpoints for total of 5 control conditions (canny for SDXL well as depth, normal, segmentation, openpose for 1Dv1.5). We follow the same evaluation setting by randomly sampling 1000 images from COC02017 val, and FID for image quality metric and SSIM/mAP for control quality metrics. As we can see from Table 11, CTRL-Adapter is strictly better than Control-LoRA on both generated image quality and control quality. Therefore, compared with Control-LoRA which sacrifices image quality and control quality for higher efficiency, our CTRL-Adapter achieves efficient training without trade-off in performance. We would like point out that the main contribution of our paper is efficient training without trade-off in performance instead of only focusing on training/inference efficiency. The lightweight design of Control-LoRA makes it achieves higher efficiency compared with ControlNet, but suffers from large image quality and control quality drop.
Table 11: Comparison of CTRL-Adapter with Control-LoRA,
Method
Depth
Canny
Segmentation
Normal
Openpose
FID
SSIM
SSIM (f)
FID (L)
SSIM (t)
FID (L)
SSIM
FID
SSIM
Control-LoRA
21.71
0.747
16.02
0.451
19.95
0.595
24.29
0.775
31.07
0.312
CTRI-Adapter
15.12
0.839
13.97
0.560
15.67
0.673
14.94
0.818
18.74
0.521
Phttps://huggingffan stabi httpai/ccoorroo--or///rreeee
30

---

Published as a conference paper at ICLR 2025
F.4 A UNIFIED BULTI-TASK ADAPTER V.S. INDIVIDUAL
ADAPTERS
In Table 12, we provide an extended version of the results as shown in Sec. 4.7. Specifically, we added that trained solely depth but applied other conditions, and diffusion without ControlNet as a baseline for comparison. Following Table 12, we report FID and SSIM evaluation metrics for visual quality and spatial control. As we can see from the tables below, zero-shot results with SDXL+Ctrl- SD Adaaterr trained on depth and applied to other conditions achieve worse performance compared individual and unified adapters (which is expected), but are much better than pure SDXL without ControlNet. This proves the effectiveness of zero-shot transferability of Ctrl-Adapter.
with plain
Table 12: Training a unified CTRI-Adapter with SDXL backbone achieves comparable FID/SSIM to training individual CTRL-Adapters evaluated on IK samples from COCO val2017.
Depth
Canny
Softelge
Soguematiio
SSIM (1)
SSIM (1)
(?)
(1)
FID (1)
(1)
CTAL Adaptes
Trained Solely Depth
SDXL ConmolNet
17.49
0.7079
17:49
0.2916
17:49
0
17.49
0.3241
17.49
0.4624
17.49
0.6306
31

---

Published as a conference paper at ICLR 2025
G ADDITIONAL QUALITATIVE ANALYSIS
G.1 VISUALIZATION OF SPATIAL FEATURE MAPS
As mentioned in Sec. and Appendix E.2, the spatial features encoded the U-Net of ControlNets and the DiT blocks are structurally different. We visualize this difference in Fig. 22. For the DiT- based model, we use PixArt-a as representative. We follow the visualization method mentioned in Tumanyan et al. (2023). Specifically, we first extract the spatial features from different DiT blocks and U-Net middle/output blocks at the last denoising step during inference. For each block, we applied PCA to the extracted features and visualized the top three leading components.
As shown in Fig. 22, almost all 28 DiT blocks capture global and semantic information about the object "cactus". This observation is consistent with the findings in Guo & Yue (2024). On the other hand, the U-Net blocks in ControlNet demonstrate a coarse-to-fine pattern as the feature map size increases. This indicates that mapping output blocks A/B of ControlNet to DiT blocks is better option compared to using middle or output blocks C/D of the ControlNet.
Feature Maps Visualization In Each Transformer Block
Block
Block 9
Block 10
Block
Block
Block
Block 14
Input Prompt
Output Image
cactus happy the Sahara desert
Block
16
Block
Block
Block
Block 20
21
Block 22
Block 23
Block 24
Block Block
Block 27
Block 28
Input Prompt
Feature Maps Visualization In Each ControlNet U-Net Block
Block
Output Blocks
Output Blocks C
Output Image
hoppy face
Saharo desert
Canny Edge
Output Blocks
Output Blocks A
Figure 22: Visualization of spatial feature maps in PixArt-a and canny edge ControlNet. We first extract the spatial features from different DiT blocks and U-Net middle/output blocks the last denoising step during inference. For each block, we applied PCA to the extracted features and visualized the top three leading components. Almost all 28 DiT blocks capture global and semantic information about the object "cactus". while the U-Net blocks in ControlNet demonstrate a coarse-to- fine pattern as the feature map size increases.
32

---

Published as a conference paper at ICLR 2025
G.2 FAST TRAINING CONVERGENCE
In addition to the quantitative results shown in Fig. 2 and Fig. 9, and the qualitative comparison on PixArt-a in Fig. 8, we provide additional visualization for SDXL depth ControlNet + CTRL-Adapter training. The training speed test is performed on 4 A 100 80GB GPUs, with a batch size of per GPU. As shown in Fig. 23, for relatively easy examples (i.e., bedroom, sandwich, bus), our CrRt-Adapter training can converge within 4.5 GPU hours (which is equivalent to around 1.125 hours measured in training clock time). For complex examples and those requiring fine details (Lee,, surfing man, group of kids), our CTRL-Adapter can also converge within around 6 to 7.5 GPU hours (which is equivalent to 1.5 to 1.875 hours measured in training clock time), which proves the training efficiency of our CTRL-Adapter.
Train GPU Hours
Input Prompt
Input Depth Map
1.5
3.0
h.5
6.0 h
7.5
A bed bad
A bus stops to plck passengers
man surfing on lange wave in the ocean
group posing court
A sandwich with a bite taken
Figure 23: Training efficiency of CTRL-Adapter on SDXL backbone. Total training GPU hours are measured on 4 A100 80GB GPUs, with batch size per GPU equal to 1.
33

---

Published as a conference paper at ICLR 2025
H ADDITIONAL VISUALIZATION EXAMPLES
We provide more qualitative examples in this section.
H.1 VIDEO GENERATION VISUALIZATION EXAMPLES
In Fig. 24, we show video generation results on COCO val2017 7 split using depth map and canny edge control conditions. We visualize baseline methods as well as CTRI-Adapters built on top of Hotshot-XL (Mullan et al., 2023), SVD (Blattmann al., 2023), 12VGen-XL (Zhang et al., 2023d), and Latte (Ma et al., 2024b).
Fig. 25 and Fig. 26, we show video generation results with 12VGen-XL. using depth map and canny edge extracted from videos from Sora and the internet.
In Fig. 27, we show video generation results with Latte using soft edge extracted from videos from Sora and the internet.
Input Image/ Prompt
Prompt: Snowboard
Prompt:
Prompt:
hockey
Depth Map
Canny Edge
Input Control Conditions
ControlVideo
Control-A-Video
Text2Video-Zero
Hotshot-XL
CTRL-Adapter
Latte
CTRL-Adapter
CTRL-Adapter
12VGen-XL
CTRL-Adapter
Figure 24: Videos generated from different video control methods and CTRL-Adapter on DAVIS 2017, using depth map (left) and canny edge (right) conditions.
"httpss:/opeeaii.com www.eeaaa.cccco/oooemmmooeeeeeee coma sora "https://opena" com/ sora
34

---

Published as a conference paper at ICLR 2025
Input Image/ Prompt
poklish swimming
Input Condition
12VGen-XL
CTAL-Adapter
Input Image/ Prompt
This closé-up shot of Victoria crowned pigeon shoucases is stricing blue plumago and red chast. crest made delicate, lacy feathers, while eve strlingg red color. The bird's head titad slighdy the sid,, giving the impression of looking regai and majestic. The backgro und blumed, driwing attention to the sidd's siiking appesrance.
Input Condition
12VGen-XL
CTAL-Adapter
while and orange ge tabby cat seen happll darting through dense ganden, as chasing something Is eyes happy fonward glants, captured path nanow closely, gaing perspictive. with shallow
Input Image/ Prompt
Towers lcaves
accentuatin
desth fiels.
Input Condition
12VGen-XL + CTRL-Adapter
Input Image/ Prompt
Christmas vilagg snow-covered glowing trees, snowmen, tiny figurines quaint, holiday- diorama evaking cozy, celebratory winter atmosphere
Input Condition
12VGen-XL
+ CTRL-Adapter
Figure 25;
Video generation with 12VGen-XL + CTRL-Adapter using depth map as control
condition.
35

---

Published as a conference paper at ICLR 2025
Input Image/ Prompt
This close-up shot chameleon showcases strkkng color changing ca capabilless The background is blured, drawing attention to the animals strking appearance.
Input Condition
12VGen-XL
CTAL-Adapter
Input Image/ Prompt
bird flying over forest.
Input Condition
12VGen-XL
+ CTAL-Adapter
Input Image/ Prompt
Reflections in the window of train traveling through the Tokyo suburbs.
Input Condition
12VGen-XL + CTRL-Adapter
Input Image/ Prompt
woman wearing blue jeans and white tshirt taking pleasant stroll Mumbal India beauntful
Input Condition
12VGen-XL
+ CTRL-Adapter
Figure 26:
Video generation with 12VGen-XL CTRL-Adapter using canny edge control
condition.
36

---

Published as a conference paper at ICLR 2025
This
futuristic cyberntic shopherd
Input Prompt
lead have rololie modiication la uaiking black caror with Edturaii digital altarcal Marcallers the slopa hoad ther rightly to the side, giving the impression of looking right and majestic the neon baceground Mumed drawing ataanion the cogs at king appuarance
Input Condition
Latte
CTRL-Adapter
Input Prompt
A Aigrr, lowaring dloud in the stupa of man lcaru over the warth. The dloud man anauta lighting botta doom to the worth
Input Condition
Latte
CTRL-Adapter
Input Prompt
historiaa bace
tial
Input Condition
Latte
CTRL-Adapter
Input Prompt
abstroct japanese anmottio here drops creatures that
Bowwnng tmeegging
Input Condition
Latte
CTRL-Adapter
Input Prompt
A Shiitts dog weating a beret and black futtennce
Input Condition
Latte
CTRL-Adapter
Figure 27: Video generation with Latte CTRI--Adapter using soft edge as control condition.
37

---

Published as a conference paper at ICLR 2025
Fig. 28 shows example videos generated with single and multiple conditions. While all videos correctly capture the high-level dynamics 'a woman wearing purple strolling during sunset', , the videos generated with more conditions show more robustness in several minor artifacts. For example, when only depth map is given (Fig. 28 a), the building behind the person is blurred. When depth map and human pose given (Fig. 28 b), the color of the purse changes from white to purple. When four conditions (depth map, canny edge, human pose, and semantic segmentation) are given, such artifacts removed (Fig. 28 c). In Fig, 29, we show multi-condition control examples with 12VGen-XL.
Input Image/Prompt
Input Conditions: Depth Map, Human Pose, Canny Edge, Segmentation Mask
Prompt:
waman wenring purple overalls pleasont shall zehormeeburg South Africo nanng becuritu surtee
(o) Depth Map Pose Canny Edge
(a) Depth Map
(b) Depth Map + Human Pose
Semantic Segmentation
Figure 28: Video generation from single and multiple conditions with CTRL-Adapter on 12VGen-XL. (a) single condition: depth map: (b) 2 conditions: depth map + human pose; (c) 4 conditions: depth map- + human pose canny edge + semantic segmentation. Adding more conditions can help fix several minor artifacts (e.g., in (a) building is blurred; in (b) purse color changes).
Input Image// Prompt
Prompt: child adult standing in shallow ocean waters along the beach
Prompt: man dancing
Input Conditions
Depth Map
Canny Edge
Semantic Segmentation
12VGen-XL
CTRL-Adapter
Figure 29: Video generated with 12VGen-XI. CTRI-Adapter from 4 control conditions: depth map + human pose canny edge semantic segmentation map.
38

---

Published as a conference paper at ICLR 2025
H.2 IMAGE GENERATION VISUALIZATION EXAMPLES
In Fig. 30 and Fig. 31, we show image generation results on COCO val2017 split using depth map and canny edge control conditions.
In Fig. 32, Fig. 33, and Fig. 34, we show image generation results on prompts from Lexica 12 using depth map, canny edge, and soft edge as control conditions.
Input Prompt
Input Depth Map
SDXL ControlNet
CTRL-Adapter (Ours)
X-Adapter
police officer motorcycle drives through parade.
middle aged black woman is standing behind table full of bananas,
A pair children stand fence together
A sitting in restaurant photographs sandwich.
reception hall holds drink
woman skiing down a snowy hill
A vegetable vendor
organizing his food for sale
Figure 30: Image generation from different SDXL-based image control methods and CTRL-Adapter on COCO val2017 split using depth map as control condition.
Phttps://texica.arv
39

---

Published as a conference paper at ICLR 2025
Input Prompt
Input Depth Map
SDXL ControlNet
CTRL-Adapter (Ours)
X-Adapter
man in suit is surfing.
A woman tennis player getting ready to be served the
Baseball player number 22 holding batting gloves and wearing helmet
22
person who is on their matorcycle the air
vase full irises with pitcher on an end table
are
arranged in a vase sitting on table
dog on top of bed in room
Figure 31: Image generation from different SDXL--based image control methods and CTRI-Addatee on COCO val2017 split using canny edge as control condition.
40

---

Published as a conference paper at ICLR 2025
lacp vulk chordig du nesuth ited redect greay, scoseeeec surecces Jowr
rainbow
cute hanyy hadgensg lkkig bia chocel, can sterr stye Battstrr
Figure 32: Image generation with SDXL CTRL-Adapter using depth map control condition.
Cute fluffy corgi dag in the city anime style
Astronaut walking on water
Cute Iody frog in dress and crown dressed in gown in cinematic environment
Cute and super adomble mouse black and red chef coat chef hat holding steaming entreo.
Figure 33: Image generation with SDXL + CTRI-Adapter using canny edge as a control condition.
cheesecaka, lightin, food photography
Amioro-tnn city pot ctr saly planted
Araccoon family hoving
Figure 34: Image generation with PixArtaa CTRI-AAdapter using soft edge as a control condition.
41

---

Published as a conference paper at ICLR 2025
H.3 VISUALIZATION EXAMPLES FOR ADDITIONAL DOWNSTREAM TASKS
Here, we describe in detail how our CTRL-Adapter can be seamlessly integrated into a wide variety of downstream tasks including video editing, video style transfer, and text-guided motion control.
Video editing, Video editing can be achieved by combining image and video CTRL-Adapters. The procedure is follows:
Firstly, given source video, first extract the control condition(s). We either extract single control condition (e.g., depth map), or multiple control conditions (e.g., depth map, canny edge, segmentation, efc.) to enhance performance (as observe in Tab. 3, Fig. 28 and Fig. 29 that multi-condition control improves spatial control accuracy).
Next, given a user sreviprovided prompt together with the extracted control condition(s), we can use image CTRI--dapter (i.e., SDXL CTRt.-Adapter) generate the first frame of the video.
Finally, can use video CTRL- (i.e., 12VGen-XL. CTRL-Adapter), with the generated first frame image, text prompt, and extracted control conditions as inputs for final video generation.
In Fig. 35, we provide additional visualizations of the camel example our main paper.
SDXL CTRL-Adapter Output
12VGen-XL CTRL-Adapter
Output
Prompt
zebra stripped camel walking
Source Video
camel with rainbow fur
Extracted Control Condition
walking, van gogh-style
camel walking, style
Figure 35: Video editing by combining SDXL and 12VGen-XL, where both models are equipped with spatial control via CTRL-Adapter. First, extract conditions (e.g., depth map) from the original video. Next, we create the initial frame with SDXL CTRL-Adapter. Lastly, we provide the newly generated initial frame and frame-wise wise conditions to 12VGen-XLL CTRL-Adapter obtain the final edited video. This video editing framework can edit both object and background.
42

---

Published as a conference paper at ICLR 2025
Text-Guided Motion Control. This task can be achieved by combining video CTRL-Adapter with inpainting ControlNet. We train such CTRp-Adapter as follows:
Firstly, for each training video, we randomly select random block the image, with the width and height of the block uniformly sampled from 0.25 to 0.75 of the image size.
Next, we color the block area of the video frames as black color (these processed frames can be regarded control condition sequences like depth maps).
Finally, train CTRt-Adapter with the frozen inpainting ControlNet similar as other types of CTRL-Adapters mentioned our main paper.
In Fig. 36, we provide additional visualizations of text-guided image amination.
Prompt
12VGen-XL CTRL ATapter
A white and orange tabby cat darting through garden, chasing something
A medium sized friendly looking dog walks through industrial parking lot
elk with impressive antlers grazing on the snow-covered ground
Figure 36: Text-guided motion control by combining inpainting ControlNet with 12VGen-XL. + CTRI-Adapter. Specifically, inpainting ControlNet takes the masked frames well text prompt as inputs. The output feature maps inpainting ControlNet are then given to CTRI-Adapter built on top of 12VGen-XL to generate the final video. Object(s) in the masked area follow the motion described in the text prompt. The unmasked area can be either static or dynamic.
43

---

Published as a conference paper at ICLR 2025
Video style transfer. This task can be achieved by combining video CTRL-Adapter with shuffle ControlNet. We train such CTRL-Adapter as follows:
Firstly, for the first frame of each training video, we apply the content shuffle detector implemented in the cont ro I net_aux library, to get shuffled image.
Next, we repeat this shuffled image N times, with N equal to the number of output frames of the backbone video diffusion model. These repeated images can regarded control condition sequences like depth maps.
Finally, we can train CTRL-Adapter with the frozen shuffle ControlNet similar as other types of CTRI--Adapters mentioned in main paper.
in Fig. 37, we provide additional visualizations of video style transfer.
Shuffle
Prompt
Output Video
miniature Christmas village with snow-covered houses, glowing windows, decorated trees, festive snowmen, and figurines quaint, haliday thamed cioroma evoking a cosy, celebratory winter otmosspheee
+
Stop motion of colorful paper flower blooming
ornate histerical massive tidal wave peaks and begins to crash
+
meticulously crafted diarama dapicting Edo-period Japan
+
Beautiful, snowy Tokyo bustling
Figure 37: Video style transfer by combining shuffie ControlNet with Latte + CTRL-Adapter Specifically, shuffle ControlNet takes the shuffied image as well as text prompt as inputs. The output feature maps of shuffle ControlNet are then given to CTRL-Adapter built on top of Latte generate the final video. The generated videos maintain similar style as the input image before shuffling.
Phttps:// thub. /ttttt..c///ggggggoffoeeee/o/e net_aux
44

---

Published as a conference paper at ICLR 2025
1 ETHICS STATEMENT
CTRI-Adapter is motivated by the fact that training ControlNets for new diffusion models, especially video diffusion models that need consider temporal consistency, can be huge burden for many users. As shown in Fig. 2, by adopting pretrained ControlNets, training CTRL-Adapter can be significantly faster than training other controllable image or video generation methods. For example, with the same type of compute (i.e, A100 80GB GPUs), CTRL-Adapte trained on SDXL depth ControlNet for GPU hours can outperform SDXL ControlNet trained for 700 GPU hours. This drastically reduces the carbon emissions footprint by over 70 times. Therefore, we believe that our work can strong contribution efficient and controllable image and video generation.
While our framework can benefit numerous applications in controllable generation, similar to other image and video generation frameworks, can also be used for potentially harmful purposes (e.g.. creating false information misleading videos). Therefore, it should be used with caution in real-world applications.
Note that since CTRL-Adapter is method to equip current open-source image and video diffusion models with better control, its performance, quality, and potential visual artifacts largely depend on the capabilities (e.g., motion styles and video length) of the backbone models used. For example, if diffusion model cannot handle complex motions, CTRL-Adapter built on top of this backbone might lead to non-optimal complex motion control.
REPRODUCIBILITY STATEMENT
We include our training and inference code in the supplementary materials. We explained in detail our architecture design in Sec. 2 and Appendix B, experimental setup details in Appendix D, and training and inference details in Appendix C.
K
LICENSE
We standard licenses from the community and provide the following links the licenses for the datasets, codes, and models that we used in this paper. For further information, please refer to the specific link.
PyTorch (Ansel et al., 2024): BSD-style
HuggingFace Transformers (Wolf et al., 2020); Apache License 2.0
HuggingFace Diffusers (von Platen et al., 2022): Apache License 2.0
ControlNet (Zhang et al., 2023c): Apache License 2.0
SDXL (Podell et al., 2024): MiT License
PixArt-ax (Chen et al., 2024c): AGPL-3.0 License
12VGen-XL. (Zhang et al., 2023d): MIT License
Stable Video Diffusion (SVD) (Blattmann al., 2023): MiT License
Latte (Ma et al., 2024b): Apache License 2.0
Hotshot-XL (Mullan et al., 2023): Apache License 2.0
LAION dataset (Schuhmann & Bevan, 2023): MiT License
Panda70M dataset (Chen et al., 2024d): License
COCO dataset (Lin al., 2014): CC BY 4.0
DAVIS 2017 dataset (Pont-Tusee et al., 2017): CC BY 4.0
45


Reviews:
[{"review_id": "iGS1SiqE6P", "reviewer": "Reviewer_6nvp", "summary": "The paper proposes an efficient method for adapting pretrained ControlNets to various image and video diffusion frameworks. The authors also proposed a mixture of expert method for multi conditioning, and a mapping between discrete and continuous diffusion timesteps. Overall, the paper address an important problem (adapting pre-trained ControlNets efficiently), provide useful insights and achieves superior results.", "questions": "Additional comments (no need to address):\n- typo in in L069 \"Controlnet a cannot\"", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The results proposed in the paper are interesting and convincing. \n- Adapting pre-trained ControlNets to new diffusion frameworks including videos is interesting and the method is novel. \n- The paper is well written and the experiments to validate the proposed method are significant. \n- The code is provided.", "weaknesses": "- Poor presentation: the main paper is packed with experiments but lacks many essential details about the method such as the adapter architecture, more details about the MoE method, etc. \n- Unjustified architecture choices: the authors proposed to use a 1-1 mapping between the ControlNet layer to ones with similar dimension in the new diffusion model. This assumes that these layers learn similar features but the authors did not touch on this topic or justified their choices. \n- What is the rational behind conditioning the CTRL-Adapter with the first frame? How does the proposed method perform without such conditioning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9CbWBYwi0h", "reviewer": "Reviewer_iSfd", "summary": "This paper introduces a new framework called CTRL-Adapter, aimed at enhancing the control capabilities of image and video diffusion models. By adapting pretrained ControlNets, CTRL-Adapter allows for the efficient integration of various control conditions into new diffusion models, supporting image control, video control, and sparse frame control. The results demonstrate that CTRL-Adapter maintains high performance on various datasets with significantly reduced computational costs compared to training ControlNets from scratch.", "questions": "Please see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The presentation and writing quality of this manuscript are are generally strong and well-structured.\n+ The framework supports a wide range of applications, from image and video control to video editing and style transfer.\n+ CTRL-Adapter requires fewer computational resources. This method significantly reduces the training time, outperforming baselines in less than 10 GPU hours\n+ The authors provide comprehensive experiments that validate the performance of CTRL-Adapter.", "weaknesses": "+ While skipping latents can increase efficiency, it may inadvertently lead to loss of fine-grained detail in frames without direct conditioning. I'm wondering if there are any issues with this method that I'm concerned about?\n+ Providing more detailed results from an ablation study about Weaknesses 1 could offer valuable insights into the impact of this strategy. It could reveal conditions where skipping latents is beneficial versus scenarios where it may compromise quality.\n+ Whether skipping latents reduces the ability of the CTRL-Adapter to generalize between certain control types.\n+ In my experience, the most significant advantage of this work is the value of a wide range of applications. In terms of technical contributions, it seems to extend multiple adapters in various generative models. Could the authors provide more elaboration on the technical contribution or insights?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q6U7QcpGi5", "reviewer": "Reviewer_QcBv", "summary": "CTRL-Adapter is an efficient and versatile framework designed to integrate diverse controls into image and video diffusion models by adapting pretrained ControlNets. This method achieves comparable performance to training ControlNet from scratch while significantly reducing training costs. It introduces fine-grained, patch-level MoE (Mixture of Experts) routing to compose ControlNet features more effectively, as opposed to prior methods that operate only at the image or frame level. CTRL-Adapter is compatible with both UNet-based and DiT-based backbones, continuous timestep samplers, and varying noise scales. Extensive experiments demonstrate that CTRL-Adapter performs well on image and video generation and can be applied to various tasks, including video editing, style transfer, and text-guided object motion control.", "questions": "1. What is the additional inference cost associated with using multiple condition ControlNets along with their corresponding control adapters, for example, when handling seven different conditions in Table 3? While the paper mentions significant reductions in training costs by focusing on adapting image-based ControlNets for video tasks, how does the increase in inference complexity, due to maintaining multiple controls, impact overall efficiency and performance? \n\n2. Would integrating control adapters directly inside each ControlNet simplify the adaptation process and make it more efficient, like a fine-tuning task from image to video ControlNet? Could this approach be a viable and efficient baseline, particularly for single-condition scenarios?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method stands out for its simplicity and efficiency. It presents a cost-effective approach to adapting image-based ControlNet models for video-based applications. It also offers valuable insights into merging multiple conditions using three strategic choices. It incorporates a fine-grained, patch-level Mixture of Experts (MoE) routing to enhance the generation process by enabling precise and adaptive control over features.\n\n2. The evaluation is thorough and well-executed, providing extensive comparisons across both single-condition and multi-condition video generation. Additionally, it demonstrates the model's versatility by showcasing its performance on tasks such as video editing, style transfer, and text-guided object motion control, highlighting its wide applicability.\n\n3. The paper is exceptionally well-written, with a logical and coherent structure that ensures ease of understanding.", "weaknesses": "1. The comparison of training costs might not be entirely fair. Since the CTRL-Adapter leverages a pre-trained image ControlNet and then adapts it for video tasks, it benefits from the substantial pre-training cost in the image control task. Given that the image ControlNet is already proficient at extracting information from various conditions, this adaptation should inherently be faster. Therefore, when evaluating training efficiency, it is crucial to account for the advantages gained from the pre-trained ControlNet. \n\n2. The explanation of how multiple conditions are integrated in the proposed method could be clearer and more detailed. The paper briefly outlines three possible integration strategies in a single paragraph, accompanied by a figure, which might be insufficient for comprehending the intricacies involved. This section deserves more space and elaboration in the main text rather than being relegated to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k503HX1CWG", "reviewer": "Reviewer_pa78", "summary": "This work Ctrl-Adapter extends X-Adapter to a more general setting: the pretrained ControlNet and downstream diffusion models can now differ in modality (images, videos) and architecture (e.g., UNet or Transformers). The authors show in extensive experiments that Ctrl-Adapter works well in many scenarios and outperforms X-Adapter.", "questions": "See weakness part. Overall, I lean to accept the paper. I need more explanation compared to X-Adapter in the technical side: (1) the architecture difference; (2) the training difference; (3) explain what is the performance and training speed come from.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The major technical contribution is to extend X-Adapter by involving temporal convolutions for the support of adapting from image to video. \n\n- There are a bunch of engineering contributions to improve performance: (1) the utilization of Q-former; (2) skipping latents; (3) handling noise scheduling shift.\n\n- A really extensive experiments to demonstrate the strength of their work in various scenarios: single condition, multi conditions, zero-shot, from image to image, from image to video.", "weaknesses": "1.\tOverall, the writing could be improved to enhance clarity. The methodology section should be more focused and closely aligned with the key contributions. Currently, it is cluttered with engineering details and lacks mathematical formulation, which makes it harder to grasp the technical novelty.\n\n2.\tDoes skipping latent require retraining ControlNet? \n\n3.\tThe Multi-ControlNet setup is unclear:\n\t•\tAre six conditions trained simultaneously in each iteration?\n\t•\tIn Section 4.5, Figure 8 (zero-shot results), the zero-shot inferences under different conditions do not appear to align accurately with the intended conditions, which is understandable since they were not trained on these. However, could you provide an example without ControlNet for comparison? Could you also add quantitative results for the zero-shot experiment in Table 4? (Consider adding a row for models trained solely on depth but applied to other conditions, and a row with plain diffusion without ControlNet as a baseline for comparison.)\n\n4.  What is architecture difference with X-Adapter? What is the training speed improvement over X-Adapter come from?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)