CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
Authors: Yifeng Xu
Zhenliang He
Shiguang Shan
Xilin Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Controllable Image Generation
Image-to-Image Generation
ControlNet
LoRA
Resource-Efficient Adaptation
Primary area: generative models

Abstract: Recently, large-scale diffusion models have made impressive progress in text-to-image (T2I) generation. To further equip these T2I models with fine-grained spatial control, approaches like ControlNet introduce an extra network that learns to follow a condition image. However, for every single condition type, ControlNet requires independent training on millions of data pairs with hundreds of GPU hours, which is quite expensive and makes it challenging for ordinary users to explore and develop new types of conditions. To address this problem, we propose the CtrLoRA framework, which trains a Base ControlNet to learn the common knowledge of image-to-image generation from multiple base conditions, along with condition-specific LoRAs to capture distinct characteristics of each condition. Utilizing our pretrained Base ControlNet, users can easily adapt it to new conditions, requiring as few as 1,000 data pairs and less than one hour of single-GPU training to obtain satisfactory results in most scenarios. Moreover, our CtrLoRA reduces the learnable parameters by 90% compared to ControlNet, significantly lowering the threshold to distribute and deploy the model weights. Extensive experiments on various types of conditions demonstrate the efficiency and effectiveness of our method. Codes and model weights will be released at
https://github.com/xyfJASON/ctrlora.

Published as a conference paper at ICLR 2025
CTRLORA: AN EXTENSIBLE AND EFFICIENT FRAME- WORK FOR CONTROLLABLE IMAGE GENERATION
Yifeng Xu Zhenliang He Shiguang Shan' 1.2 Xilin Chen 1.2
Key Lab of AI Safety, Institute of Computing Technology, CAS, China
University of Chinese Academy of Sciences, China yifeng xu@vipl ict ac.cn, (hezhenliang, sgshan, (ichenJaict wwt.aiiccc..o.om ac. cn
ABSTRACT
Recently, large-scale diffusion models have made impressive progress in text-to- image (T2I) generation. To further equip these T2I models with fine-grained spa- tial control, approaches like ControlNet introduce an extra network that learns follow condition image. However, for every single condition type, Con- trolNet requires independent training on millions of data pairs with hundreds of GPU hours, which is quite expensive and makes it challenging for ordinary users to explore and develop new types of conditions. To address this problem, we propose the CirLoRA framework, which trains Base ControlNet to learn the common knowledge of image-to-image generation from multiple base con- ditions, along with condition-specife LoRAs capture distinct characteristics of each condition. Utilizing our pretrained Base ControlNet, users can easily adapt it to new conditions, requiring as few as 1,000 data pairs and less than one hour of single-GPU training to obtain satisfactory results in most scenarios. Moreover, our CtrLoRA reduces the learnable parameters by 90% compared to ControlNet, significantly lowering the threshold to distribute and deploy the model weights. Extensive experiments on various types of conditions demonstrate the efficiency and effectiveness of method. Codes and model weights will be released pectps://qithee com/ www.As////t.ooo........
light
Segmertatoo
Pixel
Oii Paintina
Caisos Ink
Pescil
Figure 1: Our results of single- condil-co generation, multi-conditional generation, style transfer.
INTRODUCTION
In recent years, diffusion models (Sohl-Dickstein al., 2015 Song & Ermon 2019 Ho et al.
2020) have become one of the most popular generative models for visual generation and editing tasks. The superior performance and scalability of diffusion models encourage researchers train large ones on billions of text-image pairs (Schuhmann et al. [022], resulting in powerful and influ-
ential text-to-image (T21) base models (Rombach et al. 2022 Saharia al. 2022 Ramesh al.

---

Published as conference paper at ICLR 2025
2022 Betker al. 2023 Chen al. 2024 Xue al. 2024). Further, by combining these base models with parameter-effect efficient hne-tuning (PEFT) methods such as LoRA (Hu et al. 2022 Ryu 2022). + users can obtain personalized models without the need for large amount of training data and computational resources, which significantly lowers the threshold for extending a T21 base model the creation of various kinds of art. Leveraging the "Base + PEFT" paradigm, especially "Sta- ble Diffusion + LoRA", numerous individuals from both technical and non-technical backgrounds including artists, have embraced this methodology for artistic creation, forming a huge community and technological ecosystem.
However, is challenging for T2I models to accurately control spatial details such as layout and pose, as text prompts alone are not precise enough to convey these specifics. To solve this problem, ControlNet (Zhang et al. 2023) adds an extra network that accepts condition image, turning a T21 model into an image-to-image (121) model. In this manner, ControlNet is able to generate images according a specific kind of condition image such as canny edge, significantly improving the controllability. However, for each condition type, independent ControlNet needs to be trained from scratch with large amount of data and computational resources. For example, the ControlNet conditioned canny edge is trained on 3 million images for around 600 A100 GPU hours. This substantial budget makes it challenging for ordinary users to create a ControlNet for a novel kind of condition image, hindering the growth of the ControlNet community compared the flourishing Stable Diffusion community]] Therefore, it is worth figuring out a simple and economical solution extend the promising ControlNet handle novel kinds of condition images.
To address this problem, inspired by the "Base + PEFT" paradigm, we propose a CtrLoRA frame- work that allows users conveniently and efficiently establish ControlNet for customized type of condition image. As illustrated in Fig. 2 a), we first train a Base ControlNet on a large-scale
septh-to-image and dataset across multiple base condition-to-ine -o-mmnng tasks such as canny-to-image conditions. Mcan- skeleton-to- skeleton-to-image, where the network parameters are shared by all these base while, for each base condition, we add condition-speciftt LoRA the Base ControlNet. this manner, the condition-specifie LoRAs capture the unique characteristics of the corresponding condi- tions, allowing the Base ControlNet to focus on learning the common knowledge of image-to-imant (12I) generation from multiple conditions simultancously. Therefore, well-trained Base Control- Net with general I2I ability can be easily extended to any novel condition by training new LoRA layers, shown in Fig. (b). With our framework, in most scenarios, type of condition with as few as 1,000 training data and less than one hour of training on a single GPU. Moreover, our method requires only 37 million LoRA parameters per new condition, sig- nificant reduction compared to the 361 million parameters required by the original ControlNet for each condition. In a word, our method substantially lowers the resource requirements compared to the original ControlNet, as detailed in Table
learn customized
lorge scale data
much less data, fewer devices, shorter time
Canny
Depth
Skeleton
Linearl
Palette
De-raindrop
fixed Base
CN
CN
CN
CN
CN
CN
CN
(a) Train Basc on basc conditions
(b) Train LoRA on novel conditions
Figure 2: Overview of the CtrLoRA framework. "CN" denotes Base ControlNet, "L"" denotes LoRA.
(a) We first train a shared Base ControlNet in conjunction with condition-speeiff LoR As on large- scale dataset that contains multiple base conditions. (b) The trained Base ControlNet can be easily adapted to novel conditions with significantly less data, fewer devices, and shorter time.
As of September 24, 2024, on civitai.com, one of the most popular repositories for Al models, there 1024 models tagged with Stable Diffusion whereas only 56 tagged with ControlNet.
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of model size, dataset size, and training time cost. For N conditions, the total number of parameters is 361M x N for ControlNet and 360M + 37M x N for our CtrLoRA.
Method
Condition
# Params.
Dataset Size
GPU Hours
Canny
361M
3M (Internet)
600 (A (A100)
Depth
361M
3M (Internet)
500 (A (A100)
ControlNet
Skeleton
361M
200K (Openpose)
300 (A100)
CtrLoRA (Base)
9 base conditions
360M
20M (MultiGen)
6000 (4090)
Lineart
37M
1K (Custom)
0.17 (4090)
CirLoRA (LoRA) for novel conditions
Palette
37M
1K (Custom)
0.83 (4090)
De-raindrop
37M
863 (Raindrop)
0.83 (4090)
Our contributions are summarized below:
1. We propose effective and efficient framework for extensible image-to-image genera- tion. This framework utilizes a shared Base ControlNet to learn the common knowledge of image-to-image generation, while employing condition-specific LoRAs to capture unique characteristics of each image-to-image task.
2. Our Base ControlNet can be easily and economically adapted to novel conditions by train- ing new LoRA layers, which requires significantly fewer resources compared to the original ControlNet, including reduced training data, shortened training time, and decreased model size. As result, our method considerably lowers the barrier for ordinary users create customized ControlNet.
3. Without extra training. our Base ControlNet and LoRAs can seamlessly integrated into various Stable Diffusion based models from the public community. Moreover, the LoRAs trained for different conditions can be easily combined for finer and more complex control.
4. We optimize the design and initialization strategy of the condition embedding network, which significantly accelerates the training convergence. Furthermore, in this way, we do not observe the phenomenon of sudden convergence that appears in the original ControlNet.
RELATED WORK
Diffusion models. Diffusion models, originally introduced by Sohl-Dickstein al. 2015) and substantially developed by (Song & Ermon 2019); Ho et al. (202)); Song et al. (2021b) Dhariwal & Nichol (202T): Ho Salımans (2022); Bao al. (2023): ; Peebles Xie (2023). ). etc., have gained widespread popularity as type of generative model. To further enhance the expressiveness of diffusion models, researchers proposed to model the diffusion process in the latent space (Vahdat et al. | Rombach et al.) 2022) of a variational autoencoder (Kingma & Welling. 2013). which enables high-resolution image generation. The proposed CtrLoRA in this paper is built upon Stable Diffusion (Rombach et al. 2022), widely used latent diffusion model.
Conditional generation. To advance text-to-image (T2I) generation, researchers incorporate the text embedding from CLIP (Radford et al. 2021) T5 (Raffel et al. 2020) into diffusion mod- els, leading to powerful large-scale T2I models (Nichol et al. [022]] Ramesh et al.) 2022 Rombach al. 2022 Balaji et 2022; Saharia al. 2022). To facilitate more fine- une-ggained control, several methods (Li et al. 2023 Zhang et al. 2023) inject spatial conditions into the model, significantly
enhancing the controllability. For example, ControlNet (Zhang et al., (023) introduces an auxil-
iary network to process the condition images and integrates this network into the Stable Diffusion model. However, training a ControlNet for each single condition requires large amounts of data and time, creating considerable burden. To address this problem, T2I-Adapter (Mou et al. 2024). SCEdit (Jiang et al. 2024). ) and ControlNet-XS (Zavadski et al. 2023) design efficient network ar- chitectures reduce the model size, but training these models still requires large-scale datasets and devices. ControlLoRA (Hecong) 2023) directly trains LoRAs with conditional input on Stable Dif- fusion, but suffers from suboptimal performance with limited data. X-Adapter (Ran al. 2024)
3

---

Published as a conference paper at ICLR 2025
and Ctrl-Adapter (Lin et al. 2024 leverage pretrained ControlNets and efficiently adapt them to up- graded backbones. UniControl (Qin et al. 2024) and Uni- Controntrol (Zhao et al. 2024) train a uni- fied model to manage multiple conditions, significantly reducing the number of models. However, these two methods lack a straightforward and convenient manner for users to add new conditions, which limits their practicality in real-world scenarios. In contrast, our method can efficiently learn new conditions with significantly less data and fewer resources.
Low-Rank Adaptation. Low-Rank Adaptation (LoRA) is well-known technique for parameter- efficient fine-tuning of large language models (Hu et al. 2022) and image generation models (Ryu) 2022]. This method follows the assumption that the updates to the model weights have a low "intrin- sic rank" during fine-tuning, which can be represented by low-rank decomposition AW = BA, where B E Rd A E grxd and r < d. In practice, LoRA significantly reduces the number of optimizable parameters while maintaining decent performance. Although LoRA is widely used in conjunction with Stable Diffusion (Rombach et al., 2022) for customized image generation, rarely used with another prominent image generation technique, ControlNet (Zhang al.) (202)]. We think the main reason is that the ControlNet is trained separately for different conditions, making it unsuitable serve foundation model that can be shared across various conditions. In this pa- per, we investigate a novel method to train a Base ControlNet as a foundation model in cooperation with the LoRA technique.
METHOD
In this section, we introduce the design and training strategy of our CtrLoRA framework for tensible image-to-image (12I) generation. In Section 3.1 we present the fundamental formulations and clarify the associated notations. Section 3.2. propose Base ControlNet that serves foundation model for various 12I generation tasks. In Section 3.3 we illustrate how to efficiently
adapt our Base ControlNet to new conditions with LoRAs. In Section 3.4 we explain our design of the condition embedding network to accelerate the training convergence.
3.1 PRELIMINARIES
In diffusion models (Ho et al. 2020 Rombach et al. 2022), each data sample xo is diffused into Gaussian noise through Markov process, while generative model is trained to reverse this process with the following loss function:
[|Ie Ea (x:)))
( 1a
where x, / 0o + v1 crE denotes the noised sample at step I, and eg is a neural network designed to predict the diffusion noise e.
For conditional generation, the loss function can be modified as follows (Ho & Salimans 2022
()) = E (xxx.)) (U.
[|c ca (x,, e)]
where e denotes the conditional signal such as text for text-to-image generation and image for image- to-image (12I) generation Specifically, ControlNet (Zhang et al. (023)) for 12I generation designs the noise prediction network (x c) as follows:
6p (Xe, c) D (E(x:), (a Cx(x..
where & and D denote the encoder and decoder of the UNet pretrained Stable Diffusion (Rombach et al. 2022), Co denotes the ControlNet, and Fo denotes the condition embedding network.
In the original ControlNet, Cp Eq. 3 is independently trained for each type of condition image and cannot be shared across different conditions, which leads to a huge demand for training data and computational resources. the following sections, we introduce our CtrLoRA framework that trains Co as shared and extensible Base ControlNet, and explain how to efficiently extend it to various new conditions with much less data and fewer devices.
In the following, we focus on 12I generation, and the text condition is assumed to be the default. Therefore, simplify the notation, we omit the condition and use to represent the image condition.
4

---

Published as a conference paper at ICLR 2025
3.2 BASE CONTROLNET FOR EXTENSIBLE I2I GENERATION
A generalizable model for various image-to-image (12l) generation tasks necessitates comprehen- sive understanding of 121 generation. To this end, propose train shared Base ControlNet across multiple types of condition images simultaneously. in order to acquire common knowledge of diverse I2I tasks. Meanwhile, ent conditions, we propose to add condition-spectte LoRA layers to every linear layer of the Base Net In this manner, different condition specific LoRAs are responsible for capturing unique characteristics of corresponding tasks, and therefore the shared Base ControlNet can concentrate on generation. The whole schema shown Fig. Bia).
prevent the Base ControlNet from being confused by differ-
the common knowledge
Specifically, suppose we have distinct types of base conditions {c(*) fk== with corresponding data subsets {9(*) 1k=1 and let Cg denote the Base ControlNet and Lyo: denote the condition -specifcc
LoRA responsible for the 1thh condition. this context, we propose to adapt the noise prediction network from Eq. as follows:
E0, (X,, (k)) D(E(()), (V.,sc(
(4)
where Co. = Co @ Lypc refers to the Base ControlNet Co equipped with the htth LoRA L200,, and we use VAE(c(A)) as the condition embedding network to achieve faster training convergence (explained in Section 3.4). To optimize the Base ControlNet simultaneously on K base conditions,
adapt Eq. 2 to the following loss function:
L(0, ((0,))
Σ R (t) [EL-U
K
|| - - 60,0000) (x,, c(t))
[|]]]]]]]]
(5)
k=1
In practice, only single condition is selected in each training batch and different conditions are iterated batch-wise, therefore all conditions can be optimized with an equal number of training iterations. For each batch, the LoRA layers corresponding to the current condition are switched on and updated, as shown Fig. aa).
To ensure the effectiveness and generalizability of the Base ControlNet, the training process is ducted on 9 base conditions with millions of data (Qin et al.] 2024) and takes around 6000 GPU hours. Although resource-consuming. this process paves the way for efficient adaptation to novel conditions as demonstrated Section
3
Input
HED
Depth
Skeleton
Segmentation
Lineart
Palette
Grnyscale
batch #1
batch
batch
batch HK
SD VAE
VAE
SD VAE
49
CN
CN
condition image
condition
(b) Training of new LoRAs for conditions
SD VAE *
SD VAE #
HED
Palette
Oulpul
CN
SD
81
#2
83
#K
SD VAE
SD VAE
CN
CN
SD
Output
(u) Training of the Base ControlNet on base conditions
(c) Inference for multi-conditional generation
Figure 3: Training and inference of CırLoRA framework. "SD" denotes Stable Diffusion, "CN" denotes Base ControlNet, and "L"s in different colors denote LoRAs for different conditions.
5

---

Published as a conference paper at ICLR 2025
3.3 EFFICIENT ADAPTATION TO NOVEL CONDITIONS
Since the well-trained Base ControlNet learns sufficient general knowledge of 121 generation, it can be efficiently adapted new conditions via parameter-effiient fine-tuning. Similar to the LoRAs for the base conditions in Section 3.2] we can also train a new LoRA for any new condition while freezing the Base ControlNet, as shown in Fig. 3b)). As result, there are only 37 million optimiz- able parameters when setting the LoRA rank as T28, substantial reduction compared to 360 million parameters for full- full-paramin para-meteeeer fine-tuning. Moreover, most scenarios, as few as 1,000 data pairs and less than one hour of training on a single RTX 4090 GPU are sufficient for satisfactory results.
In addition, the LoRAs trained for different conditions can composed for multi-conditional gen- eration. Specifically, we can generate images that satisfy multiple conditions by summing up the outputs of the Base ControlNet equipped with corresponding LoRAs, as shown in Fig. 3(c..
3.4 DESIGN OF CONDITION EMBEDDING NETWORK
In the original ControlNet (Zhang al. 2023). simple convolutional network with random ini- tialization is employed to map the condition image into embedding, which is referred to as the condition embedding network. However, randomly initialized network cannot extract any useful information from the condition image at the beginning of training and thus causes slow convergence.
To solve this problem, instead of a randomly initialized network, we propose to employ the pre- trained VAE of Stable Diffusion (Rombach et al. 2022) as the condition embedding network, as shown in Fig. 3 and Eq. 4). For one thing, since the pretrained VAE has been proven to be power- ful to represent and reconstruct an image (Rombach et al. 2022), it can already extract meaningful embedding from the condition image without extra learning. For another, since the Base Con- trolNet is initialized as trainable copy of the Stable Diffusion encoder, the embedding space of the pretrained VAE seamlessly matches the initial input space of the Base ControlNet. In a word, compared randomly initialized network, using the pretrained VAE as the condition embedding network requires no extra effort to learn a suitable embedding space and therefore achieves much faster convergence. Besides, utilizing this method, the sudden convergence phenomenon associated with the original ControlNet is not observed anymore.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets To train the Base ControlNet, we employ a large-scale dataset, MultiGen-20M (Qin et al.) (0024), which contains over 20 million image-condition pairs across 9 image-to-image tasks. To train the LoRAs for new conditions, we create multiple types of image-condition pairs based on COC02017 (Lin al. 2014) training set. For all quantitative evaluations, we employ COC02017 validation set. Additionally, we use WazeWorld dataset (Xu et al., (023) for dehazing task, Raindrop dataset QQan al. 2018 for de-raindrop, the dataset from Yang al. (2020) for low-light image enhancement, and Danbooru2019 dataset (Branwen et al. 2019 for anime generation.
Evaluation metrics We use LPIPS (Zhang et al. 2018) to measure the faithfulness of the gener- ated images to the condition images in two scenarios. For conditions including Canny, HED, Sketch, Depth, Normal, Segmentation, Skeleton, Lineart, and Densepose, the target is to generate images that match the condition images. Thus re-extract the conditions from the generated images and compare them with the real condition images. For conditions including Outpainting, Inpainting, and Dehazing, the target generate high-fidelity images from degraded images. Therefore, we com- pare the generated images with the ground-truth images. Besides, we use FID score (Heusel et al. 2017 to evaluate the image quality.
Implementation details For fair comparison with other methods, we use Stable Diffusion v1.5 in all experiments. We set the LoRA rank to 128 for each base task when training the Base Con- trolNet. The Base ControlNet is trained with AdamW optimizer (Loshchilov & Hutter 2017) for 700k steps with a learning rate of 1x10-1 and a batch size of 32, which takes around 6000 GPU hours on 8 RTX4090 GPUs. For all new conditions, the LoRA rank is set 128. Besides,

---

Published as a conference paper at ICLR 2025
also fine-tune the normalization layers and the zero-convolutions. We AdamW optimizer with a learning rate of 1x10- 15 and a batch size of At this stage, only one GPU is needed, which is much more affordable than the requirements for training the original ControlNet. For sampling, apply DDIM (Song et al.) 2021a) sampler with 50 steps. The weight of classifier-free guidance (Ho) & Salimans 2022)
7.5 and the strength of ControlNet is set 1.0. We do not any additional prompts or negative prompts for the quantitative evaluation.
4.2 COMPARISON WITH EXISTING METHODS
Performance on base conditions To demonstrate the capacity of the Base ControlNet, we evalu- ate its performance on the base conditions, as shown in Table and Fig. 4 We compared the results UniControl (Qin et al.] 2024). state-of-the-ar method that trains unified model to manage all base conditions, similar to our Base ControlNet. As can be seen, for base conditions, our base ControlNet performs on par with the state-of-the-arr UniControl, demonstrating its robust funda- mental capabilities. Furthermore, our base ControlNet can be easily and efficiently extended to new conditions, which is not straightforward using UniControl.
Table 2: Quantitative comparison on base conditions. Each cell represents "LPIPS FIDA".
HED
Shetch
Degth
Nonnal
Segnecmation
Skelase
Ouppainting
Rounding
UniContel
6.176/
21
0.319
22 02
0.527
0.292
CarLoRA (ours)
14.25
19.17
21.13
Coodition
UuiCommol
ChLoRA (ours)
Conditiu
Unioomol
ChiLoRA
Condition
UnxCootrol
CaLoRA
Conduice
UniCoomo
ChLAARA
FRN
Nexed
Oupariin
Figure 4: Visual comparison on base conditions.
Table 3: Quantitative comparison
conditions. Each cell represents "LPIPS /FIPS.
Lincart
Densepose
Inpainting
Dehazing
training images
1k images
100% images
Ik images
100% images
images
100k images
Ik images
100k images
ControlNer
0.622 / 22 29
0.264 / 14,10
0.367 36 80
D. D.140 /3.36
0.785 / 09
0.465 12.79
0.75% / 54.07
0.348 22.85
ControlNei-LITV
0.623 / 23.00
0.267 15.24
0.368 / 36.70
0.152 / 34.51
0.785 / 88
0.530/ / 14.86
0.761 /6042
0.409/ 27.54
ControlNet-XS
0.623 /
0.245/ 15.33
0,368 36.46
0.148 /3.50
0.784
/2
0.503 13.37
0,761
59
0.397 25.83
CtrLoRA (ours)
0.305 / 16.12
0,247 13.47
0.159 / 35.18
0,126 / 32,80
0.326 9.972
0.246 / 8.214
0,255 / 15.44
0.178 10.55
Condition
ControlNer
ContoNNas-LITE CostoNNet-XS CurLaRA (ours)
ControlNet
ControlNat-LITE ControlNet-S8 Cnt (ours)
Deracpone
Iepaining
Ik training images
100k training images
Figure 5: Visual comparison on new conditions.
7

---

Published as a conference paper at ICLR 2025
Adaptation to conditions For new conditions, we compare method with Control- Net (Zhang et al. 2023). ControlNet-LITE (Zhang et al. 2023), , and ControlNet-XS (Zavadski et al. lightweight alternatives ControlNet, aiming to optimize the network chitecture and accelerate the training process. To evaluate the data efficiency and scalability, we 1 and 100k training images respectively, as shown in Table and Fig. [5] With a limited training set (IK), CtrLoRA consistently outperforms the competitors by large mar- gin, highlighting its superiority in quickly adapting to new conditions. With a large training set (100k), CtrLoRA achieves better or comparable results. In summary, regarding the adaptation to new conditions, our CtrLoRA is not only highly data-efficient but can also achieve satisfactory per- formance as the data scale increases.
2023). The latter
conduct experiments
Convergence rate We visualize the results with respect to training steps and plot the convergence curve in Fig. [ As can be seen, CtrLoRA starts to follow the condition after just 500 training steps, while the other methods take more than 10,000 steps to reach convergence.
ConniNN
Inpeiming
LeNeS
OztakA irand ComalNaasS
Training
100
5000
7,000
10,000
20,0000
StUIOn
100,000
Traning
Figure 6: Visual comparison of convergence rate.
4.3 ABLATION STUDY
Effect of each component We validate the effect of our components by starting with the original ControlNet and adding our proposed components one by one, resulting in three incremental settings (A-C) in Table 4 In setting (A) of Table 4 and Fig. 7 we evaluate the effect of using the pretrained VAE as the condition embedding network, as proposed in Section 3.4, As can be seen, using the pretrained VAE improves both LPIPS and FID score as well as speeding up the training conver- gence. In setting (B), we further switch the initialization of ControlNet from Stable Diffusion to well-trained Base ControlNet and perform full full-parameter fine-tuning. in order to validate the gener- alizability of our Base ControlNet. As shown, new conditions with a limited training set (Ik images) and still achieves leading performance with large training set (100k images). This result demonstrates that our Base ControlNet learns sufficient general knowledge of 12I generation and indeed helps the adaptation to novel conditions. At last in setting (C), replace the nall-parmmeter training with condition specific LoRAs, which represents the complete implementation of our method. As shown, although the LoRAs reduce the optimizable parameters by 90%, it does not lose much performance and maintains the second-best performance in most situations, demonstrating the effectiveness and efficiency of our CtrLoRA framework.
Base ControlNet can be more quickly adapted to
Table 4: Effect of the proposed components. Each cell represents "LPIIS FIDL".
Lineart
Densepase
Impainiing
Dehazing
triniing images
Ik images
100k images
Ik images
100k images
1k images
100% images
1k images
100% images
(O) ControiNer
0 / 23 20
B.264
14.10
0.367/ 36.80
0.140/ $3.36
0.885 22.09
0.465 / 12.79
0417
0.348 / 22.85
(A) + Pretrained VAE
0. 393 2000
0 13.85
0,17/ 36
/ 33
0.489/ 19.05
0.257/8.94
0.350/27.07
0.180/ 11.07
(8)
Bose Conteel
0,300/ 14.25
0,228
12.71
0 34.02
/
0.296 9.450
8,251 / 3.266
0.221 / 13.47
0.169 10.08
LoRA (full CuLoRA)
0,205 16.12
0,247 13.67
0.159 35.18
0.136 /2.80
0.326 0.526/9.722
0.246 8.214
0.255 15.48
0.178 10.55
The best and seoud wemitt hightigghee holdfaace snderlieed respertivoly.
8

---

Published as a conference paper at ICLR 2025
Sclin(A)
Training strp
200
500
1,000
7,000
3,000
4,000
10,000
20,900
40,000
$0,000
100/000
Figure 7: Convergence rate comparison between ControlNet and setting (A).
Effeet of LoRA rank We evaluate the CtrLoRA performance with LoRA ranks of 32, 64, 128. and 256, and we also evaluate the fall-parameter training as the upper bound. As shown in Fig, g LPIPS improves as the rank increases, while FID score plateaus the rank of 64. To balance the performance and number of optimizable parameters, we choose a rank of 128 for all conditions.
Effeet of training set size We train our CtrLoRA respectively on datasets containing 1k, 3k, 5k, 10k, and 50k images, running 5 epochs for each dataset size. As shown in Fig. 9 both LPIPS and FID improve as the dataset size increases. Nevertheless, our practice for most new conditions, a small amount of training data (Ik images) is generally sufficient for satisfactory visual perception.
rank
128
# params.
17.9M
24.2M
36.9M
62.4M
360.2M
Figure 8: Effect of LoRA rank.
Dataset size
1,000
3,000
5,000
10,000
50,000
Figure 9: Effect of training set size.
4.4 OTHER EXPERIMENTS
More novel conditions We provide visual results of more novel conditions including Palette, Lin- eart with color prompt, Pixel, De-raindrop. Low-light image enhancement, and Illusion in Fig. TO Despite the significant differences among these conditions, method yields decent results across all of them, which demonstrates the generalizability of our CtrLoRA to wide range of conditions.
Integration into community models Our CtrLoRA can be directly applied to the community models based on Stable Diffusion 1.5. In Fig. aa), we integrate our CirLoRA into four community models with markedly distinct styles. The results exhibit different styles but remain consistent with the given conditions, suggesting that our method can be flexibly used as plug-and-play module without extra training.
Combine multiple conditions By equipping the Base ControlNet with different LoRAs and sum- ming their outputs, we can perform nulti-conditional generation without extra training. The weight assigned to each condition be manually adjusted to control its effect on the final result, with an equal weight of typically sufficient in most scenarios. As shown in Fig, I[(b,, our CtrLoRA can generate visually appealing images that comply with both conditions simultaneously.

---

Published as a conference paper at ICLR 2025
Palette
Lineart with color prompt
Pixel
Condition
Outpat
De-raindrop
Low-light image enhancement
Illusion
Condition
Sch
CtrLORA
Output
ScI
Figure 10: Visual results of our CtrLoRA for various novel conditions.
Condition
SDI.5
Realistic Vision
Mistoon Anime
Painting
Condition
Condition
Output
+
(+) Integration into community models w/o extra training
(b) Mulii-onndiional generation w/o extra training
Figure 11: Without extra training, well-trained CirLoRA be directly integrated into various community models and combined for multi-conditional generation.
5 CONCLUSION AND LIMITATIONS
In this paper, we propose CtrLoRA, framework aimed at developing a controllable generation model for any condition with minimal data and resources. In this framework, first train Base ControlNet along with condition-specific LoRAs to capture the common knowledge of image- to-image generation, and then adapt new conditions by training new LoRAs. Compared to ControlNet, our approach significantly reduces the requirement for data and computational resources and greatly accelerates training convergence. Furthermore, the trained models can be seamlessly integrated into community models and combined for multi multi-conditionall generation without further training. By lowering the development threshold, we hope our research will encourage more people to join the community and facilitate the development of controllable image generation.
We empirically found that color-related conditions, such Palette and Lineart with color prompts, tend to converge more slowly than conditions involving only spatial relationships. This phenomenon seems to be a common issue that not only appears in our methods but also other ControlNet- based competitors. We speculate this issue might originate from the capabilities of the network architectures, specifically the architectures of VAE, UNet-based Stable Diffusion, and ControlNet. To enhance the capabilities of our framework, it is worth developing CtrLoRA using more advanced bas-d (Peebles & Xie 2023) backbones such as Stable Diffusion V3 (Esser et al. 2024) and Flux. 1, which leave for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partially supported by the National Natural Science Foundation of China (No. 62461 160331 and 62406311), the Postdoctoral Fellowship Program of CPSF (No. GZB20230774), and the Innovation Funding of ICT, CAS (No. E361020).
REFERENCES
Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, al. ediff-i: Text-to-inee tomee diffusion models with an ensemble of expert denoisers. arXiv preprint M00Xiv:2211.013222 2022.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22669922679, 2023.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-. pdf. 2(3):8, 2023.
Gwern Branwen, Anonymous, and Danbooru Community. Danbooru2019 portraits: A large-scale anime head illustration dataset. https://gwwm.nnt//rrre
Gaibomu2011--pprraatts March 2019. URL ptttp:://wwen..et/cccc
Bar.boouu2119-potrraiiz
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 1188780-8794 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
Wu Hecong. ControlLoRA: A Lightweight Neural Network To Control Stable Diffusion Spatial Information, 2023. URL http::ttt///c httt:::///ithuse com/ Httm//iiicccc/Cottoooooooo
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. vrXiv:2207.12599 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, and Jingfeng Zhang. Scedit: Efficient and
trollable image diffusion generation via skip connection editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8995-9004, 2024.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding
variational bayes
arXiv preprint
11

---

Published as a conference paper at ICLR 2025
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Han Lin, Jaemin Cho, Abhay Zala, and Mohit Bansal. Cirl-adapter: An efficient and versatile framework for adapting diverse controls to any diffusion model. arXiv preprint arXiv:2404.0990) 099677, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference. Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.0510,, 2017.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4296 4304, 2024.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16804. PMLR, 17-23 Jul 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2482-2491, 2018.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for con- trollable visual generation the wild. Advances in Neural Information Processing Systems, 36, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 67, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Lingmin Ran, Xiaodong Cun, Jia- Jia-Wei Liu, Rui Zhao, Song Zijie, Xintao Wang, Jussi Keppo, and Mike Zheng Shou. X-adapter: Adding universal compatibility of plugins for upgraded dif- fusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8775-8784, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Simo Ryu. Low-rank adaptation for fast -inttt---mmge diffusion fine-tuning, 2022. URL https: //gi wttph....ooccccooee...eeeoe moo 1ora
12

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kam- yar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photo- realistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-3644 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 25:25278-25294 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems. 32, 2019.
Yang Song. Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021b.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in neural information processing systems. 34:1 1287-11213,,, 2021.
Jiaqi Xu, Xiaowei Hu, Lei Zhu, Qi Dou. Jifeng Dai, Yu Qiao, and Pheng-Am Heng. Video de- hazing via a multi-range temporal alignment network with physical prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information Processing Systems, 36, 2024.
Wenhan Yang, Shiqi Wang, Yuming Fang. Yue Wang, and Jiaying Liu. From fidelity to perceptual quality: A semi- semi-supervised approach for low-light image enhancement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Denis Zavadski. Johann- Friedrich Feiden, and Carsten Rother. Controlnet-xs: Designing an ficient and effective architecture for controlling text-to-image diffusion models. arXiv preprint arXiv:2312.06573, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3836-3847, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A SUDDEN CONVERGENCE PHENOMENON
Sudden convergence an intriguing phenomenon observed in the original ControlNet (Zhang al. 2023), where the generated images suddenly converge to match the condition image in only 33 training steps (6100 to 6133). The original paper attributes this phenomenon to the use of zero- convolutions. However, we found that the condition embedding network, originally implemented as a randomly initialized convolutional network, also contributes to this phenomenon. As elaborated in Section 3.4 we propose using the pretrained VAE as the condition embedding network to accelerate the training convergence, the effect of which is shown in Table and Fig. (Setting (A)) of the main paper. Furthermore, we found that our design also alleviates the sudden convergence phenomenon. Specifically, in Fig. T2 we display the generated images every 25 training steps from 2,000 to 3,000, given the same condition. As can be seen, the results oscillate between compliance and non- compliance with the given condition, which corresponds to an oscillation near the local minimum rather than sudden convergence. This result implies that improper design of the condition embedding network is indeed one cause of the sudden convergence phenomenon, and using the pretrained VAE in our paper is effective solution.
2150
2225
2250
22.75
2350
2395
2400
2425
2450
2475
2500
2550
2600
2625
2650
2675
2700
2225
2750
2775
2500
2825
28.50
2875
2900
2925
2950
2315
2000
Figure 12: Generated images between 2,000 and 3,000 training steps.
B ADDITIONAL DISCUSSION ON RELATED EFFICIENT METHODS
T21-Adapter (Mou et al. 2024 and SCEdit (Jiang al. 2024 are two efficient alternatives of ControlNet that mainly focus on decreasing the model size. However, the data and GPU resources needed to train these models are still beyond the reach ordinary users. For example, T21-Adapter is trained on 164k to 600k images with 4 V100 GPUs for around 3 days, and SCEdit is trained on 600k images with 16 A GPUs. On the contrary, our method can achieve satisfactory performance by training about 1,000 images with single RTX 4090 GPU within I hour, while keeping the model size comparable or even smaller. thereby significantly lowering the cost for ordinary users to create their customized NentrolNets.
14

---

Published as a conference paper at ICLR 2025
C MORE QUANTITATIVE RESULTS
Controllable generation benchmark To establish a comprehensive benchmark on controllable
image-to-inas -mage generation, compare method with several representative community models, including official models of ControlNet (Zhang et al. (2023) and T2I-Adapter (Mou et al. (2024). As shown in Table and Table T our CtrLoRA outperforms fully trained ControlNet and T2I-Adapter
for both base and new conditions. The visual comparisons are shown in Fig. 17 and Fig. 18
Table 5: Benchmark for base conditions. Each cell represents "LPIPS FID.".
Canny
Depth
Segmentation
Skeleton
ControlNet
0,438 / 17.80
0,232 20.09
0,488 / 20.83
0,134 50.79
T21-Adapter
0.447 / 18.45
0.305 23.81
0.636 / 21.59
0.137 52.92
UniControl
0.273 / 18.58
0.216 21.29
0,467 / 22.02
0.129 53.64
CirLoRA (ours)
0.388 / 16.65
0.222 19.34
0.465 / 21.13
0.132 51.40
Table 6: Benchmark for novel conditions. Each cell represents "LPIPS| / FIDL".
Lineart
Densepose
Inpainting
Dehazing
ControlNet 0.254 / 15.04
0.140/ / 33.36
0.465 12.79
0.348 / 22.85
T21-Adapter
0.498 / 20.53
CuLLRA (ours))
0.247 / 13.47
0.126 32.80
0.246 / 8.214
0.178/ / 10.55
ControlNet for Densepose, Inpainting, and Dehazing are trained by ourselves on 100k images. Our CHrLoRAs trained 100k images for each novel condition.
Necessity of our Base ControlNet To demonstrate the necessity and adaptability of our Base Con- trolNet, we compare our method to directly fine-tuning& pretrained ControlNet and UniControl et al. (2024). We also compare with ControlLoRA (Hecong) (023) that directly trains LoRAs with conditional input on Stable Diffusion. We limit the training set to 1,000 images to assess the effec- tiveness of each method in quickly adapting to new conditions. As shown in Table 7 our method significantly outperforms these methods in adapting to new conditions, showing the effectiveness of our Base ControlNet and the potential of our idea to learn the general knowledge of 12I generation.
Table 7: Fine-tuning with limited data (1,000 images). Each cell represents "LPIPS FIID
Lineart
Densepose
Inpainting
Dehazing
ControlNet (canny) + LoRA
0.356 / 16.74
0.198 / 36.11 14
0.602 17.63
0.618 / 51.55
UniControl + LoRA
0.316 / 17.05
0.164 / 41.20
0,558 15.84
0.508 37.83
ConArolLoRA CtrLoRA (ours)
0.362 / 17.28
0.295 32.37
0.614 21.92
0.472 41.96
0.305 / 16.12
0.159 / 35.18
0.326 9.972
0.255 15.44
Effeet of the number of base conditions To investigate how the number base conditions affects the adaptability of our Base ControlNet to learn new conditions, train three Base ControlNets on 3, 6, and 9 base conditions respectively and fine-tune them to new conditions. As shown in Table the overall performance on conditions gets better when more base conditions are included to train the Base ControlNet, demonstrating that the Base ControlNet can extract better general knowledge from more conditions.
Table 8: Effect of the number of base conditions, Each cell represents "LPIPS| FIDI".
# Base conditions
Lineart
Densepose
Inpainting
Dehazing
0.348 / 15.71
0.161 / 35.63
0.461 14.63
0.312 / 23.16
6 9
0.324 / 15.59
0.159 / 35.25
0.343 10.73
0.262 / 17.14
0.307 15.06
0.157 35.31
0.337 10.84
0.248 16.23
3 buse conditions include Canny, Depth, Skeleton
6 base conditions inciude Canny, Depth, Skeleton, Segmentation, Bounding Bax, Ouspainting base conditions include Canny, Depth, Skeleton, Segmontation, Bounding Bax, Ouspainting, HED, Sketch, Normal
15

---

Published as a conference paper at ICLR 2025
D MORE VISUAL RESULTS
In this part, we provide more visual results our CtrLoRA, including Fig. 14 for base condi- tions, Fig. [5] for new conditions, Fig. [6 for multi-conditional generation, Fig. 17 and Fig. [8] for visual comparison with officially released models of ControlNet (Zhang et al. 2023 and T2I- Adapter (Mou 2024).
CTRLORA FOR STYLE TRANSFER
As demonstrated in Section 4.4 our CtrLoRA can be flexibly integrated into stylized community models and perform conlitiandiional generation without extra training. These features open the door exploring more creative and exciting applications such as style transfer. Given reference image, we first choose stylized model to specify the target style. Then, we combine the well- trained LoRAs for Palette and Lineart to control the color and shape of the output image, according to the reference. The overall process is shown in Fig. 13 and exemplar results are shown in Fig. 19
Copy
Pixel Style SD
Inpuat
Inkpunk Style SD
Lineart
Lineart
CN
Anime Style SD
Figure 13: The overall process style transfer.
16

---

Published as a conference paper at ICLR 2025
Photo of a squirrel.
big tree beside lake. best quality...
Canny
dog, quality... ...
forest, quality...
HED
Various kinds of fruitss.
Photo of bird, best quality...
Sketch
A modem city skyline.
dog sirting bench, best quality...
Dopdi
glass of coffee, RAW photo...
Beauriful sccnery, cliffs, RAW photo...
Normal
An ailpaiating, best quality...
Photo mountains and trees, RAW photo...
Segmentation
A fushion model, besi quality...
The stermtreoper barman iron man, best quality...
Ske leton
deg.
An apartment. RAW photo
Outpainting
sitting tablo, best quality...
bedroom, best quality....
Rounding Box
Figure 14: More visual results of base conditions.
17

---

Published as a conference paper at ICLR 2025
A miniature model a car, best quality...
Photo of a beautiful jelly fish under the sea, bost quality... ....
Lineart
girl walking on road.
An anime boy.
An anime girl standing
An anime girl room.
Lineart with prompt
robot walking the quality...
Two shaking hands with cach other, best quality...
Densepose
rocket Rying up to the sky. best quality...
A traditional Chinese building. best quality...
Illusion
Close-up photo of'a menarch buttertly. best quality...
A glass of cocktail,, best quality...
Palette
red panda glasses, best quality...
books, masterpiece, best quality...
Impainting
prompt)
prompt)
swimming pool
prompt)
Low- light
racing car.
A goat on rocks.
photo of the street.
Cars running on the street.
Dehozing
(no prompt)
(no prompt)
(no prompt)
(no prompt)
De-mindroo
(no prompi)
prompt)
(no prompi)
prompi)
Desnow
Figure 15: More visual results of new conditions.
18

---

Published as a conference paper at ICLR 2025
Condition 1
Output 1
Condition 2
Output 2
Output 1+2
flower, best quality...
Photo parrot, best quality...
haan/womn//copll, outdoors, best quality...
Beautiful scenery at sunset, best quality....
An adventurer the cliff,, best quality...
A hot balloon hovering over the city, best quality...
An cagle flying over mountains, best quality...
sitting on floor, best quality... .....
Figure 16: More visual results of multi-condiiional generation.
19

---

Published as a conference paper at ICLR 2025
Condition
ControlNet
T2I-Adapter
UniContral
CtrL.oRA A (ours)
Canny
Gears.
Canny
Three macarons stacked the table.
Depth
binocular.
Depth
Photo several apples on table.
Segmentation
Snowy mountains hikers.
Segmentation
Bedroom seene with bookease, blue comforter window.
Skeleton
man sitting at the top of'a mountain.
Skeleton
woman is skiing.
Figure 17: Visual comparison with fully trained community models base conditions.
20

---

Published as a conference paper at ICLR 2025
Condition
ControlNet
T21-Adapter
UniControl LoRA CtrLoRA (ours)
Lineart
of dog.
Lineart
beautiful actress with dark hair.
Densepose
unreleased
woman is doing yoga.
Densepose
unreleased
man wearing dancing.
Inpainting
unreleased
An alien spaceship.
Inpainting
unreleased
The man's hands are cupped, holding an apple.
Dehazing
unrelcused
Streetview.
Dehazing
unrelcased
Street.
Figure 18: Visual comparison with fully trained community models on new conditions.
21

---

Published as a conference paper at ICLR 2025
Input
Pixel
Inkpunk
Chinese Ink
Oil Painting
Pencil
Comic
Anime
Figure 19: Results of style transfer.
22

---

Published as a conference paper at ICLR 2025
F PROMPTS AND MODELS FOR VISUAL RESULTS
Since this paper mainly focuses on 121 generation, we omit the text prompts in the main paper for simplicity. For a comprehensive presentation, here we provide the prompts and the base models corresponding to the visual results of the main paper, as shown in Table 9
Table 9: Prompts and base models for visual results in the main paper.
Figure
Stable Diffusion
Prompt
SDI.5
A dirty dog sits on the front patio of a home.
SD1.5
A parked motorcycle sitting on a dirty road.
SD1.5
A man riding skis while flying through the air.
SD1.5
A cat observing a computer screen next to laptop and a cordless phone.
Fig 4
SD1.5
The painting is a vase of flowers on a table.
SD1.5
A large white bowl of many green apples. The red, double decker bus driving past other huses. There are bananas around another piece of fruit.
SDI.5
SD1.5
SDI.5
A passenger bus pulling
the side of street, best quality...
SD1.5
A man and women posing next to one another in front of a table, RAW photo... a cat wearing a hrown cowhoy hat, best quality...
Fig S
SDI.5
SDI.5
An ancient Chinese building.
Fig 6
SD1.5
dog
Fig.
SD1.5
Basehall game, RAW photo...
Fig. 8
SD1.5
A girl is eating dessert at the table, picnic, RAW photo...
Fig
SD1.5
girl wearing white dress dancing ballet
SD1.5
a flower, best quality...
SD1.5
close up photo a green scedling breaks out the ground, RAW photo...
SD1.5
mountain and trees winter, best quality... anime girl, best quality an anime girl, outdoors, best quality
Mistoon Anime
Mistoon Anime
Mistoon Anime
anime girl, best quality a cute rabbit
Mistoon Anime
Realistic Vision
city
Mistoon Anime
a girl
Fig. 10
SDI.5
bench.
SDI.5
(no prompt)
SDI.5
(no prompt)
Realistic Vision
streetview, hest quality...
Realistic Vision
strectview, best quality-..
Realistic Vision
buildings, best quality.... forest at night, best quality...
Dreamshaper
Realistic Vision
garden
Realistic Vision
sky with stars, RAW photo...
SD1.5
girl with brown hair necklace wearing a cowl-neck shirt, quality... girl with brown hair and necklace wearing a cowl-neck shirt, best quality...
Realistic Vision
Mistoon Anime
girl with brown hair and necklace wearing a cowl-neck shirt, quality...
Oil Painting
girl with brown and necklace wearing a cowl-neck shirt, quality...
Fig. 11aa)
SDI.5
An old man, best quality... An old man, best quality... An old man, best quality... An old man, best quality...
Realistic Vision
Mistoon Anime Oil Painting
SDI.5
Photo a parrot, best quality a couple, outdoors, best quality... Beautiful scenery at sunset, best quality...
Fig (bbb
Realistic Vision
Realistic Vision
23


Reviews:
[{"review_id": "plGX58BjjM", "reviewer": "Reviewer_UthF", "summary": "The paper proposes CtrLoRA for better controllability of the conditional image generation. This framework trains a Base ControlNet for the general image-to-image generation and then uses the LoRA fine-tuning for specific user instructions. Experiments show the effectiveness of the proposed method.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-organized and easy to follow.\n- The authors conduct sufficient ablation studies to evaluate the proposed modules.\n- The experiments demonstrate the training efficiency of the proposed method and its capability to unify various visual conditions for generation.", "weaknesses": "- The authors train a base ControlNet for the subsequent LoRA fine-tuning. However, why not directly fine-tune a pre-trained ControlNet or Uni-ControlNet?\n\n- Lack of comparison to: ControlNet++[1].\n\n- The paper does not explore whether this method can be generalized to other diffusion models such as SDXL and Pixart.\n\n[1] Li M, Yang T, Kuang H, et al. ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback[C]//European Conference on Computer Vision. Springer, Cham, 2025: 129-147.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bLJcpQYlY3", "reviewer": "Reviewer_GM4E", "summary": "This paper proposes CtrlLoRA, a two-stage parameter-efficient fine-tuning pipeline, to ease the original ControlNet's computation burden in terms of different conditions. The authors evaluate CtrlLoRA through extensive experiments by both the quality and the computation efficiency.", "questions": "1. The results in Figure11b demonstrate that the different conditions are effectively disentangled, with a direct summation module according to Figure 3c. Could you clarify why this module is effective, such as presenting the results of two elements both separately and after sum-up.\n\n2. A detail, why not presenting all 9 base-condition results comparison to UniControl in Table 2?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper focus on an important problem, extending ControlNet to a lightweight manner.\n2. Experimental results are impressive, especially the convergence experiment.", "weaknesses": "1. In line 70, the authors state that ControlNet with Canny edges requires 3 million images over 600 GPU hours for one condition. In contrast, line 244 indicates that Base ControlNet necessitates millions of images for 6000 GPU hours for 9 conditions. Although it is not fair enough, but it implies that the proposed method does not significantly reduce the computational burden.\n\n2. In line 239, the mechanism of training with 9 conditions is not clear enough. As different conditions have different levels of sparse information of input images, why they have equal training iterations? And continuous shifting between different conditions may make the training hard.\n\n3. the motivation why the new conditions are not trained as the Base ControlNet by a shifting mechanism is not clear enough.\n\n4. Most results are from \"Base CN + CtrlLoRA'', and results from \"Community Model + CtrlLoRA\" in Figure 11a are rare, not enough to convince that CtrlLoRA is effective when transferring to other community models.\n\n5. pretrained-VAE seems to be only an interesting trick.\n\n6. putting all the prompts in the appendix makes reading inconvenient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "omndiGxOav", "reviewer": "Reviewer_YjAj", "summary": "This paper draws on the idea of combining a base model with PEFT (Parameter-Efficient Fine-Tuning) for controllable generation. It trains a Base ControlNet obtained through several condition-specific training processes, and then fine-tunes it with a small amount of data for newly introduced conditions to obtain different condition-specific LoRAs. This approach improves the efficiency of training new condition generators at a lower cost.", "questions": "- Regarding the discussion of \"Adaptation to new conditions,\" while training a comparison method from scratch with a small amount of data may indeed result in slow convergence, what would be the results if we used a pre-trained conditional model (analogous to possessing a Base ControlNet) for fine-tuning?\n\n- I'm curious about the performance between a pre-trained controlnet model available in the community and a model trained using proposed \"Base + LoRA\" with same conditions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- To address the high cost of separately training different models for conditional generation tasks, this paper proposes a training method that transitions from a base controlnet model to a lightly fine-tuned lora model. This approach ensures generation quality while achieving a faster convergence rate.\n\n- The paper shows many analyses of the proposed method and presents the results generated for a total of more than a dozen conditions.\n\n- The paper is well structured and easy to follow.", "weaknesses": "- The paper primarily aims to improve the training efficiency of all kinds of conditional models, hence it employs a series of LoRAs to train the newly introduced conditions based on the \"Base ControlNet\". However, there is relatively little comparison and discussion of existing methods that efficiently train ControlNet, such as T2I-Adapter, ControlLoRA, and SCEdit.\n\n- There currently exists a viable **controlnet-union** model, which can handle different conditions using a single model. This may be a higher-level representation of the training of the \"Base ControlNet\" model discussed in the paper. On the other hand, the use of LoRA for fine-tuning is relatively straightforward and has been implemented in previous community works, such as ControlLoRA. In comparison, the overall innovativeness of the paper is limited.\n\n- The paper does not discuss how many conditions to use or how to select conditions for training the \"Base ControlNet\" to achieve optimal knowledge transfer effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TcjBdfJYW6", "reviewer": "Reviewer_97kn", "summary": "In this paper, the authors propose a CtrloRA framework. This framework starts by training a basic ControlNet that handles various image conditions efficiently. With this trained network, one can quickly fine-tune it to adapt to new conditions using a task-specific LoRA—specifically, fine-tuning requires only 1,000 paired images and less than an hour on a single GPU. The experimental results confirm that this method greatly speeds up the training process for new image conditions. Based on these impressive findings, I recommend a weak acceptance. However, there are some unclear points and missing experiments in the paper (see the Question section), and my final decision will depend on the authors' responses to these issues.", "questions": "1. Consider specifying 1-2 new image conditions and key metrics (e.g., adaptation speed, data efficiency, performance) for comparing UniControl [1] fine-tuning to CtrLoRA. This would provide a clear, focused comparison.\n2. Additional baselines are required for each base image condition. Comparisons should be made with a fully trained ControlNet, which has been trained exclusively under a single image condition, to establish a more comprehensive benchmark.\n3. Similarly, for the new condition, it is essential to compare the performance of CtrLora against ControlNet when ControlNet has been fully trained on a single modality. This will provide a clearer understanding of their relative efficiencies.\n4. It would be beneficial to explore how the number of image conditions used during the training of the base ControlNet affects its ability to learn new conditions. Insights into the scalability and adaptability of the base network could prove crucial for future applications.\n5. I have noted that CtrloRA can perform low-level image enhancement tasks, such as low-light image enhancement. Could the authors demonstrate how CtrloRA performs in comparison to other diffusion models for low-light image enhancement? This could highlight potential advantages or unique features of CtrloRA in practical applications.\n\n[1] UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The CtrloRA framework introduced in this paper allows users to quickly and efficiently fine-tune the ControlNet to new image conditions, with minimal resource consumption. The experimental results validate the effectiveness of this method. Additionally, the paper is well-structured and clearly written.", "weaknesses": "There are some unclear points and missing experiments in the paper (see the Question section), and my final decision will depend on the authors' responses to these issues.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)