CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation

Authors: Yifeng Xu, Zhenliang He, Shiguang Shan, Xilin Chen
Primary area: generative models
Keywords: Controllable Image Generation, Image-to-Image Generation, ControlNet, LoRA, Resource-Efficient Adaptation

Abstract:
Recently, large-scale diffusion models have made impressive progress in text-to-image (T2I) generation. To further equip these T2I models with fine-grained spatial control, approaches like ControlNet introduce an extra network that learns to follow a condition image. However, for every single condition type, ControlNet requires independent training on millions of data pairs with hundreds of GPU hours, which is quite expensive and makes it challenging for ordinary users to explore and develop new types of conditions. To address this problem, we propose the CtrLoRA framework, which trains a Base ControlNet to learn the common knowledge of image-to-image generation from multiple base conditions, along with condition-specific LoRAs to capture distinct characteristics of each condition. Utilizing our pretrained Base ControlNet, users can easily adapt it to new conditions, requiring as few as 1,000 data pairs and less than one hour of single-GPU training to obtain satisfactory results in most scenarios. Moreover, our CtrLoRA reduces the learnable parameters by 90% compared to ControlNet, significantly lowering the threshold to distribute and deploy the model weights. Extensive experiments on various types of conditions demonstrate the efficiency and effectiveness of our method. Codes and model weights will be released at
https://github.com/xyfJASON/ctrlora.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729685953287

thread_type:
official_review

<thread_object>
creator: reviewer_97kn

<content>
summary:
In this paper, the authors propose a CtrloRA framework. This framework starts by training a basic ControlNet that handles various image conditions efficiently. With this trained network, one can quickly fine-tune it to adapt to new conditions using a task-specific LoRA—specifically, fine-tuning requires only 1,000 paired images and less than an hour on a single GPU. The experimental results confirm that this method greatly speeds up the training process for new image conditions. Based on these impressive findings, I recommend a weak acceptance. However, there are some unclear points and missing experiments in the paper (see the Question section), and my final decision will depend on the authors' responses to these issues.

strengths:
The CtrloRA framework introduced in this paper allows users to quickly and efficiently fine-tune the ControlNet to new image conditions, with minimal resource consumption. The experimental results validate the effectiveness of this method. Additionally, the paper is well-structured and clearly written.

weaknesses:
There are some unclear points and missing experiments in the paper (see the Question section), and my final decision will depend on the authors' responses to these issues.

questions:
1. Consider specifying 1-2 new image conditions and key metrics (e.g., adaptation speed, data efficiency, performance) for comparing UniControl [1] fine-tuning to CtrLoRA. This would provide a clear, focused comparison.
2. Additional baselines are required for each base image condition. Comparisons should be made with a fully trained ControlNet, which has been trained exclusively under a single image condition, to establish a more comprehensive benchmark.
3. Similarly, for the new condition, it is essential to compare the performance of CtrLora against ControlNet when ControlNet has been fully trained on a single modality. This will provide a clearer understanding of their relative efficiencies.
4. It would be beneficial to explore how the number of image conditions used during the training of the base ControlNet affects its ability to learn new conditions. Insights into the scalability and adaptability of the base network could prove crucial for future applications.
5. I have noted that CtrloRA can perform low-level image enhancement tasks, such as low-light image enhancement. Could the authors demonstrate how CtrloRA performs in comparison to other diffusion models for low-light image enhancement? This could highlight potential advantages or unique features of CtrloRA in practical applications.

[1] UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730211102176

thread_type:
official_review

<thread_object>
creator: reviewer_yjaj

<content>
summary:
This paper draws on the idea of combining a base model with PEFT (Parameter-Efficient Fine-Tuning) for controllable generation. It trains a Base ControlNet obtained through several condition-specific training processes, and then fine-tunes it with a small amount of data for newly introduced conditions to obtain different condition-specific LoRAs. This approach improves the efficiency of training new condition generators at a lower cost.

strengths:
- To address the high cost of separately training different models for conditional generation tasks, this paper proposes a training method that transitions from a base controlnet model to a lightly fine-tuned lora model. This approach ensures generation quality while achieving a faster convergence rate.

- The paper shows many analyses of the proposed method and presents the results generated for a total of more than a dozen conditions.

- The paper is well structured and easy to follow.

weaknesses:
- The paper primarily aims to improve the training efficiency of all kinds of conditional models, hence it employs a series of LoRAs to train the newly introduced conditions based on the "Base ControlNet". However, there is relatively little comparison and discussion of existing methods that efficiently train ControlNet, such as T2I-Adapter, ControlLoRA, and SCEdit.

- There currently exists a viable **controlnet-union** model, which can handle different conditions using a single model. This may be a higher-level representation of the training of the "Base ControlNet" model discussed in the paper. On the other hand, the use of LoRA for fine-tuning is relatively straightforward and has been implemented in previous community works, such as ControlLoRA. In comparison, the overall innovativeness of the paper is limited.

- The paper does not discuss how many conditions to use or how to select conditions for training the "Base ControlNet" to achieve optimal knowledge transfer effects.

questions:
- Regarding the discussion of "Adaptation to new conditions," while training a comparison method from scratch with a small amount of data may indeed result in slow convergence, what would be the results if we used a pre-trained conditional model (analogous to possessing a Base ControlNet) for fine-tuning?

- I'm curious about the performance between a pre-trained controlnet model available in the community and a model trained using proposed "Base + LoRA" with same conditions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730472296786

thread_type:
official_review

<thread_object>
creator: reviewer_gm4e

<content>
summary:
This paper proposes CtrlLoRA, a two-stage parameter-efficient fine-tuning pipeline, to ease the original ControlNet's computation burden in terms of different conditions. The authors evaluate CtrlLoRA through extensive experiments by both the quality and the computation efficiency.

strengths:
1. This paper focus on an important problem, extending ControlNet to a lightweight manner.
2. Experimental results are impressive, especially the convergence experiment.

weaknesses:
1. In line 70, the authors state that ControlNet with Canny edges requires 3 million images over 600 GPU hours for one condition. In contrast, line 244 indicates that Base ControlNet necessitates millions of images for 6000 GPU hours for 9 conditions. Although it is not fair enough, but it implies that the proposed method does not significantly reduce the computational burden.

2. In line 239, the mechanism of training with 9 conditions is not clear enough. As different conditions have different levels of sparse information of input images, why they have equal training iterations? And continuous shifting between different conditions may make the training hard.

3. the motivation why the new conditions are not trained as the Base ControlNet by a shifting mechanism is not clear enough.

4. Most results are from "Base CN + CtrlLoRA'', and results from "Community Model + CtrlLoRA" in Figure 11a are rare, not enough to convince that CtrlLoRA is effective when transferring to other community models.

5. pretrained-VAE seems to be only an interesting trick.

6. putting all the prompts in the appendix makes reading inconvenient.

questions:
1. The results in Figure11b demonstrate that the different conditions are effectively disentangled, with a direct summation module according to Figure 3c. Could you clarify why this module is effective, such as presenting the results of two elements both separately and after sum-up.

2. A detail, why not presenting all 9 base-condition results comparison to UniControl in Table 2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730658098065

thread_type:
official_review

<thread_object>
creator: reviewer_uthf

<content>
summary:
The paper proposes CtrLoRA for better controllability of the conditional image generation. This framework trains a Base ControlNet for the general image-to-image generation and then uses the LoRA fine-tuning for specific user instructions. Experiments show the effectiveness of the proposed method.

strengths:
- The paper is well-organized and easy to follow.
- The authors conduct sufficient ablation studies to evaluate the proposed modules.
- The experiments demonstrate the training efficiency of the proposed method and its capability to unify various visual conditions for generation.

weaknesses:
- The authors train a base ControlNet for the subsequent LoRA fine-tuning. However, why not directly fine-tune a pre-trained ControlNet or Uni-ControlNet?

- Lack of comparison to: ControlNet++[1].

- The paper does not explore whether this method can be generalized to other diffusion models such as SDXL and Pixart.

[1] Li M, Yang T, Kuang H, et al. ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback[C]//European Conference on Computer Vision. Springer, Cham, 2025: 129-147.

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732090075189

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers and Area Chair,

We sincerely appreciate your effort and valuable review. For a better understanding of our paper, we wish to clarify our motivation and contribution.


### **1. Our goal** 

Although ControlNet is powerful and popular, developing it for **new conditions** is still an extremely heavy burden for an **ordinary user**, considering its huge consumption of data, GPUs, training time, and model sizes. To this end, our **goal is to provide an I2I foundation model and corresponding solution that allows ordinary users to create their customized ControlNets at an affordable cost** (similar to the role that Stable Diffusions plays in the T2I generation community). 


### **2. Existing methods related to the goal**

Three categories of existing methods are most related to our goal:

+ Methods such as UniControl [1], Uni-ControlNet [2], controlnet-union [3] unify multiple conditions into one single model, decreasing the number of models. However, they lack a straightforward method that extends the unified model to new conditions with limited data and GPUs. Although we could naively use LoRA to fine-tune these models, they perform worse than our method due to the inconsistency between pre-training and fine-tuning.

+ Methods such as T2I-Adapter [4] and SCEdit [5] improve the training efficiency and decrease the model sizes. However, the data and GPU resources required to train these models are still unaffordable for ordinary users.

+ Community model ControlLoRA [6] directly trains LoRA with controlling input on Stable Diffusion, which seems to be the most affordable method. However, without a powerful base model like our Base ControlNet, this method cannot obtain satisfactory performance.


### **3. How far we step towards the goal**

Existing related methods still fall far short of this goal, whereas our method makes a significant advancement towards the goal. Through extensive testing, our method enables users to create a customized ControlNet with limited data (≈1000), only 1 GPU, and within 1 hour. Besides, the LoRA size is small (≈37M params.), making it easy to distribute and deploy. As far as we know, this is **the most affordable solution for ordinary users to develop their own ControlNets with satisfactory results, which represents our contribution to practical applications**.


### **4. The main challenge of the goal**

To achieve our goal, we employ the LoRA technique for new conditions, which "seems to be straightforward". However, LoRA itself is not the challenge; **the real challenge is how to make a new LoRA perform well with limited data (1000 images in our paper), as it is difficult for an ordinary user to collect a large customized dataset**. This challenge is not straightforward to solve and existing methods cannot handle it well. To this end, we propose to train a Base ControlNet with a shifting condition scheme to capture the general knowledge of I2I generation. With our well trained Base ControlNet, 1000 data samples are sufficient to learn a LoRA for a new conditon with satisfactory results.


### **5. Why is our Base ControlNet better for learning new LoRAs**

Our Base ControlNet is trained with shifting base conditions, and these conditions themselves correspond to condition-specific LoRAs. In other words, the Base ControlNet is trained to be adapted for the LoRA form. Therefore, it's naturally suitable for learning new LoRAs.

---

[1] Qin, Can, et al. "UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild." Advances in Neural Information Processing Systems 36 (2024).

[2] Zhao, Shihao, et al. "Uni-controlnet: All-in-one control to text-to-image diffusion models." Advances in Neural Information Processing Systems 36 (2024).

[3] xinsir, et al. "controlnet-union-sdxl-1.0." Hugging Face.

[4] Mou, Chong, et al. "T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.

[5] Jiang, Zeyinzi, et al. "Scedit: Efficient and controllable image diffusion generation via skip connection editing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[6] Wu, Hecong. "ControlLoRA: A Lightweight Neural Network To Control Stable Diffusion Spatial Information." GitHub.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732459870744

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Updates

comment:
Dear Reviewers and Area Chair, 

Thanks for your precious and careful review. We have uploaded a revised paper based on your feedback, with updates below:

**Major updates**
- **Related Work & Appendix Section B:** add discussions on T2I-Adapter, SCEdit, and ControlLoRA. [Reviewer **YjAj**]
- **Appendix Section C:** add more quantitative results, including a controllable generation benchmark [Reviewer **YjAj**, **97kn**], comparison with directly fine-tuning a pretrained ControlNet or UniControl [Reviewer **UthF**, **YjAj**, **97kn**], and the effect of the number of base conditions [Reviewer **YjAj**, **97kn**].
- **Appendix Figure 17:** show more visual results of multi-conditional generation, with both separate and sum-up results of two conditions. [Reviewer **GM4E**]

**Other updates**
- **Table 2 & Appendix Figure 14:** add the quantitative and visual results of Bounding Box condition. [Reviewer **GM4E**]
- **Table 3 & 4:** fix the LPIPS results of Lineart condition since the results in the first version is not accurate. This update does not affect any conclusions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes proposes a novel approach to augment text-to-image diffusion models with spatial control for multiple tasks. A base controlNet is learned, which is adapted to different tasks using low-rank adapters for each task, facilitating adapting to new tasks with additional LoRA components using relatively few examples.   
Strengths of the paper mentioned in the reviews include: paper organization, good ablations, efficient aggregation of several conditioning forms in single model + LoRA. 
Weaknesses: why need the base controlnet, missing comparison to several baselines, no exploration of other diffusion backbones (SDXL, PixArt-alpha), lack of clarity in places.