Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models
Authors: Shihao Zhao
Dongdong Chen
Yen-Chun Chen
Jianmin Bao
Shaozhe Hao
Lu Yuan
Kwan-Yee K. Wong
Venue: NeurIPS 2023 poster (2023)
Keywords: computer vision
diffusion model
text-to-image generation

Abstract: Text-to-Image diffusion models have made tremendous progress over the past two years, enabling the generation of highly realistic images based on open-domain text descriptions. However, despite their success, text descriptions often struggle to adequately convey detailed controls, even when composed of long and complex texts. Moreover, recent studies have also shown that these models face challenges in understanding such complex texts and generating the corresponding images. Therefore, there is a growing need to enable more control modes beyond text description. In this paper, we introduce Uni-ControlNet, a unified framework that allows for the simultaneous utilization of different local controls (e.g., edge maps, depth map, segmentation masks) and global controls (e.g., CLIP image embeddings) in a flexible and composable manner within one single model. Unlike existing methods, Uni-ControlNet only requires the fine-tuning of two additional adapters upon frozen pre-trained text-to-image diffusion models, eliminating the huge cost of training from scratch. Moreover, thanks to some dedicated adapter designs, Uni-ControlNet only necessitates a constant number (i.e., 2) of adapters, regardless of the number of local or global controls used. This not only reduces the fine-tuning costs and model size, making it more suitable for real-world deployment, but also facilitate composability of different conditions. Through both quantitative and qualitative comparisons, Uni-ControlNet demonstrates its superiority over existing methods in terms of controllability, generation quality and composability. Code is available at https://github.com/ShihaoZhaoZSH/Uni-ControlNet.

Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models
Shihao Zhaot
Dongdong Chen' Microsoft addlyf@gmail.cm yen- chun. chen@miccoosoft.ccmm
Yen-Chun Chen Microsoft
The University of Hong Kong shzhao@cs hku. hk
Jianmin Bao Microsoft
Shaozhe Hao
Lu Yuan
The University of Hong Kong szhaoθcs. hku.
Microsoft
j jiamiin.. brosmocroooo...oomm com
kuyuan@microsoft..com com
Kwan- Keen-Yee K. Wong" The University of Hong Kong kykwongêcs hku. hk
Abstract
Texa-to-Image diffusion models have made tremendous progress over the past two years, enabling the generation of highly realistic images based open-domain text descriptions. However, despite their success, text descriptions often struggle to adequately convey detailed controls, texts. Moreover, recent studies have also shown that these models face challenges in understanding such complex texts and generating the corresponding images. Therefore, there is growing need to enable more control modes beyond text description. In this paper, introduce Cni-ControlNet, a unified framework that allows for the simultancous utilization of different local controls (e.g., edge maps, depth map, segmentation masks) and global controls (e.g., CLIP image embeddings) in flexible and composable manner within one single model. Unlike existing methods, Uni-ControlNet only requires the fine-tuning of two additional adapters upon frozen pre- rre-trained text-to-image diffusion models, eliminating the huge cost of training from seratch. Moreover, thanks to some dedicated adapter designs, Uni-ControINet only necessitates constant number (i.e., 2) of adapters, regardless of the number of local or global controls used. This not only reduces the fine-tuning costs and model size. making it more suitable for real- hol-world deployment, but also facilitate composability qualitative comparisons, Uni-ControINet demonstrates its superiority over existing methods in terms of controllability, generation quality and composability. Code is available /tttps://gitane com/ SaaihoohhazzSi/uiiiCootttaa
when composed of long and complex
different conditions. Through both quantitative and
Introduction
In recent two years, diffusion models [T-00] have gained significant attention due their remarkable performance in image synthesis tasks. Therefore, text-to- text-to-image (T2I) diffusion models 6 have emerged popular choice for synthesizing high-que righquuaitt images based on textual inputs. By training on large-scale datasets with large models, these T2I diffusion models demonstrate exceptional ability creating images that closely resemble the content described in text descriptions, and facilitate
7
19
"Corresponding Author, Intern Microsoft
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the connection between textual and visual domains. The substantially improved generation quality in capturing intricate texture details and complex relationships between objects, makes them highly suitable for various real-works world applications, including but not limited to content creation, fashion design, and interior decoration.
However, text descriptions often prove to be either inefficient or insufficient to accurately convey detailed controls upon the final generation results, e.g., control the fine-grained semantic layout of multiple objects, not to mention the challenge in understanding complex text descriptions for such models. As a result, there is a growing need to incorporate more additional control modes (e.g.. user-drawn sketch, semantic mask) alongside the text description into such T21 diffusion models. This necessity has sparked considerable interest from both academia and industry, as broadens the scope of T2I generation from a singular function to a comprchensive system.
studying controllable T21 diffusion models. One representative work, Composer [20), explores the integration of multiple different control signals together with the text descriptions and train the model from scratch on billion-scale datasets. While the results are promising, it requires massive GPU resources and incurs huge training cost, making it unaffordable for many researchers in this field. Considering there are powerful pretrained T21 diffusion models (e.g., Stable Diffusion [6]) publicly available, ControlNet (20, GLIGEN [23
Very recently, there some attempts 20 22
and T2I-Adapter [22 directly incorporate lightweight adapters (or extra modules) into frozen T2I diffusion models to enable additional condition signals. This makes
more affordable.
However, one drawback that they need one independent adapter for each single condition, resulting in linear increase in fine-tuning cost and model size along as the number of the control conditions grows, even though many conditions share similar characteristics. Additionally, this also makes composability among different conditions remains a formidable challenge.
In this paper, we propose Uni-ControINet, a new framework that leverages lightweight adapters to enable precise controls over pre-trained T21 diffusion models. As shown in Table [ Uni-ControlNet can not only handle different conditions within one single model but also supports composable control. By contrast, the existing methods fail to achieve this unified framework within one single model. Besides, even for those methods that support composite control, they perform poorly in terms of composability as illustrated in Section 4
Unlike previous methods, Nei categorizes various conditions into two distinet groups: local conditions and global conditions. Accordingly, we only add two additional adapters, regardless of the number of local and global controls involved. This design choice not only significantly reduces both the whole fine-tuning cost and the model size, making it highly efficient for deployment, but also facilitates the composability of different conditions. To achieve this, we dedicatedly design the adapters for local and global controls. Specifically, for local controls, we introduce a multi-scale condition injection strategy that uses a shared local condition encoder adapter. This adapter first converts the local control signals into modulation signals, which are then used to modulate the incoming noise features. And for global controls, we employ another shared global condition encoder to convert them into conditional tokens, which are concatenated with text tokens to form the extended prompt and interacted with the incoming features via -ttennattnn mechanism. Interestingly, we find these two adapters can be separately trained without the need of additional joint training, while still supporting the composition of multiple control signals. This finding adds the flexibility and ease of use provided by Uni-Contrr Nett
By only training on 10 million text-image pairs with epoch, our Uni-ControlNet demonstrates highly promising results in terms of fidelity and controllability. Figure ! provides visual examples showcasing the effectiveness of Eni-ControlNet when using either one or multiple conditions. To gain further insights, we perform in-depth ablation analysis and compare our newly proposed adapter designs with those of ControlNet (21), GLIGEN (3 and T21-Adapter [22]. The analysis results reveal the superiority of our adapter designs, emphasizing their enhanced performance over the counterparts offered by ControlNet, GLIGEN and 12I-Adapter.
2 Related Work
Text-to-Imaae Generation is an emerging field that aims to generate realistic images from text descriptions. To address this challenging task, various approaches have been proposed in the past years. Early works (24 primarily adopted Generative Adversarial Networks (GANs) and were
2

---

Canny
Sketch
Open
Depth
Segmentation
motorcycle the mountains
A girl in the room, oif painting
Figure 1: Visual results of our proposed Uni-Contnn Nni-ControlNet. The top and bottom two results for single condition and multi-conditions respectively.
Table Comparisons of different controllable diffusion models. N is the number of conditions. We define the fine-tuning cost as the number of times the model needs to be fine-tuned on conditions. As Composer is trained from scratch, both fine-tuning cost and adapter number are not applicable. For T21-Adapter, (+1) indicates that further joint fine-tuning is required on the N N-based adapters along with additional fuser to achieve composable conditions.
Fine-tuning Composable Control
Fine-tuning Cost
Adapter Number
Composer
ControlNet
N
N
GLIGEN
X
N
N
T2I-Adapter
N(+1)
N(+1)
Uni-ControlNet (Ours)
2
2
often trained on specific domains. However, they faced two key challenges, i.c., training instability and poor generalization ability to open-domain scenarios. Motivated by the success of GPT models 27.30] recent works B1S4 have explored the use of autoregressive models for text-to- text-to-image generation and train web-scale image- text pairs, which start to show strong generation capability under the zero-shot setting for open-domain scenarios. Another approach is the diffusion models [1 41, originally proposed by 2 Diffusion models comprise forward process that gradually adds noise to natural images and a backward process that learns to denoise them back generate clean output. They demonstrate stronger capability in modeling fine-grained structures and texture details compared to autorcgressive models. Recently, vast variants of diffusion models have been developed, such DALLE-2 [Т], which uses one prior model and one decoder model to generate images from CLIP latent embeddings, Another phenomenal T2I diffusion model is Stable Diffusion (SD), which scaled up the latent diffusion model 6 with larger model and data scales, and made the pre-trained models publicly available. In this paper, we use SD base model and explore how enable more control signals beyond the text description for pre- pre-trained T2I diffusion models in an efficient and composable way.
Controllable Diffusion Models are designed to enable T21 diffusion models accept more user controls for guiding the generation results. They have garnered increasing attention very recently.

---

Extended Prompt
Text Takens
Global Conditions
Middle Block
Noise
Encoder
Decoder
Global Control
Add
Concet
Adapter
Noise
Base Difiusion Model
Local Control Adapter
Local
Local Conditions
Extended Prompt
Clubal Control
Extended Prompt
Figure 2: The overall framework of our proposed Uni-ControlNet.
Broadly speaking, there are two strategies for implementing controllable diffusion models: training from scratch (20) and fine-tuning lightweight adapters [2] [2 on frozen pretrained T2I diffusion
models. In the case of training from scratch, Composer [20] trains one big diffusion model from scratch to achieve great controllability for both single and multi-conditions. It obtains remarkable generation quality but comes with huge training cost. In contrast, ControlNet 211. GLIGEN (23) and T21-Adapter [22 propose to introduce lightweight adapters (or extra modules) into publicly available SD models. By only fine-tuming the adapters while keeping original SD models frozen, they significantly reduce the training cost and make it affordable for the research community. However, all the ControlNet, GLIGEN and T2I-Adapter utilize independent adapters for each condition, resulting in increased fine-tuning cost and model size when handling increased number of conditions. Moreover, GLIGEN does not support composite control over different conditions. And different adapters in Multi-ControlNG [2I]]. a version of ControlNet that allow composite control, are isolated from one another, limiting their composability. By testing CoAdapter [22], which is jointly trained using different T21-Adapters, we find that it also exhibits inadequate performance in generating composable conditions. Our proposed Uni-ControlNet follows the second line of fine-tuning adapters and is much less expensive than Composer, while addressing the above limitations of ControlNet, GLIGEN and T2I-Adapter. It groups conditions into two groups, i.c., local controls and global controls, and only requires two additional adapters accordingly. Thanks to newly designed adapter structure, Uni-ControlNet is not only efficient in terms of training cost and model sizes, but also surpasses ControlNet, GLIGEN and T21-Adapter in controllability and quality.
3 Method
3.1 Preliminary
A typical diffusion model involves two processes: forward process which gradually adds small amounts of Gaussian noise onto the sample in T steps, and a corresponding backward process containing learnable parameters to recover input images by estimating and eliminating the noise. In this paper, we use SD as our example base model illustrate how enable diverse controls with our Uni- SD incorporates the UNet-like structure (42 as its denoising model, which consists of an encoder, middle block, and a decoder, with 12 corresponding blocks in each of the encoder and decoder modules. For brevity, we denote the encoder as F, the middle block as M, and the decoder as G. with fi and gi denoting the output of the i-th block in the encoder and decoder, and denoting the output of the middle block, respectively. It is important to note that, due to the adoption of skip connections UNet, the input for the i-th block in the decoder given by:
concut(m: fz)
where 11
i+j=10 13.
(oncat(gg-11553 ff) where 2
i+j=13
Skip connections allow the decoder directly utilize features from the encoder and thereby help minimize the information loss. In SD, cross-attention layers are employed to capture semantic information from the input text description. Here we use Z to denote the incoming noise features and to denote text token embeddings encoded by the language encoder. The Q, K, V in cross-attention can be expressed as:
Q = @=W W2) K W2(v), V W.(y), where Wg. Wi and W. are projection matrices.
(2)

---

Natu Tunursz
Naise
Cont
Conv
Feature Denormalization (FDN)
3
Local tecal
Copled Encoder F
Text Tokens
Zaro Sanvolutionn
Feedlfowwrd Layer
XN
Extractor
1
Ckubal Coodd Marta
Local Conditions
Local Control Adapter
Global Control Adapter
Figure 3: Details the local and global control adapters.
3.2
Control Adapter
In this paper, we consider seven example local conditions, including Canny edge 43 MLSD edge [44), HED boundary [45], sketch [46 477. Openpose 488, Midas depth 44), and segmentation mask [50]. We also consider one example global condition, i.e., global image embedding of one reference content image that is extracted from the CLIP image encoder [5T]. This global condition goes beyond simple image features and provides a more nuanced understanding of the semantic content of the condition image. By employing both local and global conditions, we aim provide comprehensive control over the generation process. We show the overview of our pipeline in Figure and the details of local control adapter and global control adapter are given in Figure
Local Control Adapter:
For our local control adapter, we have taken inspiration from ControlNet. Specifically, fix the weights of SD and copy the structures and weights of the encoder and middle incorporate the information from the local control adapter during the decoding process. To achieve it, we ensure that all other elements remain unchanged while modifying the input of the i-th block of the decoder
block, designated F' and M' respectively. Thereafter,
concat(m + m', + zero(f)))
where 1,
i j 13.
co + zero(ff))
where 2 S 12,
i+j= = 13.
where sero represents one zero convolutional layer whose weights increase from zero to gradually integrate control information into the main SD model. In contrast to ControlNet that adds the conditions directly to the input noise and sends them to the copied encoder, we opt for a multi- scale condition injection strategy. Our approach involves injecting the condition information at all resolutions. In detail, we first concatenate different local conditions along the channel dimension and then use feature extractor H (stacked convolutional layers) extract condition features different resolutions. Subsequently, we select the first block of each resolution (i.c., 64 x 64, 32 x 32, 16 x 16, 8 in the copied encoder (i.e., the Copied Encoder in Figure 3 for condition injection. For the injection module, we take the inspiration from SPADE (52) and implement Feature Denormalization (FDN) that uses the condition features modulate the normalized (i.e.,norm(() input noise features:
FDN, (z) norm(Z; (Z,) (1+ (2rx,(2ero(, + convy(zero(h,
(4)
where Z, denotes noise features at resolution is the concatenated local conditions, h, represents the output of the feature extractor at resolution r, and conv., and conv3 refer to learnable convolutional layers that convert condition features into spatial-sensitie scale and shift modulation coefficients. We will ablate different local feature injection strategics in following sections.
For global controls, we use the image embedding of one condition image extracted from CLIP image encoder as the example. Inspired by the fact that the text description in T2I diffusion models can be also viewed as one kind of global control without explicit spatial guidance, we project the global control signals into condition embeddings by using a condition encoder hg. The condition encoder consists of stacked feedforward layers, which aligns the global control signals with the text embeddings SD. Next, we reshape the projected condition embeddings into K global tokens (K 4 by default) and concatenate them with the original Ko text tokens
Global Control Adapter:
5

---

Rasd
Ruom
Edba ranger
Doy, wild
vain ruseiig strout
City
RRaliór myy
Golden vatrieer
fuolding whitz hoard
dig sitring By raldly boor
A the mountains
Helo Werld
Helts ifoidd
Condition-1
Sample
Sample
Condition-2
Sample
Figure More visual results of Uni-ControlNer.. The top two rows show results single condition, with columns 1-7 for local conditions and columns 8-9 for global condition. 3rd row shows the results of combining two local conditions, while row 4-th shows the results of integrating local condition with global condition. There is no text prompt for the examples in 4-th row.
to create an extended prompt Vezt (total token number is K + Kg) which serves as the input to all cross-attention layers in both main SD model and control adapters:
Vext [ui, w2- Vk. : X vi À v2. X vkl where yf hhg(c))((--)) -i. [1, K]
(5)
where y and y represent the original text tokens and global condition tokens respectively. and is hyper-parameter that controls the weight of the global condition. Co denotes the global condition and is the dimension of text token embedding. hgg-)[(. ~ ie] represents the sub-tensor of hg(.) that
contains elements from the i,-th to the ie-th positions. Finally, the Q, K, cross- operation all pross-attention layers is changed to:
Q W2 (2),), K
WA (y V= Wo(Wen)),
3.3 Training Strategy
As the local control signals and global control signals often contain different amounts of condition information, empirically find that directly joint fine-tuning these two types of adapters will produce poor controllable generation performance. Therefore, we opt to tne-tune these two types of adapters separately that both of them be sufficiently trained and contribute effectively to the final generation results. When fine-tuning each adapter, we employ predefined probability to randomly dropout each condition, along with additional probability to deliberately keep drop all conditions. For the dropped conditions, we set the value of the corresponding input channels to This can facilitate the model learn generating the results based one or multiple conditions simultancously. Interestingly, by directly integrating these two separately trained adapters during inference, Uni-ControlNet can already well combine global and local conditions together in composable way. without the need of further joint fine-tuning. In Section 4.3 detailed analysis about different fine-tuning strategies.
we will provide more
Experiments
Implementation Details. fine-tune our model, we randomly sample million text-image pairs from the LAION dataset [53 and fine-tune Uni-ControlNet for epoch. We use the AdamW optimizer [54] with a learning rate of 10-5 and resize the input images and local condition maps to 512 x 512. As described, the local and global control adapters fine-tuned separately by

---

default. During inference, we merge the two adapters and adopt DDIM 55 for sampling, with the number of time steps set to 50 and the classifier free guidance scale [56] set to 7.5. During training. the hyper-parameter À in Equation 6 is with fixed value 1. At inference time, when there is no text prompt, À remains at 1, while when there a text prompt, the value adjusted to around 0.75, depending on the intented weight between the text and global condition. As explained in Section 3.2 we employ local conditions (Canny edge, MLSD edge, HED boundary, sketch, Openpose, Midas depth, and segmentation mask) and global control condition (CLIP image embeddings) for control. As annotating a sketch dataset can be challenging. in our experiment, we initially obtain the HED boundary [45] of an image and subsequently utilize sketch simplification method [46 47] to generate the sketch for the training sample. Regarding the pose condition, as not all images in the dataset include humans, condition is fully trained. Detailed structures of global and local condition adapters can be found in the appendix.
opt not drop the pose condition during training to ensure that the pose
4.1 Controllable Generation Results
In Figure 4 we provide more controllable generation results of Uni-ControlNet in both single and multi-condition setups. Notably, for visualization purposes, we use the original condition images to denote their CLIP image embeddings. It can be seen that our Uni-ControlNet can produce very promising results in terms of both controllability and generation fidelity. For example, in the case of a single sketch condition with the text prompt "Dog, wild" (rows 1-2, column 4), the resulting image accurately depicts a vivid dog and a background of grass and trees that align well with the given sketch condition. Similarly, when presented with the global CLIP image embedding conditions with the prompt "Golden retriever" (rows 1-2, columns 8-9), our model can seamlessly change the background of the dog from the wild to a room. Moreover, our model also handles multi-condition settings well, as demonstrated in the example of "A man on the mountains" (row 3, columns 7-9), where the combination of a sketch and a pose produces a cohesive and detailed image of a man on a mountainside. When presented with local depth map and global CLIP image embeddings without any prompt (row 4, columns 1-3), our model produces an image a forest, taking the contour of an elephant, which harmonizes with both the depth map and the content of the source global image.
4.2 Comparison with Existing Methods
Here compare our Uni- Coi-ControlNet with ControlNet T21-Adapter (CoAdapter) [22]. Since Composer (20 is not open open-sourced and trained from scratch, we do not include it in comparisons.
211. GLIGEN 23 and
Quantitative Comparison: For quantitative evaluation, we use the validation set of COC02017 (57 resolution of 512 x 512. Since this set contains 5k images, and each image has multiple captions, we randomly select one caption per image resulting in 5k generated images for our evaluation. is important to note that for quantitative comparison, we limit testing to different single conditions only. Additionally, we use Style'CConten Content to represent the global condition as there are different settings in the ControlNet, GLIGEN and T21-Adapter. For the ControlNet, the content condition refers to the content shuffle ControlNet- V1.1. As T2I-Adapter does not take the MLSD and HED conditions into account, it has no results for MLSD and HED. Similarly, GLIGEN does not consider the MLSD and sketch conditions, resulting in the absence of results for MLSD and sketch.
To evaluate the generation quality, We report the FID (58 in Table 2 We can find that our model reveals superior performance across most conditions quantitatively compared to existing approaches. We also use quantitative metrics to assess the controllability. We employed the following metrics for single-condition generation:
SSIM (Structural Similarity) for Canny, HED, MLSD, and sketch conditions,
mAP (mean Average Precision) based on OKS (Object Keypoint Similarity) for pose condition,
MSE (Mean Squared Error) for depth map.
mIoU (Mean Intersection over Union) for segmentation map,
CLIP score for content condition.
7

---

Table 2: FID on different controllable diffusion models. The best results are in bold.
Canny
MLSD
HED
Sketch
Pose
Depth
Segmentation
Style),Contenl
ControlNet 18.90
31.36
26.59
22.19
27.84
21.25
23.08
31.17
GLIGEN
24.74
28.57
24.57
21.46
27.39
25.12
T2I-Adapter
18.98
18.83
29.57
21.35
23.84
28.86
Ours
17.79
26.18
17.86
20.11
26.61
21.20
23.40
23.98
Table 3: Quantitative evaluation of the controllabiltyy The best results are in bold.
Canny (SSIM)
MLSD
HED (SSIM)
Sketch (SSIM)
Pose (mAP)
Depth (MSE)
Segmentation (mloU)
Style's ConteeCnntent (CLIP Score)
(SSIM)
ControlNet 0.4828
0.7455
0,4719 0.4015
0.3657
0,4359
87.57
0.4431
0.6765
GLIGEN
0.4226
0.1677
88.22
0.2557
0.7458
T21-Adapter
0,4422
0.5148
0,5283
89.82
0,2406
0.7078
Ours 0.4911
0.6773
0.5197
0.5923
0.2164
91.05
0.3160
0.7753
To calculate these metrics, we compare the extracted conditions from the natural image (the ground truth) and the corresponding generated image. And we report the results in Table 3 Our method outperforms other baseline methods in 4 out of evaluation metrics, Notably, ControlNet achieves the best performance in 3 out of 8 metrics, while T21-Adapter only excels in out of 8 metrics. However, should be noted that all of ControlNet, GLIGEN and T2I-Adapter employ different models for different conditions, allowing each model to be well-trained for its corresponding condition. In contrast, we only use a single model and achieved even overall superior results.
To provide more comprehensive comparison of various controllable models, we also include a comparison on CLIP score Table and present the results of user studies in Section G in the appendix.
Qualitative Comparison: We further provide qualitative comparison of single and composed multi- conditions in Figure annnd Figure 6respectively. For single conditions, as GLIGEN not considering the sketch condition, we use GLIGEN's results the HED boundary as the second case in the first row for showcase. We find that our Uni-ControlNet, ControlNet, GLIGEN and T21-Adapter can all perform overall well in single condition setting, and our results show slightly better alignments with input conditions. Notably, we only fine-tune 2 adapters for all conditions, whereas ControlNet, GLIGEN and T2I-Adapter fine-tune eight adapters for eight different single conditions.
Since GLIGEN does not support composed multi-conditions, we only compare Uni-ControlNet with Culti-ConntooNet and CoAdapter under the Multi-ComtroINet and CoAdapter show poorer composability when dealing with two local conditions, e.g., missing the podium in the first example and no car in the second example. In contrast, our model can fuse the two conditions much better. As for composing a local condition with a global condition, Netti-ConttroNet is also not that good as shown in the second row in Figure 6 And CoAdapter performs okay in the case of combing a sketch of a cup and global condition of a cat. However, when the two conditions are not that related, e.g., the example where there is a Canny edge of a Minion and a global condition of a bus in London, the image generated by CoAdapter appears to be unrealistic and the two elements are not well integrated. And our model effectively creates a Minion-shaped bus with car windows and vivid background.
setting. As shown in Figure 6
4.3 Ablation Analysis
For ablation study, we fine-tune our model using a smaller dataset for resource consideration. In detail, we utilize the I million subset of the 10 million dataset and fine-tune single epoch, while keeping all other settings unchanged.
For local conditions, we compare our proposed injection method with two other strategies. The first strategy is to directly use SPADE to inject the conditions, which involves resizing the conditions to the corresponding resolutions using interpolation. We call this Injection-S1. The second strategy is similar to Composer, ControlNet and T21-Adapter, where the conditions are only sent to the adapter or the main model at the input layer, which we refer to as Injection-S2. When using these two strategies, all other parts of our method will remain unchanged. We follow the setting in Section 4.2 and evaluate the FID on different local condition injection strategies. The quantitative
Condition Injection Strategy:
8

---

Deer
Living rorm
mun is playing Sanbethall
Riegle
Gtraffi sky
Alwerss
Condition
ControlNet
GUIGEN
-ddapee
Ours
Condition
ContralNee
GUIGEN
T2l Addaperr
Ours
Figure Comparison of existing controllable diffusion models on different single conditions.
Table 4: FID on condition injection methods and training strategies. The best results are in bold.
Canny
MLSD
Sketch
Openpose
Depth
Segmentation
Content
Injection-S]
25.89
27.22
23.51
24.71
26.25
Injection-S2
22.22
27.08
21.94
22.74
26.56
24.21
24.42
Injection-S3 S3
27.06
Training-S S1 Training-S2
21.21
27.20
20.78
23.22
27.83
25.01
24.99
28.51
18.80
26.40
19.12
20.91
27.17
21.59
23.93
24.84
Ours
18.24
26.91
18.61
20.32
27.76
21.97
23.51
24.86
and qualitative results presented in Table 4 and the upper part of Figure 7 The quantitative results show our proposed condition injection strategy performs better under most settings. For the visual results, observe that for Injection-S1, the alignment with the conditions is poor. This may be because direct interpolation significantly destroys condition information. As for Injection-S2, renders unsatisfactory results for composite control. For instance, in the "An elephant in the temple" case, the lanterns on the top of the image are not accurately aligned with depth condition. Moreover, the composite results are as harmonious case, the gorilla's 's eyes and glasses are not well-merged. This may be because that if the condition information is only provided in the input layer of the adapter, the model may lose some information of the conditions in deeper layers, leading to poor alignment among the combined controls. In contrast, our proposed FDN employs multi-scale injection strategy that provides condition information at different levels, resulting in richer condition information. Furthermore, our feature extractor projects the conditions to the corresponding latent spaces of different layers, which allows for better alignment between the conditions and noise features.
ours. For example, in the "Gorilla wearing glasses"
For the global condition, condition into control adapter but not the main SD model, and we denote this injection strategy as Injection-S3, As shown the lower part Figure 7 without using the extended prompt in the main SD model, this method cannot inject the global condition into the final generated results.
compare method to way in which only add the global
Strmtaggur's feture is the forest
nice car on the country roal
Condition1
Cundition22
MrltiCCnerrllttt
CoAdleterr
Durs
Cundit und
Condition22
Muh--Cattedhar
CuAclapter
Durs
Figure 6: Comparison of different controllable diffusion models composable multi-conditions.

---

Room with colling windows
elegbaant in the tample
Foods in the plate
Gariilia wearing glasses
Injection-51
Injection-S1
Injection
Condition
Condition
Injection-53
Figure 7: Ablation results on different condition injection strategies.
Rozt
fiewer sink
Condition
Training-S1
Training
Ours
Condition-1
Condition-2
Training S1
Training S2
Ours
Figure 8: Ablation results on different training strategies.
Training Strategy: As above mentioned, fine-tune the local and global control adapters separately and merge them at inference without any further joint fine-tuning by default. Here, we also investigate alternative training strategies: joint fine-tuning together ("Train-S1"), where we fine-tune both adapters together from scratch; 2) further joint fine-tuning after separate fine-tuning ("Train-S2"), where we further fine-tune the adapters together after separate fine-tuning.
The quantitative FID results are shown in Table 4 We find that our default strategy and Training-S3 perform much better consistently than Training- S1, but further joint fine-tuning in Train S2 does not bring obvious performance gain in most cases. Some visual results are given in Figure Note that, in order to better assess the controllability of the global condition, for the cases with global condition. As we described before, the reason why Training-S1 gets poor controllability the global condition is that the global control adapter does not learn much as local adapter even equally treated during joint fine-tuning. One possible explanation is that the local conditions often contain more rich guidance information than global conditions, leading the model to pay less attention to the global condition.
do provide text prompts
5 Conclusion and Social Impact
In this paper, we propose Uni- Uni-ControlNet, a solution that enhances the capabilities of text- to-image diffusion models by enabling efficient integration of diverse local and global controls. With better adapter designs, our Uni-ControlNet only requires two adapters for different conditions while existing methods often require independent adapters for each condition. The new design of Mni-ControlNet not only saves both fine-tuning cost and model size, but also facilitates composability, allowing for the simultaneous utilization of multiple conditions. Extensive experiments validate the effectiveness of Lni-ControllNe, showcasing its improved controllability, generation fidelity, and composability. While our system empowers artists, designers, and content creators to realize their creative visions with precise control, crucial to acknowledge that can arise from misuse abuse, similar to other image generation and editing Al models. To address these concerns, responsible deployment practices, ethical regulations, and the inclusion of special flags in generated images to enhance transparency vital steps towards responsible usage.
potential negative social impact
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS 2020,
[2] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.025002 2020.
[3] Florinel- Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. TPAMI 2023.
[4] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. ICLR 2021.-
[5] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurlPS 202/.
[6] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR 2022.
[7] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Iya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint
2021.
[8] Jinniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.0657/ 2023.
[9] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Vidcocomposer. Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02018 2023.
[10] Pandeng Li, Chen-Wer Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. In NeurlPS 2023.
[11] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
[12] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. CVPR 2022,
[13] Wan-Cyual Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang. Frido: Feature pyramid diffusion for complex scene image synthesis. In AAAI 2023.
[14] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. arXiv preprint arXiv:2305. 18295, 2023.
[15] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR 2023.
[16] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang. Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05/25 2023.
[17] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR 2022.
[18] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control, arXiv preprint arXiv:2208.0/e: 026600 2022.

---

[19] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR 2023.
[20] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Com- poser: Creative and controllable image synthesis with composable conditions. arXiv preprini arXiv:2302.09778, 2023.
[21] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
[22] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprinl 2xXi:2330200843. 2023.
[23] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR 2023,.
[24] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In /CML 2016.
[25] Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang. Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In 2017.
[26] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. CVPR 2018.
[27] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. NeurlPS 2020.
[28] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. ICML 2020.
[29] Ziyu Wan, Jingho Zhang, Dongdong Chen, and Jing Liao. High- figh-fidelity pluralistic image completion with transformers. In ICCV 2021.
[30] Qiankun Liu, Zhentao Tan, Dongdong Chen, Qi Chu, Xiyang Dai, Yinpeng Chen, Mengehen Liu, Lu Yuan, and Nenghai Yu. Reduce information loss in transformers for pluralistic image inpainting. In CVPR 2022.
[31] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML 2021.
[32] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. In NeurIPS 2021.
[33] Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding. Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. M6: A chinese multimodal pretrainer. arXiv preprint arXiv:2103.00m 000.3 2021.
[34] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 2022.
[35] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar XrXiv:2288.01 0/618, 2022.
[36] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. arXiv preprint 237.02300
12

---

[37] Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan-Yee K Wong. Vico: Detail-preserving visual condition for personalized -itat generation. arXiv preprint (XXi:23066009771 2023.
[38] Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Texaee diffusion models with an ensemble of expert denoisers. arXiv preprint ArXiv:2211.001 013224, 2022.
[39] Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. ICLR 2022.
[40] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR 2022.
[41] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Ani- matediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0472. 2023.
[42] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI 2015.
[43] John Canny, A computational approach to edge detection. TPAMI 1986.
[44] Geonmo Gu, Byungsoo Ko, SeoungHyun Go, Towards light- and real-time line segment detection. In AAAI 2022.
Lee, Jingeun Lee, and Minchul Shin.
[45] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In 2015.
[46] Edgar Simo-Serra, Satoshi lizuka, Kazuma Sasaki, and Hiroshi Ishikawa. Learning to simplify: Fully convolutional networks for rough sketch cleanup. In SIGGRAPH,, 2016.
[47] Edgar Simo-Serra, Satoshi lizuka, and Hiroshi Ishikawa. Mastering sketching: Adversarial augmentation for structured prediction. In TOG 2018.
[48] Zhe Cao. Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR 2017.
[49] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI 2020.
[50] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. ECCV 2018.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML 2021.
[52] Taesung Park, Ming-Yt Yu Liu, Ting-Chun Wang, and Jun- Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR 2019.
[53] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
[54] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR 2015.
[55] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR 2021..
[56] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications.
13

---

[57] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV 2014.
[58] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS 2017.
14

---

A The Weight of the Global Condition
In the global control module, we have implemented a hyper-parameter This hyper-parametee plays a role in determining the influence of the global condition while concatenating the projected global condition to the text. Illustrating the effect of varying A, we present two representative visualization results in Figure P It can be seen that the value of À plays significant role in displaying the elements of the global conditions the generated images. As the value of À increases, the global condition takes precedence over the original text content, leading to a decrease in the influence of the text prompt on the results. For instance, in the first case, appears that the forest has experienced reduction in coverage area as compared to an increase in the number of houses with an increase in the value of x. Similarly, in the second case, it is evident that the city is shrinking while the desert's coverage is expanding with the rise of À value. In the real-world applications, we can adjust the hyper-parameter to generate our desired results with flexibility.
City
Condition
0.5
0.6
0.7
0.8
0.9
1.0
Figure 9: The effect of the hyper-parameter A. On the left side are the textual prompts and global conditions provided. On the right side are the images generated under increased A values.
B Condition Conflicts
Since our Uni-ControINet can support multiple conditions simultaneously, we are curious about its behavior when providing multiple conflicting conditions. Need that, this is very rare in the real-world applications, and this experiment is just for the analysis purpose. For example, we consider the case of providing the model with two local conditions that are fundamentally incompatible, such as the conditions of two dogs shown in Figure [0] Through this experiment, we can possibly evaluate the relative importance of each condition and learn how Eini-ControlNet resolves conflicts, which may help us design more robust integration of conditions that can adequately handle diverse and ambiguous situations.
To provide a comprehensive analysis of different condition compositions, we have assigned each condition in the first column number and each condition in the first row a number 2. This allows us to refer to the dog in the first row as dog-1 and the dog in the first line as dog-2. The results depicted in Figure T0 demonstrate that HED the most powerful condition, with generated images closely following the HED boundary when depicting text. Other conditions can only influence areas that do not overlap. For instance, when we combine the HED boundary of dog-2 with the Canny edge map of dog-I, I, the resulting image adopts the HED boundary of dog-2 but recognizes the head of dog-1 as a small element positioned near the head of dog-2. Similarly, when dog-2's HED boundary is combined with the sketch of dog-1, 1, the model fails to identify the head of dog-1 1 even though it does not conflict with the HED boundary. Among the other conditions, the Canny edge map the next most powerful, followed by the sketch, depth, MLSD, and segmentation map. The Openpose condition the weakest, whereby the model generally disregards it in the event of a conflict. Only when combined with the segmentation map, the Openpose condition produces recognizable human elements. For better visualization, we have reordered the conditions based on their strength, which implies that the upper and left conditions have greater influence.
15

---

Canny
MISO
Segmentation
Openpose
Cavditio-112 112
HED
Canny
Skettch
Depeth
MISO
Operpoos Sigmenettion
Figure 10: Study for the cases where composed conditions contradict each other. We choose the detection results of room the condition for MLSD and a man as the condition for Openpose. To test other conditions, we select two different dogs, which allows us to observe the model's output when given different dog-shaped conditions. We use "room" the prompt for MLSD, "man" for Openpose, and "dog" for other conditions. When combining two types of conditions, we integrate their prompts, such "dog". "dog and room", and "dog and man".
16

---

C Hand-drawn Sketches
For the sketch condition, as mentioned in the main paper, we first get the HED boundary 45 of the training images. Then, we employ sketch simplification method [46] 47 to generate the sketches for the model training. One question is, how does our model perform on hand-drawn sketches? We show the results of our model on hand-drawn sketches in Figure [1 We can find that although there are distribution gaps between hand-drawn sketches and the model model-generated sketches, our model can handle hand-drawe sketches pretty well.
A bicycle in the forot
Mountains
A book and pencef on the daik
A cottage in the countryssic
Condition
Sample
Condition
Sample
Condition
Sample
Condition
Sample
Figure 11: Visualization results on hand-drawn sketches.
D Extension to New Conditions
To extend trained Uni-ControlNer support new conditions, conducted experiment in two steps for comparison and analysis purpose. Firstly, we train local adapter specific to N conditions. Next, we introduce new type of condition and extend the trained adapter (N+1) conditions. The adaptation process involved modifying the input channel of the Uni-ControINet's first convolutional layer within the feature extractor. Then, we try to retrain the local adapter with different retraining strategies (R1-4) to accommodate the new conditions:
Retraining the entire feature extractor (R1),
Only retraining the pre-feature extractor, which is the part that projects the condition from resolution 512 to 64 (R2),
Only retraining the first convolutional layer in the feature extractor (R3),
Without retraining, i.c., random initialization the first convolutional layer in the feature extractor (R4).
During the retraining process, we ensure that the weights of the copied encoder in the local adapter
remain fixed. We utilize training dataset of 300k samples for the retraining. We show the extension
from [MLSD + HED + Sketch + OpenPose + Depth Seg] to [MLSD + HED + Sketch + OpenPose + Depth + Seg + Canny]. The results of this extension process are presented in Figure T2 We
surprisingly observe that retraining solely the first convolutional layer in the feature extractor can
already adequately enable the Uni-ControlNet to handle the newly added conditions.
foreet
Stormrrayye'' feture forss
Condition
R1
R2
86
Condition11
Condrtion22
R1
R3
Figure 12: Study for extending trained Uni-ControlNet to newly added conditions.
E Composite Control of Conditions with the Same Type
In real-world applications, users can actually composite two/multiple conditions of the same type easily before feeding them to the model, e.g., draw the sketch of multiple objects in one canvas. However, how to achieve composite control of two conditions with the same type an interesting research point. We try simple strategy called "Uni-Channelss Specifically, we augment the input
17

---

by adding three extra condition channels. For instance, if the original inputs had 21 channels (3 for each condition, totaling 7 local conditions), with Uni- Chi-Channels, we now have 21 + channels for the inputs.
During training, feed the Uni-Channels with randomly selected types of conditions of the input natural images. We observe that, as the shared extra condition channels for different condition types, Uni- Uni-Channels can perform well for two-condition composition of the same condition type. The visualization results are depicted in Figure 13
dog
fiaach
bird perchind
Condition-1
Condition-2
Sample
Condition-1
Condition-2
Sample
Condition-1
Condition-2
Sample
Figure 13: Visualization results of composite control of two conditions with the same type.
Comparison with Stable Diffusion 2.1
We compare our method with two Stable Diffusion models, Stable Diffusion 2 depth ("SD2-depth") and Stable Diffusion unclip ("SD2-unclip") which support the inputs of depth map and reference image respectively. The visualization results are shown in Figure 14 Additionally, we provide the quantitative results in Table Table
Table 5: FID on Uni-ControlNet and Stable Diffusion 2.1. The best results are in bold.
FID
Depth Content
SD2-depth
17.76
SD2-unclip
24.12
Ours
21.20
23.98
Table 6: CLIP score on Uni-ControlNet and Stable Diffusion 2.1.
The best results are in bold.
CLIP Score
Depth Content
SD2-depph 0.2516
SD2-unclip
0.2497
Ours
0.2561
0.2402
is important note that for SD2-depth and SD2-unclip. the whole model is fine-tuned learn the depth map or the reference images instead of only fine-tuning adapters, which is the key factor contributing to their great performance. Additionally, when compared other controllable diffusion models like ControlNet, GLIGEN and T21-Adapter, SD22-epthh and SD2-unclip outperform them, as demonstrated in Table and Table
G More Quantitative Results
CLIP Score: Besides FID, we also test CLIP score for comparing different controllable diffusion models, and ablating condition injections methods and training strategies. We follow the settings in the Section 4.2 and Section 4.3 in the main paper. The results are shown in the Table[aam Table g respectively. Our model demonstrates superior performance quantitatively across most conditions when compared to existing controllable diffusion models. Moreover, for different condition injection methods and training strategies, our method, along with Training-S2, consistently outperforms other strategies. However, joint fine-tuning in Training-S2 does not yield obvious performance gains in most cases. These finds are consistent with those presented in the Section 4.2 and Section 4.3 of the main paper.
18

---

Village
City
Bizyele
Golden Retriever
Golden Retriever
Tlowers
Condition
511-depth//unc
Durs
502 122-deeh//uccipppp
432-depth/undcip
Figure 14: Comparison between Uni-ControlNet and Stable Diffusion 2.1.
Table CLIP score on different controllable diffusion models. The best results are bold.
Canny
MLSD
HED
Sketch
Pose
Depth
Segmentation
ControlNet 0.2538
0.2481
0,2530
0.2499 0.2572 0.2558
0.2531
0.2352
GLIGEN
0.2493
0.2403
0.2534 0.2526
0.2456
0.2401
T21-Adapter 0.2513
0.2584 0.2608 0.2559
0.2478
0.2366
Ours 0.2539 0.2485 0.2556
0.2542
0.2514 0.2561
0.2540
0.2402
User Study: As FID and CLIP score may be not always consistent with human preference, further conduct user study to quantitatively compare our approach with the baseline methods Control- Net (21), GLIGEN (33 and T21-Adapter [22]. More specifically,
carry out tests in both single and multi-condition settings, with 20 cases for each setting. Each case is evaluated based on three metrics: the quality of generated images, the match with the given text, and the alignment with the given conditions. Users should select the best one for each metric from the generated images of NontrolNet, GLIGEN, T21-Adapter, and
Net We collect responses from 20 users and analyze the total number of votes for each metric under each setting.
The results presented in Figures 15 and 16 It can be seen that, our approach outperforms both Con,rolNet, GLIGEN and A2I-Adapter in the single condition setting, demonstrating clear advantage. Additionally, in the multi-conditee setting, our approach performed significantly better than Multi-ControlNe and CoAdapter.
Generation Quality
Match with Text
Match with Condition
22.05
22.34
21.8%
(01)
(69)
34.5%
[96]
(138)
25.3%
71.1%
[99]
(16)
(91)
Contralees GL A2i-Adagorr
Carmdiiee G.00D
Figure 15: User study of the preference rate for the single condition setting.
H More Visualization Results
In this section, we present additional qualitative results. Figure 17 illustrates the results for
single-condition setting, while Figure shows the results for the multi-conditions setting. Moreover, we demonstrate our performance on more challenging case where there are four conditions, as seen in rows 7-8 of Figure 18
19

---

Table 8: CLIP score on condition injection methods and training strategies. The best results are in bold.
Canny
ML.DD
HED
Sketch
Openpose
Depth
Segmentation
Content
Injection-SI
0.2513
0,2497
0.2518
0.2507
0.2527
0.2525
0.2508
Injection-S2
0.2504
0.2506
0.2518
0.2523
0.2527
0.2544
0.2540
Injection-Se
0.2502
Training-S
0,2506
0,2504
0.2511
0.2510
0.2529
0.2538
0.2526
0.2478
Training-S2
0.2528
0.2504
0.2530
0.2537
0.2533
0.2547
0.2545
0.2421
Ours
0.2528
0.2483
0.2535
0.2539
0.2503
0.2549
0.2522
0.2420
Generation Quality
Match with Text
Match with Condition
12.0%
25.5%
17.0%
20.5%
56
19.0%
67
63.2%
(253)
Cohlaatar
Milll Controlie:
Coldaparr
Mall Contralee
Figure 16: User study of the preference rate for the multi-conditions setting,
I
Adapter Details
We provide the details of our proposed local control adapter and global control adapter in Figure 19 Figure 20 and Figure 21
20

---

Tools on the ground
beautifull butterffly on the bud
Canny
Houses on the street at sunset
Books on the shelf
MLSD
Toddy bears are reading book, sunny day
Spagherti with toasts on the plate
HED
A rea crab on the beach
A tram in an old city
Sketch
A woman is rumning
A man is dancing the street
Openpose
A red apple the books
An elephinnt the forest
Depth
room with sunsfifne
Plowers in the basket, oil painting
Segmentation
Sea and beach
Content
Figure 17: More visual results of Eni-ControlNet for single condition setting.
21

---

A man walking on the street, sumny day
A yellow flower in white plate with a fork and a kinifa
OO
A house on fills, blue sky
cat on the sefa an room, art painting
Local
Local
and balioons
A yhoto of comloy riding horse the
Local
Global
Condition-1
Condition-2
Sample
Condition-1
Condition-2
Sample
A man and his car the mountair, hig moon in the sky
sefa and deer in the forest
Condition-1
Condition-2
Condition-3
Condition-4
Sample
Figure 18: More visual results of Uni-ControlNet for multi-conditions setting.
22

---

Convolutiona Layer
SIUU
Convolutional Layer
Copied Encoder
SIUU
Convolutional Layer
SIUU
Convolutional Layer
Noise
SIUU
Convolutional Layer
Convolutional Layer
SIUU
Zero
Convolutional Layer
Resblock
FDN
Zero
SIUU
Layer
ResBlock
Block-3
Resolution: 64x64
Attention Layer
Zero
Block-4
Down Sample
Zero
Convolutional Layer
ResBlock
FDN
Zero
SIU
Layer
ResBlock
Block
Resolution: 32x32
Attention Layer
Zero
Block-7
Down Sample
Zero
Convolutional Layer
ResBlock
FDN
Zero
SIU
Block-8
Attention Layer
Zero
Block-9
Resblock
Resolution: 16x16
Attention
Zero
Block-10
Down Sample
Zero
Convolutional Layer
Block-11
ResBlock
FDN
Zero
SIUU
Zero
Block-12
Resblock
Zero
Resolution: 8x8
Copied
ResBlock
Feature Extractor
Middle
Attention Layer
Block
ResBlock
Zero
Figure 19: Details of the local control adapter.
Nobs
FDN
Skip
Convolutional Layer
Local Condition Features
SILU
Convolutional Layer
Figure 20: Details of the ResBlock with FDN in local control adapter.
23

---

Glubal tond ldi
LayerNorm
Feedforward Layer-1
GEGLU
Linear
LayerNarm
Feedforward Layer-2
GEGLU
Provectdd Condirion Embendiings (lind)
Reshape
Tokon T (6)
Token 2
Tokio ((()
Figure 21: Details the global control adapter. is the dimension of text token embedding and K the number of the global tokens.
24


Reviews:
[{"review_id": "VPku7Ki8NK", "reviewer": "Reviewer_WFYK", "summary": "The authors proposed Uni-ControlNet, a novel approach that allows for the simultaneous utilization of different local controls and global controls. It uses two additional adapters (local and global) and injects their outputs into the frozen pretrained diffusion models, and only the parameters in adapters need training. \n\nThrough both quantitative and qualitative comparisons, Uni-ControlNet demonstrates its superiority over existing methods in terms of controllability, generation quality, and composability. ", "questions": "1. What's the inference time cost of Uni-controlNet?\n2. How can we extend a trained Uni-ControlNet to other types of control? Do we need to train the adapters with all types of control together from scratch?", "limitations": "see weakness and questions. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. By training with multiple conditions simultaneously, Uni-controlnet is able to perform various kinds of control with only one model. \n2. Uni-controlnet only adds 2 adapters, which is efficient in both training and inference. \n3. By concatenating clip image embedding with text embedding (condition), the method can control the style of generated image. ", "weaknesses": "1. The clarification of the dataset construction is unclear. For example, is the skeleton/sketches generated by model, or manually collected? If it's automatically generated by models, the performance will be bounded by the accuracy of those models, and may suffer from distribution gaps if the control map  are painted by human during inference. Otherwise, it's very hard to anotate such a complex dataset.  \n2. Insufficient ablation study. As mentioned in L14, \"Uni-ControlNet only necessitates a constant number (i.e., 2) of adapters, regardless of the number of local or global controls used.\" Is it because of the structure design? If so, a normal ControlNet with multiple controls trained together should be compared with. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your valuable comments!**\n\n**Q1: How to get the training data of sketches?**\n\n**Answer**: Great question! Indeed, annotating a sketch dataset can be challenging. In our experiment, we initially obtain the HED boundary detection of an image and subsequently utilize a sketch simplification method to generate the sketches for the training samples. Although there are distribution gaps between the hand-drawn sketches and the model-generated sketches, we have observed that our model can handle hand-drawn sketches pretty well (Figure 1 of the rebuttal PDF). Please note that, how to further boost the generation quality by bridging the gap between hand-drawn sketches and model-generated sketches is beyond the scope of this paper.\n\n**Q2: Ablation study on structure design.**\n\n**Answer**: In our ablation study, we extensively investigated the impact of structure design. One aspect we explored is, as you mentioned, training a ControlNet with multiple controls simultaneously, referred to as Injection-S2 in our main paper. However, this design yielded relatively poor results, as illustrated in Figure 7 and Table 3 in the main paper, as well as Table 2 in the supplementary material.\n\nFurthermore, we conducted a comparison with another structure design called \"Injection-S1\", which directly utilizes SPADE. It can be seen that, Injection-S1 produced inferior results compared to both our proposed method and Injection-S2.\n\n**Q3: Inference cost.**\n\n**Answer**: We conducted a test by evaluating 100 samples for each condition and calculated the average inference time per sample:\n\n| | ControlNet | T2I-Adapter | Ours |\n|:---|:---:|:---:|:---:|\n| Seconds | 9.02 | **6.21** | 9.16 |\n\nThe results indicate that the inference cost of ControlNet and Our model is approximately the same, around 9 seconds per sample on average. On the other hand, the T2I-Adapter demonstrates a faster inference time, achieving 6 seconds per sample on average. This can be attributed to the lightweight nature of a single adapter in the T2I-Adapter model.\n\n**Q4: Extending a trained Uni-ControlNet to newly added Conditions.**\n\n**Answer**: Super insightful question! To extend a trained Uni-ControlNet to support new conditions, we conducted an experiment in two steps for comparison & analysis purpose. Firstly, we train a local adapter specific to N conditions. Next, we introduce a new type of condition and extend the trained adapter to (N+1) conditions. The adaptation process involved modifying the input channel of the Uni-ControlNet's first convolutional layer within the feature extractor.  Then, we try to retrain the local adapter with 4 different retraining strategies (R1-4) to accommodate the new conditions:\n\n1. Retraining the entire feature extractor (R1), \n2. Only retraining the pre-feature extractor, which is the part that projects the condition from resolution 512 to 64 (R2),\n3. Only retraining the first convolutional layer in the feature extractor (R3),\n4. Without retraining, i.e., random initialization of the first convolutional layer in the feature extractor (R4).\n\nDuring the retraining process, we ensure that the weights of the copied encoder in the local adapter remain fixed. We utilize a training dataset of 300k samples for the retraining. We show the extension from [MLSD + HED + Sketch + OpenPose + Depth + Seg] to [MLSD + HED + Sketch + OpenPose + Depth + Seg + **Canny**]. The results of this extension process are presented in Figure 2 of the rebuttal PDF. We surprisingly observe that retraining solely the first convolutional layer in the feature extractor can already adequately enables the Uni-ControlNet to handle the newly added conditions. This is a great feature that enables our model to quickly expand to new conditions!"}, {"review_id": "SZZAiCjp0b", "reviewer": "Reviewer_RHVi", "summary": "This paper proposes Uni-ControlNet for the simultaneous utilization of various local controls and global controls within a single model in a flexible and composable manner. This is achieved by fine-tuning of two additional adapters on top of pre-trained text-to-image diffusion models, eliminating the significant cost of training from scratch. ", "questions": "1.\tIt is unclear from the information provided whether the LAION dataset underwent any filtering, such as using OpenPose. Not all the images includes human to detect.\n\n2.\tThe training time and GPU usage are not provided in the given information. These details are important for understanding the computational requirements and resource usage of the models.\n\n3.\tIs the last condition in Figure 5 image condition(global condition)? It is unclear how ControlNet implements this condition. The paper should provide a clear explanation of the condition and how it is implemented in ControlNet to ensure transparency and understanding of the model.\n\n4.\tDuring training, are all seven conditions of each image simultaneously inputted to the network for training, or are some conditions selectively set to empty?\n5.\tThe evaluation of clip scores is not discussed, which is important for text-driven generation.\n\n6. why Feature Denormalization is considered superior to using SPADE (Injection-S1) or ControlNet (Injection-S2) directly? Could you give some explanation?\n", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper propose a new framework that leverages lightweight adapters to enable precise controls in a single model. ", "weaknesses": "1.\tThe training sets for the different models in table 2 are not the same. It raises the question of fairness in comparisons between the models. \n\n2.\tThe author should compare with the simple baseline Multi-controlnet: https://huggingface.co/blog/controlnet.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your valuable comments!**\n\n**Q1: Fairness in comparisons.**\n\n**Answer**: It's worth noting that the training set used for the compared models is not publicly available (we already made the request by email, but no response or data not sharable). Additionally, the specific training settings for their released models are not fully disclosed as well, which makes it hard to guarantee absolute fairness in the comparison. To minimize this gap, we conducted experiments to ablate the model design between our method and other models in Section 4.3 in the main paper. Through these control-experiments, we were able to demonstrate the effectiveness of our method.\n\n\n**Q2: Comparison with Multi-ControlNet.**\n\n**Answer**: Thanks for your suggestions!  Please refer to the common Q2.\n\n\n**Q3: The construction of the training set.**\n\n**Answer**: We did not employ any filtering method for the LAION dataset, as stated in line 186 in the main paper, and used a random subset of the dataset for training. As also mentioned in line 178-180 in the main paper, we adopted a dropout strategy for conditions during training. \n\nRegarding the pose condition, it is true that not all images in the dataset include humans. To ensure that the pose condition is fully trained, we opted to not drop the pose condition during training. This means that we always keep the pose condition if it is available, allowing the model to learn the full range of pose information. In the future, if there is only a very small portion of data existing for some special conditions, advanced resampling strategies may be needed to guarantee balance and enough training.\n\n\n**Q4: Training time and GPU usage.**\n\n**Answer**: Good question! We train our model by using 64 NVIDIA Tesla 32G-V100 GPUs. We trained on 10 million data for 1 epoch, with a batch size of 192 for the local adapter and 256 for the global adapter. It took approximately 3 days to train the local adapter, and around 1.5 days to train the global adapter. However, as illustrated in Section 4.3 in our main paper, training on 1 million data for 1 epoch is sufficient to achieve great results.\n\n\n**Q5: How does ControlNet implement the content condition?**\n\n**Answer**: Yes, the last condition in Figure 5 of the main paper represents the image condition. ControlNet v1.1 achieves control through content, using an image-to-image method that differs from our embedding-to-image approach. To implement this, they first shuffle the content by remapping the image based on a random flow and then use the shuffled content to control the generation process.\n\n\n**Q6: The keep and drop of conditions during training.**\n\n**Answer**: As mentioned in line 149 in our main paper, we concatenate all the local conditions along the channel dimension to enable simultaneous use of different conditions. During training, as stated in line 178-180 in the main paper, we adopt a predefined probability to randomly drop each condition, along with an additional probability to deliberately keep or drop all conditions. For the dropped conditions, we set the value of the corresponding input channels to 0. \n\n\n**Q7: Evaluation on CLIP score.**\n\n**Answer**: Due to space constraints in the main paper, we have provided the evaluation results in terms of CLIP score in Table 1 and Table 2 of the supplementary material. In addition to the FID and CLIP score, we have also conducted a user study to evaluate the results, which can be found in Figure 3 and Figure 4 of the supplementary material. These evaluation measures provide a comprehensive assessment of the performance of our model.\n\n\n**Q8: Why is FDN better than using SPADE or ControlNet directly?**\n\n**Answer**: It is a great question! Directly using SPADE to inject conditions involves resizing the conditions to the corresponding resolutions using interpolation which we call Injection-S1 in our main paper. This direct interpolation significantly destroys condition information, leading to poor alignment with the conditions, as illustrated in Figure 7, Table 3 of the main paper. Additionally, when directly resizing the conditions and sending them to the model, the resized conditions cannot align well with the latent space where they were injected. In contrast, our FDN employs a multi-scale injection strategy that provides condition information at different levels, resulting in richer condition information. Furthermore, our feature extractor projects the conditions to the corresponding latent spaces of different layers, which allows for better alignment between the conditions and noise features.\n\nWhen using ControlNet or T2I-Adapter directly, which only provide condition information in the input layer of the adapter (which we refer to as Injection-S2 in our main paper), the model may lose some information of the conditions in deeper layers. This is illustrated in Figure 7 of the main paper, where Injection-S2 cannot handle composite controls effectively, and the generated samples do not align well with the combined conditions, or the conditions are not well merged. \n\nIn contrast, our FDN method provides multi-scale injection, which allows for better preservation of condition information and alleviates the condition forgetting issue in deeper layers. We demonstrate that this design results in superior performance in handling composite controls within a unified framework. \n"}, {"review_id": "H8bO3GirD8", "reviewer": "Reviewer_WHfG", "summary": "This paper proposed a method to do controlleble t2i generation from a pretrained diffusion model. The main contribution is that they only have two adapters one local (e.g., edge map, keypoint etc) and one global (e.g., image). For local, they use the controlnet, but concatenate conditions as input. For global, they extract image feature and treat it as text tokens. \n\n", "questions": "NA", "limitations": "See weakness \n\n\n\n\n\n====================================================================================\n\nThey addressed my main concern which is they only evaluated image quality, but not controllability.\nI strongly encourage them to add results table in the rebuttal to their paper, thus will be served as a baseline for future controllable image generation work. \n\nBased on this point, I am willing to raise my score despite that I feel novelty is a bit weak   ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea is simple and writing is clear ", "weaknesses": "There are several weakness for this paper:\n\n1, the technique novelty is incremental. Although, they tried something such as SPADE like injecting information for local branch and combine image and text tokens for global etc, but they are very straightforward. \n\n2, missing baseline GLIGEN [Li et al, CVPR, 2023] which also supports conditions studied in this paper. \n\n3, one more weakness for this paper is missing evaluation for condition correspondence. They only reported FID as a metric, which only reflects image quality. But they should also study how well the generated images are corresponded with input conditions. For example, in GLIGEN, they use mask-rcnn to detect keypoints from generated images and compare with input keypoint, so that we can know how well the model following the input. I understand that for certain conditions such as edge map, maybe it is hard to evaluate, but as least for keypoint, semantic map, depth map, it is easy to come up with some metrics.\n\n \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your valuable comments**\n\n**Q1. The technique novelty.**\n\n**Answer**: Thanks for your suggestions! Please refer to the common Q1.\n\n\n**Q2: The comparison with the GLIGEN.**\n\n**Answer**:  Thanks for your suggestions! GLIGEN is an excellent paper that introduces a model conditioned on bounding boxes with caption groundings. It also explores alternative forms of grounding, such as edge maps and pose information. While our original paper does not consider GLIGEN as a baseline due to its primary focus on bounding boxes condition and lack of support for composable control, comparing our method to GLIGEN can enhance the comprehensiveness of our study and show performance differences.\n\nTo facilitate its incorporation into the final version, we directly utilize the samples presented in Figure 5 of the main paper. We showcase the qualitative comparison results in Figure 4 of the rebuttal PDF. From the results, we can easily observe some shortcomings in GLIGEN's output. For example, the detail of the deer's face is not very good, the depiction of the forest lacks realism, and the sky is missing in the case of segmentation map condition.\n\nWe further conduct the quantitative comparison in terms of FID and CLIP Score:\n\n| FID | Canny | MLSD | HED | Sketch | Pose |Depth | Segmentation | Content |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| GLIGEN | 24.74 | / | 28.57 | / | **24.57** | 21.46 | 27.39 | 25.12 |\n| Ours | **17.79** | **26.18** | **17.86** | **20.11** | 26.61 | **21.20** | **23.40** | **23.98** |\n\n| CLIP score | Canny | MLSD | HED | Sketch | Pose |Depth | Segmentation | Content |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| GLIGEN | 0.2493 | / | 0.2403 | / | **0.2534** | 0.2526 | 0.2456 | 0.2401 |\n| Ours | **0.2539** | **0.2485** | **0.2556** | **0.2542** | 0.2514 | **0.2561** | **0.2540** | **0.2402** |\n\nTo evaluate the perception quality, we further conduct a user study, following the settings in Section 3 - User Study in the supplementary material.\n\n| | Generation Quality | Match with Text | Match with Condition |\n|:---|:---:|:---:|:---:|\n| GLIGEN | 30.3% (121) | 44.2% (168) | 29.5% (118) |\n| Ours | **69.7% (279)** | **55.8% (212)** | **70.5% (282)** |\n\nWe can find that our method outperforms GLIGEN in nearly all evaluations.\n\n\n**Q3: Evaluation of the controllability.**\n\n**Answer**: Really great question! How to evaluation of the controllability is a common and important problem in controllable diffusion models. And we believe user study is the most accurate way to evaluate the controllability from the user perception view, and we already include this metric during the user study (provided in supplementary material). However, following your suggestion, we also try some automatic controllability evaluation metrics. Please refer to the common Q3."}, {"review_id": "WINITE6KfA", "reviewer": "Reviewer_n8So", "summary": "This paper proposes Uni-ControlNet that leverages lightweight local and global adapters to enable precise controls over pre-trained T2I diffusion models. ", "questions": "See Weakness.", "limitations": "The authors have adequately addressed the limitations. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written and organized.\n2. The idea of local/global adapter to achieve all-in-one control is reasonable and interesting.\n3. The results seem good.\n\n", "weaknesses": "Since this paper is easy to follow and self-consistent, I have only a few minor questions:\n\n1. Please compare the training costs of Uni-ControlNet with those of other methods (T2IAdapter, ControlNet).\n2. I notice that the different conditions are concatenated as inputs to the adapter. If we want to add other control conditions, does the adapter need to be retrained?\n\nOverall, although this work has some limitations, I think it meets the bar of NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Comparing the training cost of Uni-ControlNet with other methods.**\n\n**Answer**: Great suggestions! Since the scale of the training set and training epochs varies across different methods, we present the time cost of a single training step as a measure. The reported result represents the average time cost across different conditions for each method:\n\n| | ControlNet | T2I-Adapter | Ours |\n|:---|:---:|:---:|:---:|\n| Seconds | 5.01 | **3.73** | 5.16 |\n\nThe results show that both ControlNet and Our model have similar training costs, with an average of around 5 seconds per training step. In contrast, the T2I-Adapter exhibits a lower training cost, which can be attributed to the lightweight nature of a single adapter within the T2I-Adapter model.\n\nIt is important to note that, regardless of the number (N) of conditions, we only need to train a unified single model. However, for ControlNet and T2I-Adapter, their training cost will increase linearly with N, as they require training a dedicated model for each specific condition.\n\n\n**Q2: Extending a trained Uni-ControlNet to newly added Conditions.** \n\n**Answer**: Super insightful question! To extend a trained Uni-ControlNet to support new conditions, we conducted an experiment in two steps for comparison & analysis purpose. Firstly, we train a local adapter specific to N conditions. Next, we introduce a new type of condition and extend the trained adapter to (N+1) conditions. The adaptation process involved modifying the input channel of the Uni-ControlNet's first convolutional layer within the feature extractor.  Then, we try to retrain the local adapter with 4 different retraining strategies (R1-4) to accommodate the new conditions:\n\n1. Retraining the entire feature extractor (R1), \n2. Only retraining the pre-feature extractor, which is the part that projects the condition from resolution 512 to 64 (R2),\n3. Only retraining the first convolutional layer in the feature extractor (R3),\n4. Without retraining, i.e., random initialization of the first convolutional layer in the feature extractor (R4).\n\nDuring the retraining process, we ensure that the weights of the copied encoder in the local adapter remain fixed. We utilize a training dataset of 300k samples for the retraining. We show the extension from [MLSD + HED + Sketch + OpenPose + Depth + Seg] to [MLSD + HED + Sketch + OpenPose + Depth + Seg + **Canny**]. The results of this extension process are presented in Figure 2 of the rebuttal PDF. We surprisingly observe that retraining solely the first convolutional layer in the feature extractor can already adequately enables the Uni-ControlNet to handle the newly added conditions. This is a great feature that enables our model to quickly expand to new conditions!"}, {"review_id": "vzIY7pFg2h", "reviewer": "Reviewer_MFNW", "summary": "This paper presents Uni-ControlNet, a model that aims to enhance Text-to-Image diffusion techniques by allowing the concurrent use of multiple local and global controls. It only requires two additional adapters, regardless of the number of controls used, and circumvents the necessity of training from scratch. The authors assert that Uni-ControlNet performs favorably in terms of controllability, generation quality, and composability.  ", "questions": "-\tHave you considered introducing a metric that specifically evaluates the controllability of Uni-ControlNet? FID primarily assesses image quality, but it doesn't provide a measure for controllability.", "limitations": "yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method allows for the simultaneous utilization of different local and global controls within one model, making it flexible and composable.\n\n- It eliminates the need for training from scratch, reducing costs and making it suitable for real-world deployment.", "weaknesses": "-\tLimited Novelty and Contribution: The paper's primary contributions are its condition injection strategy and training approach. However, the condition injection strategy appears to be derived from SPADE, and the training strategy seems to be based primarily on empirical evidence, without providing much novel insight or theoretical explanation.\n\n-\tInsufficient Detail in Discussion: The description of the training strategy and the inference process, stated as major contributions, are not sufficiently clear. It remains unclear how the authors handle other conditions when only one condition is being utilized. It seems problematic to set the local conditions' values to zero with the intent of rendering them empty. Moreover, it seems that Uni-ControlNet cannot handle multiple conditions of the same type, as suggested in Figure 2 of the Supplementary Materials.\n\n-\tIncomplete Comparisons in Experiments: The experimental comparison does not seem comprehensive. For instance, it's known that Stable Diffusion 2.1 unclip can also accept CLIP image embeddings as inputs, like global condition in this paper. Additionally, ControlNet can accommodate multiple conditions. Furthermore, the quantitative results do not appear to be particularly impressive - with the method achieving the best result in only 4 out of 8 instances in Table 2, and only 2 out of 8 in Table 1.  Same for CLIP score in Supplementary Materials.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your valuable comments!**\n\n**Q1. Novelty and Contribution.**\n\n**Answer**: Thanks for your comments!  Please refer to the common Q1.\n\n\n**Q2: Training strategy and inference process.**\n\n**Answer**: As mentioned in line 149 of our main paper, we concatenate all the local conditions along the channel dimension to enable simultaneous use of different conditions. During training, as stated in line 178-180 in the main paper, we adopt a predefined probability to randomly drop each condition, along with an additional probability to deliberately keep or drop all conditions. For the dropped conditions, we set the value of the corresponding input channels to 0. This simple strategy has been shown to be very effective, because the feature extractor in the local adapter can learn to understand the user intention by the supervision loss during training. \n\n\n**Q3: How to achieve composite control of 2 conditions with the same type.**\n\n**Answer**: Really good point! In real-world applications, user can actually composite two/multiple conditions of the same type easily before feeding them to the model, e.g., draw the sketch of multiple objects in one canvas. However, considering it as an interesting research point, we try one simple strategy called \"Uni-Channels\". Specifically, we augment the input by adding three extra condition channels. For instance, if the original inputs had 21 channels (3 for each condition, totaling 7 local conditions), with Uni-Channels, we now have 21 + 3 channels for the inputs.\n\nDuring training, we feed the Uni-Channels with randomly selected types of conditions of the input natural images. We observe that, as the shared extra condition channels for different condition types, Uni-Channels can perform well for two-condition composition of the same condition type. The visualization results are depicted in Figure 5 of the rebuttal PDF.\n\n\n**Q4: Comparison with Stable Diffusion 2.1.**\n\n**Answer**: Thanks for your suggestions! We compare our method with 2 Stable Diffusion models, Stable Diffusion 2 - depth (\"SD2-depth\") and Stable Diffusion 2 - unclip (\"SD2-unclip\") which support the inputs of depth map and reference image respectively. The visualization results are shown in Figure 6 in the rebuttal PDF. Additionally, we provide the quantitative results below:\n\n| FID | Depth | Content |\n|:---|:---:|:---:|\n| SD2-depth | **17.76** | / |\n| SD2-unclip | / | 24.12 |\n| Ours | 21.20| **23.98** |\n\n| CLIP score | Depth | Content |\n|:---|:---:|:---:|\n| SD2-depth | 0.2516 | / |\n| SD2-unclip | / | **0.2497** |\n| Ours | **0.2561** | 0.2402 |\n\nIt is important to note that for SD2-depth and SD2-unclip, the whole model is fine-tuned to learn the depth map or the reference images instead of only fine-tuning adapters, which is the key factor contributing to their great performance. Additionally, when compared to other controllable diffusion models like T2I-Adapter and ControlNet, SD2-depth and SD2-unclip outperform them, as demonstrated in Table 2 of the main paper and Table 1 of the supplementary material.\n\n\n**Q5: Comparison with Multi-ControlNet.**\n\n**Answer**: Please refer to the common Q2.\n\n\n**Q6: Quantitative results are not particularly impressive?**\n\n**Answer**: Thanks for your valuable comments, but we respectively disagree with this point. Regarding the quantitative results of the comparison with other methods presented in Table 2 in the main paper, our approach achieves the best performance in 6 out of 8 metrics (or 4 out of 6, considering that T2I-Adapter does not consider MLSD and HED) even with one unified single model. In the ablation study, it is important to note that Training-S2 is also our method. Training-S2 involves additional joint fine-tuning after our separate fine-tuning. Therefore, in Table 3, we achieve the best results in 7 out of 8 metrics. The CLIP score provided in the supplementary file also reflects our great performance. \n\nFurthermore, our method demonstrates good results in the user study in the supplementary material, which provides reliable and straightforward indications of our superior performance from the user perception perspective.\n\n\n**Q7:Evaluation of the controllability.**\n\n**Answer**: Really great question! How to evaluation of the controllability is a common and important problem in controllable diffusion models. And we believe user study is the most accurate way to evaluate the controllability from the user perception view, and we already include this metric during the user study (provided in supplementary material). However, following your suggestion, we also try some automatic controllability evaluation metrics. Please refer to the common Q3."}]

Author rebuttal:
**We would like to thank all the reviewers for the valuable feedback!** Here we first address some common questions.

**Q1. Re-clarification of Contribution and Novelty.**

**Answer**:  We would like to emphasize that our primary contribution is proposing a new unified controllable diffusion model that can not only handle different conditions within **one single model** but also supports composable control, as illustrated in Table 1 in the main paper. By contrast, existing methods fail to achieve this unified framework within one single model. Besides, even for those methods that support composite control, their composability is much worse than ours. Through extensive qualitative and quantitative evaluations, our method demonstrates even overall better results with a unified single model thanks to our newly proposed designs.

From the technique side, we have explored some existing techniques and found they are inadequate for achieving our goal of a unified framework. For example, the injection strategy employed by ControlNet and T2I-Adapter is insufficient, as it will suffer from information loss. Similarly, directly using SPADE will also result in poor performance, as resizing condition features through direct interpolation to low resolutions causes significant loss of condition information. Therefore, we develop a new multi-scale injection strategy through FDN to achieve better alignment between condition features and latent noise features across different layers of the local adapter, which is not explored in previous methods.

**Q2: Comparison with Multi-ControlNet.**

**Answer**: Great suggestion and thanks for the reminder! We want to explain that we missed the comparison with Multi-Controlnet because it is not included in the original ControlNet paper. However, we acknowledge the importance of comparing our method with Multi-ControlNet. To facilitate its incorporation into the final version, we utilize the samples presented in Figure 6 of the main paper and present the comparison results in Figure 3 of the rebuttal PDF.  Our observations are that Multi-ControlNet has much worse composability. For example, similar to T2I-Adapter, it misses the podium and the car in the first 2 samples in Figure 3 of the rebuttal PDF respectively. In addition, the composite generation of a local condition and a global condition is also not that good.

As Multi-ControlNet is designed for composite control, we conducted a similar user study  evaluation following the settings in Section 3 - User Study of the supplementary material. The results are presented below:

| | Generation Quality | Match with Text | Match with Condition |
|:---|:---:|:---:|:---:|
| Multi-ControlNet | 27.0% (108) | 39.5% (79) | 10.8% (43) |
| Ours | **73.0% (292)** | **60.5% (121)** | **89.2% (357)** |

It can be seen that our method demonstrates a clear advantage compared to the Multi-ControlNet on all the three metrics in the user study.

**Q3:Evaluation of the controllability.**

**Answer**: Really great question! Evaluating the controllability of different methods is a crucial aspect of the controllable diffusion models. We firmly believe that human perception provides the most effective and accurate measure in this regard, especially for the multi-condition scenarios. Therefore, we conducted a user study that encompassed both single condition and multi-condition controls, allowing participants to select the results they deemed to best match the given conditions. The results of our ablation study can be found in the supplementary material, specifically in Figure 3 and Figure 4.

While human perception serves as the most important evaluation metric, we also recognize the importance of utilizing other quantitative metrics as the auxiliary metrics to assess the controllability. Following the reviewers' suggestion, we employed the following metrics for single-condition generation:

1. SSIM (Structural Similarity) for Canny, HED, MLSD, and sketch conditions.
2. mAP (mean Average Precision) based on OKS (Object Keypoint Similarity) for pose condition.
3. MSE (Mean Squared Error) for Depth map.
4. mIoU (Mean Intersection over Union) for segmentation map.
5. CLIP score for content condition.

To calculate these metrics, we compare the extracted conditions from the natural image (the ground truth) and the corresponding generated image. We follow the settings in Section 4.2 Comparison with Existing Methods - Quantitative Comparison of the main paper and here are the results:

|  | Canny-SSIM | MLSD-SSIM | HED-SSIM | Sketch-SSIM | Pose-mAP |Depth-MSE | Segmentation-mIoU | Content-CLIP score |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| ControlNet | 0.4828 | **0.7455** | 0.4719 | 0.3657 | 0.4359 | **87.57** | **0.4431** | 0.6765 |
| GLIGEN | 0.4226 | / | 0.4015 | / | 0.1677 | 88.22 | 0.2557 | 0.7458 |
| T2I-Adapter | 0.4422 | / | / | 0.5148 | **0.5283** | 89.82 | 0.2406 | 0.7078 |
| Ours | **0.4911** | 0.6773 | **0.5197** | **0.5923** | 0.2164 | 91.05 | 0.3160 | **0.7753** |

Our method outperforms other baseline methods in 4 out of 8 evaluation metrics. Notably, ControlNet achieves the best performance in 3 out of 8 metrics, while T2I-Adapter only excels in 1 out of 8 metrics. However, it should be noted that such methods employ different models for different conditions, allowing each model to be well-trained for its corresponding condition. In contrast, we only use a single model and achieved even overall superior results.

By utilizing both human perception (user study) and quantitative metrics, we aim to provide a comprehensive evaluation of the controllability achieved by our method and enable a thorough understanding of its performance.

Meta-review: This paper was reviewed by five knowledgeable referees. The initial concerns raised by the reviewers included (1) the novelty of the proposed approach, which was perceived as incremental (WHfG, MFNW); (2) the experimental evidence, which was considered insufficient (missing ablations and comparisons, extensions to new controls unclear, fairness of the comparisons, quantitative evaluation based on FID) (WFYK, RHVi, WHfG, n8So, MFNW); (3) the dataset collection, which was not sufficiently clear (WFYK). The reviewers also raised questions about the inference cost of the proposed approach (WFYK). The rebuttal adequately addressed the reviewers' concerns by reporting conditioning-generation consistency metrics, by emphasizing the contribution, adding the requested baseline comparisons, discussing the inference cost, and clarifying the extension of the proposed approach to new controls. The reviewers acknowledge the effort that the authors put in the rebuttal and discussion and unanimously lean towards acceptance. The AC agrees with the reviewers' assessment and therefore recommends to accept. The AC encourages the authors to include the results from the rebuttal in the final version of the manuscript. The AC also encourages the authors to add results on COCO2014 validation set in the final version of the paper -- COCO2017 validation set is small and FID is sensitive to the number of samples (see results reported by GLIGEN).
Decision: Accept (poster)