SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Quantitative Comparison: For quantitative evaluation, we use the validation set of COCO2017 [57] at a resolution of  $512 \times 512$ . Since this set contains 5k images, and each image has multiple captions, we randomly select one caption per image resulting in 5k generated images for our evaluation. It is important to note that for quantitative comparison, we limit our testing to different single conditions only. Additionally, we use Style\Content to represent the global condition as there are different settings in the ControlNet, GLIGEN and T2I-Adapter. For the ControlNet, the content condition refers to the content shuffle in ControlNet-V1.1. As T2I-Adapter does not take the MLSD and HED conditions into account, it has no results for MLSD and HED. Similarly, GLIGEN does not consider the MLSD and sketch conditions, resulting in the absence of results for MLSD and sketch.
To evaluate the generation quality, We report the FID [58] in Table 2. We can find that our model reveals superior performance across most conditions quantitatively compared to existing approaches. We also use quantitative metrics to assess the controllability. We employed the following metrics for single-condition generation:
- SSIM (Structural Similarity) for Canny, HED, MLSD, and sketch conditions,
- mAP (mean Average Precision) based on OKS (Object Keypoint Similarity) for pose condition,
- MSE (Mean Squared Error) for depth map,
- mIoU (Mean Intersection over Union) for segmentation map,
- CLIP score for content condition.
Table 2: FID on different controllable diffusion models. The best results are in bold.
Canny	MLSD	HED	Sketch	Pose	Depth	Segmentation	Style\Content
ControlNet	18.90	31.36	26.59	22.19	27.84	21.25	23.08	31.17
GLIGEN	24.74	-	28.57	-	24.57	21.46	27.39	25.12
T2I-Adapter	18.98	-	-	18.83	29.57	21.35	23.84	28.86
Ours	17.79	26.18	17.86	20.11	26.61	21.20	23.40	23.98
Table 3: Quantitative evaluation of the controllability. The best results are in bold.
Canny(SSIM)	MLSD(SSIM)	HED(SSIM)	Sketch(SSIM)	Pose(mAP)	Depth(MSE)	Segmentation(mIoU)	Style\Content(CLIP Score)
ControlNet	0.4828	0.7455	0.4719	0.3657	0.4359	87.57	0.4431	0.6765
GLIGEN	0.4226	-	0.4015	-	0.1677	88.22	0.2557	0.7458
T2I-Adapter	0.4422	-	-	0.5148	0.5283	89.82	0.2406	0.7078
Ours	0.4911	0.6773	0.5197	0.5923	0.2164	91.05	0.3160	0.7753
To calculate these metrics, we compare the extracted conditions from the natural image (the ground truth) and the corresponding generated image. And we report the results in Table 3. Our method outperforms other baseline methods in 4 out of 8 evaluation metrics. Notably, ControlNet achieves the best performance in 3 out of 8 metrics, while T2I-Adapter only excels in 1 out of 8 metrics. However, it should be noted that all of ControlNet, GLIGEN and T2I-Adapter employ different models for different conditions, allowing each model to be well-trained for its corresponding condition. In contrast, we only use a single model and achieved even overall superior results.
To provide a more comprehensive comparison of various controllable models, we also include a comparison on CLIP score in Table 7 and present the results of user studies in Section G in the appendix.
Qualitative Comparison: We further provide qualitative comparison of single and composed multi-conditions in Figure 5 and Figure 6 respectively. For single conditions, as GLIGEN not considering the sketch condition, we use GLIGEN's results on the HED boundary as the second case in the first row for showcase. We find that our Uni-ControlNet, ControlNet, GLIGEN and T2I-Adapter can all perform overall well in single condition setting, and our results show slightly better alignments with input conditions. Notably, we only fine-tune 2 adapters for all conditions, whereas ControlNet, GLIGEN and T2I-Adapter fine-tune eight adapters for eight different single conditions.
Since GLIGEN does not support composed multi-conditions, we only compare Uni-ControlNet with Multi-ControlNet and CoAdapter under the multi-condition setting. As shown in Figure 6, Multi-ControlNet and CoAdapter show poorer composability when dealing with two local conditions, e.g., missing the podium in the first example and no car in the second example. In contrast, our model can fuse the two conditions much better. As for composing a local condition with a global condition, Multi-ControlNet is also not that good as shown in the second row in Figure 6. And CoAdapter performs okay in the case of combing a sketch of a cup and a global condition of a cat. However, when the two conditions are not that related, e.g., the example where there is a Canny edge of a Minion and a global condition of a bus in London, the image generated by CoAdapter appears to be unrealistic and the two elements are not well integrated. And our model effectively creates a Minion-shaped bus with car windows and vivid background.
4.3 Ablation Analysis
For ablation study, we fine-tune our model using a smaller dataset for resource consideration. In detail, we utilize the 1 million subset of the 10 million dataset and fine-tune a single epoch, while keeping all other settings unchanged.
Condition Injection Strategy: For local conditions, we compare our proposed injection method with two other strategies. The first strategy is to directly use SPADE to inject the conditions, which involves resizing the conditions to the corresponding resolutions using interpolation. We call this Injection-S1. The second strategy is similar to Composer, ControlNet and T2I-Adapter, where the conditions are only sent to the adapter or the main model at the input layer, which we refer to as Injection-S2. When using these two strategies, all other parts of our method will remain unchanged. We follow the setting in Section 4.2 and evaluate the FID on different local condition injection strategies. The quantitative
Figure 5: Comparison of existing controllable diffusion models on different single conditions. Table 4: FID on condition injection methods and training strategies. The best results are in bold.
Canny	MLSD	HED	Sketch	Openpose	Depth	Segmentation	Content
Injection-S1	25.89	27.22	22.48	23.51	27.89	24.71	26.25	-
Injection-S2	22.22	27.08	21.94	22.74	26.56	24.21	24.42	-
Injection-S3	-	-	-	-	-	-	-	27.06
Training-S1	21.21	27.20	20.78	23.22	27.83	25.01	24.99	28.51
Training-S2	18.80	26.40	19.12	20.91	27.17	21.59	23.93	24.84
Ours	18.24	26.91	18.61	20.32	27.76	21.97	23.51	24.86
and qualitative results are presented in Table 4 and the upper part of Figure 7. The quantitative results show our proposed condition injection strategy performs better under most settings. For the visual results, we observe that for Injection-S1, the alignment with the conditions is poor. This may be because direct interpolation significantly destroys condition information. As for Injection-S2, it renders unsatisfactory results for composite control. For instance, in the "An elephant in the temple" case, the lanterns on the top of the image are not accurately aligned with depth condition. Moreover, the composite results are not as harmonious as ours. For example, in the "Gorilla wearing glasses" case, the gorilla's eyes and glasses are not well-merged. This may be because that if the condition information is only provided in the input layer of the adapter, the model may lose some information of the conditions in deeper layers, leading to poor alignment among the combined controls. In contrast, our proposed FDN employs a multi-scale injection strategy that provides condition information at different levels, resulting in richer condition information. Furthermore, our feature extractor projects the conditions to the corresponding latent spaces of different layers, which allows for better alignment between the conditions and noise features.
For the global condition, we compare our method to one way in which we only add the global condition into control adapter but not the main SD model, and we denote this injection strategy as Injection-S3. As shown in the lower part of Figure 7, without using the extended prompt in the main SD model, this method cannot inject the global condition into the final generated results.
Figure 6: Comparison of different controllable diffusion models on composable multi-conditions.
Figure 7: Ablation results on different condition injection strategies.
Figure 8: Ablation results on different training strategies.
Training Strategy: As above mentioned, we fine-tune the local and global control adapters separately and merge them at inference without any further joint fine-tuning by default. Here, we also investigate two alternative training strategies: 1) joint fine-tuning together ("Train-S1"), where we fine-tune both adapters together from scratch; 2) further joint fine-tuning after separate fine-tuning ("Train-S2"), where we further fine-tune the adapters together after separate fine-tuning.
The quantitative FID results are shown in Table 4. We find that our default strategy and Training-S2 perform much better consistently than Training-S1, but further joint fine-tuning in Train-S2 does not bring obvious performance gain in most cases. Some visual results are given in Figure 8. Note that, in order to better assess the controllability of the global condition, we do not provide text prompts for the cases with global condition. As we described before, the reason why Training-S1 gets poor controllability on the global condition is that the global control adapter does not learn as much as local adapter even equally treated during joint fine-tuning. One possible explanation is that the local conditions often contain more rich guidance information than global conditions, leading the model to pay less attention to the global condition.
5 Conclusion and Social Impact
In this paper, we propose Uni-ControlNet, a new solution that enhances the capabilities of text-to-image diffusion models by enabling efficient integration of diverse local and global controls. With better adapter designs, our Uni-ControlNet only requires two adapters for different conditions while existing methods often require independent adapters for each condition. The new design of Uni-ControlNet not only saves both fine-tuning cost and model size, but also facilitates composability, allowing for the simultaneous utilization of multiple conditions. Extensive experiments validate the effectiveness of Uni-ControlNet, showcasing its improved controllability, generation fidelity, and composability. While our system empowers artists, designers, and content creators to realize their creative visions with precise control, it is crucial to acknowledge the potential negative social impact that can arise from misuse or abuse, similar to other image generation and editing AI models. To address these concerns, responsible deployment practices, ethical regulations, and the inclusion of special flags in generated images to enhance transparency are vital steps towards responsible usage.
[57] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dálár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV 2014.
[58] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS 2017.
A The Weight of the Global Condition
In the global control module, we have implemented a hyper-parameter  $\lambda$ . This hyper-parameter plays a role in determining the influence of the global condition while concatenating the projected global condition to the text. Illustrating the effect of varying  $\lambda$ , we present two representative visualization results in Figure 9. It can be seen that the value of  $\lambda$  plays a significant role in displaying the elements of the global conditions in the generated images. As the value of  $\lambda$  increases, the global condition takes precedence over the original text content, leading to a decrease in the influence of the text prompt on the results. For instance, in the first case, it appears that the forest has experienced a reduction in coverage area as compared to an increase in the number of houses with an increase in the value of  $\lambda$ . Similarly, in the second case, it is evident that the city is shrinking while the desert's coverage is expanding with the rise of  $\lambda$  value. In the real-world applications, we can adjust the hyper-parameter  $\lambda$  to generate our desired results with flexibility.
Figure 9: The effect of the hyper-parameter  $\lambda$ . On the left side are the textual prompts and global conditions provided. On the right side are the images generated under increased  $\lambda$  values.
B Condition Conflicts
Since our Uni-ControlNet can support multiple conditions simultaneously, we are curious about its behavior when providing multiple conflicting conditions. Need to that, this is very rare in the real-world applications, and this experiment is just for the analysis purpose. For example, we consider the case of providing the model with two local conditions that are fundamentally incompatible, such as the conditions of two dogs shown in Figure 10. Through this experiment, we can possibly evaluate the relative importance of each condition and learn how Uni-ControlNet resolves conflicts, which may help us design more robust integration of conditions that can adequately handle diverse and ambiguous situations.
To provide a comprehensive analysis of different condition compositions, we have assigned each condition in the first column a number 1 and each condition in the first row a number 2. This allows us to refer to the dog in the first row as dog-1 and the dog in the first line as dog-2. The results depicted in Figure 10 demonstrate that HED is the most powerful condition, with generated images closely following the HED boundary when depicting text. Other conditions can only influence areas that do not overlap. For instance, when we combine the HED boundary of dog-2 with the Canny edge map of dog-1, the resulting image adopts the HED boundary of dog-2 but recognizes the head of dog-1 as a small element positioned near the head of dog-2. Similarly, when dog-2's HED boundary is combined with the sketch of dog-1, the model fails to identify the head of dog-1 even though it does not conflict with the HED boundary. Among the other conditions, the Canny edge map is the next most powerful, followed by the sketch, depth, MLSD, and segmentation map. The Openpose condition is the weakest, whereby the model generally disregards it in the event of a conflict. Only when combined with the segmentation map, the Openpose condition produces recognizable human elements. For better visualization, we have reordered the conditions based on their strength, which implies that the upper and left conditions have greater influence.
Figure 10: Study for the cases where composed conditions contradict each other. We choose the detection results of a room as the condition for MLSD and a man as the condition for Openpose. To test other conditions, we select two different dogs, which allows us to observe the model's output when given different dog-shaped conditions. We use "room" as the prompt for MLSD, "man" for Openpose, and "dog" for other conditions. When combining two types of conditions, we integrate their prompts, such as "dog", "dog and room", and "dog and man".
C Hand-drawn Sketches
For the sketch condition, as mentioned in the main paper, we first get the HED boundary [45] of the training images. Then, we employ a sketch simplification method [46, 47] to generate the sketches for the model training. One question is, how does our model perform on hand-drawn sketches? We show the results of our model on hand-drawn sketches in Figure 11. We can find that although there are distribution gaps between the hand-drawn sketches and the model-generated sketches, our model can handle hand-drawn sketches pretty well.
Figure 11: Visualization results on hand-drawn sketches.
D Extension to New Conditions
To extend a trained Uni-ControlNet to support new conditions, we conducted an experiment in two steps for comparison and analysis purpose. Firstly, we train a local adapter specific to N conditions. Next, we introduce a new type of condition and extend the trained adapter to  $(\mathrm{N} + 1)$  conditions. The adaptation process involved modifying the input channel of the Uni-ControlNet's first convolutional layer within the feature extractor. Then, we try to retrain the local adapter with 4 different retraining strategies (R1-4) to accommodate the new conditions:
- Retraining the entire feature extractor (R1),
- Only retraining the pre-feature extractor, which is the part that projects the condition from resolution 512 to 64 (R2),
- Only retraining the first convolutional layer in the feature extractor (R3),
- Without retraining, i.e., random initialization of the first convolutional layer in the feature extractor (R4).
During the retraining process, we ensure that the weights of the copied encoder in the local adapter remain fixed. We utilize a training dataset of 300k samples for the retraining. We show the extension from  $\mathrm{[MLSD + HED + Sketch + OpenPose + Depth + Seg]}$  to  $\mathrm{[MLSD + HED + Sketch + OpenPose + Depth + Seg + Canny]}$ . The results of this extension process are presented in Figure 12. We surprisingly observe that retraining solely the first convolutional layer in the feature extractor can already adequately enable the Uni-ControlNet to handle the newly added conditions.
Figure 12: Study for extending a trained Uni-ControlNet to newly added conditions.
E Composite Control of Conditions with the Same Type
In real-world applications, users can actually composite two/multiple conditions of the same type easily before feeding them to the model, e.g., draw the sketch of multiple objects in one canvas. However, how to achieve composite control of two conditions with the same type is an interesting research point. We try one simple strategy called "Uni-Channels". Specifically, we augment the input
by adding three extra condition channels. For instance, if the original inputs had 21 channels (3 for each condition, totaling 7 local conditions), with Uni-Channels, we now have  $21 + 3$  channels for the inputs.
During training, we feed the Uni-Channels with randomly selected types of conditions of the input natural images. We observe that, as the shared extra condition channels for different condition types, Uni-Channels can perform well for two-condition composition of the same condition type. The visualization results are depicted in Figure 13.
Figure 13: Visualization results of composite control of two conditions with the same type.
F Comparison with Stable Diffusion 2.1
We compare our method with two Stable Diffusion models, Stable Diffusion 2 - depth ("SD2-depth") and Stable Diffusion 2 - unclip ("SD2-unclip") which support the inputs of depth map and reference image respectively. The visualization results are shown in Figure 14. Additionally, we provide the quantitative results in Table 5 and Table 6.
Table 5: FID on Uni-ControlNet and Stable Diffusion 2.1. The best results are in bold.
FID	Depth	Content
SD2-depth	17.76	-
SD2-unclip	-	24.12
Ours	21.20	23.98
Table 6: CLIP score on Uni-ControlNet and Stable Diffusion 2.1. The best results are in bold.
CLIP Score	Depth	Content
SD2-depth	0.2516	-
SD2-unclip	-	0.2497
Ours	0.2561	0.2402
It is important to note that for SD2-depth and SD2-unclip, the whole model is fine-tuned to learn the depth map or the reference images instead of only fine-tuning adapters, which is the key factor contributing to their great performance. Additionally, when compared to other controllable diffusion models like ControlNet, GLIGEN and T2I-Adapter, SD2-depth and SD2-unclip outperform them, as demonstrated in Table 2 and Table 7.
G More Quantitative Results
CLIP Score: Besides FID, we also test CLIP score for comparing different controllable diffusion models, and ablating condition injections methods and training strategies. We follow the settings in the Section 4.2 and Section 4.3 in the main paper. The results are shown in the Table 7 and Table 8 respectively. Our model demonstrates superior performance quantitatively across most conditions when compared to existing controllable diffusion models. Moreover, for different condition injection methods and training strategies, our method, along with Training-S2, consistently outperforms other strategies. However, joint fine-tuning in Training-S2 does not yield obvious performance gains in most cases. These finds are consistent with those presented in the Section 4.2 and Section 4.3 of the main paper.
Figure 14: Comparison between Uni-ControlNet and Stable Diffusion 2.1.
Table 7: CLIP score on different controllable diffusion models. The best results are in bold.
Canny	MLSD	HED	Sketch	Pose	Depth	Segmentation	Style\Content
ControlNet	0.2538	0.2481	0.2530	0.2499	0.2572	0.2558	0.2531	0.2352
GLIGEN	0.2493	-	0.2403	-	0.2534	0.2526	0.2456	0.2401
T2I-Adapter	0.2513	-	-	0.2584	0.2608	0.2559	0.2478	0.2366
Ours	0.2539	0.2485	0.2556	0.2542	0.2514	0.2561	0.2540	0.2402
User Study: As FID and CLIP score may be not always consistent with human preference, we further conduct user study to quantitatively compare our approach with the baseline methods ControlNet [21], GLIGEN [23] and T2I-Adapter [22]. More specifically, we carry out tests in both single and multi-condition settings, with 20 cases for each setting. Each case is evaluated based on three metrics: the quality of generated images, the match with the given text, and the alignment with the given conditions. Users should select the best one for each metric from the generated images of ControlNet, GLIGEN, T2I-Adapter, and our Uni-ControlNet. We collect responses from 20 users and analyze the total number of votes for each metric under each setting.
The results are presented in Figures 15 and 16. It can be seen that, our approach outperforms both ControlNet, GLIGEN and T2I-Adapter in the single condition setting, demonstrating a clear advantage. Additionally, in the multi-conditions setting, our approach performed significantly better than Multi-ControlNet and CoAdapter.
Figure 15: User study of the preference rate for the single condition setting.
In this section, we present additional qualitative results. Figure 17 illustrates the results for the single-condition setting, while Figure 18 shows the results for the multi-conditions setting. Moreover, we demonstrate our performance on a more challenging case where there are four conditions, as seen in rows 7-8 of Figure 18.
Table 8: CLIP score on condition injection methods and training strategies. The best results are in bold.
Canny	MLSD	HED	Sketch	Openpose	Depth	Segmentation	Content
Injection-S1	0.2513	0.2497	0.2518	0.2507	0.2527	0.2525	0.2508	-
Injection-S2	0.2504	0.2506	0.2518	0.2523	0.2527	0.2544	0.2540	-
Injection-S3	-	-	-	-	-	-	-	0.2502
Training-S1	0.2506	0.2504	0.2511	0.2510	0.2529	0.2538	0.2526	0.2478
Training-S2	0.2528	0.2504	0.2530	0.2537	0.2533	0.2547	0.2545	0.2421
Ours	0.2528	0.2483	0.2535	0.2539	0.2503	0.2549	0.2522	0.2420
Figure 16: User study of the preference rate for the multi-conditions setting.
I Adapter Details
We provide the details of our proposed local control adapter and global control adapter in Figure 19, Figure 20 and Figure 21.
Tools on the ground
Houses on the street at sunset
Teddy bears are reading a book, sunny day
MLSD
A red crab on the beach
HED
Openpose
A red apple on the books
A warm room with sunshine in
Figure 17: More visual results of Uni-ControlNet for single condition setting.
A beautiful butterfly on the bud
Books on the shelf
Spaghetti with toasts on the plate
A tram in an old city
A man is dancing on the street
Flowers in the basket, oil painting
Sea and beach
A man walking on the street, sunny day
A house on hills, blue sky
A cute cat and balloons
A yellow flower in a white plate with a fork and a knife
A cat on the sofa in an old room, art painting
A photo of a cowboy riding a horse on the street
Condition-1
Condition-2
A man and his car on the mountain, a big moon in the sky
A sofa and a deer in the forest
Condition-3
Condition-4
Figure 18: More visual results of Uni-ControlNet for multi-conditions setting.
Figure 19: Details of the local control adapter.