UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
Authors: Can Qin
Shu Zhang
Ning Yu
Yihao Feng
Xinyi Yang
Yingbo Zhou
Huan Wang
Juan Carlos Niebles
Caiming Xiong
Silvio Savarese
Stefano Ermon
Yun Fu
Ran Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Image Generation
Multi-modal
HyperNet

Abstract: Achieving machine autonomy and human control often represent divergent objectives in the design of interactive AI systems. Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages. However, they often fall short in generating images with spatial, structural, or geometric controls. The integration of such controls, which can accommodate various visual conditions in a single unified model, remains an unaddressed challenge. In response, we introduce UniControl, a new generative foundation model that consolidates a wide array of controllable condition-to-image (C2I) tasks within a singular framework, while still allowing for arbitrary language prompts. UniControl enables pixel-level-precise image generation, where visual conditions primarily influence the generated structures and language prompts guide the style and context. To equip UniControl with the capacity to handle diverse visual conditions, we augment pretrained text-to-image diffusion models and introduce a task-aware HyperNet to modulate the diffusion models, enabling the adaptation to different C2I tasks simultaneously. Trained on nine unique C2I tasks, UniControl demonstrates impressive zero-shot generation abilities with unseen visual conditions. Experimental results show that UniControl often surpasses the performance of single-task-controlled methods of comparable model sizes. This control versatility positions UniControl as a significant advancement in the realm of controllable visual generation.

UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild
Can Qin Shu Zhang!, Ning Yu!, Yihao Feng Xinyi Yang', Yingbo Zhou', Huan Wang Juan Carlos Niebles", Caiming Xiong', Silvio Savarese Stefano Ermon Yun Fu", and Ran Xu
'Salesforce Al Research, "Northeastern University, 'Stanford Univeristy,
qin. cawnortheastern edu, ermonêcs... crrnnesssstaforrdd sww.sorrd.....m edu, yunfuβece neu neu. edu, {shu. zhang, ning-yu yihaof, y.yang, yingbo zhou, huan, wang, jniebles, cxiong, ssavarese, ran. ran.xu)@saleesooceeco com
Abstract
Achieving machine autonomy and human control often represent divergent ohjec- tives in the design of interactive Al systems. Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages. However, they often fall short in generating images with spatial, structural, or geometric controls. The integration of such controls, which can accommodate various visual conditions in a single unified model, remains an unaddressed challenge. In response, we introduce UniControl, new generative foundation model that consolidates wide array of controllable condition- tirtiion-t--mageeee (C2I) tasks within a singular framework, while still allowing for arbitrary language prompts. UniControl enables pixel-level-prectie generation, where visual conditions primarily influence the generated structures and language prompts guide the style and context. To equip UniControl with the capacity to handle diverse visual conditions, we augment pretrained eex-et--maaee diffusion models and introduce a task-aware HyperNet to modulate the diffusion models, enabling the adaptation to different C2I tasks simultaneously. Trained on nine unique C2I tasks, UniControl demonstrates impressive zero-shot generation abilities with unseen visual conditions. Experimental results show that UniControl often surpasses the performance of ble model sizes. This control versatility positions UniControl as significant advancement in the realm of controllable visual generation.
image
controlled methods of compara-
1 Introduction
Generative foundation models are revolutionizing the ways that humans and AT interact in natural language processing (NLP) (1-6), computer vision (CV) (410), audio processing (AP) [2]. and robotic controls [5), to name a few. In NI.,, generative foundation models such InstructGPT or GPT-4, achieve excellent performance
wide range of tasks, e.g., question answering, summariza- tion, text generation, or machine translation within a single-unified model. Such multi-tasking ability is one of the most appealing characteristics of generative foundation models. Furthermore, generative foundation models can also perform zero-shot or few-shot learning on unseen tasks [ 16 17].
For generative models in vision domains 9 18 20 such multi-tasking ability is less clear. Stable Diffusion Model (SDM) [ has established itself the major cornerstone for text-conditioned image generation. However, while text descriptions provide a very flexible way to control the generated images, their ability to provide pixel-level precision for spatial, structural, or geometric controls
Code: https:/ggiill..oomaaaasssccruuun
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

UniControl Diffusion Model
Pre Pre-training Tasks
Oana
Pre training Tasks
Canny Edge Image
HED
Depth to Image
Normal Map to Image
Commytdo
Image
"Brathh of Wild compreseed COND art"
Segmentation
Bounding Box to Image
Skeleton to Image
Image Outpainting
to Image
NED AO- Imagg
Zero-shot Combination
Segmentation Skeleton to Image
Depth Map Skeleton to Image
Chocolate mousee cale
Zero- shot Tasks
Dapoth douape
Gray Image Colorization
Image Deblur
Image Inpainting
Image Deblanieg
sebras gratigg gross
Narmal Image
"iihes Nott Pork Three Gendis With Mi Way The Baciground
Short Ha Women"
Sleletore Aaaggaa
Image
Image Outpohhdin
ca choeri
Zero-shet
"Oresay fruits
Sag Timmn
"tekgrund: Old el bife, foregrannd: & set of tiins who are teöin ball""
zhot Task: Depth and Shoicton-to to image
pancling 3d
BRO 1O Image
Baciground White House, Forground: A groupp womer sttigge bench"
gOSEES wozring poses picuue"
Figure 1: UniControl trained with multiple tasks with a unified model, and further demonstrates promising capability in zero-shot tasks generalization with visual example results shown above.
often inadequate. A recent work, ControlNet (21), was proposed to augment SDM to enable visual conditions (e.g., edge maps, depth maps). With the additional visual conditions, ControlNet can achieve explicit spatial, structural, geometric control over generated structures, without losing the semantic control from textual captions. Unfortunately, unlike language prompts that a unified module such CLIP [22] can handle, each ControlNet model only handle specific control modality that it was trained on (e.g., edge map). Retraining a separate model is necessary to handle a different modality of visual conditions, incurring non-trivial time and spatial complexity costs.
To overcome the limitation of previous works, we present UniControl, a unified diffusion model for controllable visual generation in the wild, which is capable of simultancously handling both

---

language and various visual conditions. Naturally, UniControl can perform multi-tasking and can encode visual conditions from different tasks into a universal representation space, seeking a common representation structure among tasks. The unified design of UniControl allows us to enjoy the advantages of improved training and inference efficiency, as well as enhanced controllable generation. On the one hand, the model size of UniControl does not significantly increase as the number of tasks scales up. On the other hand, UniControl derives advantages from the inherent connections between different visual conditions [e.g. 2.) These relationships, such as depth and segmentation mapping, leverage shared geometric information to enhance the controllable generation quality.
The unified controllable generation ability of UniControl relies on two novel designed modules, mixture of expert (MOE)-style adapter and task- HypenNer (26] (27) The MOE-style adapter can learn necessary low-level feature maps from various visual conditions, allowing UniControl to capture unique information from different visual conditions. The task-aware HyperNet, which takes the task instruction as natural language prompt inputs, and outputs a task-aware embedding. The output embeddings can be incorporated to modulate ControlNet (1 for task-aware visual condition controls, where each task corresponds to particular format of visual condition. As result, the task-aware HyperNet allows UniControl to learn weea-knoowledge across various tasks, and obtain abilities to generalize to unseen tasks. As Tab. T UniControl has significantly compressed the model size compared with its direct baseline, i.e., Multi-ControlNet,, by unifying nine tasks into ONE model.
Table 1: Architecture and Model Size (#rams)) UniControl vs. Multi-ControlNy
Stable Diffusion
ControlNet MoE-Adapter TaskHyperNet
Total
UniControl Multi-ControlNet
1065.7M
361M
0.06M
12.7M
1.44B
1065.7M
361M 9
4.32B
To obtain multi-tasking and zero-shot learning abilities, we pre-train UniControl on nine distinct tasks across five categories: 1) edges (Canny, HED, User Sketch); 2) region-wise maps (Segmentation Maps, Bounding Boxes); 3) skeletons (Human Pose Skeletons); 4) geometric maps Depth, Surface Normal); 5) editing (Image Outpainting). We build MultiGen-20M dataset, comprising over 20 million high-quality triplets of original images, language prompts, and visual conditions for all the tasks. Then UniControl is trained for over 5,000 GPU hours on NVIDIA A A100-40G hardware that is comparable with the overall training cost of different NontrolNets. Moreover, UniControl exhibits a remarkable capacity for zero-shot adaptation to new tasks, highlighting its potential for deployment in rorl-world applications. Our contributions are summarized below:
We present UniControl, a unified model capable of handling various visual conditions for the controllable visual generation.
We collect a new dataset for multi-condition visual generation with more than 20 million image- text-condition triplets over nine distinct tasks across five categories.
We conduct extensive experiments to demonstrate that the unified model UniControl outperforms each single-task controlled image generation, thanks to learning the intrinsic relationships between different visual conditions.
UniControl shows the ability to adapt to unseen tasks in a zero-shot manner, highlighting its versatility and potential for widespread adoption in the wild.
2
Related Works
Diffusion-based Generative Models. Diffusion models were initially introduced in 28 that yield favorable outcomes for generating images [8] 21). Improvements have been made through various training and sampling techniques such score-basee diffusion [29] 30), Denoising Diffusion Proba- bilistic Model (DDPM) [II), and Denoising Diffusion Implicit Model (DDIM) 32). When training
U-Net denoisers 33 with high-resolution images, researchers involve speed-up techniques including
pyramids 334. multiple stages (20), or latent representations [9. In particular, UniControl leverages Stable Diffusion Models (SDM) [ the base model to perform multi-tasking.
Text-to-Image Diffusion. Diffusion models emerge to set up a cutting-edge performance in text-to-image generation tasks 19), by cross-attending U-Net denoiser in diffusion generators with CLIP 22 or T5-pretrained (2 text embeddings. GLIDE (35] is another example of a text- guided diffusion model that supports image generation and editing. UniControl and closely related
3

---

ControlNet 21 are both built upon previous works on diffusion based text-to-in -image generation [9]
36 introduces the compositional conditions to guide visual generation.
Prage-to-Image Translation. Image-to-image (12I) translation task was initially proposed in Pix2Pix (37] focusing learning mapping between images in different domains. Recently, diffusion-based approaches [38] 339 [1 set the new state of the art results. Recent diffusion- based image editing methods show outstanding performances without requiring paired data, e.g.. Edict 42 Other image editing examples include various diffusion bridges and flows 45 classifier guidance 33 based methods for colorization, super- resolution [4]. inpainting 48 and etc. ControlNet [21 takes both visual and text conditions and achieves new state-of-the art controllable image generation. Our proposed UniControl unifies various visual conditions of ControlNet, and is capable of performing zero-shot learning on newly unseen tasks. Concurrently, Prompt Diffusion [49] introduces visual prompt [50) from image inpainting to controllable diffusion models, which requires two additional image pairs as the in-context example for both training and inference. By contrast, UniControl takes only a single visual condition while
SDEdit 40), prompt-to-prompl
still capable of both multi-tasking and zero-shot learning.
3 UniControl
In this section, describe the training and the model design of our unified controllable diffusion model UniControl. Specifically, we first provide the problem setup and training objectives in Sec. 3.1 and then show the novel network design UniControl in Sec. 3.2 Finally, we explain how to perform zero-shot image generation with the trained UniControl in Sec. 3.3
3.1 Training Setup
Different from the previous generative models such as Stable Diffusion Models (SDM) 9 or ControlNet [21] where the image generation conditions are single language prompt, or single type of visual condition such canny, UniControl is required take wide range of visual conditions from different tasks, as well as the language prompt.
To achieve this, we reformulate the training conditions and target pairs for UniControl. Specifically, suppose we have a dataset consisting of K tasks = {Di U...U DD ). and for each task training set DR. denote the training pairs by ([Crext, Ctask] Ie, with Crask being the task instruction that indicates the task type, Ctext being the language prompt describing the target image. Ie being the visual conditions, and z being the target image. With the additional task instruction, UniControl can differentiate visual conditions from different tasks. A concrete training example pair is the following:
Task- Aware Vision-LLngunnn Language Condition
Target output X
Visual Condition
Ie
Language Prompt Ctext:
"Camp on mountain top: Birthday Presents, Adventure, Outdoor, Mountain Camps, Great View, Places, Hiking, Momiinss Lights, Himalayan Sunn"
Task Instruction Ctask: "Canny Edge to Image"
where the task is to translate the canny edge to real images following language prompt. With the induced training pairs (x, [Ctasks Crext,; Ic), we define the training loss for task k following LDM
( (0) = Ez,,, frraa Preset Tc [[|s- E (Zz, ł, Ctask, Crext, Ie) with ([Crask: Crext), Ic, x) ~ D
where represents the time step, 2, is the noire-co latent tensor time step 20 E(z). and 0 is the trainable parameters of UniControl. We also apply classifier-free guidance [51] to randomly drop 30% text prompts to enhance the controllability of input visual conditions. We train UniControl uniformly on the K tasks. To be more specific, we first randomly select a task & and sample a mini-match from DR, and optimize with the calculated loss (²))).
3.2 Model Design
Since our unified model UniControl needs achieve superior performance on a set of diverse tasks, necessary ensure the network design enjoys the following properties: 1) The model can overcome

---

UniControl Model Overview
Task-Aware Langon-L Condition
MOR Adapter
Teursen
Test&
Time
your or
Impur uas so vnser cxadition
Tasl: Iratracikon
Dige
Task-Aware HyperNet
Tase Insouction "neerea! survcc menge"
Random Norse
MOE Adupter
CLIP Racode
Modulated Zero Cons
Hepernet
Thinnale Weights
HyperNet
Perilid Lizaer Layerz
SD Encoder
SD Encader
(Trainable
Lincar Layer
Cops)
Takk Enbedling
Middlle
SD Middle
(Taiinabe Copyy
Coodition
Zero Casvilution
Task Enbeediing
Modulated Zero
SD Decoder
Modulatted Zero
N
so Nock
Bock
Zero Casvelation
Thak Embeiliing
Generated
Images
Figure 2: This figure shows proposed UniControl method. To accommodate diverse tasks, we're 'e designed a Mixture of Experts (MOE) Adapter, containing roughly ~00K #params for each task, and Task-aware HyperNet (~12M #ppaams) to modulate N (i.e.. 7) zero-conv layers. This structure allows for multi-task functionality within singular model, significantly reducing the model size compared to equivalent stack of single-task models, each with around 1.4B #params.
the misalignment of low-level features from different tasks; 2) The model can learn meta-knowledge across tasks, and adapt to each task effectively.
The first property can ensure that UniControl can learn necessary and unique information from all tasks. For instance. if UniControl takes the segmentation map as the visual condition, the model might ignore the 3D information. As result, the feature map learned may not be suitable for the task that takes the depth map images as visual condition. The second property would allow the model to learn the shared knowledge across tasks, as well the differences among them.
We introduce novel designed modules, MOE-style adapter and task-aware HyperNet, that allows UniControl enjoys the above two properties. An overview of the model design for UniControl is in Fig. We describe the detailed designs of these modules below.
we devise group of convolution modules serve as the adapter for UniControl capture features of various low-level visual conditions. Precisely. the designed adapter module can be expressed as
MOE-Style Adapter. Inspired by the design of Mixture-of-Experts (MOEs) 52
K
F FAdapter (I:) = Σ 1(i== = k) F(()
Covl o F(O (I!),
where 1(-) is the indicator function, Ih is the conditioned image from task k. and (+) are the convolution layers of the i-th module of the adapter. We remove the weights of the original MOEs since our designed adapter is required to differentiate various visual conditions. Meanwhile, naive MOE modules can not explicitly distinguish different visual conditions when the weights are learnable. Moreover, such task-specific MOE adapters facilitate the zero-shot tasks with explicit retrieval of the adapters of highly related pre-training tasks. Besides, the number of parameters for each convolution module is approximately 70K, which is computationally efficient.
z()
Cov1 Cov2
5

---

techgn
MOE Hybrid Tasks Generalization
MOE Zero-Shot New Task Generalization
Figure 3: Illustration of MOE's behaviors under zero-shot scenarios. The left part shows the capacity of the MOE to generalize to hybrid task conditions, achieved through the integration of outputs from two pertinent convolution layers. The right part illustrates the ability of the MOE-style adapter to generalize to unseen tasks, facilitated by the aggregation of pre-traines tasks using estimated weights.
Task-Aware HyperNet. The task-aware HyperNet modulates the zero-convolution modules of ControlNet (21) with the task instruction condition Ctask. As shown in Figure 2 our hyperNet first projects the task instruction Crask into task embedding with the help of CLIPText encoder. Then similar in spirit of style modulation in StyleGAN2 [53], we inject the task embedding into the trainable copy of ControlNet, by multiplying the task embedding to each zero-conv layer. In specific, the length of the embedding is the same as the number of input channels of the zero-conv layer, and each element scalar the embedding multiplied to the convolution kernel per input channel. We also show that our newly designed task-aware HyperNet can also efficiently learn from training instances and task supervision following a similar analysis as in ControlNet (21].
3.3 Task Generalization Ability
With the comprehensive pretraining on the MultiGen-20X dataset, UniControl exhibits zero-shot capabilities on tasks that were not encountered during its training, suggesting that Unicontrol possesses the ability transcend in-domain distributions for broader generalization. We demonstrate the zero- shot ability of UniControl in the following two scenarios:
Hybrid Tasks Generalization. As shown in the left side of Fig. g We consider two different visual conditions as the input of UniControl, hybrid combination of segmentation maps and human skeletons, and augment specific keywords "background" and "foreground" into the text prompts. Besides, we rewrite the hybrid task instruction as blend of instructions of the combined two tasks such "segmentation map and human skeleton to image".
Zero-Shot New Tasks Generalization. As shown the right side of Fig. UniControl needs to generate controllable images on newly unseen visual condition. To achieve this, estimating the task weights based on the relationship between unseen and seen pre-trained tasks is essential, The task weights can be estimated by either manual assignment or calculating the similarity score of task instructions in the embedding space. The example result in Fig. [3]]5 (d) is generated by our manually assigned MOE weights as "depth: 0.6, seg: 0.3, canny: 0.1" for colorization. The MOE-style adapter can be linearly assembled with the estimated task weights to extract shallow features from the newly unseen visual condition.
Experiments
We empirically evaluate the effectiveness and robustness of UniControl. We conduct series of comprehensive experiments across various conditions and tasks, utilizing diverse datasets to challenge the model's adaptability and versatility. Experimental setup, methodologies, and results analysis are provided in the subsequent sections.
4.1 Experiment Setup
Implementation. The UniControl is illustrated as Fig. 2 with Stable Diffusion, ControlNet, MOE Adapter, and Task-aware HyperNet consisting w1.5B parameters. MOE Adapter consists of parallel convolutional modules, each of which corresponds to one task. The aware HyperNet inputs the CLIP text embedding [22] of task instructions and outputs the task embeddings to modulate weights of zero-conv kernels. We implement our model upon the ControlNet]] []] We take the
AdamW (54) as the optimizer based on PyTorch Lightning [55] The learning rate assigned as
1 x 10-5 Our full-version UniControl model is trained on Nvidia-A 100 GPUs with the batch size of 4, requiring ~ 5, 000 GPU hours. We have also applied
safeguards of results.

---

Conditiee
talk
Candition
Sinllstook Contrntton
Hummn image
Hoall Twinta
Sunvio
Object Dww. to
HED ITARE
Nomoa Inaaaa
Corma Ain Nendola Fecape
Anarwnad mg ane bendinar 1ilk chuctin phons
Imae Outpaitting
Agunn teeell cimgore Beli
SWET mage
Doeth to intrare
Eacosbaon Schour 1olln Se-houn 14. fellww
Figure 4: Visual comparison between official or re-implemented task-specific ControlNet and our proposed model. The example data is collected from testing set sampled from COCO and Laion.
Data Collection. Since the training of ControlNet is currently unavailable, initiate own data collection process from scratch and name it as MultiGen-20M. We use subset of Laion- Aesthetics V2 [56] with aesthetics ratings over six, excluding Now-resolution images smaller than 512. This yields approximately 2.8 million image-text pairs. Subsequently, we process this dataset for nine distinet tasks across five categories (edges, regions, skeletons, geometric maps, real images):
Canny (288M): Utilize the Canny edge detector with randomized thresholds.
HED (2.8M): Deploy the Holistically-nested edge detection (88 for robust boundary determination.
Depth (2.8M): Employ the Midas [59] for monocular depth estimation.
Normal (288M): Use the depth estimation results from the depth task estimate scene or object surface normals.
Segmentation (2.8M): Implement the Uniformer 60 model, pre-trained the ADE20K [61 dataset, to generate segmentation maps across 150 classes
Object Bounding Box (874K): Utilize YOLO V4 62 pre-trained on the COCO 63 dataset for bounding box labelling across 80 object classes.
Human Skeleton (1.3M): Employ the pre-trained Openpose 64 model to generate human
skeleton labels from source images.
Image Outpainting (288M): Create boundary masks for source images with random masking percentages from 20% 80%.
Further processings are carried out on HED maps using Gaussian filtering and binary thresholding to simulate user sketching. Overall, we amass over 20 million mmge--roopptccondtton triplets. Task instructions were naturally derived from the respective conditions, with each task corresponding to specific instruction, such as "canny edge to image" for the canny task. We maintain one-to- one correspondence between tasks and instructions without introducing variance to ensure stability during training. We have additionally collected testing dataset for evaluation with 100-300 image- condition ceddtion-pproppt triplets for each task. The source data is collected from Laion and COCO. We will open-source training and testing data to contribute to the community.
Benchmark Models. The most straightforward comparison for UniControl comes from task-specific ControlNet models. Six tasks overlap with those presented in ControlNet, so their official models are chosen as baselines for these tasks. For fair comparison, we re-implement the ControlNet model (single task) using our collected data. Our unified multi-task UniControl is compared against these

---

Photograph, Tometo fome "Background dld of Telx, Foreground A twins who are taking bath" colerfal shatt standing pedriss
Conditie Depth
Condetion 2: Human Skeleton
Segmentation
(a) Zero Shot Combination: and Human Skeleton Imagee
(b) Zero Shot Combination: Segmentation and Human Skeleton to Image
"Danici Dae Gin smilss at camcea"
Quiuut
(d) Zero Shot Task: Colorization Example Result
Out
"Weddling hair and maksup with worm tonee and Ippe"
(e) Zero-Shot Imagee Deblurring Example
Image Inpointing Example
Figure (a)-(b): Example results of UniControl over hybrid (unseen combination) conditions with key words "background" and "foreground" attached in prompts. (c)-(e): Example results of UniControl on three unseen tasks (deblurring, colorization, inpainting).
Cotie to. Irage
HED-50 Hagg
Depth ta Irooe
Serrotts irage
Haver Sie Strettstss
Figure 6: User study between our method and official ControlNet checkpoints on six tasks. Our method outperforms ControlNet on all tasks.
task-aware models for each task. We apply default sampler as DDIM 32 with guidance weight and steps 50. All single-task models used for comparison are trained by 100K iterations and our multi-task model is trained around 900K with similar iterations for each task ensure fairness. The efficiency and compact design of our proposed model are evident in its construction. The total size of UniControl is around 1.5B #params and a single task ControlNet+SDM takes 1.4B. In order to achieve the same nine-task functionality, single-task strategy would require the ensemble of SDM with nine specifc ControlNet models, amounting to approximately 4.3B #params total.
4.2 Visual Comparison
We visually compare different tasks (Canny, HED, Depth, Normal, Segmentation, Openpose, Bound- ing Box, and Outpainting) in Fig. 4 Our method consistently outperforms the baseline ControlNet model. This superiority in terms of both visual quality and alignment with conditions prompts.
For the Canny task, the results generated by our model exhibit higher degree of detail preservation and visual consistency. The outputs of UniControl maintain a faithful reproduction of the edge information (i.e,, round table) compared ControlNet. the HED task, our model effectively captures the robust boundaries, leading to visually appealing images with clear and sharp edge transitions, whereas ControlNet results appear to be non-factual. Moreover, our model demonstrate more subtle understanding of 3D geometrical guidance of depth maps and surface normals than ControlNet. The depth map conditions produce visibly more accurate outputs. the Normal task, our model faithfully reproduces the normal surface information (i.c., ski pole), leading to more realistic

---

Canmy-to-mmage
HED-to-mage
Normale--tmage
Depth-to-image
50
50
40
60-
43
30
30
20
10
Ourssegg Bbox-to-lmage
Ours-Mul
OuraSnigl
Outpaining-t--Ira
501
18
00-
30
30-
20
20
20-
10
10
30
90 Curedarge
Ta
OunoMaaii Oursdingle
T
Gure-Sinie
T
Figure User study between our multi-task model (Ours-multi) and single task model (Ours-single) on eight tasks. Our method outperforms baselines on most of tasks, and achieves big performance gains on tasks of seg-to-image and outpainting-to-ine in all cases is computed as 0.0028 that is statistically significant according to the criteria of <0.05.
Moreover, the p-value voting Ours-multi
and visually superior outputs. During the Segmentation, Openpose, and Object Bounding Box tasks, the produced images generated by our model are better aligned with the given conditions than that by ControlNet, ensuring a higher fidelity to the input prompts. For example, the re-implemented ControlNet-BBox misunderstands "a woman near statue", whereas our outputs exhibit high degree accuracy and detail. In the Outpainting task, our model demonstrates its superiority by generating reasonable images with smooth transitions and natural-looking textures. It outperforms the ControlNet model, which produces less coherent results "a bear missing one leg". This visual comparison underscores the strength and versatility of our approach across diverse of tasks.
4.3
Quantitative Evaluation
User Study. We compare the performance of our method with both the released ControlNet model and the re-implemented single-task ControlNet on our training set. As shown in Fig. 6 our approach consistently outperforms the alternatives in all cases. In the HED-to-in to-image generation task, our method significantly surpasses ControlNet. This superiority is even more pronounced in overwhelmingly favor
the depth and normal surface to image generation tasks, where method, demonstrating its ability to handle complex geometric interpretations. When compared to the re-implemented single-task model, Fig. revells that our approach maintains smaller advantage, yet still demonstrates its benefits by effectively discerning image regions to guide content generation. Even in the challenging outpainting task, our model outperforms the baseline, highlighting its robustness and capacity to generalize.
Image Perceptual Met-
Table 2: Image Perceptual Distance
ric. We evaluate the dis- tance between our output and the ground truth im- ControlNet
Canny
HED
Normal
Depth
Pose
Segmentation
UniControl
0.346
0.466
0.623
0.654
0.741
0.693
0.577
0,582
0.778
0.700
0.747
0.693
age. As aim to obtain a structural similar image to the ground truth image, we adopt the perceptual metric in 665 where a lower value indicates more similar images. As shown in Tab. 2 UniControl outperforms ControlNet on five tasks, and obtains the same image distance to ControlNet on Segmentation.
Fréchet Inception Distance (FID). We've 'e've further conducted quantitative analysis with FID 666
to include more classic single-takk-contrllee methods such as GLIGEN [67] and T21-adapter (68] With a collection of over 2,000 test samples sourced from Laion and COCO, we've ve assessed wide range of tasks covering edges (Canny, HED), regions (Seg), skeletons (Pose), and geometric maps (Depth, Normal). The Tab. demonstrates that our UniControl consistently surpasses the baseline methods across the majority of tasks. Notably, UniControl achieves this while maintaining a more compact and efficient architecture than its counterparts.
Ablation Study. We've 'e've conducted an ablation study, specifically focusing on the MoB-Style Adapter and TaskHyperNet in Tab. 4 with FID scores reported as the previous part. It is noticeable that the full-version UniControl (MoE-Style Adapter + TaskHyperNet) significantly outperforms the ablations which demonstrates the superiority of proposed MoE-Style Adapter and TaskHyperNet.

---

Table Quantitative Comparison (FID)
Canny
HED
Depth
Normal
Seg
Pose !
GLIGEN
67
24.9
27.8
25.8
27.7
T21-Adapter
68
23.66
25.4
27.1
28.9
ControlNet 21
22.7
25.1
25.5
28.4
26.7
28.8
UniControl
22.9
23.6
21.3
23.4
25.5
27.4
Table 4: Ablation Study (FID)
MoE-Adapter
TaskHyperNet
Canny
HED
Depth
Normal
Seg
Pose
Avg |
27.2
29.0
27.6
29.1
30.2
28.7
24.5
26.1
23.7
24.8
26.9
28.3
25.7
22.9
23.6
21.3
23,4
25.5
27.4
24.0
4.4 Zero-shot Generalization
We further showcase the surprising capabilities of our method undertake the zero-shot challenge of hybrid conditions combination and unseen tasks generalization.
Hybrid Tasks Combination. This involves generating results from two distinct conditions si- multaneously. Our model's zero-shot ability is tested with combinations such as depth and human skeleton or segmentation map and human skeleton. The results are shown in Fig. 1s-55co (a)-(b). When the background is conditioned on depth map, the model effectively portrays the intricate 3D structure of the scene, while maintaining the skeletal structure of the human subject. Similarly, when the model is presented with combination of segmentation map and human skeleton, the output skillfully retains the structural details of the subject, while adhering to the segmentation boundaries. These examples illustrate our model's s adaptability and robustness, highlighting its ability to handle complex hybrid tasks without any prior explicit training.
Unseen Tasks Generalization. To evaluate the zero-shot ability to generalize to unseen tasks such as gray image colorization, image deblurring, and image inpainting, we conduct the case analysis Fig. (c)-(e).. The model skillfully handles the unseen tasks, producing compelling results. This capability is deeply rooted in the shared attributes and implicit correlations among pre-training and new tasks, allowing our model to adapt seamlessly. For instance, the colorization task leverages the model's understanding of image structures from the segmentation task and depth estimation task, while deblurring and inpainting tasks benefit from the model's 's familiarity with edge detection and outpainting ones.
Conclusion and Discussion
We introduce UniControl, novel unified model for incorporating wide range of conditions into the generation process of diffusion models. UniControl has been designed to be adaptable to various tasks through the employment of two key components: a Mixture-of-Exppee Experts (MOE) style adapter and task-aware HyperNet. The experimental results have showcased the model's robust performance and adaptability across different tasks and conditions, demonstrating its potential for handling complex text-to-image generation tasks.
Limitation and Broader Impact. While UniControl demonstrates impressive performance, it still inherits the limitation of diffusion-based image generation models. Specifically, it is limited by our training data, which is obtained from subset of the -esthhetssse datasets. We observe that there is a data bias in this dataset. Although we have performed keywords and image based data filtering methods, we are aware that the model may generate biased low-fidelity output. Our model is also limited when high-quality human output is desired. UniControl could be improved if better open-source datasets are available to block the creation of biased, toxic, sexualized, or other harmful content. We hope work motivate researchers to develop visual generative foundation models.
10

---

References
[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188)99 2019.
[2] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Rex., 21(140): 2020.
[3] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetundd language models. arXiv preprint arXiv:2210. 2022.
[4] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 35:27730-27744, 2022.
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
[6] OpenAl. Gpt-4 technical report, 2023.
[7] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 10555 2023.
[8] Haoxuan You, Mandy Guo, Zhecan Wang, Kai-Wei Chang, Jason Baldridge, and Jiahui Yu. Co- bit: A contrastive bi-directional image-text generation model. arXiv preprint arXiv:2303. 13455, 2023.
[9] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[10] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. arXiv preprint arXiv:2304.02644 2023.
[11] Bo Li, Dongseong Hwang, Zhouyuan Huo, Junwen Bai, Guru Prakash, Tara N Sainath, Khe Chai Sim, Yu Zhang, Wei Han, Trevor Strohman, et al. Efficient domain adaptation for speech foundation models. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.
[12] Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong. Jiawei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint Xii::2304411999. 2023.
[13] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as i can, not as say: Grounding language in robotic affordances. arXiv preprint wrXiv:224.00669) 2022.
[14] Yilun Dai, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuur- mans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.00111 001 2023.
[15] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor: Chatgpt for roboties: Design principles and model abilities. Technical Report MSR-TR-2023-8, Microsoft, February 2023. URL. http:///ha..aaaooooooooooaaaaaaa
chatgpt tincc-forrrbbooooo -ww.ooooooossdeeeeeeeeeeeeeeeeee ottp::/eeeee.ii.ooooooooooooonno -ndedddddde---iiiitteeee/

---

[16] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55, 2022.
[17] OpenAl. Chatgpt. http:////openaa..ommb/ologo
2022.
[18] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780-8799 2021.
[19] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:225.111487, 2022.
[20] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[21] Lvmin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. arXiv preprini arXiv:2302.05544; 2023.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103, 00000 2021.
[23] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pages 3712-3722, 2018.
[24] Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung-Yi Boung-Yi e Lin. Multi-task self- training for learning general representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8856-888 000-8885,222 2021.
[25] Shikun Liu, Edward Johns, and Andrew J Davison. End-to-end multi learning with attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1871-1880, 2019.
[26] David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations, 2017.
[27] Johannes Von Oswald, Christian Henning, Benjamin Grewe, and João Sacramento. Continual 2019.
learning with hypernetworks. arXiv preprint
[28] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In ICML, pages 2256-2265. PMLR, 2015.
[29] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[30] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33:6840-6851, 2020.
[32] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arx:v:2010.02500, October 2020. http:httts///rxxttt org/ 200b0 02502
[33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241. Springer, 2015.
12

---

[34] Jonathan Ho. Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23(47):1-33, 2022.
[35] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Iya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 10741, 2021.
[36] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423 439. Springer, 2022.
[37] Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
[38] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee. Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[39] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv: 12952, 2022.
[40] Chenlin Meng, Yutong He, Yang Song. Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[41] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen- Or. Prompt-to-prontt to--mpppppptt image editing with cross attention control. arXiv preprint 220::2208.01666, 2022.
[42] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled transformations. arXiv preprint arXiv:2211. 12446, 2022.
[43] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In The Eleventh International Conference on Learning Representa- tions, 2022.
[44] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.0274: 2022.
[45] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arX0v.2209.03003, 2022.
[46] Xingehao Liu, Lemeng Wu, Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffusion generative models. arXiv preprint arXiv:2208, 14699, 2022.
[47] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima Anandkumar. 12sh: Image-to-image schrödinger bridge. arXiv preprint arXiv:2302.05872. 2023.
[48] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2022.
[49] Zhendong Wang, Yifan Jiang, Yadong Lu, Yelong Shen, Pengcheng He, Weizhu Chen, Zhangyang Wang, and Mingyuan Zhou In-context learning unlocked for diffusion models. arXiv preprint arXiv:2305.01 2023. URL URL/htpps: :tsttps://arxvvv org/ abs/2305 01115
[50] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexii Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35: 25005-25017, 2022.
[51] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. 222.5 2022.
arXiv preprint
13

---

[52] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate mixture-of-experts layer. arXiv preprint arXiv: 7701.06338 2017.
[53] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pages 8110-8119; 2020.
[54] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[55] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS. NeurlPS, 2019.
[56] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2111.02114. 2021.
[57] John Canny. A computational approach edge detection. IEEE Transactions on pallern analysis and machine intelligence, pages 679-698, 1986.
[58] Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In Proceedings of the IEEE international conference on computer vision, pages 1395-1403, 2015.
[59] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623- 1637, 2020.
[60] Kunchang Li, Yali Wang. Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint 2rXiv:2201.09450 2022.
[61] Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-32 221122311222222 2019.
[62] Alexey Bochkovskiy, Chien-Yas Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
[63] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer VisV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[64] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291-7299, 2017.
[65] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition, 2018
[66] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NeurlPS, 30, 2017.
[67] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv:2301 007093 2023.
14

---

[68] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08455 2023.
15

---

Appendix
A Details of Implementation
A.1 MOE-Style Adapter
The MOE adapter is implemented set of parallel ConvNets composed of three consecutive convolution and non-linear activation layers. The entire model is comprised of nine individual MOE adapters, each of which consumes 70K parameters. Task keys designated to each adapter, ensuring that they align with the corresponding visual conditions. Once the MOE adapter processes the input, the remaining model parameters become shared across all tasks. This architecture facilitates task adaptability while promoting parameter efficiency.
A.2 Task-aware HyperNet
The task-aware hypernet applied to modulate the parameters of zero-conv layers in the NontrolNet. Since the ControlNet can be considered the hypernet of Stable Diffusion (fixed copy). Our idea can be concluded as the control over control or meta-control to let the task-aware hypernet learn the universe representation that is generalizable across different tasks. To implement it, firstly map the task keys to instruction with mapping function as: {"hed": "hed edge to image",
"canny": "canny edge to image", "seg": "segmentation map to image", "depth": "depth map to image", "normal": "normal surface map to image", "pose": "human pose skeleton to image", "hedsketch": "sketch image",
"bbox": "bounding box to image", "outpainting": "image outpainting"); Then, such instructions will be projected as text embeddings with the help of language model (we adopt CLIPText in our implementation). The Task-aware HyperNet takes these task instruction embeddings, and projects them into different shapes to match the size of different zero-conv kernels, which will be modulated by these task embeddings accordingly. We would fix the parameters of task-aware hyperNet in the later stage of model training ensure the stability of dynamics.
A.3 Data Collection
We have collected a large amount of training (MultiGen-20M) including 20M condition- image-prompt triplets across nine different tasks. We firstly download 3/4 of Laion-Aesthetics- V2 with score over six and filter out low-resolution (<512) images. As result, 2.8M images selected as source images. Then we apply the visual condition extractors as described in the main paper to collect Canny, HED, Sketch, Depth, Normal Surface, Seg Map, Object Bounding Box, Human Skeleton and Outpainting.
B Numerical Analysis of Task-Aware Modulated ControlNet
We show that proposed task-aware modulated ControlNet preserves the properties of the original ControlNet structure. Specifically, we show 1) The new task-aware modulated ControlNet preserves the zero-initialization property of ControlNet; 2) The parameters of the task-aware modulated Controlnet can be updated once we start to train the model.
Denote the input feature map by æ. the frozen SD Block in Fig. 2 by FSD. the extra condition by 6, two zero convolution operators by 2/, (.) and 2%, (-), the trainable copy SD Block by 9 (-), the task instruction by Ctask, and the task-aware hyperNet by Ho, Then the output of the new task-aware modulated Controlnet can be expressed as
ye = Fz)(z) + Zg, (G (x + (c)(c)
(c) Hew (Crask))) How (Crask).
(I)
Property Zero Initialization. Similar to ControlNet (21. the weights and biases of the convolution layers are initialized zeros. As a result, we have 2/, (.) = 0 and ye Fz) regardless of the initialization of Ho. (.).
16

---

Gradient Analysis. We analyze the gradient of the modulated part
Vo (Zo, (1) Hop (Crask)) = Hax (Crask) V L3, (1) + Zb. (1) Vox Hox (Crask),
(2)
where I is the input of the zero convolution layer.
When we start to train the network, the first part of the RHS of ( follows similar analysis of ControlNet (21] since Ho, (Ceask) is constant when we analyze the gradient Va Zo (7). Since the parameters of Hox (Ctask) are not initialized to zero, it is known that Ho, (Ctask) * 0. So the gradient dynamic follows the analysis of ControlNet. Therefore, we conclude that Z (7) A 0 after the first gradient update, and that the network can start to learn and update the following standard dynamics of stochastic gradient descent.
As for the second part of the RHS of 2 2 (1) 0 before the first gradient update, so the gradient
zero for 0z. However, after the first gradient update of 01, we know Z (T) o 0, and fy can be updated with non-zero gradients.
To conclude, the new task-aware Modulated ControlNet can still be efficiently updated and learned even if the convolution layers are initialized to zero.
C Zero-shot-task Results and Analysis
We show more zero-shot-task results in this section, where the tasks have not been trained on. In Fig. [1]] we show zero-shot deblurring results guided by the keywords. Our deblurred images can successfully recover the fine- -ne-graieed details of the images without training on such data. We note that some details are still missing, egg., the details in the painting in the first row are still not clear enough. In Fig. [4] we illustrate two zero-shot image colorization results. We believe that most parts of the generated images are acceptable, though the clothes of the second woman do not look the same to the input blurred image. In Fig. [5] we observe impressive zero-shot inpainting results. In the first row, the duck that is inputted in the text has been successfully generated in the inpainted image. The second row obtains acceptable results well, though the faces do not look perfect. The overall zero-shot quality of UniControl is remarkable.
While inpainting and outpainting might appear related, they are fundamentally distinet. Inpainting heavily leverages the contextual information from unmasked regions, necessitating a precise match. Conversely, outpainting has more freedom, with the generative model prioritizing prompts to envision new content. As shown in Fig. g. directly using outpainting model for inpainting tasks can be challenging since the model tends to leave a sharp change over the mask boundaries. Our pretrained UniControl, thanks to intensive training across multiple tasks, has learned edge and ongion-to-imae mappings, which assists preserving contextual information.
Our model also demonstrates a promising capacity to generalize under scribble conditions, showing parallels to the ControlNet's ss ability, even though UniControl hasn't 't been directly trained using scribble data. Fig. provides results illustrating the criibllett-immae generation.
D Details of User Study
In the evaluation steps, we use Amazon Mechanical Turk (Mtrr)]to perform user study. Specifically, we ask three Mturk master workers to select the best output result for each input condition. As shown in Fig. 10, we provide instructions on guidelines to select the best generated image. The annotators are provided the condition map and the text that describes the image, and are required to select the better output between the two generated images. Considering that images can both in good or bad qualities, we provide the tie option as well. We use the majority vote to determine the result of each image. which means that image is considered for it. We use 294 images for the tasks of Canny, HED, Surface Normal, Depth, Segmentation, User Sketch, and Outpainting. We adopt 100 images for the task of Human Skeleton and 187 images for the task of Bounding Box. In summary, we totally obtain 7,035 voting results for all nine tasks. 2/3 of source images in testing set collected from MSCOCO with the remaining 1/3 from Laion. And it includes a very diverse range of topics including indoor scene, outdoor scene, oil painting, portrait, pencil sketch, animation, cartoon, etc.
better image if two more annotators vote
thtpss//www.mturk.com
17

---

Input (Masked Image)
Ours-SingleOuutaanntnng
UniControl
"Wedding hair and makeup with warm tones and red lips"
"Contemporary Bedroom Designs 2015 modern bedroom designs intended design"
Figure 8: Visual comparison of Oun-ssingle-uutpaannnng and UniControl the inpainting task. The single outpainting model cannot well address the zero-shot inpainting task whereas UniControl demonstrates promising capacity.
Input (Scribble)
ControlNet-Scribble Results
UniControl Results (zero-shot)
"a turtle river"
"a digital painting of a hot air balloon"
"a door on wall"
Figure Visual comparison Searroleet-Seribbll and UniControl on the scribble data. ControlNet- Scribble is trained by the scribble data which, however, are unseen for UniControl.
18

---

Depth lmage
Gononsted Irage
Genostted Irage?
Euge cuccon on Brindy
dirtentdd Immap
Equilyy good or
Camps Crea nes These, liir Daringg
Figure 10: Mturk interface to select the better generated image.
Sketch-to-lmage
80
60
17%
Multi
Ours-Single
Tie
Figure 11: User study results of User Sketch to image generation.
Failure Cases
We illustrate some failure cases in Fig. 12 In the first row, although generated image successfully aligns the Bounding Box condition, the generated human has a distorted body. In the second row, our generated image looks similar the ground truth; however, the human faces blurred. We think that the reason is that UniControl inherits the data and model bias of Stable Diffusion, where the generated human commonly have issues. In the third row, the generated image does not look realistic. We believe that the training data can be improved both quantitatively and qualitatively.
Additional Results
We illustrate more visualized results in this section on tasks Canny (Fig. I6 HED (Fig. Depth (Fig. (8). Surface Normal (Fig. 19 Human Skeleton (Fig. 20 Bounding Box (Fig. 21).
Segmentation (Fig. 22) and Outpainting (Fig. 23). These results further demonstrate the effectiveness of our proposed method. Moreover, due to the space limitation in the main paper, we report results of the last task, User Sketch. Given sketched image, UniControl is able to achieve promising realistic
images. The visualized results are in Fig. 24 The user study result can be found in Fig. []] where it observed that UniControl obtains significantly more votes than the single task model.
19

---

Visual Condition
Our Result
Ground Truth
"La tricoteuse Realism William Adolphe Bouguereau Oil Paintings"
"A man and woman in ski gear standing in front of a mountain.
"The Taj Mahal mirrored by water fountain's 's reflection. Agra, Uttar Pradesh, India Daily Travel Photos"
Figure 12: Failure Cases: distorted body (row one); blurred faces (row two); incorrect creation (row three).
20

---

Blurred Image
Our Result
"Bedroom Colour Ideas 25 Paint Colours With Impact Living"
"Christa McAuliffe (right, sat with her backup crew member Barbara Morgan) was a social studies teacher who had won NASA's Teacher in Space contest and earned herself spot on the mission"
"Mountain Village in the Alps Canvas print - Bedroom"
Figure 13: More zero-shot-task deblurring results.
21

---

Gray Image
Our Result
"Long White Casual Wedding Dress"
"Pixie Cropped Short Layered Synthetic Wig for Women-KAMI WIGS"
"Early morning view over the town of Tinerhir, south of the Todra Gorge, Morocco, North Africa, Africa"
Figure 14: More zero-shot-tee gray-to-RGB colorization results.
22

---

Cropped Image
Our Result
"A lone duck basks in the calm lake's mirror reflection of the Chugach mountain valley"
"Chancellor of the Exchequer Rishi Sunak was the most high-profile, and unexpected, appointment of the day"
"Contemporary Bedroom Designs 2015 modern bedroom designs intended design*
Figure 15: More zero-shot-tase image in-painting results. The in-painting MOE adapter weights are directly inherited from outpainting.
23

---

Input Image
Our Methad Output (a) "Two arms typing on laptop and hand on a mouse"
Input Image
Our Method Output
"Two people walking along a walk next to train on the tracks."
Method Output (c) "A close up of glazed donuts that plain with chocolate."
Image
Input Image (d) "A group elephants with water in front and trees behind."
Our Method Output
Figure 16: Canny to Image Generation
24

---

Image
Methad Output
"A person skis makes her way through the snow"
Image (b) "Three zebras grazing in grassy area near shrubs"
Method Output
dralla
Inout Image cc) "The Taj Mahal mirrored by water fountain's reflection. Agra, Uttar Pradesh, India Daily Travel Photos"
Method Output
Input Image
Our Method Output
(d) "A young girl who is brushing her teeth with toothbrush."
Figure 17: HED to Image Generation
25

---

Input Image
Method Output
(a) "The Ta Prohm Temple Located at Angkor in Cambodia by Kylc Hammons"
Our Method Output (b) "A brown dog standing wooden bench lemon tree."
Input Image
image
Method Output
Polar walks toward water, while large bird lands on the opposite bank." "
AOSH AOSHAUW RAVAAEES
Input Image
Method Output
(d) "A display of vintage animal toys on the floor." Figure 18: Depth to Image Generation
26

---

ala0 OARiGO
Tnt Mon mnnmam
Input Image
Our Method Output
(a) "A line of small teddy bears are in front several DVD cases."
Input Image
Our Method Output
(b) "A in the kitchen standing with his dog"
Input Image
Method Output
(c) "A hot dog sitting top of bun in wrapper"
Input Image
Our Output
"Several people waiting the side of train tracks train with it's lights comes down the track"
Figure 19: Surface Normal to Image Generation
27

---

Input Image
Our Method Output
(a) "Photo of handsome man in black leather jacket"
Input Image
Output
(b) "A woman sitting near prominent landmark"
Input Image
Output
(c) "A man that has ski's and is standing in the snow."
Input Image
Our Method Outout
(d) "A woman is sitting in front of a desk"
Figure 20: Human Pose Skeleton Image Generation
28

---

Input Image
Our Method Output
(a) "A woman walking two dogs the snow"
Input Image
Output
(b) "Simone Righi frasi glasses linen suit menswear streetstyle icon fashion florence."
Mathod Output
(c) "The large has wooden table with chairs and couch."
Input Image
Our Method Output
(d) "Two black bags placed standing the ground"
Figure 21: Bounding Box (by YOLO-V4-MSCOCO) to Image Generation
29

---

Input Image
Our Method Output
(a) "A bench at the beach next the sea"
Input Image
Our Method Output
(b) "Water traffic along the Thames by Big Ben"
Input Image
Our Method Output
(c) "A well-lit and well-decorated living room shows glimpse of glass front door through the corridor."
Input Image
Our Method Output
(d) "Blue Hour Barley"
Figure 22: Segmentation Map (by UnifO))er-ADE20K) to Image Generation
30

---

Input Image
Our Method Output
(a) "St, Michaels Mount, Cornwall"
Input Image
Our Method Output
"Melhoure by teckay 72"
Input Image
Our Method Output
(c) "Lady Black Kimono Paint by Diamonds"
Input Image (d) "Beautiful kitchen grand scale living pinterest for Kitchen cabinets lowes with old world metal wall art"
Method Outout
Figure 23: Image Outpainting
31

---

Input Image
Our Method Output
(a) "A Limited Edition, Fine Art photograph of beautiful sunrise at Lake Jackson in Sebring. Florida. Available Fine Art print"
Input Image
Our Method Output
(b) "Giraffes in Lake Manyara national park"
Input Image
Our Method Output
(c) "Wild Ones 1991 Limited Edition Print Frank McCarthy"
Input Image
Our Method Output
(d) "Superhero watching over city. No transparency used. Basic (linear) gradients. A4 proportions."
Figure 24: User Sketch to Image Generation
32


Reviews:
[{"review_id": "YrN2Ggb5Fo", "reviewer": "Reviewer_hjKo", "summary": "The paper presents UniControl, that unifies multiple visual controlling condition into a single unified model. To achieve this, the authors introduce a task-aware HyperNet to modulate the diffusion models, enabling adaptation to different condition-to-image (C2I) tasks simultaneously. UniControl is trained on nine unique C2I tasks and demonstrates impressive controlled generation quality. It also demonstrates zero-shot generation abilities with unseen visual conditions, including condition combination and new conditioning. User studies show that UniControl often surpasses the performance of single-task-controlled methods of comparable model sizes.", "questions": "- Can the authors please clarify if the difference between Ours-Single and ControlNet is only the training data / training schedule or there are other differences? From the results in Fig. 6 and Fig. 7, it seems that Ours is much better than ControlNet, while Ours is only slightly better than Ours-Single.\n\n- Where does the performance can mainly come from ControlNet to Ours-Single? Is it because of the better dataset and/or longer training? Note that this is not a criticism, so I put it here in the questions. But I think a clarification or analysis in this regard can be helpful for the readers and the research community.\n\n- Cost comparison with ControlNet. In L38-39, the authors mention \"Retraining a separate model is necessary to handle a different modality of visual conditions, incurring non-trivial time and spatial complexity costs.\" However, UniControl requires 5000 GPU hours on A100, while ControlNet training for a single model is usually just 100-300 hours. This somehow invalidates this point.", "limitations": "The authors discuss the limitations in the main papar.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an interesting and important research topic, unifying the controllability of different models into a single model.\n- The authors compare the results with ControlNet and demonstrate better qualitative results.\n- The zero-shot task generalization and instruction combination aspects of the proposed method are intriguing and valuable.", "weaknesses": "- The paper lacks quantitative evaluations for the alignment of the generated content and conditional inputs. For segmentation and bounding box, pretrained detectors could be used for evaluating the alignment, following the configuration in [1]. Although ControlNet does not handle bounding box conditions, a baseline for object detection instruction-following would be [1].\n\n[1] Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., ... & Lee, Y. J. (2023). Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22511-22521).\n\n- Zero-shot task generalization seems underexplored, and there is a lack of detailed analysis.\n    - Inpainting results are not surprising as they share a strong alignment in terms of task instruction with the pretraining with outpainting.\n    - For deblurring and colorization, it would be better to illustrate how the model can perform zero-shot task generalization. Does the capability mainly come from the task instruction encoder?\n    - If so, it would be more intriguing to demonstrate results with more tasks that can really benefit from the zero-shot task generalization. It is easy to obtain data for colorization, inpainting. It would be valuable to demonstrate results on tasks that may be hard to collect training data.  For example, for scribble to image, ControlNet utilizes strong human-crafted data augmentation to synthesize the scribbles. Can UniControl generalize well to this case?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable time to review this paper. We sincerely appreciate your constructive comments. Your concerns are addressed below.\n\n**Q1:  More quantitative evaluations for the alignment of the generated content and conditional inputs**\n\nThank you for highlighting this. We agreed with your feedback and have incorporated additional quantitative evaluations. We've included FID metrics and the average precision (AP) metrics, adopting the methodology from GLIGEN [1], and utilizing pre-trained detectors.\n\nIn addition to GLIGEN [1] as you suggested, we've also incorporated T2I-adapter [2] and ControlNet [3] as baselines, given their foundational contributions to this domain. Our experimental setup adheres to Section 4.3 of the main paper, and for sampling, we employ DDIM with a guidance score of 9. We've collected over 2,000 test samples from Laion and COCO, and conducted evaluations across six diverse tasks: these span edges (Canny, HED), regions (Seg), skeletons (Pose), and geometric maps (Depth, Normal).\n\nOur FID table, presented subsequently, showcases evaluations related to visual quality. A clear observation from the table is that our UniControl model consistently surpasses the baseline methods across the majority of tasks. Notably, UniControl also offers a more dense architecture compared to other single-task methods, which typically needs multiple checkpoints.\n\nFID Scores\n\n|  | GLIGEN | T2I-adapter  | ControlNet | UniControl |\n|--|--|--|--|--|\n| Canny | 24.9 | 23.6 | **22.7** | 22.9 |\n| HED | 27.8 | - | 25.1 | **23.6**  |\n| Depth | 25.8 | 25.4 | 25.5 | **21.3** |\n| Normal | 27.7 | - | 28.4 | **23.4** |\n| Seg | - | 27.1 | 26.7 | **25.5** |\n| Pose | - | 28.9 | 28.8 | **27.4** |\n\nTaking your advice, we've integrated the pre-trained YOLO-v4 detector. We deployed our UniControl model to generate images corresponding to the bounding box masks and captions from the COCO14-Val dataset. The summarized AP scores in the ensuing table demonstrate that our method is superior over GLIGEN.\n\nAP Scores on COCO (YOLO-V4)\n\n|  | GLIGEN | UniControl |\n|--|--|--|\n| AP | 24.0 | **26.2** |\n| AP_50 | 42.2 | **45.0** |\n| AP_75 | 24.1 | **26.3**  |\n\n[1] GLIGEN: Open-Set Grounded Text-to-Image Generation. CVPR 23.\n\n[2] T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. arXiv 2302.08453.\n\n[3] Adding Conditional Control to Text-to-Image Diffusion Models. arXiv 2302.05543.\n\n**Q2:  More details and exploration of the zero-shot tasks**\n\n*Inpainting and Outpainting:* While inpainting and outpainting might appear related, they are fundamentally distinct. Inpainting heavily leverages the contextual information from unmasked regions, necessitating a precise match. Conversely, outpainting has more freedom, with the generative model prioritizing prompts to envision new content. Directly using outpainting model for inpainting tasks can be challenging since the model tends to leave a sharp change over the mask boundaries. Our pretrained UniControl, thanks to intensive training across multiple tasks, has learned edge and region-to-image mappings, which assists in preserving contextual information. We've incorporated visual comparisons in the attached pdf for further clarity.\n\n*Deblurring and Colorization:* UniControl processes blurred or grayscale images as visual cues, relying on textual prompts and specific task instructions like \"grey to image\" or \"image deblurring\" during inference. The MoE-style adapters are structured based on the similarities observed among the pre-trained tasks, as explained in Sec. 3.3. For instance, the colorization task uses weights as \"depth: 0.6, seg: 0.3, canny: 0.1\". This adaptability stems from a blend of task-specific instructions and the MoE-style adapters. Misaligned adapters or inappropriate task instructions compromise the model's performance.\n\n\n*Scribbles:* Thank you for your suggestion. Indeed, our model demonstrates a promising capacity to generalize under scribble conditions, showing parallels to the ControlNet's ability, even though UniControl hasn't been directly trained using scribble data. Our pdf file provides additional results illustrating the scribble-to-image generation.\n\n**Q3:  Ours-Single vs ControlNet**\n\nThe only difference between Ours-Single and ControlNet is the training data. The authors of ControlNet have not released their training data. As a result, it is necessary to reimplement the ControlNet model using our collected dataset, MultiGen-20M. Notably, MultiGen-20M is set to be the pioneering open-sourced conditional visual generation dataset in this area. The variance observed between Fig. 6 and Fig. 7 results from these differing datasets. We have filtered images with higher resolutions (>=512) and aesthetic scores (>6) from Laion whose quality would likely be better than the original ControlNet’s training data. The data quality is essential for visual performance. Therefore, Ours-single is better than the official ControlNet on these tasks.\n\n**Q4:  Fair comparison of training cost**\n\nThank you for your careful observation.  Indeed, the ~5000 GPU Hours we've stated are similar to the cumulative training cost of multiple ControlNets. For our work, we utilized the A100-40G, whereas ControlNets employed the A100-80G. Let's break down ControlNets' training costs:\n|  | Canny | HED | Pose | Seg| Depth| Normal|Sketch | Total|\n|--|--|--|--|--|--|--|--|--|\n| Hours |600 | 300 | 300 | 400 | 500 |  200| 150 | 2450 |\n| GPU Type |A100-80G  | A100-80G | A100-80G | 3090TI | A100-80G |A100-80G  | A100-80G |   |\n\nAs we can see, most ControlNets are trained by A100-80G whose total training cost is ~2500 GPU Hours.  In contrast, our use of the A100-40G, which has been shown to be less powerful than the A100-80G as per [4].\n\n[4] https://www.topcpu.net/en/gpu-c/a100-sxm4-40-gb-vs-a100-pcie-80-gb\n"}, {"review_id": "LI13ncXuMJ", "reviewer": "Reviewer_J9bc", "summary": "This paper presents a method for controlling the output of a diffusion\nmodel with multiple modalities of reference images, e.g. edges,\nsegmentation, depth, etc. It can be seen as proposing a multi-task\nversion of ControlNet. Experiments in the paper show the multi-task\napproach outperforms the single-task, and also allows zero-shot\napplicability to novel tasks such as colorization.", "questions": "Please refer to the weaknesses section.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper effectively demonstrates architecture modifications for multi-task controlling model\n  and shows that it is generally better than the single-task (Table\n  1).\n- The model generalization to tasks that were not in the training set such\n  as colorization, deblurring or inpainting is quite remarkable.\n- The paper introduces a dataset with 20M multi-modal condition\n  training pairs.", "weaknesses": "The paper presents a well engineered solution to achieving a\nmulti-task version of ControlNet, and show generalization to some new\ntasks. Some decision justifications are not backed up by ablation\nexperiments and the experiments on task generalization are\ndemonstrated only with a few visual examples.\n\n- If I understood correctly, there \"Mixture of Experts\" component is\n  manually selecting the encoder for each modality. In that case\n  calling this module a MOE is justified, since it could be misleading.\n- Unless I missed it, the different components in the design are not\n  ablated. What is the contribution of the Hypernet task embedding?\n- How do you explain the poor performance of ControlNEt in\n  Normal-to-Image (Fig. 6)\n- Quantitative comparison for the generalization to new tasks is\n  limited to a few visual examples. Some of the examples are even\n  repeated three times (segmentation+skeleton); for that case, it may\n  be more convincing if the paper showed three different results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments and suggestions for our submission.\n\n**Q1:  Missing Ablation Study**\n\nThank you for pointing this out. We've conducted an ablation study with FID scores as follows. Our experimental setup adheres to Sec. 4.3 of the main paper, employing DDIM as the sampler and a guidance score of 9. Drawing from a collection of over 2,000 test samples sourced from Laion and COCO, we applied six tasks for evaluation, spanning edges (Canny, HED), regions (Seg), skeletons (Pose), and geometric maps (Depth, Normal). It's evident that the UniControl model (incorporating both MoE-Style Adapter and TaskHyperNet) consistently surpasses the results of its ablations.\n\n| MoE-Adapter | TaskHyperNet | Canny | HED | Depth | Normal  | Seg  |  Pose | Avg|\n|--|--|--|--|--|--|--|--|--|\n| x | x | 27.2 | 29.0 | 27.6 | 28.8 |29.1  | 30.2 | 28.7 |\n| ✓ | x | 24.5 | 26.1 | 23.7 | 24.8 | 26.9 | 28.3 | 25.7 |\n| ✓ | ✓ | **22.9**  | **23.6** | **21.3** | **23.4** | **25.5** | **27.4** | **24.0** |\n\n**Q2: UniControl is an engineered solution for multi-task version of ControlNet**\n\nUniControl is deeply influenced by the principles of HyperNetwork [1] and multi-task visual learning as Taskonomy [2]. It embodies the concept of \"Control over the Control\" or \"Meta Control.\" We believe that unifying diverse visual modalities and tasks within a single framework is not just an engineering endeavor, but a significant scientific challenge since it is required to keep both single-task discriminability and multi-task generality.\n\n[1] HyperNetworks. ICLR 17.\n\n[2] Taskonomy: Disentangling Task Transfer Learning. CVPR 18.\n\n\n\n\n**Q3: Justification of MoE**\n\nYes. It is not the MoE as its assembly is not learnable. Because of this distinction, we name it MoE-style adapter instead of MoE in the paper. But the high-level ideas parallel since each module catering to a specific modality, effectively functioning as an \"Expert\" network. When dealing with zero-shot tasks, the adapter's weights can be autonomously determined through the computation of similarity scores derived from task embeddings.\n\nThe disparity in results can be attributed to the differing training data used for ControlNet and UniControl. Since ControlNet hasn't made its training data public, we were compelled to create our own datasets for both training and testing. In our approach, we selected images from Laion with a high resolution (>=512) and aesthetic scores exceeding 6. It's plausible that the quality of these images surpasses that of the original ControlNet’s training data for such tasks.\n\n**Q4: Question of Fig. 6 about the poor result of Normal-to-Image ControlNet**\n\nThis Can be attributed to the differing training data used for ControlNet and UniControl. Since ControlNet hasn't made its training data public, we were compelled to create our own datasets for both training and testing. In our approach, we selected images from Laion with a high resolution (>=512) and aesthetic scores exceeding 6.  It's plausible that the quality of these images surpasses that of the original ControlNet’s training data for these tasks.\n\n\n**Q5: More results on zero-shot tasks**\n\nThank you for your suggestion.  We've incorporated additional zero-shot results in the Appendix. It's worth noting that the zero-shot capabilities of UniControl are a welcome byproduct. An exhaustive exploration of arbitrary zero-shot success deserves its own comprehensive study and extends beyond the scope of our current focus. We remain hopeful that future research in this direction will exhibit the reliable zero-shot generalization capacities akin to those observed in LLMs.\n"}, {"review_id": "Bm08K6MR4k", "reviewer": "Reviewer_Haxt", "summary": "This paper introduces UniControl, a new generative foundation model that consolidates a wide array of controllable condition-to-image (C2I) tasks within a singular framework. UniControl enables pixel-level-precise image generation, where visual conditions primarily influence the generated structures and language prompts guide the style and context. For this purpose, the authors augment pretrained text-to-image\ndiffusion models (ControlNet) and introduce a task-aware HyperNet to modulate the diffusion models, enabling the adaptation to different C2I tasks simultaneously. UniControl was trained on nine unique C2I task and demonstrated excellent zero-shot generation abilities with unseen visual conditions.", "questions": "Here are my concerns:\n1. The paper claim in the abstract that: \"...UniControl often surpasses the performance of single-task-controlled methods of comparable model sizes\". Maybe I missed something, but could the authors elaborate more on this statement? What are the other 'single-task-controlled methods' they refer to? I only found comparison with ControlNet.\n2. Could you adapt your approach to work with other Image Generation software (besides Stable Diffusion)?", "limitations": "Limitations are addressed in the paper in a dedicated section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n2. The related work section covers the most relevant papers in the field.\n3. The approach produces excellent image generation results.\n4. The experimental evaluation is convincing.", "weaknesses": "1. It relies too much on existing image generation methods (ControlNet).\n2. A more scientific contribution would have been expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time to review this paper. Your concerns are addressed below.\n\n\n**Q1: More comparison with the single-task-controlled methods**\n\nThank you for your valuable suggestion. In response, we've expanded our quantitative analysis to include classic single-task-controlled methods such as GLIGEN [1], T2I-adapter [2], and ControlNet [3]. Our experimental setup remains consistent with Sec. 4.3 of the main manuscript, utilizing DDIM as the sampler with a guidance score of 9. With a collection of over 2,000 test samples sourced from Laion and COCO, we've assessed a wide range of tasks covering edges (Canny, HED), regions (Seg), skeletons (Pose), and geometric maps (Depth, Normal).\n\nThe following FID table demonstrates that our UniControl consistently surpasses the baseline methods across the majority of tasks. Notably, UniControl achieves this while maintaining a more compact and efficient architecture than its counterparts.\n\n\nFID Scores\n|  | GLIGEN | T2I-adapter  | ControlNet | UniControl |\n|--|--|--|--|--|\n| Canny | 24.9 | 23.6 | **22.7** | 22.9 |\n| HED | 27.8 | - | 25.1 | **23.6**  |\n| Depth | 25.8 | 25.4 | 25.5 | **21.3** |\n| Normal | 27.7 | - | 28.4 | **23.4** |\n| Seg | - | 27.1 | 26.7 | **25.5** |\n| Pose | - | 28.9 | 28.8 | **27.4** |\n\n[1] GLIGEN: Open-Set Grounded Text-to-Image Generation. CVPR 23.\n[2] T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. arXiv 2302.08453.\n[3] Adding Conditional Control to Text-to-Image Diffusion Models. arXiv 2302.05543.\n\n\n**Q2: Adapt UniControl method to work with other Image Generation software (besides Stable Diffusion)**\n\nYes. UniControl is versatile and can be adapted to various diffusion-based models, including the likes of Deep-floyd [1]. To facilitate this integration, we project the embeddings from UniControl onto the new model's backbone using cross-attention layers or linear mapping. However, it's essential to note that this integration necessitates re-training UniControl to ensure seamless alignment with the new backbone.\n\n\n[1] https://github.com/deep-floyd/IF\n"}, {"review_id": "gVnilDcK0V", "reviewer": "Reviewer_y5r6", "summary": "UniControl is a diffusion-based image generation model that can condition on natural language input as well as multiple types of visual inputs (e.g. edge map, depth map). The framework is built upon components of Stable Diffusion Models, an MOE adapter and a ControlNet modulated by a task-aware HyperNet. The model demonstrates abilities of conditioning on one or multiple visual inputs at a time, as well as visual inputs that it has not seen during training.", "questions": "- HyperNet should be cited?", "limitations": "The authors have discussed the limitations and broader impact of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- UniControl extends ControlNet to work with multiple tasks and shows that the tasks help each other so as to improve performance on single task metrics as well.\n- the MOE adapter set up easily allows the model to condition on multiple visual inputs.", "weaknesses": "- The model has a certain complexity, as it involves multiple modules such as the MOE adapter, the ControlNet and a HyperNet.\n- It needs to maintain two sets of SDM parameters. (This is the same as ControlNet, though.)\n- MOE adapter means more parameters to be added with each new task added.\n- Task instruction is needed for both training and inference, and is handled by a separate module than the language prompt, even though both are text prompts. This can be a downside if the task instruction is unknown or not well defined.\n- It is not clear what $c_{task}$ is used for the task-aware hypernet during hybrid-task or zero-shot new task inference.\n\nAblation study would be helpful to show the importance/usefulness of the proposed components:\n- model performance without the MOE adapter.\n- model performance without conditioning on $c_{task}$, or merge $c_{task}$ into $c_{text}$ (in a thoughtful way). This will show whether the task-aware hypernet is needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your suggestions to enrich our paper. Your concerns and questions are addressed as follows.\n\n**Q1:  Complexity of UniControl**\n\nComponents and #params of whole UniControl Model and Multi-ControlNet\n|  | Stable Diffusion | ControlNet  | MoE-style Adapters | TaskHyperNet | Total|\n|--|--|--|--|--|--|\n| UniControl| 1065.7M  | 361M | 0.06M | 12.7M | 1.44B|\n| Multi-ControlNet| 1065.7M  | 361M * 9  |- | - | 4.315B|\n\nCompared with the original ControlNet Model (incorporating Stable Diffusion + ControlNet), the complexity of UniControl has seen a marginal increase of 0.09%, amounting to an addition of 12.76M parameters. Despite this modest augmentation, UniControl's capability extends to nine distinct tasks. In contrast, the Multi-ControlNet (an ensemble of several ControlNets) demands a separate Stable Diffusion plus nine individual ControlNets to accomplish the same tasks. This underscores the efficiency of UniControl, which at 1.44B parameters, significantly trims the complexity when compared to the 4.315B parameters of Multi-ControlNet, all the while achieving a comparable objective.\n\n\n**Q2:  MoE adapter means more parameters to be added with each new task added.**\n\nMoE-style adapters are exceptionally lightweight. It has ~0.06M #params whose size and associated computation cost are marginal compared with the entire model. \n\n\n**Q3:  Downside of task instruction**\n\nAs explained in Sec. A.2 of the Appendix, we utilize a predefined set of task instructions that can be systematically matched to each respective task, ensuring a consistent and transparent process. When it comes to unknown tasks, our model demonstrates robustness towards a spectrum of new instructions. This is largely attributed to the clustering of their embeddings, due to their semantic similarities.\n\nTaskHyperNet is efficient in size, comprising just 12.7M parameters. And there are some techniques for acceleration. One such technique involves offline collection of task embeddings. By doing so, we can bypass the need for instruction-to-embedding inference and instead directly retrieve the pre-calculated task embeddings, expediting the process.\n\n\n\n**Q4:  How does Task-aware HyperNet use in the zero-shot tasks?**\n\nIn the context of zero-shot tasks, we defined the task instruction in a straightforward way. Examples include designations like \"image inpainting\" or \"segmentation map and human skeleton to image.\" These instructions are subsequently processed by CLIPText and the Task-aware HyperNet to derive the requisite task embeddings.\n\n\n**Q5:  Missing Ablation Study**\n\nThank you for pointing this out. In response, we've conducted an ablation study, specifically focusing on the MoE-Style Adapter and TaskHyperNet. The table contains the FID scores. It is noticeable that the full-version UniControl (MoE-Style Adapter + TaskHyperNet) constantly outperforms the ablations.\n\n\n| MoE-Adapter | TaskHyperNet | Canny | HED | Depth | Normal  | Seg  |  Pose | Avg|\n|--|--|--|--|--|--|--|--|--|\n| x | x | 27.2 | 29.0 | 27.6 | 28.8 |29.1  | 30.2 | 28.7 |\n| ✓ | x | 24.5 | 26.1 | 23.7 | 24.8 | 26.9 | 28.3 | 25.7 |\n| ✓ | ✓ | **22.9**  | **23.6** | **21.3** | **23.4** | **25.5** | **27.4** | **24.0** |\n\n**Q6:  Cite HyperNet**\n\nThank you for your suggestion. It is our mistake. We have updated these two references [1,2] in our latest manuscript.\n\n[1] HyperNetworks. ICLR 17.\n\n[2] Continual Learning with Hypernetworks. ICLR 20.\n"}, {"review_id": "p0BCK4NXp9", "reviewer": "Reviewer_HFTZ", "summary": "The paper proposes a unified diffusion framework, UnitControl for a more fine-grained controllable generation based on input visual conditions, utilizing two modules mixture-of-experts adaptor that extract features from different visual conditions, and a task-aware HyperNet that extracts language-based task embedding to control visual generation.", "questions": "1. How are both hybrid tasks generalization and zero-shot new task generalization done? For example, a detailed procedure (i.e., sampling technique) will be useful for better understanding.\n2. There are 6 tasks overlapped with ControlNets, each of which is trained for 100K iterations. However, there are 8 tasks in your multi-task model, but it is trained for 900K, which in fact is not a fair comparison. If I misunderstood, feel free to point that out.\n3. The method mainly uses modalities that are spatial, so is it capable of using global context, e.g., color, texture, to guide image generation?", "limitations": "The authors have included limitations and broader impact such as training data bias.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper conducts a relatively comprehensive evaluation of the proposed method, including human evaluations.\n2. The framework can be useful as it unified different modalities for visual controllable generation using diffusion models.\n3. The authors promise to open-source 20M triple training datasets that can benefit the whole community.", "weaknesses": "1. **The writing is not so well written and missing details that can hinder understanding of readers**. For example, Figure 2 should also mention parameters of encoder and decoder since I initially thought the introduced parameters are only from MOE adaptors and Hypernet. The authors should also highlight the differences and **similarities** (e.g., encoder and decoder are copied) between their work and ControlNet. In Section 3.3, how is hybrid task generalization done? Since your model only conditions on one visual modality when computing the score in one forward pass, do you compute scores individually for each condition for generation? If so, what sampling technique did you use? One sampling technique I can think of is the conjunction operator from Composable Diffusion [1] or do you use some other existing techniques? For zero-shot task generalization, I also find it confusing and don't quite understand how it is done. Mentioning it in the supplementary is not sufficient since it is meant to provide details that don't affect your understanding overall.\n\n2. **The claims are quite shaky.** The authors claim that the model need to tackle the misalignment of low-level features from different tasks. However, based on Figure 4, I don't see too much of misalignment from baselines, i.e., ControlNet. Besides the visual quality, I don't see much of advantages over ControlNet. Even sometimes single-task ControlNet is better in some aspects. For example, for the object bounding box to image, control net seems to understand there should be two separate benches instead of one long bench generated by the proposed model. In addition, since one unified model is trained for longer time, model could be more robust, directly improving overall visual quality.\n\n3. **The zero-shot generalization ability could be misleading.** Suppose in training, there doesn't exist any instructions that perform colorization. I don't expect the language itself can bridge the gap between visual signals and language to perform such unseen tasks. For example, in Figure 5, such zero-shot results doesn’t necessarily reflect the true reality that the model understands given instructions. One way to check this is to reverse the whole process - for example, can you do image decolorization using your prompt or blur the image instead. Since training images are mostly high resolution, and colored images, the model could automatically utilize learned prior to make images higher quality, which doesn’t necessarily mean they are doing what they are told to do. If so, then the model just does general reconstruction when prompted something unknown.\n\n4. **Missing quantitative evaluations.** As stated, one contribution is to show that the visual quality of such controllable generation can be improved, then it is needed to include quantitative comparisons for image fidelity using metrics such as FID and KID, though I don't think they are good metrics but it provides enough insights.\n\n5. **Lacking relevant baselines.** There are also other existing adaptor methods for such generation, for example, T2I-adaptor [2] and GLIGEN [3] seem to be highly relevant, and both of them are not used as a comparison in the paper.\n\n5. **Related work.** If one of the contributions is to combine different modalities for compositional generation, then it would be important to add existing works related to compositional generation.\n\n[1] Liu et al., Compositional Visual Generation with Composable Diffusion Models (ECCV 2022) \\\n[2] Mou et al., T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. \\\n[3] Li et al., GLIGEN: Open-Set Grounded Text-to-Image Generation (CVPR 2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your suggestions and questions of our paper. Your concerns are addressed as follows.\n\n**Q1:  UniControl vs ControlNet**\n\nComponents and #params of whole UniControl Model and Multi-ControlNet\n|  | Stable Diffusion | ControlNet  | MoE-style Adapters | TaskHyperNet | Total|\n|--|--|--|--|--|--|\n| UniControl| 1065.7M  | 361M | 0.06M | 12.7M | 1.44B|\n| Multi-ControlNet| 1065.7M  | 361M * 9  |- | - | 4.315B|\n\n\nCompared with the original ControlNet Model (Stable Diffusion + ControlNet)), the increase in UniControl's size is  0.09%, amounting to an additional 12.76M parameters. Notably, UniControl's versatility spans nine distinct tasks. In contrast, Multi-ControlNet (assembly of multiple ControlNets) needs a single Stable Diffusion and nine separate ControlNets to achieve comparable results. UniControl (1.44B #params)  has greatly reduced the complexity compared with the Multi-ControlNet (4.315B #params) to achieve the same goal with better generation quality. The Stable Diffusion Model (including U-Net, CLIPText, VQVAE) is directly copied from its official checkpoint that is the same as the ControlNet's Stable Diffusion part.\n\n\n\n**Q2:  Details of zero-shot hybrid and new task inference**\n\nThank you for your suggestions. The work presented in [1] indeed offers a novel perspective on compositional generation via multiple conditions. However, UniControl adopts a distinct approach, detailed as follows:\n\n*Hybrid Task:* As illustrated in the left subfigure of Fig. 3, the features from two condition inputs are integrated through an addition operation. This fused feature set is consolidated into a single tensor, which is subsequently fed into the ensuing ControlNet modules that are interleaved with modulated zero-convs. The entire inference procedure follows the standard DDIM protocol the same as ControlNet and single task UniControl. \n\n*New Task:* Similar to the hybrid task, we feed the novel condition into a weighted ensemble of MoE-style adapters. An example configuration might be “depth: 0.6, seg: 0.3, canny: 0.1” for the colorization task. This process yields one feature tensor that is fed to subsequent modules. Again, the inference follows ControlNet's standard methodology, employing the regular DDIM.\n\nWe appreciate the reference to [1] and plan to incorporate it into our forthcoming manuscript.\n\n\n**Q3: More quantitative evaluations and baseline methods**\n\nThank you for your suggestion. We've conducted quantitative analysis to include classic baselines such as GLIGEN [1], T2I-adapter [2], and ControlNet [3]. Our experimental setup remains consistent with Sec. 4.3 of the main manuscript, utilizing DDIM as the sampler with a guidance score of 9. With a collection of over 2,000 test samples sourced from Laion and COCO, we've assessed a wide range of tasks covering edges (Canny, HED), regions (Seg), skeletons (Pose), and geometric maps (Depth, Normal).\n\n\nFID Scores\n|  | GLIGEN | T2I-adapter  | ControlNet | UniControl |\n|--|--|--|--|--|\n| Canny | 24.9 | 23.6 | **22.7** | 22.9 |\n| HED | 27.8 | - | 25.1 | **23.6**  |\n| Depth | 25.8 | 25.4 | 25.5 | **21.3** |\n| Normal | 27.7 | - | 28.4 | **23.4** |\n| Seg | - | 27.1 | 26.7 | **25.5** |\n| Pose | - | 28.9 | 28.8 | **27.4** |\n\n[1] GLIGEN: Open-Set Grounded Text-to-Image Generation. CVPR 23.\n\n[2] T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models. arXiv 2302.08453.\n\n[3] Adding Conditional Control to Text-to-Image Diffusion Models. arXiv 2302.05543.\n\n\n\n**Q4: Claims are shaky.**\n\nControlNet trains separate models for different tasks, thus there is no parameter sharing for each task, and no misalignment for ControlNet. MoE-adapter is specifically designed to solve the misalignment issue for UniControl. Notably, UniControl's versatility spans nine distinct tasks. In contrast, Multi-ControlNet (assembly of multiple ControlNets) needs a single Stable Diffusion and nine separate ControlNets to achieve comparable results. UniControl (1.44B #params)  has greatly reduced the complexity compared with the Multi-ControlNet (4.315B #params) to achieve the same goal with better generation quality. \n\n\nFor the bench example you mentioned (Fig. 4), the prompt is “A women sitting on a bench near a statue, checking her phone”. UniControl generates a real woman checking her phone, while ControlNet generates two statues, not checking her phone.\n\n\n\n**Q5: Zero-shot generalization ability is misleading**\n\nDirectly using outpainting model for inpainting tasks can be challenging since the model tends to leave a sharp change over the mask boundaries as shown in pdf file. Currently we think the generalization ability mainly comes from the combination of different visual conditions by assigning different weights of the visual input. If we want to have more stronger generalization ability on zero-shot tasks, we agree we need to specifically train it. \n\n**Q6: More related works [1,2,3]**\n\nThanks for pointing out the highly related works, we included [2,3] for comparison and will cite and discuss [1] in our upcoming version. \n\n\n\n**Q7: Fair comparison with ControlNet in training cost**\n\nActually, the iterations are not fair to compare since the different configurations of model training. Instead, we compare the training cost on the GPU Hours. UniControl is trained by ~5000 GPU Hours by A100-40G. This is comparable to the overall training cost of different ControlNets with ~2500 GPU Hours on A100-80G. \n\n|  | Canny | HED | Pose | Seg| Depth| Normal|Sketch | Total|\n|--|--|--|--|--|--|--|--|--|\n| Hours |600 | 300 | 300 | 400 | 500 |  200| 150 | 2450 |\n| GPU |A100-80G  | A100-80G | A100-80G | 3090TI | A100-80G |A100-80G  | A100-80G |   |\n\n\n\n\n**Q8: Inclusion of global context such as color and texture as condition**\n\nWe plan to include new conditions to UniControl similar as the T2I-Adaper-Color. Moreover, we can integrate the LoRA or Dreambooth to UniControl to control the style or texture of generative results.\n"}]

Author rebuttal:
Thanks to all the reviewers for your valuable time in reviewing this paper. We sincerely appreciate your constructive comments and questions to make this paper better. Below, we respond to each concern in the order.

Meta-review: This paper aims to consolidate different control conditions within a single model, distinguishing itself from ControlNet's approach of training separate models for each condition. It aligns more with the category of system papers and integrates a combination of existing techniques, including the original ZeroConv design from ControlNet, hard-gate Mixture of Experts (MOE), and HyperNet.

During the initial review phase, reviewers raised several concerns about the paper, primarily centered on the need for more comprehensive evaluation, analysis, and comparisons. This led to a varied range of feedback. However, the rebuttal effectively addressed most of these concerns, resulting in some reviewers who initially leaned towards a borderline rejection modifying their scores to borderline acceptance.

In summary, this paper hovers on the borderline of acceptance. After discussion, Area Chair finally recommends accept.
Decision: Accept (poster)