SlowGuess/NeurIPS_2023_OCR huggingface dataset:
task-aware models for each task. We apply default sampler as DDIM [32] with guidance weight 9 and steps 50. All single-task models used for comparison are trained by 100K iterations and our multi-task model is trained around 900K with similar iterations for each task to ensure fairness. The efficiency and compact design of our proposed model are evident in its construction. The total size of UniControl is around 1.5B #params and a single task ControlNet+SDM takes 1.4B. In order to achieve the same nine-task functionality, a single-task strategy would require the ensemble of a SDM with nine task-specific ControlNet models, amounting to approximately 4.3B #params in total.
4.2 Visual Comparison
We visually compare different tasks (Canny, HED, Depth, Normal, Segmentation, Openpose, Bounding Box, and Outpainting) in Fig. 4. Our method consistently outperforms the baseline ControlNet model. This superiority is in terms of both visual quality and alignment with conditions or prompts.
For the Canny task, the results generated by our model exhibit a higher degree of detail preservation and visual consistency. The outputs of UniControl maintain a faithful reproduction of the edge information (i.e., round table) compared to ControlNet. In the HED task, our model effectively captures the robust boundaries, leading to visually appealing images with clear and sharp edge transitions, whereas ControlNet results appear to be non-factual. Moreover, our model demonstrates a more subtle understanding of 3D geometrical guidance of depth maps and surface normals than ControlNet. The depth map conditions produce visibly more accurate outputs. In the Normal task, our model faithfully reproduces the normal surface information (i.e., ski pole), leading to more realistic
Figure 7: User study between our multi-task model (Ours-multi) and single task model (Ours-single) on eight tasks. Our method outperforms baselines on most of tasks, and achieves big performance gains on tasks of seg-to-image and outpainting-to-image. Moreover, the p-value of voting Ours-multi in all cases is computed as 0.0028 that is statistically significant according to the criteria of  $< 0.05$ .
and visually superior outputs. During the Segmentation, Openpose, and Object Bounding Box tasks, the produced images generated by our model are better aligned with the given conditions than that by ControlNet, ensuring a higher fidelity to the input prompts. For example, the re-implemented ControlNet-BBox misunderstands "a woman near a statue", whereas our outputs exhibit a high degree of accuracy and detail. In the Outpainting task, our model demonstrates its superiority by generating reasonable images with smooth transitions and natural-looking textures. It outperforms the ControlNet model, which produces less coherent results - "a bear missing one leg". This visual comparison underscores the strength and versatility of our approach across a diverse set of tasks.
**User Study.** We compare the performance of our method with both the released ControlNet model and the re-implemented single-task ControlNet on our training set. As shown in Fig. 6, our approach consistently outperforms the alternatives in all cases. In the HED-to-image generation task, our method significantly surpasses ControlNet. This superiority is even more pronounced in the depth and normal surface to image generation tasks, where users overwhelmingly favor our method, demonstrating its ability to handle complex geometric interpretations. When compared to the re-implemented single-task model, Fig. 7 reveals that our approach maintains a smaller advantage, yet it still demonstrates its benefits by effectively discerning image regions to guide content generation. Even in the challenging outpainting task, our model outperforms the baseline, highlighting its robustness and capacity to generalize.
Image Perceptual Metric. We evaluate the distance between our output and the ground truth image. As we aim to obtain
Table 2: Image Perceptual Distance
Canny ↓	HED ↓	Normal ↓	Depth ↓	Pose ↓	Segmentation ↓
UniControl	0.546	0.466	0.623	0.654	0.741	0.693
ControlNet	0.577	0.582	0.778	0.700	0.747	0.693
a structural similar image to the ground truth image, we adopt the perceptual metric in [65], where a lower value indicates more similar images. As shown in Tab. 2, UniControl outperforms ControlNet on five tasks, and obtains the same image distance to ControlNet on Segmentation.
Frechet Inception Distance (FID). We've further conducted quantitative analysis with FID [66] to include more classic single-task-controlled methods such as GLIGEN [67] and T2I-adapter [68]. With a collection of over 2,000 test samples sourced from Laion and COCO, we've assessed a wide range of tasks covering edges (Canny, HED), regions (Seg), skeletons (Pose), and geometric maps (Depth, Normal). The Tab. 3 demonstrates that our UniControl consistently surpasses the baseline methods across the majority of tasks. Notably, UniControl achieves this while maintaining a more compact and efficient architecture than its counterparts.
Ablation Study. We've conducted an ablation study, specifically focusing on the MoE-Style Adapter and TaskHyperNet in Tab. 4 with FID scores reported as the previous part. It is noticeable that the full-version UniControl (MoE-Style Adapter + TaskHyperNet) significantly outperforms the ablations which demonstrates the superiority of proposed MoE-Style Adapter and TaskHyperNet.
Table 3: Quantitative Comparison (FID)
Canny ↓	HED ↓	Depth ↓	Normal ↓	Seg ↓	Pose ↓
GLIGEN [67]	24.9	27.8	25.8	27.7	-	-
T2I-Adapter [68]	23.6	-	25.4	-	27.1	28.9
ControlNet [21]	22.7	25.1	25.5	28.4	26.7	28.8
UniControl	22.9	23.6	21.3	23.4	25.5	27.4
Table 4: Ablation Study (FID)
MoE-Adapter	TaskHyperNet	Canny ↓	HED ↓	Depth ↓	Normal ↓	Seg ↓	Pose ↓	Avg ↓
X	X	27.2	29.0	27.6	28.8	29.1	30.2	28.7
✓	X	24.5	26.1	23.7	24.8	26.9	28.3	25.7
✓	✓	22.9	23.6	21.3	23.4	25.5	27.4	24.0
We further showcase the surprising capabilities of our method to undertake the zero-shot challenge of hybrid conditions combination and unseen tasks generalization.
Hybrid Tasks Combination. This involves generating results from two distinct conditions simultaneously. Our model's zero-shot ability is tested with combinations such as depth and human skeleton or segmentation map and human skeleton. The results are shown in Fig. 5 (a)-(b). When the background is conditioned on a depth map, the model effectively portrays the intricate 3D structure of the scene, while maintaining the skeletal structure of the human subject. Similarly, when the model is presented with a combination of a segmentation map and human skeleton, the output skillfully retains the structural details of the subject, while adhering to the segmentation boundaries. These examples illustrate our model's adaptability and robustness, highlighting its ability to handle complex hybrid tasks without any prior explicit training.
Unseen Tasks Generalization. To evaluate the zero-shot ability to generalize to unseen tasks such as gray image colorization, image deblurring, and image inpainting, we conduct the case analysis in Fig. 5 (c)-(e). The model skillfully handles the unseen tasks, producing compelling results. This capability is deeply rooted in the shared attributes and implicit correlations among pre-training and new tasks, allowing our model to adapt seamlessly. For instance, the colorization task leverages the model's understanding of image structures from the segmentation task and depth estimation task, while deblurring and inpainting tasks benefit from the model's familiarity with edge detection and outpainting ones.
We introduce UniControl, a novel unified model for incorporating a wide range of conditions into the generation process of diffusion models. UniControl has been designed to be adaptable to various tasks through the employment of two key components: a Mixture-of-Experts (MOE) style adapter and a task-aware HyperNet. The experimental results have showcased the model's robust performance and adaptability across different tasks and conditions, demonstrating its potential for handling complex text-to-image generation tasks.
Limitation and Broader Impact. While UniControl demonstrates impressive performance, it still inherits the limitation of diffusion-based image generation models. Specifically, it is limited by our training data, which is obtained from a subset of the Laion-Aesthetics datasets. We observe that there is a data bias in this dataset. Although we have performed keywords and image based data filtering methods, we are aware that the model may generate biased or low-fidelity output. Our model is also limited when high-quality human output is desired. UniControl could be improved if better open-source datasets are available to block the creation of biased, toxic, sexualized, or other harmful content. We hope our work can motivate researchers to develop visual generative foundation models.
[68] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023.
A Details of Implementation
A.1 MOE-Style Adapter
The MOE adapter is implemented as a set of parallel ConvNets composed of three consecutive convolution and non-linear activation layers. The entire model is comprised of nine individual MOE adapters, each of which consumes 70K parameters. Task keys are designated to each adapter, ensuring that they align with the corresponding visual conditions. Once the MOE adapter processes the input, the remaining model parameters become shared across all tasks. This architecture facilitates task adaptability while promoting parameter efficiency.
A.2 Task-aware HyperNet
The task-aware hypernet is applied to modulate the parameters of zero-conv layers in the ControlNet. Since the ControlNet can be considered as the hypernet of Stable Diffusion (fixed copy). Our idea can be concluded as the control over control or meta-control to let the task-aware hypernet learn the universe representation that is generalizable across different tasks. To implement it, we firstly map the task keys to instruction with a mapping function as: {"hed": "hed edge to image", "canny": "canny edge to image", "seg": "segmentation map to image", "depth": "depth map to image", "normal": "normal surface map to image", "pose": "human pose skeleton to image", "hedsketch": "sketch to image", "bbox": "bounding box to image", "outpainting": "image outpainting"}. Then, such instructions will be projected as text embeddings with the help of a language model (we adopt CLIPText in our implementation). The Task-aware HyperNet takes these task instruction embeddings, and projects them into different shapes to match the size of different zero-conv kernels, which will be modulated by these task embeddings accordingly. We would fix the parameters of task-aware hyperNet in the later stage of model training to ensure the stability of dynamics.
A.3 Data Collection
We have collected a large amount of training set (MultiGen-20M) including over 20M condition-image-prompt triplets across nine different tasks. We firstly download 3/4 of Laion-Aesthetics-V2 with score over six and filter out low-resolution (<512) images. As a result, 2.8M images are selected as source images. Then we apply the visual condition extractors as described in the main paper to collect Canny, HED, Sketch, Depth, Normal Surface, Seg Map, Object Bounding Box, Human Skeleton and Outpainting.
B Numerical Analysis of Task-Aware Modulated ControlNet
We show that our proposed task-aware modulated ControlNet preserves the properties of the original ControlNet structure. Specifically, we show 1) The new task-aware modulated ControlNet preserves the zero-initialization property of ControlNet; 2) The parameters of the task-aware modulated ControlNet can be updated once we start to train the model.
Denote the input feature map by  $\pmb{x}$ , the frozen SD Block in Fig. 2 by  $\mathcal{F}_{\mathrm{SD}}$ , the extra condition by  $c$ , two zero convolution operators by  $\mathcal{Z}_{\theta_1}^1(\cdot)$  and  $\mathcal{Z}_{\theta_2}^2(\cdot)$ , the trainable copy of SD Block by  $\mathcal{G}_{\theta_{\mathrm{s}}}^{\mathrm{SD}}(\cdot)$ , the task instruction by  $c_{\mathrm{task}}$ , and the task-aware hyperNet by  $\mathcal{H}_{\theta_{\mathcal{H}}}(\cdot)$ . Then the output of the new task-aware modulated ControlNet can be expressed as
$$
\boldsymbol {y} _ {c} = \mathcal {F} _ {\mathrm {S D}} (\boldsymbol {x}) + \mathcal {Z} _ {\theta_ {1}} ^ {1} \left(\mathcal {G} _ {\theta_ {\mathrm {s}}} ^ {\mathrm {S D}} (\boldsymbol {x} + \mathcal {Z} _ {\theta_ {2}} ^ {2} (c) \cdot \mathcal {H} _ {\theta_ {\mathcal {H}}} (c _ {\text {t a s k}}))\right) \cdot \mathcal {H} _ {\theta_ {\mathcal {H}}} (c _ {\text {t a s k}}). \tag {1}
$$
Property of Zero Initialization. Similar to ControlNet [21], the weights and biases of the convolution layers are initialized as zeros. As a result, we have  $\mathcal{Z}_{\theta_1}^1 (\cdot)\equiv 0$  and  $\pmb {y}_c = \mathcal{F}_{\mathrm{SD}}(\pmb {x})$  regardless of the initialization of  $\mathcal{H}_{\theta_{\mathcal{H}}}(\cdot)$
Gradient Analysis. We analyze the gradient of the modulated part
$$
\nabla_ {\theta} \left(Z _ {\theta_ {1}} ^ {1} (I) \cdot \mathcal {H} _ {\theta_ {\mathcal {H}}} \left(c _ {\text {t a s k}}\right)\right) = \mathcal {H} _ {\theta_ {\mathcal {H}}} \left(c _ {\text {t a s k}}\right) \cdot \nabla_ {\theta} Z _ {\theta_ {1}} ^ {1} (I) + Z _ {\theta_ {1}} ^ {1} (I) \cdot \nabla_ {\theta_ {\mathcal {H}}} \mathcal {H} _ {\theta_ {\mathcal {H}}} \left(c _ {\text {t a s k}}\right), \tag {2}
$$
where  $I$  is the input of the zero convolution layer.
When we start to train the network, the first part of the RHS of (2) follows similar analysis of ControlNet [21] since  $\mathcal{H}_{\theta_{\mathcal{H}}}(c_{\mathrm{task}})$  is constant when we analyze the gradient  $\nabla_{\theta}Z_{\theta_1}^1 (I)$ . Since the parameters of  $\mathcal{H}_{\theta_{\mathcal{H}}}(c_{\mathrm{task}})$  are not initialized to zero, it is known that  $\mathcal{H}_{\theta_{\mathcal{H}}}(c_{\mathrm{task}})\neq 0$ . So the gradient dynamic follows the analysis of ControlNet. Therefore, we conclude that  $Z_{\theta_1}^1 (I)\neq 0$  after the first gradient update, and that the network can start to learn and update the following standard dynamics of stochastic gradient descent.
As for the second part of the RHS of (2),  $Z_{\theta_1}^1 (I) \equiv 0$  before the first gradient update, so the gradient is zero for  $\theta_{\mathcal{H}}$ . However, after the first gradient update of  $\theta_{1}$ , we know  $Z_{\theta_1}^1 (I) \neq 0$ , and  $\theta_{\mathcal{H}}$  can be updated with non-zero gradients.
To conclude, the new task-aware Modulated ControlNet can still be efficiently updated and learned even if the convolution layers are initialized to zero.
C Zero-shot-task Results and Analysis
We show more zero-shot-task results in this section, where the tasks have not been trained on. In Fig. 13, we show zero-shot deblurring results guided by the keywords. Our deblurred images can successfully recover the fine-grained details of the images without training on such data. We note that some details are still missing, e.g., the details in the painting in the first row are still not clear enough. In Fig. 14, we illustrate two zero-shot image colorization results. We believe that most parts of the generated images are acceptable, though the clothes of the second woman do not look the same to the input blurred image. In Fig. 15, we observe impressive zero-shot inpainting results. In the first row, the duck that is inputted in the text has been successfully generated in the inpainted image. The second row obtains acceptable results as well, though the faces do not look perfect. The overall zero-shot quality of UniControl is remarkable.
While inpainting and outpainting might appear related, they are fundamentally distinct. Inpainting heavily leverages the contextual information from unmasked regions, necessitating a precise match. Conversely, outpainting has more freedom, with the generative model prioritizing prompts to envision new content. As shown in Fig. 8, directly using outpainting model for inpainting tasks can be challenging since the model tends to leave a sharp change over the mask boundaries. Our pretrained UniControl, thanks to intensive training across multiple tasks, has learned edge and region-to-image mappings, which assists in preserving contextual information.
Our model also demonstrates a promising capacity to generalize under scribble conditions, showing parallels to the ControlNet's ability, even though UniControl hasn't been directly trained using scribble data. Fig. 9 provides results illustrating the scribble-to-image generation.
D Details of User Study
In the evaluation steps, we use Amazon Mechanical Turk (Mturk) to perform user study. Specifically, we ask three Mturk master workers to select the best output result for each input condition. As shown in Fig. 10, we provide instructions on guidelines to select the best generated image. The annotators are provided the condition map and the text that describes the image, and are required to select the better output between the two generated images. Considering that images can both in good or bad qualities, we provide the tie option as well. We use the majority vote to determine the result of each image, which means that an image is considered as a better image if two or more annotators vote for it. We use 294 images for the tasks of Canny, HED, Surface Normal, Depth, Segmentation, User Sketch, and Outpainting. We adopt 100 images for the task of Human Skeleton and 187 images for the task of Bounding Box. In summary, we totally obtain 7,035 voting results for all nine tasks. 2/3 of source images in testing set are collected from MSCOCO with the remaining 1/3 from Laion. And it includes a very diverse range of topics including indoor scene, outdoor scene, oil painting, portrait, pencil sketch, animation, cartoon, etc.
$^{2}$ https://www.mturk.com
Input (Masked Image)
Ours-Single-Outpainting
"Wedding hair and makeup with warm tones and red lips"
UniControl
"Contemporary Bedroom Designs 2015 modern bedroom designs intended design"
Figure 8: Visual comparison of Ours-single-outpainting and UniControl on the inpainting task. The single outpainting model cannot well address the zero-shot inpainting task whereas UniControl demonstrates promising capacity.
Input (Scribble)
ControlNet-Scribble Results
"a turtle in river"
UniControl Results (zero-shot)
"a digital painting of a hot air balloon"
Figure 9: Visual comparison of ControlNet-Scribble and UniControl on the scribble data. ControlNet-Scribble is trained by the scribble data which, however, are unseen for UniControl.
"adooran wall
Figure 10: Mturk interface to select the better generated image.
Figure 11: User study results of User Sketch to image generation.
E Failure Cases
We illustrate some failure cases in Fig. 12. In the first row, although our generated image successfully aligns the Bounding Box condition, the generated human has a distorted body. In the second row, our generated image looks similar to the ground truth; however, the human faces are blurred. We think that the reason is that UniControl inherits the data and model bias of Stable Diffusion, where the generated human commonly have issues. In the third row, the generated image does not look realistic. We believe that the training data can be improved both quantitatively and qualitatively.
We illustrate more visualized results in this section on tasks Canny (Fig. 16), HED (Fig. 17), Depth (Fig. 18), Surface Normal (Fig. 19), Human Skeleton (Fig. 20), Bounding Box (Fig. 21), Segmentation (Fig. 22) and Outpainting (Fig. 23). These results further demonstrate the effectiveness of our proposed method. Moreover, due to the space limitation in the main paper, we report results of the last task, User Sketch. Given a sketched image, UniControl is able to achieve promising realistic images. The visualized results are in Fig. 24. The user study result can be found in Fig. 11, where it is observed that UniControl obtains significantly more votes than the single task model.
Figure 12: Failure Cases: distorted body (row one); blurred faces (row two); incorrect creation (row three).
Blurred Image
Our Result
"Bedroom Colour Ideas 25 Paint Colours With Impact Living"
"Christa McAuliffe (right, sat with her backup crew member Barbara Morgan) was a social studies teacher who had won NASA's Teacher in Space contest and earned herself a spot on the mission"
"Mountain Village in the Alps - Canvas print - Bedroom"
Figure 13: More zero-shot-task deblurring results.
Gray Image
"Long White Casual Wedding Dress"
"Pixie Cropped Short Layered Synthetic Wig for Women-KAMI WIGS"
"Early morning view over the town of Tinerhir, south of the Todra Gorge, Morocco, North Africa, Africa"
Figure 14: More zero-shot-task gray-to-RGB colorization results.
Our Result
"A lone duck basks in the calm lake's mirror reflection of the Chugach mountain valley"
"Chancellor of the Exchequer Rishi Sunak was the most high-profile, and unexpected, appointment of the day"
"Contemporary Bedroom Designs 2015 modern bedroom designs intended design "
Figure 15: More zero-shot-task image in-painting results. The in-painting MOE adapter weights are directly inherited from outpainting.
Our Method Output
(a) "Two arms typing on a laptop and one hand on a mouse"
Input Image
Input Image
(b) "Two people walking along a side walk next to a train on the tracks."
Our Method Output
(c) "A close up of glazed donuts that are plain or with chocolate."
Input Image
(d) "A group of elephants with water in front and trees behind."
Figure 16: Canny to Image Generation
(a) "A person on skis makes her way through the snow"
(b) "Three zebras grazing in a grassy area near shrubs"
Our Method Output
(c) "The Taj Mahal mirrored by a water fountain's reflection. - Agra, Uttar Pradesh, India - Daily Travel Photos"
Input Image
(d) "A young girl who is brushing her teeth with a toothbrush."
Figure 17: HED to Image Generation
(a) "The Ta Prohm Temple Located at Angkor in Cambodia by Kyle Hammons"
(b) "A brown dog standing on a wooden bench near a lemon tree."
(c) "A Polar Bear walks toward water, while a large bird lands on the opposite bank."
Our Method Output
Figure 18: Depth to Image Generation
(d) "A display of vintage animal toys on the floor."
(a) "A line of small teddy bears are in front of several DVD cases."
Input Image
(b) "A man in the kitchen standing with his dog"
Input Image
(c) "A hot dog sitting on top of a bun in a wrapper"
Input Image
(d) "Several people waiting on the side of train tracks as a train with it's lights on comes down the track"
Figure 19: Surface Normal to Image Generation
(a) "Photo of handsome man in black leather jacket"
Input Image
(b) "A woman is sitting near a prominent landmark"
Input Image
(c) "A man that has ski's and is standing in the snow."
Input Image
(d) "A woman is sitting in front of a desk"
Figure 20: Human Pose Skeleton to Image Generation
(a) "A woman is walking two dogs in the snow"
(b) "Simone Righi frasi glasses linen suit menswear streetstyle icon fashion florence."
Our Method Output
(c) "The large room has a wooden table with chairs and a couch."
Input Image
(d) "Two black bags placed standing on the ground"
Figure 21: Bounding Box (by YOLO-V4-MSCOCO) to Image Generation
(a) "A bench at the beach next to the sea"
(b) "Water traffic along the Thames by Big Ben"
Our Method Output
(c) "A well-lit and well-decorated living room shows a glimpse of a glass front door through the corridor."
Input Image
(d) "Blue Hour Barley"
Figure 22: Segmentation Map (by Uniformer-ADE20K) to Image Generation
(a) "St, Michaels Mount, Cornwall"
Input Image
(b) "Melbourne by teekay 72"
Input Image
(c) "Lady in Black Kimono Paint by Diamonds"
Input Image
(d) "Beautiful kitchen grand scale living pinterest for Kitchen cabinets lowes with old world metal wall art"
Figure 23: Image Outpainting
Figure 24: User Sketch to Image Generation