Cocktail: Mixing Multi-Modality Control for Text-Conditional Image Generation
Authors: Minghui Hu
Jianbin Zheng
Daqing Liu
Chuanxia Zheng
Chaoyue Wang
Dacheng Tao
Tat-Jen Cham
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-modality
Image Generation
Diffusion
TL;DR: We propose a novel pipeline, termed as Cocktail, that achieves multi-modality control based on the pre-trained text-conditional diffusion model.

Abstract: Text-conditional diffusion models are able to generate high-fidelity images with diverse contents.
However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. 
In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined control for text-conditional diffusion models. 
Specifically, we introduce a hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. 
gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination of modality signals, or the supplementary fusion of multiple modality signals. 
The control signals are then fused and injected into the backbone model according to our proposed ControlNorm.
Furthermore, our advanced spatial guidance sampling methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image.
We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals.

Cocktail Mixing Multi-Modality Controls for Text-Conditionll Image Generation
Minghui Hul, Jianbin Zheng", Daqing Liu², Chuanxia Zhengå, Chaoyue Wang1, Dacheng Tao and Tat-Jen Cham
Nanyang Technological University, * 'oouhh China University of Technology. UUniversity of Oxford, The University of Sydney, 3JD Explore Academy
Abstract
Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy text-guided diffusion models. In this work, we propose Cocktail, pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and spatial guidance sampling method, to actualize multi-modal and spatially-refind control for text-conditional diffusion models. Specifically, introduce hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination modality signals, the supplementary fusion of multiple modality signals. The control signals are then fused and injected into the backbone model according to our proposed ControlNorm. Furthermore, methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image. We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals. The codes are released at https: pttps://mhhiis //whh0318.h.../ github. io/ cocktail/.
advanced spatial guidance sampling
Introduction
Text-conditional diffusion models [41, 39, 42, 34] have actualized the capacity for high-quality generative capabilities. These models facilitate the generation of an array of high-calibre images through the utilization of concise textual prompts. However, linguistic representations pose inherent challenges accurately encapsulating the precise imagery anticipated by the user, owing to the potential ambiguities and subjective interpretations of verbal descriptions within the context of visual synthesis. Moreover, minor alterations to the textual prompts also yield distinct visual outputs, underscoring the absence of refined control over the generative process.
Modifying the prior is a series of existing solutions for multi- moltil control, egg., the control of the entire prior space [39, 48, 23, 22, 57] as demonstrated in Fig. ((a). These approaches centered on the whole prior lack the capacity for localised image modifications and the preservation of background elements. Moreover, these models typically require training from scratch, which demands substantial amount resources.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Modality
Modaliy
btodiaity
Moddaaay
Modality
Modaliyy a
Modality xs
Modalisy
Mudaley
Medaliyy
.2
Text-conditional Diffusion
Modaliries
Modality
Dillusion
Modality
Modality #3
Text-conditional
Decoder
Diffusion
(a) Prior Modification
(b) ControlNet/ T21 Adapter
(c) Cocktail
Comparison of various control methods. Our approach requires only one generalized model, unlike previous that needed multiple models for multiple modalities.
Figure 1:
In response the methods dealing with latent representation [29, 27], additional lightweight hyper-network introduced in [53, 33], which is designed to encode external control signals into latent vectors and subsequently inject them directly into the backbone network. Such methods effectively handle control over the single additional modality; however, they exhibit limitations when confronted with multiple modalities, as shown in Fig I(b). One issue is that each modality requires unique network, leading to computational overhead that escalates proportionally with the increase the number of modalities. Furthermore, the impact of additive coefficients between different modes on the final imaging outcomes warrants consideration. The inherent imbalance among superimposed modes makes these additive coefficients pivotal factor in determining the ultimate synthesized output. An additional challenge emerges during the sampling process when the model conducts an initial inference devoid of control signal injection. This preliminary inference step could result in object placement that potentially contradicts the control signals.
In this paper, we propose novel pipeline, shown in Fig. 1(c), termed Cocktail, which accomplishes multi-modality control through a text-conditional diffusion model. It encompasses three main components: 1) a hyper-network capable of accommodating multi-modal input, 2) a conditional normalisation method to mix the control features, and 3) sampling strategy designed to facilitate precise control over the generative process. As shown in Fig. 2, our method is proficient in generating images that meet all input conditions or any arbitrary subset thereof, utilizing only one single model.
To achieve this goal, initially trained branched network named gControlNet, which accepts multiple modalities. Upon training, the branched network can simultaneously accept arbitrary combinations of existing modalities. When multiple modalities coexist within the same region, the model capable of automatically fusing input from different modalities, balancing the disparities between them. To leverage the features of the gControlNet, we further proposed controllable normalisation (ControlNorm). We can achieve better representation of control signals in terms of semantic and spatial aspects according to the decoupling provided by ControlNorm.
Moreover, we introduce a spatial guidance sampling method order to facilitate spatial generation under the purview of multi-modal control signals. Specifically, employ distinct textual prompts to differentiate entities from the background, incorporating entities into the background via prompt editing approach. Our devised sampling approach demonstrates efficacy in circumventing the generation of undesired entities.
In summary, main contributions follows:
We introduced the Generalized ControlNet (gControlNet), a branched network capable of adaptively integrating multi-modal information, effectively addressing issues stemming from imbalances between modalities;
We proposed the Controllable Normalistion (ControlNorm) to optimize the utilization of information within branched networks to yield more effective outcomes;
We introduced spatial guidance sampling method based on the operation within the attention map to generate relevant information tailored to regional contexts, preventing the inclusion of undesired objects outside the specified regions.

---

Edge
(I) Edge+ Pose
Pose
(2) Posc* Mask
Full: Edge Mask
Edge
Pose
Full
Mask
Pose
Mask Prompt: girl holding a cat
Edge
Mask
(3) Mask-Edge
Figure 2: Examples of our model with the same prompt. Given text prompt along with various modality signals, our approach is able to synthesize images that satisfy all input conditions or any arbitrary subset of these conditions using single model. The prompt is: A girl holding a cat.
Related Work
Text-conditional Diffusion Models. Diffusion models [19, 46] has achieved great success in the area text-to-image synthesis [34, 39, 41, 3, 14]. To reduce the computational cost, diffusion models typically function within the latent space [41] produce low-resolution images that are later improved through super-resolution models [39, 3]. Fast sampling methods have also successfully reduced the number of generation steps required by diffusion models from hundreds down merely a few [45, 32, 25, 30, 111. The provision of classifier guidance during the sampling process can also significantly impact the outcomes, leading to substantial improvements in the results [9]. In addition to the widely used classifier-free guidance [18], other types of guidance are also worth
exploring [55, 13].
for Pre-trained Models. Training diffusion model is highly resource-intensive and environmentally unfriendly [41]. Fine-tuning such model can also be challenging due to the vast number of parameters involved [1]. Therefore, introducing additional tiny branched network to bias the output of the original network is a more reasonable choice [10, 2, 8J. Similar ideas have also proven to be effective in diffusion models: Hypernet [15] and LoRA [20] models capable of altering the original diffusion model's s sampling distribution by training a small branched network, which has become the most popular branched network in the current research community. More similar to our work is ControlNet [53], which is subsequently combined with different layers in the denoising U-Net provide support for various task-specific guidance. presents impressive results with various conditional inputs, yet with only one modality for each model. In contrast, our cocktail endows the ControlNet with multitasking capabilities using only single model.
Conditional Normalisation approaches have been employed across range of vision tasks, includ-
ing style transfer [24, 12], conditional generation [36, 14, 21] and image-to-image translation [40]. These techniques involve normalizing layer activations zero mean and unit deviation, followed by denormalisation through an affine transformation derived from external data. External data can in multiple formats, such style images, semantic masks, category labels.

---

Generalized CoutrolNct
Thump Arrentoo Maps for Spari
Gandance Sumpling
Aun
Encoder Bloock 64-64
Encoder Bleck 52-52
Encoder Bloekl 16-16
16-16
Encodeer Block S-8
Middle illoc 8-8
Conv
Contrel Ressangs
Curr
Pocareter Uiee
Laten Teatur
Batch Normn
Controllable Nomsalisation
Encodeer llkck 64-64
Encoder Black 52+52
Enceder BBeckk 16-16
Cutroliie Nerma barton
Cunediall Commlaatt
Concelaae Nomalattin
Decoder Block 52-12
Comroditte Namaleatim
Decdder Black Blod664-4 64v64
Encode: Block 8-8
Comrella Nauaalaaii
Divoler Bllck
7
18 prompt umbbelding
Figure 3: The network architecture Generalized ControlNet (gControlNet) with Controllable Normalisation (ControlNorm). The parameters indicated by the yellow sections are sourced from the pre-trained model and stay constant, while only those in the blue sections are updated during training, with the gradient pack-propagated along the blue arrows.
Attention Map-based Prompt Tuning. Prompt-to-Prompe [16] is method that adjusts local or global specifics in text-guided diffusion models by altering the cross-attention maps from source to target image, thereby maintaining spatial layout and geometry. Recently, numerous efforts have been made to improve the outcomes [35,37]. 37]. However, such approaches are limited to synthesized images without inversion technique. Unlike cross-attention, self-attention focuses inter-pixd relationships within the same domain [47]. Paint-with-words [3] another method that allows users specify the spatial locations of objects by selecting phrases from the text prompt.
3 Methods
In this work, our main goal is to design a controllable generator that utilizes various modalities of input, within single model. To achieve this, first propose gControlNet, which can generate control signals from different modalities using single network and adaptively weighted fuse them together. Furthermore, propose ControlNorm to inject the signals from the gControlNet into the diffusion backbone, which solves the imbalanced problem within various modalities. Finally, presence of extraneous objects in the generation process. By modifying the attention map, this approach effectively incorporates control signals into the backbone network. The whole pipeline is demonstrated in Fig. 3.
general branched network,
propose spatial guidance sampling method to avoid the
3.1 Generalized ControlNet with Controllable Normalisation
Generalized ControlNet. ControlNet [53] is method designed to influence and control the behavior of neural networks by adjusting the input conditions specific network blocks. Instead of directly modifying the parameters of the primary network, ControlNet employs auxiliary network to generate feature offsets. These offsets are then combined with different layers in the main network, such denoising U-Net, support various task-specific guidance.
Given trained backbone network block F(: 0) with parameter 0, the input feature ær can be mapped to y. For the branched part, duplicate the parameter 0 to create trainable copy O., which is

---

then trained using the supplementary modality. Preserving the original weights helps retain the information stored in the initial model after training on large-scale datasets, which ensures that the quality and diversity of the generated images do not degrade. Mathematically, the output from the trained network block can be expressed as:
y = F(;;0) 0) + (F(x + Z(mm); 0.)) + F(;;0)
(1)
where the control signal of a single modality Cm is typically processed to obtain a format identity for a, for example, through the gero-initialized layer. It is not only used in the process of handling Cm. but also serves to adjust the output of the branched network O.).
pero-initialized convolutional layer, and then added æ. 2-) represents
To accomplish the goals of accepting multiple external modalities as input and balancing signals from different modalities, we have devised modified framework that adeptly merges these varied sources of information. At the top of our network, we adopt a simple downsampling network M(-) to convert external conditional signals to the latent space, allowing the conditional signals to be directly injected into the latent space. It is worth noting that M(-) is versatile and can adapt to different types of external signals. Given & different modalities, the converted conditional features are c = M(C*).
Controllable Normalisation. Instead of directly passing the sum of conditional features via a vero-initializee layer the network block F(:: 0b), i.e., C. 2 c²), we introduce a controllable normalisation (ControlNorm) method, which has an additional layer to generate two sets of learnable parameters, y(Cm) and ((mm), conditioned on all l modalities. These two sets of parameters are used in the conditional normalisation layer to fuse the external conditional signals and the original signals. Specifically, the input to the trainable block F(::0,) 0t) becomes:
x (x) (x)
(I + Z(7 (cm))
2(3()m)) + x - 2(cm):
where Me(x)) and (x(2)) are the mean and standard deviation of the feature along the channel c, @ is Hadamard product, and r(é) and ((2) are vectors that have the same dimension as x. With the help of cero-convolution, we can preserve the identity of x: just before the start of the fine-tuning. It is worth noting that in the following layers, the internal feature in the branched network h can be integrated with the latent feature z from the original network in the same way:
(o(2)) + Z(B(h)) + + Z(cm).
(I 3(y (h))) ①
x=(2))
where h is the intermediate features from the original network and z is the intermediate features from the branched network. Specifically, we will have five sets of intermediate features from the Stable Diffusion [41] U-Net backbone and our generalized ControlNet, including four sets of features from encoder blocks and one from the middle block.
In fact, our controllable normalisation is generalized version of conditional normalisation [36, 24]. After changing the mean and variance calculation dimension and replacing the external signal è by mask image, real image, or class labels, we can derive the various forms of SPADE [36], AdaIN [24], CIN [12] and MoVQ [56]. More interestingly, our controllable normalisation method not only enables the use of external signals as conditions, but also allows intermediate-layee signals to act as constraints.
As shown in Fig. 3, only the parameters of the gControlNet require updating. Given an initial latent zo and time-step f, the diffusion process progressively introduces noise to this latent, transforming the original latent into noisy state z₁. The objective of the diffusion model is to estimate the noise adding to the x. Our proposed gControlNet shares the same objective function as the diffusion model, aiming to predict the noise added time t. The only distinction lies in the incorporation of multimodal information as conditional signals én = [chn e cm.
cm......
C = em..f.e...m.meeN/0.... ((.11 [[|e Eg (zz, t Cp. 2m 112]
3.2 Spatial Guidance Sampling
In order to leverage the control signals from generalized ControlNet and ensure that the generated objects appear within the areas of interest, we proposed a spatial guidance sampling method. Here, we mainly focus on editing the cross-attention layers of the U-Net in Stable Diffusion.
5

---

Given a noisy latent 2 at timestep as the input to the pretrained U-Net with n blocks F(") (::)(m) at different resolution, the latent vector & will be down-sampled to various dimensional z ((m) that map to the corresponding block F("), The cross-attention maps A (r) E (N....).) in each block, parameterized by ((*), are associated with the linear projection of latent feature Q(n) = =f(((=(()) and the prompt K fx (c
exp(Q)") K2)
(n) (5)
(5)
exp(Q.") TM K+)
where A(7) represents the attentional strength between the j-th prompt token (among N; tokens) and the i-th latent feature (among N₂ features). Intuitively, for each prompt token K, there exists
corresponding latent feature map A/")) with spatial information. Moreover, the corresponding feature map will contain spatial and shape information for the associated object. However, the object-deseribing tokens that contain information about the object's location and shape, some connecting words and padding tokens also convey spatial information for the overall scene [6, 43).
not only
We apply a masking strategy to the corresponding attention maps. In detail, we construct two sets of attention masks Mros(n) and M"ee(n)) (N),,N),, Each column Mp and M M) is flattened alpha mask, which is determined by the visibility of the corresponding text token Kj. The values of M (ppos()) and M (e) are determined based on the relationship between image token Q₂ and text token Ky. On the one hand, if image token Q, corresponds to region of the image that should be influenced by text token Kj, MMppa(n) is assigned the value of 1. On the other hand, if image token Qi corresponds to region of the image that should not be influenced by text token Kj, MM (eg()) is set to I. It is worth noting that we consider the feature maps corresponding to most words as negative in order to avoid generating undesired objects. The mask components MPos(n) and Mosg(n) are incorporated into the cross-attention computation process:
exp(Q,") .,,) + w Mpox(n) weg (neg(n)
i (n) ((i)))
(6)
Σ exp(Q."",
(1) Ks)
It is found that larger weights higher noise levels [3] can lead to better results, thus wos and weg are noise-level sensitive parameters, defined by:
w w/ log(1 + o) max(( (*)),
where w/ is user-provided hyper-parameter
We then substitute the feature map corresponding the object description. Contrary the image editing motivation behind conventional prompt tuning methods, it is not necessary for our method to provide reference image and an amended prompt. Recalling the framework of our generalized ControlNet, the branch architecture is identical to the encoder portion of the backbone network. We
found that the attention maps A (n)
sl0(") within the branch network also encompass object locations
and shape information. Consequently, we opt for the attention map A (n) 3i0(") associated with the description Kj from the respective layer in ControlNet as the source for the original attention map A (n) ji0(0) substitution, ensuring that the information in the substituted attention map aligns more closely with the external input signal rather than the textual information derived from the original backbone network. Specifically, we replace the attention map generated by the original backbone network (based on the corresponding object description) with the attention map from the corresponding module in the branch network:
A (n) Subsequently, can produce the spatially guided output from cross-attention layer by taking the product of (m) with V.
[A (a) ( ...;; (n)
(1n) No [ A / A (n)
N₂ p-
(8)
4 Experiments
In this section, we delve into a comprehensive experimental analysis to validate the efficacy and superiority of the proposed method through ablation studies and application demonstrations. Subse-

---

Table 1: Quantitative comparison on the COCOSK validation set.
The best result is highlighted.
Similarity (LPIPS ()
Sketch Map
Segmentation Map (mPA f)
Segmentation Map (mloU f)
Pose Map (mAP 1))
Method
(L2 Distance J)
Multi-Adapter
0.7273 +0.00120
7.93310 +0.01852
26.30 +0.242
13.98 +0.177
40.02 #0.701
0,6653 20.00145
7.59721
#0.01526
36.59 40.27s
22.70 20.229
38.19 20.781
Oury w/o ControlNorn
0,4900 #0.00141
7.18413
#0.01453
48.26 +0.357
32,66 #0.272
61.93 #0.776
Ours 1
0.4836
10.00133
7.28929
10.01385
49.20 10.250
33.27
61.99 10.778
hungress and big white wolf.
A little boy walks down the road. followed by cat.
astromaut standing on the mountain
Figure 4: Our model can generate images with the provided prompts and multi colti-modality information (e.g., edge, pose, and segmentation map) across various scales.
quently, Sec. 4.1, we put forth both quantitative and qualitative results, elucidating the comparative advantages of our approach. We also present an array of intriguing applications made possible by our gControlNet, showcasing its practical utility. Finally, See. 4.2 is dedicated to the discussion of abla- tion studies, scrutinising the impacts of varying injection and sampling methods. The experimental configurations, including the dataset specifications, implementation details, and evaluation metries, can be found in the Appendix.
4.1 Applications and Comparisions
Cocktail is proficient in seamlessly supporting multiple control inputs and autonomously fusing them, thereby eliminating the necessity for manual intervention to equilibrate diverse modalities. This unique property empowers users to easily incorporate a variety of modalities, resulting in more flexible multi-modal control. In Figure 4, we demonstrate how users can provide spatial information about multiple objects in different modalities to generate a complex scene. Notably, this entire process accomplished by single model without the need for additional branch networks.
We then compare Cocktail with two state-of- ofate-of-the-art methods: ControlNet [53] and T21-Adapter [33], in the context of rext-guided image-to- -mage translation within multiple modalities. We employ sev- eral evaluation methods, including LPIPS, mPA, mloU, mAP and L2 distance for various modalities in this section. As depicted in Table 1, our method outperformed both ControlNet and T21-Adapte across all evaluation metrics. This remarkable achievement signifies that our proposed cocktail can generate a structural image that elosely resembles the ground truth image and aligns better with the input conditions, establishing its superiority. The visualization in Fig. 6 also illustrates the superior ability of our model to harmonize with the control signals.
We further present some samples from our method to examine its effectiveness on uni-modality translation. The visual comparison the COCO validation set is showcased in Figure 5, highlighting the compelling performance. Benefiting from mixed training of multiple modalities, our method achieves remarkable generation quality, surpassing models exclusively trained for single control signal.
Our experiments show that Cocktail effectively leverages information from different modalities and exhibits outstanding multi-object generation abilities with consistent composition across various control signals.
7

---

A white dog standing on top of a wooden bench.
A clock tower part of old building.
A coo coo clock mounted the side of wall.
A teddy bear sitting on blue chair.
A man that has glasses and hat.
cute litcle blond girl sitting next a pizza.
Prompt
Control Signal
Ground truth
T2I Adapter
ControlNet
Ours
Figure 5: Qualitative comparison of Uni-Modality the COCO validation set.
4.2 Ablations
gControlNet and ControlNorm Previous methods [53] utilized a direct sum approach fuse control signals from hyper-networks with latent variables from the original network. While this method conveys spatial information, fails to consider semantic information, such as text or other modalities. Control signals decoupled through ControlNorm allow the preservation of semantic information while conveying spatial information. Another point the need for normalization to address the imbalance of different modality signals. However, typical normalization methods lead to the loss of semantic information [36]. Therefore, the control signals introduced through ControlNorm better interpret conditional information. We present some generated images in Fig. 10 to substantiate the interpretative capability of ControlNorm.
Spatial Guidance Sampling The spatial guidance sampling method not only ensures that objects are generated within controllable areas, but also minimizes the impact on other areas. An intuitive application that when we modify certain objects or modalities, other parts the generated image can remain unchanged. Figure 7 illustrates the contrast between employing spatial guidance sampling and absence. more significant variation in the overall tone of the resultant image observed when spatial guidance is not utilized, leading to inconsistencies in the details of the attire. Conversely, the incorporation of new objects and modalities with the use of spatial guidance minimally impacts the original image.

---

Kes Poies
X Key Pauss Seprertaue
Key Poiss Sureerane
Sipremase
Natioca Okjeco
X Addiiind Ofjers
Additioalll Oaject
Golla dae Sorpodd
X Sqrretaiion
Sopezitaiv
Suprazation
X Stsak
X Shrics
Skarc
X Ailiion
AdMieeil Ofjecs
Checley and
Key Signematio
Prie-
Key of Segnenatio
Sepeenake
Sirch
Strik
Objers
Objycts
Figure 6: Cocktail can address the imbalance among various modalities. Here, the "cross" symbol and the checkmark symbol denote the unmatched and matched modalities, respectively. is important note that our model accurately captures all modalities.
Conclusion
In this work, we proposed Cocktail, a pipeline to achieve fine control by fusing multi-modal signals. Firstly, introduced generalized ControlNet capable of handling signals of different modalities ing a single model. We also revealed that the semantic information of signals from the hyper-network may be lost through direct addition. However, a simple decoupling allows for better interpretation of control signals. We introduced a controllable normalisation for decoupling and integration. Finally, proposed sampling scheme that can prevent the generation of unnecessary objects outside the focus area while also achieving certain degree of editing capability and background protection.
Limitations. While we have achieved multi-modal fusion and control, there are still certain limita- tions the handling control signals from Cocktail that need to be addressed future work. Firstly, although the current spatial guidance method effectively prevent objects from being generated outside the focus area, it requires users to individually specify the area and corresponding object description during implementation. Secondly, spatial guidance can cause instability in the latent space certain situations, leading to the generated images degrading and deviating from the existing control signals. Therefore, finding stable anchor point reference during the generation process also worth exploring.
Broader Impacts. The integration of multimodal control signals inputs for synthesis greatly enhances user interaction flexibility and streamlines the utilization of text-conditional diffusion models. However, the process of fine-tuning large-scale generative models to accommodate diverse modalities requires significant amount energy, and we anticipate that the development of universal model capable of accepting multiple modalities will help mitigate this impact. However, the growing capabilities in image generation also facilitate the production of manipulated images with malicious intent, such as the creation of counterfeit or deceitful information.

---

References
[1] Armen Aghajanyan, Luke Zett and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:20/2./3255, 2020.
[2] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypemetworks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition, pages 18511 18521, 2022.
[3] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.0132 2022.
[4] Manuel Brack, Patrick Schramowski, Felix Friedrich, Dominik Hintersdorf, and Kristian Kersting. The stable artist: Steering semantics in diffusion latent space. arXiv preprint arXiv:2212.06113 2022.
[5] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. IEEE transactions on pattern analysis and machine intelligence, 43(1):172-186, 2021.
[6] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.033: 033373, 2023.
[7] Ting Chen, Mario Lucie, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. arXiv preprint 1rXiv:1810.01300 2018.
[8] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534, 2022.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-98 089944 2021.
[10] Tan M Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua. Hyperinverter: Improving stylegan inversion via hypernetwork. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11389-1153 11398, 2022.
[11] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. arXiv preprint arXiv:21 2021.
[12] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv: 2016.
[13] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv preprint 22t.::220060000022 2022.
[14] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
[15] David Ha, Andrew Dai, and Quoe V Le. Hypernetorkss. arXiv preprint 4rXiv:1609009106 2016.
[16] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. image editing with cross attention control. arXiv preprint arXiv:2208.01626,
2022.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. axXiv:2207.12598, 2022.
arXiv preprint
10

---

[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
[20] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arxiv 106.255006.09885. 2021.
[21] Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and Ponnuthurai N Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11502-11511, 2022.
[22] Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang. Zuopeng Yang. Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for simultaneous vision-language generation. arXiv preprint arXiv:2211. 14842, 2022.
[23] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Com- poser: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778, 2023.
[24] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference computer vision, pages 1501-1510, 2017.
[25] Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364, 2022.
[26] Alexander Kirillov. Mscoco keypoint evaluation metric.
https://github com/
crcodataset/cocodaattae
github. io, 2021.
[27] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210. 2022.
(28) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[29] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.070% 2023.
[30] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778, 2022.
[31] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3431-3440, 2015.
[32] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0099 009277, 2022.
[33] Chong Mou, Xintao Wang. Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint 2xXx:23322.0853, 2023.
[34] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 10741, 2021.
[35] Dong Huk Park, Grace Luo, Clayton Toste, Samaneh Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell. Shape-guided diffusion with inside-outside attention. arXiv preprint arXiv:2212.000 002100 2022.

---

[36] Taesung Park, Ming- Yu Liu. Ting- Ling-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptiv normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337-2346, 2019.
[37] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch- Elor, and Daniel Cohen-Or. Lo- calizing object-level shape variations with text-to-image diffusion models. arXiv preprint urXiv:2303.11306 2023.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[40] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: stylegan encoder for image-to-image translation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 2287-2296, 2021.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems, 35:36479-36494 2022.
[43] Vishnu Sarukkai, Linden Li, Arden Ma, Christopher Ré, and Kayvon Fatahalian. Collage diffusion. arXiv preprint 0tti::2333.002622 2023.
[44] Christoph Schuhmann. Clip+mlp aesthetic score predictor. https://github. com/ www.amaaaminmmmnnoeedeeeaa -ew.-eee------eeecttorrr 2022.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1.t.:20/1.135500 13456, 2020.
[47] Narck Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-pla -lay diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572. 2022.
[48] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205. 12952, 2022.
[49] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to- image models with human preference. arXiv preprint arXiv:2303.1444 2023.
[50] Saining Xie and Zhuowen Tu. Molistically-nestte edge detection. In International conference on computer vision, pages 1395-1403, 2015.
[51] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint a7Xh::2304.05977, 2023.
[52] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. arXiv preprint orXiv:2302.12242 2023.
[53] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXhv:3302.05543, 2023.
12

---

[54] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[55] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image immi translation via energy-guin guidedd stochastic differential equations. arXiv preprint arXiv:2207.06 06635, 2022.
[56] Chuanxia Zheng. Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412-2364 2022.
[57] Jianbin Zheng. Daqing Liu, Chaoyue Wang, Minghui Hu, Zuopeng Yang, Changxing Ding, and Dacheng Tao. Mmot: conditional image synthesis. arXiv preprint 2555922305.05992, 2023.
transformer for composed multimodal
A Experimental settings
A.1 Datasets and Annotations.
All of our experiments are performed on LAION-AESTHETICS dataset, which contains about 600K image-text pairs with predicted aesthetics scores of higher than 6.5.
Our experiment includes three types of commonly used additional control signals, and we use different methods obtain pseudo-labeling annotations:
Sketch map: The delineation of texture details within the structured control signal is accom- plished using the HED boundary detector [50]
Semantic segmentation map: To derive segmentation maps from images, we leverage SAN [52], a nigh-pprfforming. open-vocabulary segmentator.
Human pose map: The generation of comprehensive human pose map, including body, face, and hand positions, is facilitated by OpenPose [5].
It is crucial to note that our proposed Generalized ControlNet exhibits broad applicability and can be effortlessly adapted to accommodate a multitude of other modality inputs.
A.2 Implementation Details.
gControlNet is adapted from the pretrained Stable Diffusion v2.1 in this paper and trained for 20 epochs with a batch size of 64 on NVIDIA 80G-A100 GPUs within 4 days. We use the AdamW optimizer with learning rate of 3.0 c-05. All the training images in the LAION-AESTHETICS- 6.5 are first resized to 512 by the short side and then randomly cropped to 512 x 512. During inference, the sampler is DDIM, the sampling steps 50, and the classifier-free guidance scale is 9,0 by default.
A.3 Evaluation Metrics.
In order to compare the generation performance of existing methods with ours, we adopted six metrics to evaluate the quality, text-image alignment, aesthetics, and human preference of the generated images. They are Frechet Inception Distance (FID) [17], CLIP/BLIP Score [38, 28], Aesthetic Score [44], Human Preference Score (HPS) [49] and ImageRewared [51]. Specifically, FID measures the distribution distances of real and generated image sets utilizing pre-trained classification network (e.g., Inception V3 trained on ImageNet). Instead, CLIP/BLIP Score calculates cosine similarities between the corresponding image and text features extracted by the CLIP/BLIP image and text encoders, respectively. The Aesthetic Score is based on an additional MLP layer on top of a pretrained CLIP image encoder to measure the human aesthetic aspect of single image. To consider the alignment with human values and preferences, HPS and ImageReward were proposed and are based on the reward model pretrained on CLIP or BLIP.
However, these metrics do not effectively measure the fidelity of our model different modalities. Therefore, we additionally employ several other evaluation metrics, including the Learned Perceptual
13

---

Table 2: Additional quantitative comparison on the COCO validation set.
The best result is high-
lighted.
(a) Text + Edge
Method
FID |
CLIP Score f
BLIP Score 1
Aesthetic Score f
HPS f
ImageReward 1
ControlNet
18.73
0.2588
0.5338
5.39
19.53
0.3726
T21-Adapter
20.21
0.2638
0.5335
5.36
19.69
0.3344
Ours
16.66
0.2561
0.5321
5.35
19.53
0.2989
Text Segmentation Map
Method
FID 1
CLIP Score t
BLIP Score f Aesthetic Score f
HPS f
ImageReward f
ControlNet
20.53
0.2640
0.5262
5,43
19.85
0.2609
T21-Adapter
21.81
0.2626
0.5225
5.25
19.57
0.0837
Ours
23.88
0.2613
0.5260
5.64
19.89
0.3735
(c) Text + Pose
Method
FID
CLIP Score f
BLIP Score f
Aesthetic Score f
HPS f
ImageReward 1
ControlNet
35.13
0.2656
0.5171
5.49
20.17
0.3683
T2I-Adapter
35.53
0.2697
0.5211
5.45
20.19
0.38
Ours
39.64
0.2641
0.5210
5.70
20.22
0.4488
(d) Text + Sketch Segmentation + Keypoints
Method
FID 4
CLIP Score 1
BLIP Score
Aesthetic Score
HPS f
ImageReward
Ours
16.70
0.2562
0.5326
5.36
19.56
0.3118
Image Patch Similarity (LPIPS) [54] for overall image quality, the L2 distance for Holistically-Nestee Edge Detection (HED), the mean Pixel Accuracy (mPA) [31] and mean Intersection over Union (mloU) [31] for segmentation maps, and the mean Average Precision (mAP) over 10 object keypoint similarity (OKS) thresholds [26] for human pose map. Specifically, LPIPS is utilized assess the dissimilarity between the generated image and the ground-truth image. Furthermore, we extract the conditions, namely the sketch map, segmentation map, and pose map, from both the generated image and the ground-truth image. By computing the distance, specifically employing L.2 Distance, mPA, mloU, and mAP, between these extracted conditions, we can gain deeper understanding of the fidelity with respect to various modalities.
Through these metries, we can more clearly ascertain whether the generated images can follow the guidance of different modal conditions.
B Additional quantitative analysis
We compare Cocktail with two ofte-oofthhearr methods: ControlNet [53] and T2I-Adapter [33], in the context of text-guided image-to-image translation within a single modality. As demonstrated in Table 2, model not only acquires multi-modal capabilities but also demonstrates performance par with the comparative methods in single-modal tasks. worth noting that FID was measured on the basis of 5000 images from zero-shot generation.
worth noting that FID may not provide a precise evaluation as they are substantially contingent on the sample size and the characteristics of the training dataset. Our methodology prioritizes the fidelity of the generated images, a perspective that is at odds with the principles of CLIP/BLIP.
C Additional qualitative results
We provide additional generated images under various multi-conditional scenarios and compare them with Multi-ControlNer and Multi-Adapter. In Fig. 6, we have demonstrated some samples within less complexity, we further show the comparative performance of our model against other
14

---

MER With but.
Spotial Gridince Samplnng
wel Sputial Guilance Suälane Sarplix
Figure 7: With spatial guidance, our method is capable of maintaining constancy in certain regions while modifying specific objects.
Stateoof-the-an models under challenging disjoint scenarios Fig. In Fig. 10, we also conduct an ablation study on the effectiveness of our ControlNorm module. It can be observed that our method outperforms other state-of-the-ars models in terms of balancing and expressing multi-modal signals, and ControlNorm provide more stable integration performance. Besides, our model is able to accept arbitraty combiantions of the given modalities, as shown in Fig. 9.
We also provide further samples under single modality signals, e.g., Human Pose in Figs. 11& 12, Sketch Map as shown in Figs. 13& 14 and Segmentation Map in Fig. 15. can be seen that our method is more faithful to the given conditional signals.
15

---

Ground Trush
Conditions
Moldi-oomtollee
Multi-Adapeer
Ours
Ale
Kay
Key
X Segmertation
X Segmantation
Scymestation
X Stetch
X Skeehh
gire to branstfll drece
X Key Points
X Key Points
Key Points Seymenattion
X Sepmematioon
X Scymentaion
X Skenc
X Statc
Skeocs
againer
Kay Points
Key Points
Key Points
Segmemttion
X Seymentaioo
Segmerestion
Shonds
Skatch
Skerch
Figure 8: Additional results for Disjoint Multi-Modality Control.
16

---

loiking
randing with lizs.
Compose ef
Campose
Segreestation
Segientttion
Compose<22
Cusspose
Key Pains
Prints
Seprooration
Sepreetaaionn
Compose Xts
Kry Palio
Key Priets
Sammmattio Stutch
Segmenbation
Skeids
Figure 9: Our model accepts arbitrary combinations of the given modalities.
17

---

A WIZE E.M..
2GoncolNe: w/ ConneNNorn
AConceeNNes w/o ContelNorrs
Mclli ConnolNet
Moti
gir with eye glase.
&ConnelNet
gConieeNNt Cont
Figure 10: ControlNorm address the imbalance among various modalities. Note that the framed areas in magenta and cyan do not provide high-fidelity interpretation.
18

---

A young child squats down in front of bowl.
man wearing a hat and a gray jacket.
An older man with how tie happily poses for a picture.
man sitring at a table in front of plate of food.
Prompt
Control Signal
Ground truth
T21 Adapter
ControlNet
Ours
Figure 11: Additional results for Single Modality Control: Key Points
looking through a book top of table.
A skingg in an open area of snow.
A woman stands on snow covened hill.
An older woman sitting
at table with
Prompt
Control Signal
Ground truth
T21 Adapter
ControlNet
Owrs
Figure 12: Additional results for Single Modality Control: Key Points
19

---

An old wooden clock attached to the wall
row of white teddy bears shelf next to DVDs.
POON
PRO WEN
HOME
BECIIIR
HOMK
the eifiien
Sho SESIN
cAlioo
Two men smiling and taking picture together.
Prompt
Control Signal
Ground truth
T2I Adapter
ControlNet
Ours
Figure 13: Additional results for Single Modality Control: Sketch
A kitchen with counter and rable with chairs.
A little room and dining with Junnitures.
A woman bolding phone hand up
car.
close of a clgg near bowl.
car with snowboard on it's ski meck patked loggy porking next to two cats with empty aki racks.
small dog laying on bed next laptop computer.
Prompt
Signal
Adaattrr
Figure 14: Additional results for Single Modality Control: Sketch
20

---

A big and white horse in dirt cage.
A couple of pieces very nice looking luggage.
person riding skis on snowy slope.
A hand holdinng carrot for Ilaaa to chew.
female horse rider mounted near body of water and mountains.
living room filled with furniture and set of glass doors.
vase filled with flowers next smaller vase filled even more flowers.
living room that has some couches and tables in it. Prompe
Control Signal
Ground truth
TM Adapter
ControINet
Ours
Figure 15: Additional results for Single Modality Control: Segmentation Map
21


Reviews:
[{"review_id": "EFsWzNntbY", "reviewer": "Reviewer_vJhF", "summary": "This paper proposes Cocktail, a pipeline to mix various modalities into one embedding. The model is based on a variant of ControlNet and infuses control signals from disparate modalities into the pre-trained diffusion model. It is also equipped with a sampling approach named spatial guidance sampling that constructs the cross-attention weights based on the spatial location. The model can perform text-to-image generation conditioned on multiple modalities and outperforms controlnet in some metrics.\n", "questions": "1. Line 216 - 218, what figures or tables refer to “consistent composition”?\n2. Eq 7, what is $\\sigma$? How to determine its value?\n3. How to determine the correspondence between text and image token?\n", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Both the proposed controllable normalization and spatial sampling method are novel and have not been previously explored in the context of controllable generation.\n", "weaknesses": "The model does not consistently outperform the two baselines: ControlNet and T2I-Adapter, in text-guided image-to-image translation.  The author claimed the model can benefit from mixed training of multiple modalities (line 213-215) but the proposed model shows a mixed result in various metrics across different tasks. Additionally, there is no quantitative comparison in multi-modality conditioning to existing baselines. In spatial guidance, no qualitative metrics showing its effect were given as well. Overall, it is unclear what are the advantages of the proposed method over simple baselines such as controlnet. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and observations regarding our model.\n\n#### **Clarify the specific benefits of our proposed method.**\nThe metrics introduced in the main text focus on the quality of image generation. After training, the quality of the images generated by our model outperformed the baselines in most results, especially in tasks involving Segmentation Map or Pose. However, it is essential to note that the quality of the generated images is strongly related to the training dataset, particularly when it comes to the FID score. And the main objective of our task is to enhance the controllability of the generated images rather than merely improving quality.\nIn the appendix, we introduced a more effective metric to measure the control strength of our proposed scheme. For the convenience, I have attached the table: \n\n|                                                                     |        Similarity (LPIPS$\\downarrow$)                             |            Sketch Map   (L2 Distance$\\downarrow$)                       |            Segmentation Map (mPA$\\uparrow$)                      |          Segmentation Map (mIoU$\\uparrow$)                        |                Pose Map (mAP$\\uparrow$)                  |\n|---------------------------------------------------------------------|-------------------------------------|--------------------------------------|----------------------------------|----------------------------------|----------------------------------|\n| Multi-Adapter                    | 0.7273 $\\pm$ 0.00120           | 7.93310 $\\pm$ 0.01392          | 26.30  $\\pm$ 0.242          | 13.98  $\\pm$ 0.177          | 40.02  $\\pm$ 0.761          |\n| Multi-ControlNet               | 0.6653 $\\pm$ 0.00145          | 7.59721  $\\pm$ 0.01516          | 36.59  $\\pm$ 0.273          | 22.70  $\\pm$ 0.229          | 38.19  $\\pm$ 0.761          |\n| Ours w/o ControlNorm     | 0.4900 $\\pm$  0.00141         | **7.18413 $\\pm$ 0.01453**    | 48.26 $\\pm$ 0.287          | 32.66$\\pm$ 0.272          | 61.93  $\\pm$ 0.775          |\n| Ours                                 | **0.4836 $\\pm$ 0.00133** | 7.28929 $\\pm$ 0.01385          |**49.20 $\\pm$ 0.289** |**33.27 $\\pm$ 0.271** | **61.99$\\pm$ 0.778** |\n\nThese metrics employed open-source methods to process the generated images and then compared them with the given conditional information. For instance, we obtain the HED of the generated image and then calculate its L2 loss relative to the HED provided as a conditional input.  This approach allows us to focus more on the controllability aspect, which is the primary goal of our study. **It is worth noting that our Cocktail achieves much better quantitative results according to these controllable inputs.**\n\n#### **Line 216 - 218, what figures or tables refer to “consistent composition”?**\n\nThis sentence summarizes the experimental section, and in addition to Figures 5 and 6 in the main text, Figures 1 to 7 in the appendix effectively demonstrate that our approach can efficiently blend the control signals with the generated content. Specifically, **Figure 6** in the main text, along with **Figures 1 and 2 in the appendix**, more clearly showcase that *signals from different control modalities can be applied to multiple objects without conflicts, maintaining consistency.*\n\n#### **Eq 7, what is $\\sigma$? How to determine its value?**\n\n$\\sigma$ is a fixed parameter determined by the scheduler, representing the degree to which noise affects the model at different steps. For a detailed understanding, one can refer to **Equation 16 in Reference [1]**, where $\\sigma$ is used as a conventional expression.\n\n####  **How to determine the correspondence between text and image token?**\n\nThe question raised appears to be somewhat unclear. The relationship between text and image in the network can be established through a cross-attention map. By modifying the cross-attention layer, there are various methods currently available to alter objects within the generated image. In the validation phase, calculating the CLIP score is a common approach to measuring the similarity between text and image.\n\n[1] Song, J., Meng, C., & Ermon, S. (2020). Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.\n"}, {"review_id": "EWB9i6bq2m", "reviewer": "Reviewer_a1qK", "summary": "The paper presents Cocktail, a novel pipeline for multi-modal and spatially-refined control in text-conditional diffusion models. The authors address the challenge of ambiguous descriptions in linguistic representations by incorporating additional control signals. They propose three main components: gControlNet, ControlNorm, and a spatial guidance sampling method.\n\ngControlNet is a hyper-network designed to align and infuse control signals from different modalities into the pre-trained diffusion model. It can accommodate flexible modality signals and allows for the simultaneous reception of any combination of modalities or the fusion of multiple modalities. This capability eliminates the need for manual intervention and equilibrates the disparities between modalities, making the system more flexible and capable of seamlessly supporting multiple control inputs.\n\nControlNorm is a controllable normalization method that optimizes the utilization of information within branched networks. It decouples control signals, allowing for better representation of both semantic and spatial aspects. By preserving semantic information while conveying spatial information, ControlNorm overcomes the limitations of previous methods that either ignored semantic information or led to the loss of it through normalization. The proposed method effectively interprets conditional information and demonstrates its interpretative capability through generated images.\n\nAdditionally, the paper introduces a spatial guidance sampling method that ensures precise control over the generative process. By modifying the attention map, the method incorporates control signals into the backbone network, preventing the generation of undesired objects outside the specified regions. This approach enables the generation of high-quality images that closely align with the input conditions.\n\nOverall, the proposed Cocktail pipeline addresses the challenges of ambiguous descriptions in text-conditional diffusion models and achieves multi-modal and spatially-refined control. It offers a novel approach to integrating control signals from various modalities, optimizing their utilization, and generating high-quality images with fidelity to external signals.", "questions": "- How does the model handle conflicting signals provided by different modality signals, for example, if the provided pose and mask conflict with each other, how is the model supposed to handle this?\n- How does the method prevent users from misusing the model from generating sensitive images or modify images to create unethical content", "limitations": "- The authors dont specify how they control for potential misuse of their model. A model like this if open sourced, can have wide ethical impacts with people using it for unethical means like creating deep fakes and modifying images which could potentially be unethical. This should be addressed by the authors.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel pipeline called Cocktail that enables multi-modal and spatially-refined control in text-conditional diffusion models, addressing the challenge of ambiguous descriptions in linguistic representations. This is a significant contribution to the field as it tackles an important problem in generative models and expands the capabilities of text-guided image synthesis.\n\n- The proposed pipeline consists of three well-defined components: gControlNet, ControlNorm, and a spatial guidance sampling method. Each component is carefully designed to address specific challenges and effectively integrate control signals from different modalities. The approach is comprehensive and demonstrates a systematic solution to achieve high-quality synthesis and fidelity to multiple external signals.\n\n- The paper provides extensive experimental results and evaluation metrics, comparing the proposed Cocktail pipeline with state-of-the-art methods. The results consistently show that Cocktail outperforms existing approaches in text-guided image-to-image translation across multiple modalities. This empirical evidence demonstrates the effectiveness and superiority of the proposed method.\n\n- The paper emphasizes the practicality and efficiency of the Cocktail pipeline by demonstrating its ability to accomplish multi-modal control within a single model. This not only simplifies the model architecture but also reduces the computational overhead associated with multiple branch networks. The approach is both technically sound and resource-efficient, making it highly applicable in real-world scenarios.", "weaknesses": "- The authors dont specify how they control for potential misuse of their model. A model like this if open sourced, can have wide ethical impacts with people using it for unethical means like creating deep fakes and modifying images which could potentially be unethical. This should be addressed by the authors.\n- The paper does not provide a thorough discussion or analysis of the computational complexity or efficiency of the Cocktail pipeline. Considering the potential computational overhead of incorporating multiple modalities and the fusion process, it would be valuable to address the computational requirements and scalability of the proposed method\n- The evaluation section could be expanded to include a more comprehensive analysis of the results. While the paper mentions various evaluation metrics, it would be valuable to discuss the limitations of these metrics and provide additional qualitative analysis or user studies to further validate the effectiveness of the proposed method.\n- Although the paper compares the proposed Cocktail pipeline with state-of-the-art methods, it does not thoroughly discuss the limitations and failure cases of the proposed method. Understanding the shortcomings and potential failure modes is crucial for assessing the robustness and generalizability of the proposed approach.\n- Implementation details seem sparse and might hinder reproducibility for the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and questions. Your feedback helps us to address important aspects of our work that deserve more detailed consideration. Here's our response to your observations.\n\n#### **Ethical Considerations:**\nWe acknowledge your concern about the potential misuse of our model. The images that our model can generate heavily depend on the training dataset used. Currently, our model's training data strictly complies with open-source licenses and has been carefully selected, providing some degree of safeguard against misuse. In situations where the model generates certain objects (including human being), many details can be easily distinguished. Going forward, we are considering the addition of watermarks to the generated images to prevent unauthorized use, along with the implementation of a monitoring network at the output stage. Similar tools are already widely utilized in projects like Stability / DALL.E2, and we believe our model can also benefit from adopting such measures.\n\n#### **Computational Complexity:**\nOne of our key objectives is to alleviate the significant overhead associated with multi-modal control. We are pleased to provide specific details of our model's parameters to clarify the contributions of this work.\n\nFirstly, we stipulate that the parameter count for our employed SD base model stands at 865.91M, with a single ControlNet having 378.50M parameters. Our fusion network $\\mathcal{M}$ contains only 1.29M parameters per modality, and each ControlNorm adds a mere 0.31M parameters.\n\nIn the context of single modality control using ControlNet, the total parameter count is 865.91M + 378.50M. As the number of modalities increases, this overhead grows linearly, reaching a total of 865.91M + $n$ x 378.50M, where $n$ is the number of modalities.\n\nOur proposed gControlNet efficiently handles multiple modalities through a single HyperNetwork. Excluding ControlNorm, our model has a parameter count of 865.91M + 378.50M + $n$ x 1.29M. This formulation significantly reduces the computational requirements in multi-modal scenarios, underscoring the efficiency and innovation of our approach.\n\n#### **Comprehensive Analysis of Results**\nWe are more than willing to engage in further discussions regarding the metrics employed in our study. \nIn the main part of this submission, we employed various metrics such as CLIP, FID, and aesthetic scoring to assess the performance of our model. These metrics tend to emphasize the model's ability to generate image. For instance, FID evaluates the fit of the generated image to the distribution of real images, the CLIP Score represents the correlation between the generated image and corresponding text, and aesthetic scores reflect the artistic quality of the generated images.\n\nHowever, our proposed model aims to enhance controllability while maintaining image generation quality. Therefore, to provide a more comprehensive evaluation of the model's ability to control image attributes, we have introduced a set of additional metrics in the appendix.\nThese supplementary metrics process the generated images into the given control modality signals, subsequently quantifying the discrepancies between the generated images and the provided signals. These measures are instrumental in substantiating the effectiveness of our proposed solution in enhancing image controllability, aligning our evaluation more closely with the specific goals of our research.\n\n#### **Concerns about reproducibility**\nThe codes will be released for the reproducibility and publicly available, and the corresponding repo link will be attached in the manuscript for camera ready version. \n\n#### **Conflicting control signals**\nWhen confronted with conflicting signals across different modalities, the model endeavors to interpret each modality coherently. It strives to incorporate all the information conveyed by the different modalities, translating them into conditions for the generated images. The result is a theoretically reasonable representation that balances and harmonizes the conflicting inputs, ensuring that the synthesized images reflect as much of the diverse input data as possible.  However, there is a possibility that one modality may overshadow another if its representation is significantly more dominant or contains richer information. In such cases, the more expressive modality may suppress the other, leading to an unbalanced influence on the generated images. This phenomenon might result in the loss of some nuanced details conveyed by the less dominant modality.\nWe have included some example samples of such situations in the Global Rebuttal attachment.\n\n\nFinally, we would like to express our sincere gratitude for your thoughtful and comprehensive review.\n\n"}, {"review_id": "G316Dq6QBb", "reviewer": "Reviewer_KthK", "summary": "This study introduces ChatIR, a chat-based image retrieval system that engages in a conversation with the user to clarify their search intent and retrieve the desired image from a large corpus. The system leverages Large Language Models to generate follow-up questions to an initial image description and achieves a success rate of over 78% after 5 dialogue rounds, compared to 75% when questions are asked by humans and 64% for a single shot text-to-image retrieval.", "questions": "It would greatly enhance the clarity of the paper if the authors could explicitly list the implemented and supported modalities in the abstract or introduction. Currently, the detailed setup is not revealed until Section 4, with the exception of Figure 1. Providing this information earlier in the paper would help readers gain a clear understanding of the modalities that are incorporated and supported in the proposed approach. Including this information in the abstract or introduction would improve the overall organization and accessibility of the paper, allowing readers to quickly grasp the key aspects of the submission.\n", "limitations": "The authors have included discussions of the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The strength of this submission lies in its clear and impactful contribution, which proposes a pipeline that combines multiple modalities such as edge, pose, segmentation mask, and more. The authors effectively communicate the significance of this research contribution, highlighting the potential of integrating diverse modalities to enhance the proposed method's performance. Additionally, the submission features a visually appealing and clear illustrative depiction of the method, which effectively aids in explaining the approach. The selection of an attractive and informative illustration further enhances the clarity and appeal of the paper.\n", "weaknesses": "One notable weakness of the submission is the lack of sufficient explanation regarding the experimental results. While the paper goes into detail in explaining the proposed method, it falls short in providing a thorough analysis and interpretation of the results. In Table 1, there are inconsistencies in the trends of metric scores between the proposed method and the two baselines, yet the authors do not adequately explain these discrepancies. It is crucial to understand why the proposed method may not perform as well in certain metrics, whether it is due to the limitations of the metrics themselves or issues with the method's performance. Merely providing cherry-picked demonstrations is insufficient and does not substitute for a comprehensive analysis of the results. It is essential to address these gaps in the paper by providing a detailed explanation and interpretation of the observed trends and discrepancies in the experimental results.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful comments and suggestions regarding our work. In the revised version, we will restructure the manuscript to improve clarity. Moreover, I would like to address your concerns about the performance of our model across different experimental data and the supported modalities:\n\n**Concerns about the performance**\n\nThe metrics used in the main manuscript focus on the quality of image generation and the degree of text-image alignment. As can be seen in Table 1, our model outperforms the baseline in most generated results (Aesthetic Score, HPS, etc.). However, concerning text-image alignment, such as the CLIP Score, one hypothesis might be that the introduction of control signals from other modalities could weaken the control from the text part, potentially resulting in a lower CLIP Score than the baseline. And FID score strongly related to the training dataset, but due to the resource limits, we only use a subset of LAION which may decrease the diversty and lead to a higher FID. \n\nWe wish to emphasize that our work prioritizes the controllability of generated images while keeping the generated images quality, and for this reason, we introduced a new set of measures in the appendix.\nWe assess our control ability by evaluating the similarity between the processed generated image and the given condition signal. To illustrate, we obtain the HED of the generated image and then calculate its L2 loss relative to the HED provided as a conditional input. This measure effectively demonstrates our method's capability to control other modalities and shows that it performs better than the baseline even in multi-modal scenarios. We believe this provides a comprehensive understanding of the advantages of our proposed method. Please feel free to let us know if further clarification is needed. Here we attach the table, which is also included in the suppl. file of the submission. \n\n|                                                                     |        Similarity (LPIPS$\\downarrow$)                             |            Sketch Map   (L2 Distance$\\downarrow$)                       |            Segmentation Map (mPA$\\uparrow$)                      |          Segmentation Map (mIoU$\\uparrow$)                        |                Pose Map (mAP$\\uparrow$)                  |\n|---------------------------------------------------------------------|-------------------------------------|--------------------------------------|----------------------------------|----------------------------------|----------------------------------|\n| Multi-Adapter                    | 0.7273 $\\pm$ 0.00120           | 7.93310 $\\pm$ 0.01392          | 26.30  $\\pm$ 0.242          | 13.98  $\\pm$ 0.177          | 40.02  $\\pm$ 0.761          |\n| Multi-ControlNet               | 0.6653 $\\pm$ 0.00145          | 7.59721  $\\pm$ 0.01516          | 36.59  $\\pm$ 0.273          | 22.70  $\\pm$ 0.229          | 38.19  $\\pm$ 0.761          |\n| Ours w/o ControlNorm     | 0.4900 $\\pm$  0.00141         | **7.18413 $\\pm$ 0.01453**    | 48.26 $\\pm$ 0.287          | 32.66$\\pm$ 0.272          | 61.93  $\\pm$ 0.775          |\n| Ours                                 | **0.4836 $\\pm$ 0.00133** | 7.28929 $\\pm$ 0.01385          |**49.20 $\\pm$ 0.289** |**33.27 $\\pm$ 0.271** | **61.99$\\pm$ 0.778** |\n\n**Supported Modalities**\n\nWe trained the network with Segmentation map, Pose and Sketch map. It can effectively handle any number of trained modalities but is unable to deal with unknown modalities, meaning the model can process subsets and the exact set of training modalities. We will provide a more clear statement in introduction of supported Modalities. It is also worth noting that the supported modalities can be extended with sufficient dataset.\n"}, {"review_id": "OLlqI6KIuz", "reviewer": "Reviewer_u8fB", "summary": "This paper studies text-conditional diffusion models and proposes a pipeline for image generation with multi-modal control signals. The pipeline contains three new ingredients. Firstly, the paper introduces gControlNet, a generalized version of ControlNet, that can take different conditions (such as sketch, segmentation and pose maps) simultaneously as input without increasing the ControlNet size linearly with respect to the number of conditions. Secondly, the paper proposes to inject features produced by gControlNet into the backbone model via normalization, instead of a simple sum as used in the original ControlNet. Thirdly, the paper introduces spatial sampling that aims to harmonize spatial arrangements from both text prompt and image controls. Experiments show that the proposed method achieves on-par quantitative performance compared with ControlNet and T2I-Adapter, and improved visual quality especially with multiple modalities.", "questions": "1. What is the architecture of the downsampling network M in gControlNet? Can it deal with varying number of input conditions, and deal with new conditions during inference that are unseen in training?\n\n2. In spatial guidance sampling, how are M^{pos} and M^{neg} determined? Are they instance-specific or universal for all instances? It would be interesting to see, in challenging cases, how this module can help resolve conflicts among different input condition maps, and layer arrangement in complex scenes.\n\n3. How are the three types of control used in training? Are all of them or only some of them used in each update step?\n\n4. (Minor) The mapping notation in Eq. (1)-(3) is confusing.", "limitations": "The paper discusses limitations of the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. ControlNet proves to be a powerful approach for controllable text-to-image generation, while its original form requires fine-tuning a ControlNet network for each condition, which is resource consuming. This paper proposes a method to jointly train a shared network that can deal with different types of control input, which may extend the applicability of ControlNet in many cases.\n\n2. The paper has done extensive quantitative evaluations to compare the proposed method with previous work. For generative models quantitative metrics may not be able to measure the actual quality of generated images, but it is important to study what are the good metrics and how to design better ones. This paper makes an effort towards the direction.", "weaknesses": "1. The gControlNet is a crucial component in the proposed pipeline, but the paper does not discuss details of the downsampling network M, either in main text or Appendix. On the other hand, it seems the controllable normalization and spatial guidance do not show significant improvement over their counterparts in previous work. In Figure 6 gControlNet without ControlNorm has much worse image quality, but is it due to insufficient training in this case (as the original ControlNet does not use ControlNorm and performs well)? As for spatial guidance sampling, the paper shows the effect of retaining background, but not placing objects to the correct location. The latter may not be an issue since sketch, segmentation and pose maps contain spatial information already. This seems to weaken the role of spatial sampling, which requires manual design that is not present in paper.\n\n2. The paper uses Stable Diffusion 2.1 as backbone for training gControlNet, while ControlNet and T2I-Adapter used Stable Diffusion 1.5 as backbone. Therefore it may be unfair to directly compare gControlNet with ControlNet and T2I-Adapter. Also it seems that gControlNet does not show much qualitative improvement over ControlNet in single condition cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed reading and valuable insights concerning the gControlNet and its components. Allow me to clarify your concerns:\n\n#### **Regarding the downsampling network $\\mathcal{M}$:**\n$\\mathcal{M}(C^k)$ is a simple convolutional network consisting of eight convolutional layers and seven SiLU activation functions, with the final convolution layer being zero initialized. Each $\\mathcal{M}(C^k)$ has 1.29M parameters and each modality will have a corresponding  $\\mathcal{M}$, e.g., in our case we include $3$ different modalities and our $\\mathcal{M}$ will have 3.88M parameters in total.  It can effectively handle any number of trained modalities but is unable to deal with unknown modalities, meaning the model can process subsets and the exact set of training modalities.\n\n#### **On gControlNet without ControlNorm in Figure 6:**\nThe training epochs for gControlNet without ControlNorm and gControlNet with ControlNorm are the same. But it is worth noting that *gControlNet without ControlNorm* also has a mixed network  $\\mathcal{M}$, and the lack of ControlNorm in the former prevents the effective balancing of different modalities, leading to a decline in image quality. It precisely demonstrated ControlNorm significant improvement according to the conditional inputs. And unlike T2I and ControlNet trained different modalities separately, gControlNet  w/ or w/o ControlNorm co-trained $\\mathcal{M}$ with different modalities.\n\n#### **Regarding spatial guidance sampling:** \nSpatial guidance requires manual setting of keywords or latent bounding boxes of objects during inference. $M^{pos}$ and $M^{neg}$ are case-specific, but once provided, $M$ generates them based on the time step $t$. It's also important to note that we did not utilize the spatial guidance strategy in our quantitative experiments.\n\n#### **On the comparison between gControlNet, ControlNet, and T2I-Adapter:**\nWe acknowledge the different backbones used for training the models. However, as illustrated in [1], when CFG>5, the FID performance of SD2.1 is approximate to SD1.5, and the CLIP performance is inferior to SD1.5. This is consistent with Table 1 in our main text. And the human preference win rate of SD2.1 is slightly lower than SD1.5 as shown in [2]. Our paper primarily aims to demonstrate the ability of additional modalities to control image generation. The control ability of our proposed method is substantiated effectively in Appendix Table 1, thus making the comparison relatively fair.\n\n#### **The control signals usage during training:**\nThank you for raising this question, as it is indeed worth emphasizing in the paper. During the training process, we randomly select the number of modalities and apply random masking within certain areas of the chosen modalities. Specifically, assuming there are three known modalities, we randomly select 0 to 3 modalities during training, and for each modality, we choose arbitrary areas as input signals. That is to say, within the given control signal domain, there may be a single control singnal ($n=1$), or an overlap of $n \\in \\\\{2,3\\\\}$ control signals, or there may be no control signals ($n=0$).\n\n#### **The mapping notation in Eq. (1)-(3) is confusing.**\nWe will revise the manuscript to improve clarity of notation.\n\n[1] https://github.com/Stability-AI/stablediffusion, Fig: https://github.com/Stability-AI/stablediffusion/raw/main/assets/model-variants.jpg\n\n[2] https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0, Fig: https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/comparison.png"}, {"review_id": "mT3Ldgqw7Q", "reviewer": "Reviewer_H6cw", "summary": "This paper proposes an approach to enhance the controllability of generative image models through the fusion of multiple modality signals.\nThe proposed model is a strict generalization of ControlNet which is able to incorporate geometric constraints into the generation, including pose, edges, and segmentation map.", "questions": "None.", "limitations": "Limitations adequately described.\nIt is could be a good idea to include a sample of random results (not cherry picked) in the appendix in order to provide a sense of typical failure modes.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper describes a very practical generalization of a SOTA approach to controllable generative models. It is well motivated, provides a clear description of the architecture, and both quantitative and qualitative results that validate the approach.", "weaknesses": "None. The paper is very clear on the benefits and limitations of the approach, and provides a practical improvement over a SOTA approach that is relevant to the research community and downstream users of the technology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your comprehensive review and favorable assessment of our paper. Your commendation regarding the soundness, presentation, and contributions of our research not only encourages us but also validates the effort we have invested in this work.\n\nWe will try to include more samples without cherry pick."}]

Author rebuttal:
We have organized the comments from the reviewers, and below are several issues that have been highlighted by multiple reviewers.

**Clarification of Benefits and Analysis:**

In the principal section of this submission, metrics such as CLIP, FID, and aesthetic scoring were employed to gauge our model's performance. These metrics often stress the model's capacity to generate images. For example, FID measures how closely the generated images fit the distribution of real images, while the CLIP Score represents the correlation between the generated images and corresponding text, and aesthetic scores gauge the artistic quality. The FID score is highly linked to the training dataset, and due to resource constraints, using only a subset of LAION might decrease diversity, leading to a higher FID.

We emphasize that our work focuses on image controllability while maintaining quality. Thus, additional metrics have been introduced in the appendix for a more comprehensive evaluation of control over image attributes. These supplementary metrics process the images into given control modality signals, then quantify the differences between generated images and provided signals.  For example, we obtain the HED of the generated image, then calculate its L2 loss relative to the conditional HED input. These supplementary metrics process the generated images into the given control modality signals, subsequently quantifying the discrepancies between the generated images and the provided signals. We repeat our evaluation results as Tables 1 and 2 in the attachment. It can be found that our approach is superior to the baseline in controllability across all trained modalities.


**Computational Complexity and Additional Details:**

Alleviating the substantial overhead of multi-modal control is a primary goal. We're happy to clarify specific details about our model's parameters to illuminate this work's contributions.

Our employed SD base model's parameter count is 865.91M, with a single ControlNet having 378.50M parameters. Our fusion network $\mathcal{M}$ consists of only 1.29M parameters per modality, and each ControlNorm adds just 0.31M parameters.
In single modality control using ControlNet, the total parameters are 865.91M + 378.50M. With an increase in modalities, this overhead grows linearly, reaching 865.91M + $n$ × 378.50M where $n$ is the number of modalities.
Our innovative gControlNet handles multiple modalities via a single HyperNetwork efficiently. Without ControlNorm, the parameters count is 865.91M + 378.50M + $n$ × 1.29M, substantially cutting the computational needs in multi-modal contexts and highlighting our approach's efficiency.

$\mathcal{M}$ is a straightforward convolutional network with eight layers and seven SiLU activation functions, with the last layer initialized to zero. Each  $\mathcal{M}$ has 1.29M parameters, and each modality will have a corresponding $\mathcal{M}$, for instance, 3 different modalities in our case result in 3.88M total parameters. It can effectively manage any number of trained modalities but can't handle unknown ones, meaning the model can process subsets and the exact set of training modalities.

Compared with gControlNet without ControlNorm and ControlNet, there will be a $\mathcal{M}$ in the former's training process. This operation might degrade the quality of the generated images without the assistance of ControlNorm. And after passing through $\mathcal{M}$ and gControlNet will there be only a set of control signals injected into the base model while ControlNet or T2I will have multipile set of control signals.  In addition, unlike T2I and ControlNet, where different modalities are trained separately, gControlNet co-trained $\mathcal{M}$ with different modalities. 

Codes will be made available for reproduction.

**Ethical Considerations:**

We acknowledge your concern about the potential misuse of our model. The images that our model can generate heavily depend on the training dataset used. Currently, our model's training data strictly complies with open-source licenses and has been carefully selected, providing some degree of safeguard against misuse. In situations where the model generates certain objects (including human being), many details can be easily distinguished. Going forward, we are considering the addition of watermarks to the generated images to prevent unauthorized use, along with the implementation of a monitoring network at the output stage. Similar tools are already widely utilized in projects like Stability / DALL.E2, and we believe our model can also benefit from adopting such measures.

**Conflicting control signals**

When confronted with conflicting signals across different modalities, the model endeavors to interpret each modality coherently. It strives to incorporate all the information conveyed by the different modalities, translating them into conditions for the generated images. The result is a theoretically reasonable representation that balances and harmonizes the conflicting inputs, ensuring that the synthesized images reflect as much of the diverse input data as possible.  However, there is a possibility that one modality may overshadow another if its representation is significantly more dominant or contains richer information. In such cases, the more expressive modality may suppress the other, leading to an unbalanced influence on the generated images. This phenomenon might result in the loss of some nuanced details conveyed by the less dominant modality. Some samples are demonstrated as Fig.1 in the attachment.


Meta-review: This paper introduces a novel idea supported by strong empirical results. The reviewers appreciate the valuable contribution of this study. In preparation for the camera-ready version, we kindly urge the authors to incorporate the feedback provided by the reviewers, which will undoubtedly further refine the quality and impact of the work.
Decision: Accept (poster)