Title: Cocktail: Mixing Multi-Modality Control for Text-Conditional Image Generation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-modality

Image Generation

Diffusion
Abstract: Text-conditional diffusion models are able to generate high-fidelity images with diverse contents.
However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. 
In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined control for text-conditional diffusion models. 
Specifically, we introduce a hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. 
gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination of modality signals, or the supplementary fusion of multiple modality signals. 
The control signals are then fused and injected into the backbone model according to our proposed ControlNorm.
Furthermore, our advanced spatial guidance sampling methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image.
We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals.

Cocktail : Mixing Multi-Modality Controls for Text-Conditional Image Generation

Abstract

Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined control for text-conditional diffusion models. Specifically, we introduce a hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination of modality signals, or the supplementary fusion of multiple modality signals. The control signals are then fused and injected into the backbone model according to our proposed ControlNorm. Furthermore, our advanced spatial guidance sampling methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image. We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals. The codes are released at https://mhh0318.github.io/cocktail/ .

1 Introduction

Text-conditional diffusion models [41, 39, 42, 34] have actualized the capacity for high-quality generative capabilities. These models facilitate the generation of an array of high-calibre images through the utilization of concise textual prompts. However, linguistic representations pose inherent challenges in accurately encapsulating the precise imagery anticipated by the user, owing to the potential ambiguities and subjective interpretations of verbal descriptions within the context of visual synthesis. Moreover, minor alterations to the textual prompts also yield distinct visual outputs, underscoring the absence of refined control over the generative process.

Modifying the prior is a series of existing solutions for multi-modal control, e.g. , the control of the entire prior space [39, 48, 23, 22, 57] as demonstrated in Fig. 1(a). These approaches centered on the whole prior lack the capacity for localised image modifications and the preservation of background elements. Moreover, these models typically require training from scratch, which demands a substantial amount of resources.

Modality #1

Modality

#2

Modality #3

Modality

... ...

Text

-

conditional

Diffusion

(c)

Cocktail

Modality #1

Modality #5

Modality #2

Modality #3

Modality

... ...

Text

-

conditional

Diffusion

(b) ControlNet / T2I Adapter

(a) Prior Modification

Diffusion

Decoder

Modalities

Prior

Modality #1

Modality #2

Modality #3

Modality

... ...

In response to the methods dealing with latent representation [29, 4, 27], an additional lightweight hyper-network is introduced in [53, 33], which is designed to encode external control signals into latent vectors and subsequently inject them directly into the backbone network. Such methods effectively handle control over the single additional modality; however, they exhibit limitations when confronted with multiple modalities, as shown in Fig 1(b). One issue is that each modality requires a unique network, leading to a computational overhead that escalates proportionally with the increase in the number of modalities. Furthermore, the impact of additive coefficients between different modes on the final imaging outcomes warrants consideration. The inherent imbalance among superimposed modes makes these additive coefficients a pivotal factor in determining the ultimate synthesized output. An additional challenge emerges during the sampling process when the model conducts an initial inference devoid of control signal injection. This preliminary inference step could result in object placement that potentially contradicts the control signals.

In this paper, we propose a novel pipeline, as shown in Fig. 1(c), termed Cocktail, which accomplishes multi-modality control through a text-conditional diffusion model. It encompasses three main components: 1) a hyper-network capable of accommodating multi-modal input, 2) a conditional normalisation method to mix the control features, and 3) a sampling strategy designed to facilitate precise control over the generative process. As shown in Fig. 2, our method is proficient in generating images that meet all input conditions or any arbitrary subset thereof, utilizing only one single model .

To achieve this goal, we initially trained a branched network named gControlNet, which accepts multiple modalities. Upon training, the branched network can simultaneously accept arbitrary combinations of existing modalities. When multiple modalities coexist within the same region, the model is capable of automatically fusing input from different modalities, balancing the disparities between them. To leverage the features of the gControlNet, we further proposed controllable normalisation (ControlNorm). We can achieve better representation of control signals in terms of semantic and spatial aspects according to the decoupling provided by ControlNorm.

Moreover, we introduce a spatial guidance sampling method in order to facilitate spatial generation under the purview of multi-modal control signals. Specifically, we employ distinct textual prompts to differentiate entities from the background, incorporating entities into the background via a prompt editing approach. Our devised sampling approach demonstrates efficacy in circumventing the generation of undesired entities.

In summary, our main contributions are as follows:

We introduced the Generalized ControlNet (gControlNet), a branched network capable of adaptively integrating multi-modal information, effectively addressing issues stemming from imbalances between modalities;

We proposed the Controllable Normalistion (ControlNorm) to optimize the utilization of information within branched networks to yield more effective outcomes;

We introduced a spatial guidance sampling method based on the operation within the attention map to generate relevant information tailored to regional contexts, preventing the inclusion of undesired objects outside the specified regions.

(3) Mask+Edge

Edge

(1) Edge+Pose

Pose

(2) Pose+Mask

Full: Edge+Mask+Pose

Edge

(1)

(3)

Full

Pose

Pose

Mask

Edge

(2)

Mask

Prompt: A girl holding a cat

Mask

2 Related Work

Text-conditional Diffusion Models. Diffusion models [19, 46] has achieved great success in the area of text-to-image synthesis [34, 39, 41, 3, 14]. To reduce the computational cost, diffusion models typically function within the latent space [41] or produce low-resolution images that are later improved through super-resolution models [39, 3]. Fast sampling methods have also successfully reduced the number of generation steps required by diffusion models from hundreds down to merely a few [45, 32, 25, 30, 11]. The provision of classifier guidance during the sampling process can also significantly impact the outcomes, leading to substantial improvements in the results [9]. In addition to the widely used classifier-free guidance [18], other types of guidance are also worth exploring [55, 13].

HyperNetworks for Pre-trained Models. Training a diffusion model is highly resource-intensive and environmentally unfriendly [41]. Fine-tuning such a model can also be challenging due to the vast number of parameters involved [1]. Therefore, introducing an additional tiny branched network to bias the output of the original network is a more reasonable choice [10, 2, 8]. Similar ideas have also proven to be effective in diffusion models: Hypernet [15] and LoRA [20] models are capable of altering the original diffusion model's sampling distribution by training a small branched network, which has become the most popular branched network in the current research community. More similar to our work is ControlNet [53], which is subsequently combined with different layers in the denoising U-Net to provide support for various task-specific guidance. It presents impressive results with various conditional inputs, yet with only one modality for each model. In contrast, our cocktail endows the ControlNet with multitasking capabilities using only one single model.

Conditional Normalisation approaches have been employed across a range of vision tasks, including style transfer [24, 12], conditional generation [36, 7, 14, 21] and image-to-image translation [40]. These techniques involve normalizing layer activations to zero mean and unit deviation, followed by denormalisation through an affine transformation derived from external data. External data can be in multiple formats, such as style images, semantic masks, or category labels.

GeneralizedControlNet

×3

Dump Attention

Maps for Spatirl Guidance Sampling

Down-sampling

Encoder Block 64x64

Encoder Block 32x32

Encoder Block 16x16

Encoder Block 8x8

Middle Block 8x8

x1

Controllable

Normalisation

Conv

Conv

β

(·)w

Control

→

Conv

ParameterFree

Latent

ControllableNormalisation

Encoder Block 64x64

Encoder Block 32x32

Encoder Block 16x16

Encoder Block 8x8

Middle Block 8x8

x1

ControllableNormalisation

Decoder Block 8x8

×3

ControllableNormalisation

Decoder Block 16x16

ControllableNormalisation

Decoder Block 32x32

ControllableNormalisation

Decoder Block 64x64

Controllable

→

Zt

t&prompt

embedding

Attention Map-based Prompt Tuning. Prompt-to-Prompt [16] is a method that adjusts local or global specifics in text-guided diffusion models by altering the cross-attention maps from source to target image, thereby maintaining spatial layout and geometry. Recently, numerous efforts have been made to improve the outcomes [35, 37]. However, such approaches are limited to synthesized images without an inversion technique. Unlike cross-attention, self-attention focuses on inter-pixel relationships within the same domain [47]. Paint-with-words [3] is another method that allows users to specify the spatial locations of objects by selecting phrases from the text prompt.

3 Methods

In this work, our main goal is to design a controllable generator that utilizes various modalities of input, within a single model . To achieve this, we first propose a more general branched network, gControlNet, which can generate control signals from different modalities using a single network and adaptively weighted fuse them together. Furthermore, we propose ControlNorm to inject the signals from the gControlNet into the diffusion backbone, which solves the imbalanced problem within various modalities. Finally, we propose a spatial guidance sampling method to avoid the presence of extraneous objects in the generation process. By modifying the attention map, this approach effectively incorporates control signals into the backbone network. The whole pipeline is demonstrated in Fig. 3.

3.1 Generalized ControlNet with Controllable Normalisation

Generalized ControlNet. ControlNet [53] is a method designed to influence and control the behavior of neural networks by adjusting the input conditions of specific network blocks. Instead of directly modifying the parameters of the primary network, ControlNet employs an auxiliary network to generate feature offsets. These offsets are then combined with different layers in the main network, such as a denoising U-Net, to support various task-specific guidance.

Given a trained backbone network block F ( · ; θ ) with parameter θ , the input feature x can be mapped to y . For the branched part, we duplicate the parameter θ to create a trainable copy θ t , which is

then trained using the supplementary modality. Preserving the original weights helps retain the information stored in the initial model after training on large-scale datasets, which ensures that the quality and diversity of the generated images do not degrade. Mathematically, the output from the trained network block can be expressed as:

where the control signal of a single modality c m is typically processed to obtain a format identity for x , for example, through a zero-initialized convolutional layer, and then added to x . Z ( · ) represents the zero-initialized layer. It is not only used in the process of handling c m , but also serves to adjust the output of the branched network F ( · , θ t ) .

To accomplish the goals of accepting multiple external modalities as input and balancing signals from different modalities, we have devised a modified framework that adeptly merges these varied sources of information. At the top of our network, we adopt a simple downsampling network M ( · ) to convert external conditional signals to the latent space, allowing the conditional signals to be directly injected into the latent space. It is worth noting that M ( · ) is versatile and can adapt to different types of external signals. Given k different modalities, the converted conditional features are c k m = M ( C k ) .

Controllable Normalisation. Instead of directly passing the sum of conditional features via a zero-initialized layer to the network block F ( · ; θ t ) , i.e. , ˆ c m = Z ( ∑ i c i ) , we introduce a controllable normalisation (ControlNorm) method, which has an additional layer to generate two sets of learnable parameters, γ (ˆ c m ) and β (ˆ c m ) , conditioned on all k modalities. These two sets of parameters are used in the conditional normalisation layer to fuse the external conditional signals and the original signals. Specifically, the input to the trainable block F ( · ; θ t ) becomes:

where µ c ( x ) and σ c ( x ) are the mean and standard deviation of the feature x along the channel c , ⊙ is Hadamard product, and γ (ˆ c ) and β (ˆ c ) are vectors that have the same dimension as x . With the help of zero-convolution, we can preserve the identity of x just before the start of the fine-tuning. It is worth noting that in the following layers, the internal feature in the branched network h can be integrated with the latent feature z from the original network in the same way:

where h is the intermediate features from the original network and z is the intermediate features from the branched network. Specifically, we will have five sets of intermediate features from the Stable Diffusion [41] U-Net backbone and our generalized ControlNet, including four sets of features from encoder blocks and one from the middle block.

In fact, our controllable normalisation is a generalized version of conditional normalisation [36, 24]. After changing the mean and variance calculation dimension and replacing the external signal ˆ c by a mask image, real image, or class labels, we can derive the various forms of SPADE [36], AdaIN [24], CIN [12] and MoVQ [56]. More interestingly, our controllable normalisation method not only enables the use of external signals as conditions, but also allows intermediate-layer signals to act as constraints.

As shown in Fig. 3, only the parameters of the gControlNet require updating. Given an initial latent z 0 and a time-step t , the diffusion process progressively introduces noise ϵ to this latent, transforming the original latent into a noisy state z t . The objective of the diffusion model is to estimate the noise ϵ adding to the x . Our proposed gControlNet shares the same objective function as the diffusion model, aiming to predict the noise added at time t . The only distinction lies in the incorporation of multimodal information as conditional signals ˆ c m = [ c 0 m , c 1 m , . . . , c k m ] :

3.2 Spatial Guidance Sampling

In order to leverage the control signals from generalized ControlNet and ensure that the generated objects appear within the areas of interest, we proposed a spatial guidance sampling method. Here, we mainly focus on editing the cross-attention layers of the U-Net in Stable Diffusion.

Given a noisy latent z at timestep t as the input to the pretrained U-Net with n blocks F ( n ) ( · ; θ ( n ) ) at different resolution, the latent vector z will be down-sampled to various dimensional z ( n ) that map to the corresponding block F ( n ) . The cross-attention maps A ( n ) ∈ R ( N i ,N t ) in each block, parameterized by θ ( n ) , are associated with the linear projection of latent feature Q ( n ) = f Q ( ϕ ( z ( n ) )) and the prompt K = f K ( c text ) :

where A ( n ) ij represents the attentional strength between the j -th prompt token (among N t tokens) and the i -th latent feature (among N i features). Intuitively, for each prompt token K j , there exists a corresponding latent feature map A ( n ) j with spatial information. Moreover, the corresponding feature map will contain spatial and shape information for the associated object. However, it is not only the object-describing tokens that contain information about the object's location and shape, some connecting words and padding tokens also convey spatial information for the overall scene [6, 43].

We apply a masking strategy to the corresponding attention maps. In detail, we construct two sets of attention masks M pos ( n ) and M neg ( n ) ∈ R ( N i ,N t ) . Each column M pos ( n ) j and M neg ( n ) j is a flattened alpha mask, which is determined by the visibility of the corresponding text token K j . The values of M pos ( n ) ij and M neg ( n ) ij are determined based on the relationship between image token Q i and text token K j . On the one hand, if image token Q i corresponds to a region of the image that should be influenced by text token K j , M pos ( n ) ij is assigned the value of 1. On the other hand, if image token Q i corresponds to a region of the image that should not be influenced by text token K j , M neg ( n ) ij is set to 1. It is worth noting that we consider the feature maps corresponding to most words as negative in order to avoid generating undesired objects. The mask components M pos ( n ) and M neg ( n ) are incorporated into the cross-attention computation process:

It is found that larger weights at higher noise levels [3] can lead to better results, thus ω pos and ω neg are noise-level sensitive parameters, defined by:

where ω ′ is a user-provided hyper-parameter.

We then substitute the feature map corresponding to the object description. Contrary to the image editing motivation behind conventional prompt tuning methods, it is not necessary for our method to provide a reference image and an amended prompt. Recalling the framework of our generalized ControlNet, the branch architecture is identical to the encoder portion of the backbone network. We found that the attention maps A ( n ) j | θ ( n ) t within the branch network also encompass object locations and shape information. Consequently, we opt for the attention map A ( n ) j | θ ( n ) t associated with the description K j from the respective layer in ControlNet as the source for the original attention map A ( n ) j | θ ( n ) substitution, ensuring that the information in the substituted attention map aligns more closely with the external input signal rather than the textual information derived from the original backbone network. Specifically, we replace the attention map generated by the original backbone network (based on the corresponding object description) with the attention map from the corresponding module in the branch network:

Subsequently, we can produce the spatially guided output from cross-attention layer by taking the product of ˆ A ( n ) with V .

4 Experiments

In this section, we delve into a comprehensive experimental analysis to validate the efficacy and superiority of the proposed method through ablation studies and application demonstrations. Subse-

A huntress and a big white wolf.

A little boy walks down the road, followed by a cat.

An astronaut standing on themountain

quently, in Sec. 4.1, we put forth both quantitative and qualitative results, elucidating the comparative advantages of our approach. We also present an array of intriguing applications made possible by our gControlNet, showcasing its practical utility. Finally, Sec. 4.2 is dedicated to the discussion of ablation studies, scrutinising the impacts of varying injection and sampling methods. The experimental configurations, including the dataset specifications, implementation details, and evaluation metrics, can be found in the Appendix.

4.1 Applications and Comparisions

Cocktail is proficient in seamlessly supporting multiple control inputs and autonomously fusing them, thereby eliminating the necessity for manual intervention to equilibrate diverse modalities. This unique property empowers users to easily incorporate a variety of modalities, resulting in more flexible multi-modal control. In Figure 4, we demonstrate how users can provide spatial information about multiple objects in different modalities to generate a complex scene. Notably, this entire process is accomplished by a single model without the need for additional branch networks.

We then compare Cocktail with two state-of-the-art methods: ControlNet [53] and T2I-Adapter [33], in the context of text-guided image-to-image translation within multiple modalities. We employ several evaluation methods, including LPIPS, mPA, mIoU, mAP and L2 distance for various modalities in this section. As depicted in Table 1, our method outperformed both ControlNet and T2I-Adapter across all evaluation metrics. This remarkable achievement signifies that our proposed cocktail can generate a structural image that closely resembles the ground truth image and aligns better with the input conditions, establishing its superiority. The visualization in Fig. 6 also illustrates the superior ability of our model to harmonize with the control signals.

We further present some samples from our method to examine its effectiveness on uni-modality translation. The visual comparison on the COCO validation set is showcased in Figure 5, highlighting the compelling performance. Benefiting from mixed training of multiple modalities, our method achieves remarkable generation quality, surpassing even models exclusively trained for a single control signal.

Our experiments show that Cocktail effectively leverages information from different modalities and exhibits outstanding multi-object generation abilities with consistent composition across various control signals.

A white dog

standing on

top of a

wooden bench.

A clock tower

is part of an

old building.

A coo coo

clock mounted

to the side of a

wall.

A teddy bear

sitting on a

blue chair.

A man that has

glasses and a

hat.

A cute little

blond girl

sitting next to

a pizza.

Prompt

Control Signal

Ground truth

ControlNet

Ours

4.2 Ablations

gControlNet and ControlNorm Previous methods [53] utilized a direct sum approach to fuse control signals from hyper-networks with latent variables from the original network. While this method conveys spatial information, it fails to consider semantic information, such as text or other modalities. Control signals decoupled through ControlNorm allow the preservation of semantic information while conveying spatial information. Another point is the need for normalization to address the imbalance of different modality signals. However, typical normalization methods lead to the loss of semantic information [36]. Therefore, the control signals introduced through ControlNorm can better interpret conditional information. We present some generated images in Fig. 10 to substantiate the interpretative capability of ControlNorm.

Spatial Guidance Sampling The spatial guidance sampling method not only ensures that objects are generated within controllable areas, but also minimizes the impact on other areas. An intuitive application is that when we modify certain objects or modalities, other parts of the generated image can remain unchanged. Figure 7 illustrates the contrast between employing spatial guidance sampling and its absence. A more significant variation in the overall tone of the resultant image is observed when spatial guidance is not utilized, leading to inconsistencies in the details of the attire. Conversely, the incorporation of new objects and modalities with the use of spatial guidance minimally impacts the original image.

Ground Truth

Conditions

Multi-ControlNet

Multi-Adapter

Ours

The man is next to a lion

X Key Points

X Key Points

√Key Points

X Segmentation

√Segmentation

√Segmentation

X Sketch

X Additional Objects

X Additional Objects

√Additional Objects

Girl sitting next to the snow leopard

X Key Points

X Key Points

√Key Points

√Segmentation

X Segmentation

√Segmentation

× Sketch

X Sketch

√Sketch

X Additional Objects

X Additional Objects

√Additional Objects

Cowboy boy and his horse

X Key Points

X Key Points

√Key Points

√Segmentation

X Segmentation

√Segmentation

X Sketch

X Sketch

X Additional Objects

X Additional Objects

√Additional Objects

5 Conclusion

In this work, we proposed Cocktail, a pipeline to achieve fine control by fusing multi-modal signals. Firstly, we introduced a generalized ControlNet capable of handling signals of different modalities using a single model. We also revealed that the semantic information of signals from the hyper-network may be lost through direct addition. However, a simple decoupling allows for a better interpretation of control signals. We introduced a controllable normalisation for decoupling and integration. Finally, we proposed a sampling scheme that can prevent the generation of unnecessary objects outside the focus area while also achieving a certain degree of editing capability and background protection.

Limitations. While we have achieved multi-modal fusion and control, there are still certain limitations in the handling of control signals from Cocktail that need to be addressed in future work. Firstly, although the current spatial guidance method can effectively prevent objects from being generated outside the focus area, it requires users to individually specify the area and corresponding object description during implementation. Secondly, spatial guidance can cause instability in the latent space in certain situations, leading to the generated images degrading and deviating from the existing control signals. Therefore, finding a stable anchor point as a reference during the generation process is also worth exploring.

Broader Impacts. The integration of multimodal control signals as inputs for synthesis greatly enhances user interaction flexibility and streamlines the utilization of text-conditional diffusion models. However, the process of fine-tuning large-scale generative models to accommodate diverse modalities requires a significant amount of energy, and we anticipate that the development of a universal model capable of accepting multiple modalities will help mitigate this impact. However, the growing capabilities in image generation also facilitate the production of manipulated images with malicious intent, such as the creation of counterfeit or deceitful information.

References

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 , 2020.

Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition , pages 18511-18521, 2022.

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022.

Manuel Brack, Patrick Schramowski, Felix Friedrich, Dominik Hintersdorf, and Kristian Kersting. The stable artist: Steering semantics in diffusion latent space. arXiv preprint arXiv:2212.06013 , 2022.

Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. IEEE transactions on pattern analysis and machine intelligence , 43(1):172-186, 2021.

Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv preprint arXiv:2304.03373 , 2023.

Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. arXiv preprint arXiv:1810.01365 , 2018.

Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534 , 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Tan M Dinh, Anh Tuan Tran, Rang Nguyen, and Binh-Son Hua. Hyperinverter: Improving stylegan inversion via hypernetwork. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11389-11398, 2022.

Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically-damped langevin diffusion. arXiv preprint arXiv:2112.07068 , 2021.

Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv:1610.07629 , 2016.

Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv preprint arXiv:2206.09012 , 2022.

Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10696-10706, 2022.

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106 , 2016.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 , 2021.

Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and Ponnuthurai N Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11502-11511, 2022.

Minghui Hu, Chuanxia Zheng, Heliang Zheng, Tat-Jen Cham, Chaoyue Wang, Zuopeng Yang, Dacheng Tao, and Ponnuthurai N Suganthan. Unified discrete diffusion for simultaneous vision-language generation. arXiv preprint arXiv:2211.14842 , 2022.

Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778 , 2023.

Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision , pages 1501-1510, 2017.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 , 2022.

Alexander Kirillov. Mscoco keypoint evaluation metric. https://github.com/ cocodataset/cocodataset.github.io , 2021.

Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210.10960 , 2022.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint arXiv:2301.07093 , 2023.

Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 , 2022.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927 , 2022.

Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 , 2023.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Dong Huk Park, Grace Luo, Clayton Toste, Samaneh Azadi, Xihui Liu, Maka Karalashvili, Anna Rohrbach, and Trevor Darrell. Shape-guided diffusion with inside-outside attention. arXiv preprint arXiv:2212.00210 , 2022.

Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2337-2346, 2019.

Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or. Localizing object-level shape variations with text-to-image diffusion models. arXiv preprint arXiv:2303.11306 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2287-2296, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Vishnu Sarukkai, Linden Li, Arden Ma, Christopher Ré, and Kayvon Fatahalian. Collage diffusion. arXiv preprint arXiv:2303.00262 , 2023.

Christoph Schuhmann. Clip+mlp aesthetic score predictor. https://github.com/ christophschuhmann/improved-aesthetic-predictor , 2022.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572 , 2022.

Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for image-to-image translation. arXiv preprint arXiv:2205.12952 , 2022.

Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-toimage models with human preference. arXiv preprint arXiv:2303.14420 , 2023.

Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In International conference on computer vision , pages 1395-1403, 2015.

Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977 , 2023.

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. arXiv preprint arXiv:2302.12242 , 2023.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. arXiv preprint arXiv:2207.06635 , 2022.

Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems , 35:23412-23425, 2022.

Jianbin Zheng, Daqing Liu, Chaoyue Wang, Minghui Hu, Zuopeng Yang, Changxing Ding, and Dacheng Tao. Mmot: Mixture-of-modality-tokens transformer for composed multimodal conditional image synthesis. arXiv preprint arXiv:2305.05992 , 2023.

A Experimental settings

A.1 Datasets and Annotations.

All of our experiments are performed on LAION-AESTHETICS-6.5 dataset, which contains about 600K image-text pairs with predicted aesthetics scores of higher than 6.5.

Our experiment includes three types of commonly used additional control signals, and we use different methods to obtain pseudo-labeling annotations:

Sketch map : The delineation of texture details within the structured control signal is accomplished using the HED boundary detector [50]

Semantic segmentation map : To derive segmentation maps from images, we leverage SAN [52], a high-performing, open-vocabulary segmentator.

Human pose map : The generation of a comprehensive human pose map, including body, face, and hand positions, is facilitated by OpenPose [5].

It is crucial to note that our proposed Generalized ControlNet exhibits broad applicability and can be effortlessly adapted to accommodate a multitude of other modality inputs.

A.2 Implementation Details.

gControlNet is adapted from the pretrained Stable Diffusion v2.1 in this paper and trained for 20 epochs with a batch size of 64 on 4 NVIDIA 80G-A100 GPUs within 4 days. We use the AdamW optimizer with a learning rate of 3.0 e-05. All the training images in the LAION-AESTHETICS-6.5 are first resized to 512 by the short side and then randomly cropped to 512 × 512 . During inference, the sampler is DDIM, the sampling steps are 50, and the classifier-free guidance scale is 9.0 by default.

A.3 Evaluation Metrics.

In order to compare the generation performance of existing methods with ours, we adopted six metrics to evaluate the quality, text-image alignment, aesthetics, and human preference of the generated images. They are Frechet Inception Distance (FID) [17], CLIP/BLIP Score [38, 28], Aesthetic Score [44], Human Preference Score (HPS) [49] and ImageRewared [51]. Specifically, FID measures the distribution distances of real and generated image sets utilizing a pre-trained classification network ( e.g. , Inception V3 trained on ImageNet). Instead, CLIP/BLIP Score calculates cosine similarities between the corresponding image and text features extracted by the CLIP/BLIP image and text encoders, respectively. The Aesthetic Score is based on an additional MLP layer on top of a pretrained CLIP image encoder to measure the human aesthetic aspect of a single image. To consider the alignment with human values and preferences, HPS and ImageReward were proposed and are based on the reward model pretrained on CLIP or BLIP.

However, these metrics do not effectively measure the fidelity of our model to different modalities. Therefore, we additionally employ several other evaluation metrics, including the Learned Perceptual

Image Patch Similarity (LPIPS) [54] for overall image quality, the L2 distance for Holistically-Nested Edge Detection (HED), the mean Pixel Accuracy (mPA) [31] and mean Intersection over Union (mIoU) [31] for segmentation maps, and the mean Average Precision (mAP) over 10 object keypoint similarity (OKS) thresholds [26] for human pose map. Specifically, LPIPS is utilized to assess the dissimilarity between the generated image and the ground-truth image. Furthermore, we extract the conditions, namely the sketch map, segmentation map, and pose map, from both the generated image and the ground-truth image. By computing the distance, specifically employing L2 Distance, mPA, mIoU, and mAP, between these extracted conditions, we can gain a deeper understanding of the fidelity with respect to various modalities.

Through these metrics, we can more clearly ascertain whether the generated images can follow the guidance of different modal conditions.

B Additional quantitative analysis

We compare Cocktail with two state-of-the-art methods: ControlNet [53] and T2I-Adapter [33], in the context of text-guided image-to-image translation within a single modality. As demonstrated in Table 2, our model not only acquires multi-modal capabilities but also demonstrates performance on par with the comparative methods in single-modal tasks. It is worth noting that FID was measured on the basis of 5000 images from zero-shot generation.

It worth noting that FID may not provide a precise evaluation as they are substantially contingent on the sample size and the characteristics of the training dataset. Our methodology prioritizes the fidelity of the generated images, a perspective that is at odds with the principles of CLIP/BLIP.

C Additional qualitative results

We provide additional generated images under various multi-conditional scenarios and compare them with Multi-ControlNet and Multi-Adapter. In Fig. 6, we have demonstrated some samples within less complexity, we further show the comparative performance of our model against other

A man with a hat.

w/Spatial Guidance Sampling

w/o Spatial Guidance Sampling

state-of-the-art models under challenging disjoint scenarios in Fig. 8. In Fig. 10, we also conduct an ablation study on the effectiveness of our ControlNorm module. It can be observed that our method outperforms other state-of-the-art models in terms of balancing and expressing multi-modal signals, and ControlNorm provide a more stable integration performance. Besides, our model is able to accept arbitraty combiantions of the given modalities, as shown in Fig. 9.

We also provide further samples under single modality signals, e.g. , Human Pose in Figs. 11&12, Sketch Map as shown in Figs. 13&14 and Segmentation Map in Fig. 15. It can be seen that our method is more faithful to the given conditional signals.

Conditions

Multi-ControlNet

Multi-Adapter

Ours

√Key Points

√KeyPoints

√Key Points

X Segmentation

X Segmentation

√Segmentation

× Sketch

X Sketch

A girl in a beautiful dress

X Key Points

X Key Points

√KeyPoints

X Segmentation

X Segmentation

√Segmentation

× Sketch

X Sketch

√Sketch

A man in a tie leans against a chair

X Key Points

√Key Points

√Key Points

X Segmentation

X Segmentation

√Segmentation

√Sketch

X Sketch

Sketch

A man looking at a horse.

A man standing with a lion.

Compose#1

Compose#1

Key Points

Key Points

Segmentation

Segmentation

Sketch

Sketch

Compose#2

Compose#2

Key Points

Key Points

Segmentation

Segmentation

Sketch

Sketch

Compose #3

Compose#3

口

Key Points

Key Points

Segmentation

Segmentation

Sketch

Sketch

A young child

squats down

in front of a

bowl.

A man wearing

a hat and a

An older man

with a bow tie

happily poses

for a picture.

A man sitting

at a table in

front of a plate

of food.

Prompt

Control Signal

Ground truth

T2I Adapter

ControlNet

Ours

A man looking

through a

book on top of

a table.

A person

skiing in an

open area of

snow.

A woman

stands on a

snow covered

hill.

Anolder

woman sitting

at a tablewith

acake

Prompt

Control Signal

Ground truth

T2IAdapter

ControlNet

Ours

An old wooden clock attached to the wall

1

A row of white teddy bears on shelf next to DVDs.

th

the offae

le

Two men smiling and taking a picture together.

Prompt

Ours

A kitchen with a counter and a table with chairs.

A littleroom and dining room area with furnitures.

A woman holding a phone in one hand up to her ear.

a close up of a dog near abowl.

A car with a snowboard on it's ski rack parked in a foggy parking lot next to twol cars with empty ski racks.

A small dog laying on a bed next to a laptop computer.

T2IAdapter

ControlNet

Prompt

Control Signal

Ground truth

Ours

A big and

white horse in

a dirt cage.

A couple of

pieces of very

nice looking

luggage.

a person riding

skis on a

A hand is

holding a

carrot for a

llama to chew.

A female horse

rider mounted

near a body of

water and

mountains.

A living room

filled with

furniture and

set of glass

doors.

A vase filled

with flowers

next to a

smaller vase

filled with

evenmore

flowers.

a living room

thathas some

couches and

tables in it.

Prompt

Control Signal

Ground truth

T2IAdapter

ControlNet

Ours

Peer reviews:
comments: SUMMARY:
This paper proposes an approach to enhance the controllability of generative image models through the fusion of multiple modality signals.
The proposed model is a strict generalization of ControlNet which is able to incorporate geometric constraints into the generation, including pose, edges, and segmentation map.

STRENGTHS:
The paper describes a very practical generalization of a SOTA approach to controllable generative models. It is well motivated, provides a clear description of the architecture, and both quantitative and qualitative results that validate the approach.

WEAKNESSES:
None. The paper is very clear on the benefits and limitations of the approach, and provides a practical improvement over a SOTA approach that is relevant to the research community and downstream users of the technology.

QUESTIONS:
None.

LIMITATIONS:
Limitations adequately described.
It is could be a good idea to include a sample of random results (not cherry picked) in the appendix in order to provide a sense of typical failure modes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies text-conditional diffusion models and proposes a pipeline for image generation with multi-modal control signals. The pipeline contains three new ingredients. Firstly, the paper introduces gControlNet, a generalized version of ControlNet, that can take different conditions (such as sketch, segmentation and pose maps) simultaneously as input without increasing the ControlNet size linearly with respect to the number of conditions. Secondly, the paper proposes to inject features produced by gControlNet into the backbone model via normalization, instead of a simple sum as used in the original ControlNet. Thirdly, the paper introduces spatial sampling that aims to harmonize spatial arrangements from both text prompt and image controls. Experiments show that the proposed method achieves on-par quantitative performance compared with ControlNet and T2I-Adapter, and improved visual quality especially with multiple modalities.\n\nSTRENGTHS:\n1. ControlNet proves to be a powerful approach for controllable text-to-image generation, while its original form requires fine-tuning a ControlNet network for each condition, which is resource consuming. This paper proposes a method to jointly train a shared network that can deal with different types of control input, which may extend the applicability of ControlNet in many cases.\n\n2. The paper has done extensive quantitative evaluations to compare the proposed method with previous work. For generative models quantitative metrics may not be able to measure the actual quality of generated images, but it is important to study what are the good metrics and how to design better ones. This paper makes an effort towards the direction.\n\nWEAKNESSES:\n1. The gControlNet is a crucial component in the proposed pipeline, but the paper does not discuss details of the downsampling network M, either in main text or Appendix. On the other hand, it seems the controllable normalization and spatial guidance do not show significant improvement over their counterparts in previous work. In Figure 6 gControlNet without ControlNorm has much worse image quality, but is it due to insufficient training in this case (as the original ControlNet does not use ControlNorm and performs well)? As for spatial guidance sampling, the paper shows the effect of retaining background, but not placing objects to the correct location. The latter may not be an issue since sketch, segmentation and pose maps contain spatial information already. This seems to weaken the role of spatial sampling, which requires manual design that is not present in paper.\n\n2. The paper uses Stable Diffusion 2.1 as backbone for training gControlNet, while ControlNet and T2I-Adapter used Stable Diffusion 1.5 as backbone. Therefore it may be unfair to directly compare gControlNet with ControlNet and T2I-Adapter. Also it seems that gControlNet does not show much qualitative improvement over ControlNet in single condition cases.\n\nQUESTIONS:\n1. What is the architecture of the downsampling network M in gControlNet? Can it deal with varying number of input conditions, and deal with new conditions during inference that are unseen in training?\n\n2. In spatial guidance sampling, how are M^{pos} and M^{neg} determined? Are they instance-specific or universal for all instances? It would be interesting to see, in challenging cases, how this module can help resolve conflicts among different input condition maps, and layer arrangement in complex scenes.\n\n3. How are the three types of control used in training? Are all of them or only some of them used in each update step?\n\n4. (Minor) The mapping notation in Eq. (1)-(3) is confusing.\n\nLIMITATIONS:\nThe paper discusses limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study introduces ChatIR, a chat-based image retrieval system that engages in a conversation with the user to clarify their search intent and retrieve the desired image from a large corpus. The system leverages Large Language Models to generate follow-up questions to an initial image description and achieves a success rate of over 78% after 5 dialogue rounds, compared to 75% when questions are asked by humans and 64% for a single shot text-to-image retrieval.\n\nSTRENGTHS:\nThe strength of this submission lies in its clear and impactful contribution, which proposes a pipeline that combines multiple modalities such as edge, pose, segmentation mask, and more. The authors effectively communicate the significance of this research contribution, highlighting the potential of integrating diverse modalities to enhance the proposed method's performance. Additionally, the submission features a visually appealing and clear illustrative depiction of the method, which effectively aids in explaining the approach. The selection of an attractive and informative illustration further enhances the clarity and appeal of the paper.\n\nWEAKNESSES:\nOne notable weakness of the submission is the lack of sufficient explanation regarding the experimental results. While the paper goes into detail in explaining the proposed method, it falls short in providing a thorough analysis and interpretation of the results. In Table 1, there are inconsistencies in the trends of metric scores between the proposed method and the two baselines, yet the authors do not adequately explain these discrepancies. It is crucial to understand why the proposed method may not perform as well in certain metrics, whether it is due to the limitations of the metrics themselves or issues with the method's performance. Merely providing cherry-picked demonstrations is insufficient and does not substitute for a comprehensive analysis of the results. It is essential to address these gaps in the paper by providing a detailed explanation and interpretation of the observed trends and discrepancies in the experimental results.\n\nQUESTIONS:\nIt would greatly enhance the clarity of the paper if the authors could explicitly list the implemented and supported modalities in the abstract or introduction. Currently, the detailed setup is not revealed until Section 4, with the exception of Figure 1. Providing this information earlier in the paper would help readers gain a clear understanding of the modalities that are incorporated and supported in the proposed approach. Including this information in the abstract or introduction would improve the overall organization and accessibility of the paper, allowing readers to quickly grasp the key aspects of the submission.\n\nLIMITATIONS:\nThe authors have included discussions of the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Cocktail, a novel pipeline for multi-modal and spatially-refined control in text-conditional diffusion models. The authors address the challenge of ambiguous descriptions in linguistic representations by incorporating additional control signals. They propose three main components: gControlNet, ControlNorm, and a spatial guidance sampling method.\n\ngControlNet is a hyper-network designed to align and infuse control signals from different modalities into the pre-trained diffusion model. It can accommodate flexible modality signals and allows for the simultaneous reception of any combination of modalities or the fusion of multiple modalities. This capability eliminates the need for manual intervention and equilibrates the disparities between modalities, making the system more flexible and capable of seamlessly supporting multiple control inputs.\n\nControlNorm is a controllable normalization method that optimizes the utilization of information within branched networks. It decouples control signals, allowing for better representation of both semantic and spatial aspects. By preserving semantic information while conveying spatial information, ControlNorm overcomes the limitations of previous methods that either ignored semantic information or led to the loss of it through normalization. The proposed method effectively interprets conditional information and demonstrates its interpretative capability through generated images.\n\nAdditionally, the paper introduces a spatial guidance sampling method that ensures precise control over the generative process. By modifying the attention map, the method incorporates control signals into the backbone network, preventing the generation of undesired objects outside the specified regions. This approach enables the generation of high-quality images that closely align with the input conditions.\n\nOverall, the proposed Cocktail pipeline addresses the challenges of ambiguous descriptions in text-conditional diffusion models and achieves multi-modal and spatially-refined control. It offers a novel approach to integrating control signals from various modalities, optimizing their utilization, and generating high-quality images with fidelity to external signals.\n\nSTRENGTHS:\n- The paper introduces a novel pipeline called Cocktail that enables multi-modal and spatially-refined control in text-conditional diffusion models, addressing the challenge of ambiguous descriptions in linguistic representations. This is a significant contribution to the field as it tackles an important problem in generative models and expands the capabilities of text-guided image synthesis.\n\n- The proposed pipeline consists of three well-defined components: gControlNet, ControlNorm, and a spatial guidance sampling method. Each component is carefully designed to address specific challenges and effectively integrate control signals from different modalities. The approach is comprehensive and demonstrates a systematic solution to achieve high-quality synthesis and fidelity to multiple external signals.\n\n- The paper provides extensive experimental results and evaluation metrics, comparing the proposed Cocktail pipeline with state-of-the-art methods. The results consistently show that Cocktail outperforms existing approaches in text-guided image-to-image translation across multiple modalities. This empirical evidence demonstrates the effectiveness and superiority of the proposed method.\n\n- The paper emphasizes the practicality and efficiency of the Cocktail pipeline by demonstrating its ability to accomplish multi-modal control within a single model. This not only simplifies the model architecture but also reduces the computational overhead associated with multiple branch networks. The approach is both technically sound and resource-efficient, making it highly applicable in real-world scenarios.\n\nWEAKNESSES:\n- The authors dont specify how they control for potential misuse of their model. A model like this if open sourced, can have wide ethical impacts with people using it for unethical means like creating deep fakes and modifying images which could potentially be unethical. This should be addressed by the authors.\n- The paper does not provide a thorough discussion or analysis of the computational complexity or efficiency of the Cocktail pipeline. Considering the potential computational overhead of incorporating multiple modalities and the fusion process, it would be valuable to address the computational requirements and scalability of the proposed method\n- The evaluation section could be expanded to include a more comprehensive analysis of the results. While the paper mentions various evaluation metrics, it would be valuable to discuss the limitations of these metrics and provide additional qualitative analysis or user studies to further validate the effectiveness of the proposed method.\n- Although the paper compares the proposed Cocktail pipeline with state-of-the-art methods, it does not thoroughly discuss the limitations and failure cases of the proposed method. Understanding the shortcomings and potential failure modes is crucial for assessing the robustness and generalizability of the proposed approach.\n- Implementation details seem sparse and might hinder reproducibility for the paper\n\nQUESTIONS:\n- How does the model handle conflicting signals provided by different modality signals, for example, if the provided pose and mask conflict with each other, how is the model supposed to handle this?\n- How does the method prevent users from misusing the model from generating sensitive images or modify images to create unethical content\n\nLIMITATIONS:\n- The authors dont specify how they control for potential misuse of their model. A model like this if open sourced, can have wide ethical impacts with people using it for unethical means like creating deep fakes and modifying images which could potentially be unethical. This should be addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes Cocktail, a pipeline to mix various modalities into one embedding. The model is based on a variant of ControlNet and infuses control signals from disparate modalities into the pre-trained diffusion model. It is also equipped with a sampling approach named spatial guidance sampling that constructs the cross-attention weights based on the spatial location. The model can perform text-to-image generation conditioned on multiple modalities and outperforms controlnet in some metrics.

STRENGTHS:
Both the proposed controllable normalization and spatial sampling method are novel and have not been previously explored in the context of controllable generation.

WEAKNESSES:
The model does not consistently outperform the two baselines: ControlNet and T2I-Adapter, in text-guided image-to-image translation.  The author claimed the model can benefit from mixed training of multiple modalities (line 213-215) but the proposed model shows a mixed result in various metrics across different tasks. Additionally, there is no quantitative comparison in multi-modality conditioning to existing baselines. In spatial guidance, no qualitative metrics showing its effect were given as well. Overall, it is unclear what are the advantages of the proposed method over simple baselines such as controlnet.

QUESTIONS:
1. Line 216 - 218, what figures or tables refer to “consistent composition”?
2. Eq 7, what is $\sigma$? How to determine its value?
3. How to determine the correspondence between text and image token?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07