SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cocktail : Mixing Multi-Modality Controls for Text-Conditional Image Generation
Minghui Hu†, Jianbin Zheng*, Daqing Liu‡, Chuanxia Zheng§, Chaoyue Wang¶, Dacheng Tao¶, and Tat-Jen Cham†
†Nanyang Technological University, *South China University of Technology, ‡University of Oxford, †The University of Sydney, ‡JD Explore Academy
Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined control for text-conditional diffusion models. Specifically, we introduce a hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination of modality signals, or the supplementary fusion of multiple modality signals. The control signals are then fused and injected into the backbone model according to our proposed ControlNorm. Furthermore, our advanced spatial guidance sampling methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image. We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals. The codes are released at https://mhh0318.github.io/cocktail/.
Text-conditional diffusion models [41, 39, 42, 34] have actualized the capacity for high-quality generative capabilities. These models facilitate the generation of an array of high-calibre images through the utilization of concise textual prompts. However, linguistic representations pose inherent challenges in accurately encapsulating the precise imagery anticipated by the user, owing to the potential ambiguities and subjective interpretations of verbal descriptions within the context of visual synthesis. Moreover, minor alterations to the textual prompts also yield distinct visual outputs, underscoring the absence of refined control over the generative process.
Modifying the prior is a series of existing solutions for multi-modal control, e.g., the control of the entire prior space [39, 48, 23, 22, 57] as demonstrated in Fig. 1(a). These approaches centered on the whole prior lack the capacity for localised image modifications and the preservation of background elements. Moreover, these models typically require training from scratch, which demands a substantial amount of resources.
(a) Prior Modification
(b) ControlNet / T2I Adapter
Figure 1: Comparison of various control methods. Our approach requires only one generalized model, unlike previous that needed multiple models for multiple modalities.
(c) Cocktail
In response to the methods dealing with latent representation [29, 4, 27], an additional lightweight hyper-network is introduced in [53, 33], which is designed to encode external control signals into latent vectors and subsequently inject them directly into the backbone network. Such methods effectively handle control over the single additional modality; however, they exhibit limitations when confronted with multiple modalities, as shown in Fig 1(b). One issue is that each modality requires a unique network, leading to a computational overhead that escalates proportionally with the increase in the number of modalities. Furthermore, the impact of additive coefficients between different modes on the final imaging outcomes warrants consideration. The inherent imbalance among superimposed modes makes these additive coefficients a pivotal factor in determining the ultimate synthesized output. An additional challenge emerges during the sampling process when the model conducts an initial inference devoid of control signal injection. This preliminary inference step could result in object placement that potentially contradicts the control signals.
In this paper, we propose a novel pipeline, as shown in Fig. 1(c), termed Cocktail, which accomplishes multi-modality control through a text-conditional diffusion model. It encompasses three main components: 1) a hyper-network capable of accommodating multi-modal input, 2) a conditional normalisation method to mix the control features, and 3) a sampling strategy designed to facilitate precise control over the generative process. As shown in Fig. 2, our method is proficient in generating images that meet all input conditions or any arbitrary subset thereof, utilizing only one single model.
To achieve this goal, we initially trained a branched network named gControlNet, which accepts multiple modalities. Upon training, the branched network can simultaneously accept arbitrary combinations of existing modalities. When multiple modalities coexist within the same region, the model is capable of automatically fusing input from different modalities, balancing the disparities between them. To leverage the features of the gControlNet, we further proposed controllable normalisation (ControlNorm). We can achieve better representation of control signals in terms of semantic and spatial aspects according to the decoupling provided by ControlNorm.
Moreover, we introduce a spatial guidance sampling method in order to facilitate spatial generation under the purview of multi-modal control signals. Specifically, we employ distinct textual prompts to differentiate entities from the background, incorporating entities into the background via a prompt editing approach. Our devised sampling approach demonstrates efficacy in circumventing the generation of undesired entities.
- We introduced the Generalized ControlNet (gControlNet), a branched network capable of adaptively integrating multi-modal information, effectively addressing issues stemming from imbalances between modalities;
- We proposed the Controllable Normalisation (ControlNorm) to optimize the utilization of information within branched networks to yield more effective outcomes;
- We introduced a spatial guidance sampling method based on the operation within the attention map to generate relevant information tailored to regional contexts, preventing the inclusion of undesired objects outside the specified regions.
(1) Edge+Pose
(2) Pose+Mask
(3) Mask+Edge
Pose
Prompt: A girl holding a cat
Full: Edge+Mask+Pose
Figure 2: Examples of our model with the same prompt. Given a text prompt along with various modality signals, our approach is able to synthesize images that satisfy all input conditions or any arbitrary subset of these conditions using a single model. The prompt is: A girl holding a cat.
Text-conditional Diffusion Models. Diffusion models [19, 46] has achieved great success in the area of text-to-image synthesis [34, 39, 41, 3, 14]. To reduce the computational cost, diffusion models typically function within the latent space [41] or produce low-resolution images that are later improved through super-resolution models [39, 3]. Fast sampling methods have also successfully reduced the number of generation steps required by diffusion models from hundreds down to merely a few [45, 32, 25, 30, 11]. The provision of classifier guidance during the sampling process can also significantly impact the outcomes, leading to substantial improvements in the results [9]. In addition to the widely used classifier-free guidance [18], other types of guidance are also worth exploring [55, 13].
HyperNetworks for Pre-trained Models. Training a diffusion model is highly resource-intensive and environmentally unfriendly [41]. Fine-tuning such a model can also be challenging due to the vast number of parameters involved [1]. Therefore, introducing an additional tiny branched network to bias the output of the original network is a more reasonable choice [10, 2, 8]. Similar ideas have also proven to be effective in diffusion models: Hypernet [15] and LoRA [20] models are capable of altering the original diffusion model's sampling distribution by training a small branched network, which has become the most popular branched network in the current research community. More similar to our work is ControlNet [53], which is subsequently combined with different layers in the denoising U-Net to provide support for various task-specific guidance. It presents impressive results with various conditional inputs, yet with only one modality for each model. In contrast, our cocktail endows the ControlNet with multitasking capabilities using only one single model.
Conditional Normalisation approaches have been employed across a range of vision tasks, including style transfer [24, 12], conditional generation [36, 7, 14, 21] and image-to-image translation [40]. These techniques involve normalizing layer activations to zero mean and unit deviation, followed by denormalisation through an affine transformation derived from external data. External data can be in multiple formats, such as style images, semantic masks, or category labels.
Figure 3: The network architecture of Generalized ControlNet (gControlNet) with Controllable Normalisation (ControlNorm). The parameters indicated by the yellow sections are sourced from the pre-trained model and stay constant, while only those in the blue sections are updated during training, with the gradient back-propagated along the blue arrows.
Attention Map-based Prompt Tuning. Prompt-to-Prompt [16] is a method that adjusts local or global specifics in text-guided diffusion models by altering the cross-attention maps from source to target image, thereby maintaining spatial layout and geometry. Recently, numerous efforts have been made to improve the outcomes [35, 37]. However, such approaches are limited to synthesized images without an inversion technique. Unlike cross-attention, self-attention focuses on inter-pixel relationships within the same domain [47]. Paint-with-words [3] is another method that allows users to specify the spatial locations of objects by selecting phrases from the text prompt.
In this work, our main goal is to design a controllable generator that utilizes various modalities of input, within a single model. To achieve this, we first propose a more general branched network, gControlNet, which can generate control signals from different modalities using a single network and adaptively weighted fuse them together. Furthermore, we propose ControlNorm to inject the signals from the gControlNet into the diffusion backbone, which solves the imbalanced problem within various modalities. Finally, we propose a spatial guidance sampling method to avoid the presence of extraneous objects in the generation process. By modifying the attention map, this approach effectively incorporates control signals into the backbone network. The whole pipeline is demonstrated in Fig. 3.
3.1 Generalized ControlNet with Controllable Normalisation
Generalized ControlNet. ControlNet [53] is a method designed to influence and control the behavior of neural networks by adjusting the input conditions of specific network blocks. Instead of directly modifying the parameters of the primary network, ControlNet employs an auxiliary network to generate feature offsets. These offsets are then combined with different layers in the main network, such as a denoising U-Net, to support various task-specific guidance.
Given a trained backbone network block  $\mathcal{F}(\cdot ;\theta)$  with parameter  $\theta$  , the input feature  $\pmb{x}$  can be mapped to  $\pmb{y}$  . For the branched part, we duplicate the parameter  $\theta$  to create a trainable copy  $\theta_{t}$  , which is
then trained using the supplementary modality. Preserving the original weights helps retain the information stored in the initial model after training on large-scale datasets, which ensures that the quality and diversity of the generated images do not degrade. Mathematically, the output from the trained network block can be expressed as:
$$
\boldsymbol {y} = \mathcal {F} (\boldsymbol {x}; \boldsymbol {\theta}) + \mathcal {Z} \left(\mathcal {F} (\boldsymbol {x} + \mathcal {Z} \left(\boldsymbol {c} _ {m}\right); \boldsymbol {\theta} _ {t})\right) \leftarrow \mathcal {F} (\boldsymbol {x}; \boldsymbol {\theta}), \tag {1}
$$
where the control signal of a single modality  $c_{m}$  is typically processed to obtain a format identity for  $x$ , for example, through a zero-initialized convolutional layer, and then added to  $x$ .  $\mathcal{Z}(\cdot)$  represents the zero-initialized layer. It is not only used in the process of handling  $c_{m}$ , but also serves to adjust the output of the branched network  $\mathcal{F}(\cdot, \theta_t)$ .
To accomplish the goals of accepting multiple external modalities as input and balancing signals from different modalities, we have devised a modified framework that adeptly merges these varied sources of information. At the top of our network, we adopt a simple downsampling network  $\mathcal{M}(\cdot)$  to convert external conditional signals to the latent space, allowing the conditional signals to be directly injected into the latent space. It is worth noting that  $\mathcal{M}(\cdot)$  is versatile and can adapt to different types of external signals. Given  $k$  different modalities, the converted conditional features are  $\pmb{c}_{m}^{k} = \mathcal{M}(C^{k})$ .
Controllable Normalisation. Instead of directly passing the sum of conditional features via a zero-initialized layer to the network block  $\mathcal{F}(\cdot ;\pmb{\theta}_t)$ , i.e.,  $\hat{\pmb{c}}_m = \mathcal{Z}(\sum_i\pmb {c}^i)$ , we introduce a controllable normalisation (ControlNorm) method, which has an additional layer to generate two sets of learnable parameters,  $\gamma (\hat{c}_m)$  and  $\beta (\hat{c}_m)$ , conditioned on all  $k$  modalities. These two sets of parameters are used in the conditional normalisation layer to fuse the external conditional signals and the original signals. Specifically, the input to the trainable block  $\mathcal{F}(\cdot ;\pmb{\theta}_t)$  becomes:
$$
\left(\boldsymbol {I} + \mathcal {Z} \left(\gamma \left(\hat {\boldsymbol {c}} _ {m}\right)\right)\right) \odot \frac {\boldsymbol {x} - \mu_ {c} (\boldsymbol {x})}{\sigma_ {c} (\boldsymbol {x})} \oplus \mathcal {Z} \left(\boldsymbol {\beta} \left(\hat {\boldsymbol {c}} _ {m}\right)\right) \leftarrow \boldsymbol {x} + \mathcal {Z} \left(\boldsymbol {c} _ {m}\right), \tag {2}
$$
where  $\mu_c(\pmb{x})$  and  $\sigma_c(\pmb{x})$  are the mean and standard deviation of the feature  $\pmb{x}$  along the channel  $c$ ,  $\odot$  is Hadamard product, and  $\gamma(\hat{c})$  and  $\beta(\hat{c})$  are vectors that have the same dimension as  $\pmb{x}$ . With the help of zero-convolution, we can preserve the identity of  $x$  just before the start of the fine-tuning. It is worth noting that in the following layers, the internal feature in the branched network  $\pmb{h}$  can be integrated with the latent feature  $\pmb{z}$  from the original network in the same way:
$$
\left(\boldsymbol {I} + \mathcal {Z} (\boldsymbol {\gamma} (\boldsymbol {h}))\right) \odot \frac {\boldsymbol {z} - \mu_ {c} (\boldsymbol {z})}{\sigma_ {c} (\boldsymbol {z})} \oplus \mathcal {Z} (\boldsymbol {\beta} (\boldsymbol {h})) \leftarrow \boldsymbol {x} + \mathcal {Z} (\boldsymbol {c} _ {m}), \tag {3}
$$
where  $h$  is the intermediate features from the original network and  $z$  is the intermediate features from the branched network. Specifically, we will have five sets of intermediate features from the Stable Diffusion [41] U-Net backbone and our generalized ControlNet, including four sets of features from encoder blocks and one from the middle block.
In fact, our controllable normalisation is a generalized version of conditional normalisation [36, 24]. After changing the mean and variance calculation dimension and replacing the external signal  $\hat{c}$  by a mask image, real image, or class labels, we can derive the various forms of SPADE [36], AdaIN [24], CIN [12] and MoVQ [56]. More interestingly, our controllable normalisation method not only enables the use of external signals as conditions, but also allows intermediate-layer signals to act as constraints.
As shown in Fig. 3, only the parameters of the gControlNet require updating. Given an initial latent  $\mathbf{z}_0$  and a time-step  $t$ , the diffusion process progressively introduces noise  $\epsilon$  to this latent, transforming the original latent into a noisy state  $\mathbf{z}_t$ . The objective of the diffusion model is to estimate the noise  $\epsilon$  adding to the  $\mathbf{x}$ . Our proposed gControlNet shares the same objective function as the diffusion model, aiming to predict the noise added at time  $t$ . The only distinction lies in the incorporation of multimodal information as conditional signals  $\hat{\mathbf{c}}_m = [c_m^0, c_m^1, \dots, c_m^k]$ :
$$
\mathcal {L} = \mathbb {E} _ {\boldsymbol {z} _ {0}, t, \boldsymbol {c} _ {p}, \hat {\boldsymbol {c}} _ {m}, \epsilon \sim \mathcal {N} (0, 1)} \left[ \| \epsilon - \epsilon_ {\theta} \left(\boldsymbol {z} _ {t}, t, \boldsymbol {c} _ {p}, \hat {\boldsymbol {c}} _ {m}\right) \| _ {2} ^ {2} \right] \tag {4}
$$
3.2 Spatial Guidance Sampling
In order to leverage the control signals from generalized ControlNet and ensure that the generated objects appear within the areas of interest, we proposed a spatial guidance sampling method. Here, we mainly focus on editing the cross-attention layers of the U-Net in Stable Diffusion.
Given a noisy latent  $\mathbf{z}$  at timestep  $t$  as the input to the pretrained U-Net with  $n$  blocks  $\mathcal{F}^{(n)}(\cdot; \boldsymbol{\theta}^{(n)})$  at different resolution, the latent vector  $\mathbf{z}$  will be down-sampled to various dimensional  $\mathbf{z}^{(n)}$  that map to the corresponding block  $\mathcal{F}^{(n)}$ . The cross-attention maps  $A^{(n)} \in \mathbb{R}^{(N_i, N_t)}$  in each block, parameterized by  $\boldsymbol{\theta}^{(n)}$ , are associated with the linear projection of latent feature  $Q^{(n)} = f_{Q}(\phi(\mathbf{z}^{(n)}))$  and the prompt  $K = f_{K}(\mathbf{c}^{\mathrm{text}})$ :
$$
A _ {i j \mid \boldsymbol {\theta} ^ {(n)}} ^ {(n)} = \frac {\exp \left\langle Q _ {i} ^ {(n)} , K _ {j} \right\rangle}{\sum_ {k = 1} \exp \left\langle Q _ {i} ^ {(n)} , K _ {k} \right\rangle}, \tag {5}
$$
where  $A_{ij}^{(n)}$  represents the attentional strength between the  $j$ -th prompt token (among  $N_t$  tokens) and the  $i$ -th latent feature (among  $N_i$  features). Intuitively, for each prompt token  $K_j$ , there exists a corresponding latent feature map  $A_j^{(n)}$  with spatial information. Moreover, the corresponding feature map will contain spatial and shape information for the associated object. However, it is not only the object-describing tokens that contain information about the object's location and shape, some connecting words and padding tokens also convey spatial information for the overall scene [6, 43].
We apply a masking strategy to the corresponding attention maps. In detail, we construct two sets of attention masks  $M^{\mathrm{pos}(n)}$  and  $M^{\mathrm{neg}(n)} \in \mathbb{R}^{(N_i,N_t)}$ . Each column  $M_j^{\mathrm{pos}(n)}$  and  $M_j^{\mathrm{neg}(n)}$  is a flattened alpha mask, which is determined by the visibility of the corresponding text token  $K_{j}$ . The values of  $M_{ij}^{\mathrm{pos}(n)}$  and  $M_{ij}^{\mathrm{neg}(n)}$  are determined based on the relationship between image token  $Q_{i}$  and text token  $K_{j}$ . On the one hand, if image token  $Q_{i}$  corresponds to a region of the image that should be influenced by text token  $K_{j}$ ,  $M_{ij}^{\mathrm{pos}(n)}$  is assigned the value of 1. On the other hand, if image token  $Q_{i}$  corresponds to a region of the image that should not be influenced by text token  $K_{j}$ ,  $M_{ij}^{\mathrm{neg}(n)}$  is set to 1. It is worth noting that we consider the feature maps corresponding to most words as negative in order to avoid generating undesired objects. The mask components  $M^{\mathrm{pos}(n)}$  and  $M^{\mathrm{neg}(n)}$  are incorporated into the cross-attention computation process:
$$
\tilde {A} _ {i j \mid \boldsymbol {\theta} ^ {(n)}} ^ {(n)} = \frac {\exp \left\langle Q _ {i} ^ {(n)} , K _ {j} \right\rangle + \omega^ {\operatorname {p o s}} M ^ {\operatorname {p o s} (n)} - \omega^ {\operatorname {n e g}} M ^ {\operatorname {n e g} (n)}}{\sum_ {k = 1} \exp \left\langle Q _ {i} ^ {(n)} , K _ {k} \right\rangle}. \tag {6}
$$
It is found that larger weights at higher noise levels [3] can lead to better results, thus  $\omega^{\mathrm{pos}}$  and  $\omega^{\mathrm{neg}}$  are noise-level sensitive parameters, defined by:
$$
\omega^ {(\cdot)} = \omega^ {\prime} \cdot \log (1 + \sigma) \cdot \max  (\boldsymbol {A} ^ {(n)}), \tag {7}
$$
where  $\omega^{\prime}$  is a user-provided hyper-parameter.
We then substitute the feature map corresponding to the object description. Contrary to the image editing motivation behind conventional prompt tuning methods, it is not necessary for our method to provide a reference image and an amended prompt. Recalling the framework of our generalized ControlNet, the branch architecture is identical to the encoder portion of the backbone network. We found that the attention maps  $A_{j|\pmb{\theta}_t^{(n)}}^{(n)}$  within the branch network also encompass object locations and shape information. Consequently, we opt for the attention map  $A_{j|\pmb{\theta}_t^{(n)}}^{(n)}$  associated with the description  $K_j$  from the respective layer in ControlNet as the source for the original attention map  $A_{j|\pmb{\theta}^{(n)}}^{(n)}$  substitution, ensuring that the information in the substituted attention map aligns more closely with the external input signal rather than the textual information derived from the original backbone network. Specifically, we replace the attention map generated by the original backbone network (based on the corresponding object description) with the attention map from the corresponding module in the branch network:
$$
\hat {\boldsymbol {A}} ^ {(n)} = \left[ \tilde {A} _ {0 | \boldsymbol {\theta} ^ {(n)}} ^ {(n)}; \dots ; A _ {j | \boldsymbol {\theta} _ {t} ^ {(n)}} ^ {(n)}; \dots ; \tilde {A} _ {N _ {t} | \boldsymbol {\theta} ^ {(n)}} ^ {(n)} \right] \leftarrow \left[ \tilde {A} _ {0 | \boldsymbol {\theta} ^ {(n)}} ^ {(n)}; \dots ; \tilde {A} _ {j | \boldsymbol {\theta} ^ {(n)}} ^ {(n)}; \dots ; \tilde {A} _ {N _ {t} | \boldsymbol {\theta} ^ {(n)}} ^ {(n)} \right]. \tag {8}
$$
Subsequently, we can produce the spatially guided output from cross-attention layer by taking the product of  $\hat{\pmb{A}}^{(n)}$  with  $V$ .
In this section, we delve into a comprehensive experimental analysis to validate the efficacy and superiority of the proposed method through ablation studies and application demonstrations. Subse
Table 1: Quantitative comparison on the COCO5k validation set. The best result is highlighted.
Method	Similarity (LPIPS ↓)	Sketch Map (L2 Distance ↓)	Segmentation Map (mPA ↑)	Segmentation Map (mIoU ↑)	Pose Map (mAP ↑)
Multi-Adapter	0.7273 ±0.00120	7.93310 ±0.01392	26.30 ±0.242	13.98 ±0.177	40.02 ±0.761
Multi-ControlNet	0.6653 ±0.00145	7.59721 ±0.01516	36.59 ±0.273	22.70 ±0.229	38.19 ±0.761
Ours w/o ControlNorm	0.4900 ±0.00141	7.18413 ±0.01453	48.26 ±0.287	32.66 ±0.272	61.93 ±0.775
Ours ¥	0.4836 ±0.00133	7.28929 ±0.01385	49.20 ±0.289	33.27 ±0.271	61.99 ±0.778
A huntress and a big white wolf.
A little boy walks down the road, followed by a cat.
An astronaut standing on the mountain
Figure 4: Our model can generate images with the provided prompts and multi-modality information (e.g., edge, pose, and segmentation map) across various scales.