SlowGuess/NeurIPS_2023_OCR huggingface dataset:
sequently, in Sec. 4.1, we put forth both quantitative and qualitative results, elucidating the comparative advantages of our approach. We also present an array of intriguing applications made possible by our gControlNet, showcasing its practical utility. Finally, Sec. 4.2 is dedicated to the discussion of ablation studies, scrutinising the impacts of varying injection and sampling methods. The experimental configurations, including the dataset specifications, implementation details, and evaluation metrics, can be found in the Appendix.
4.1 Applications and Comparisons
Cocktail is proficient in seamlessly supporting multiple control inputs and autonomously fusing them, thereby eliminating the necessity for manual intervention to equilibrate diverse modalities. This unique property empowers users to easily incorporate a variety of modalities, resulting in more flexible multi-modal control. In Figure 4, we demonstrate how users can provide spatial information about multiple objects in different modalities to generate a complex scene. Notably, this entire process is accomplished by a single model without the need for additional branch networks.
We then compare Cocktail with two state-of-the-art methods: ControlNet [53] and T2I-Adapter [33], in the context of text-guided image-to-image translation within multiple modalities. We employ several evaluation methods, including LPIPS, mPA, mIoU, mAP and L2 distance for various modalities in this section. As depicted in Table 1, our method outperformed both ControlNet and T2I-Adapter across all evaluation metrics. This remarkable achievement signifies that our proposed cocktail can generate a structural image that closely resembles the ground truth image and aligns better with the input conditions, establishing its superiority. The visualization in Fig. 6 also illustrates the superior ability of our model to harmonize with the control signals.
We further present some samples from our method to examine its effectiveness on uni-modality translation. The visual comparison on the COCO validation set is showcased in Figure 5, highlighting the compelling performance. Benefiting from mixed training of multiple modalities, our method achieves remarkable generation quality, surpassing even models exclusively trained for a single control signal.
Our experiments show that Cocktail effectively leverages information from different modalities and exhibits outstanding multi-object generation abilities with consistent composition across various control signals.
Figure 5: Qualitative comparison of Uni-Modality on the COCO validation set.
4.2 Ablations
gControlNet and ControlNorm Previous methods [53] utilized a direct sum approach to fuse control signals from hyper-networks with latent variables from the original network. While this method conveys spatial information, it fails to consider semantic information, such as text or other modalities. Control signals decoupled through ControlNorm allow the preservation of semantic information while conveying spatial information. Another point is the need for normalization to address the imbalance of different modality signals. However, typical normalization methods lead to the loss of semantic information [36]. Therefore, the control signals introduced through ControlNorm can better interpret conditional information. We present some generated images in Fig. 10 to substantiate the interpretative capability of ControlNorm.
Spatial Guidance Sampling The spatial guidance sampling method not only ensures that objects are generated within controllable areas, but also minimizes the impact on other areas. An intuitive application is that when we modify certain objects or modalities, other parts of the generated image can remain unchanged. Figure 7 illustrates the contrast between employing spatial guidance sampling and its absence. A more significant variation in the overall tone of the resultant image is observed when spatial guidance is not utilized, leading to inconsistencies in the details of the attire. Conversely, the incorporation of new objects and modalities with the use of spatial guidance minimally impacts the original image.
Figure 6: Cocktail can address the imbalance among various modalities. Here, the "cross" symbol and the checkmark symbol denote the unmatched and matched modalities, respectively. It is important to note that our model accurately captures all modalities.
In this work, we proposed Cocktail, a pipeline to achieve fine control by fusing multi-modal signals. Firstly, we introduced a generalized ControlNet capable of handling signals of different modalities using a single model. We also revealed that the semantic information of signals from the hyper-network may be lost through direct addition. However, a simple decoupling allows for a better interpretation of control signals. We introduced a controllable normalisation for decoupling and integration. Finally, we proposed a sampling scheme that can prevent the generation of unnecessary objects outside the focus area while also achieving a certain degree of editing capability and background protection.
Limitations. While we have achieved multi-modal fusion and control, there are still certain limitations in the handling of control signals from Cocktail that need to be addressed in future work. Firstly, although the current spatial guidance method can effectively prevent objects from being generated outside the focus area, it requires users to individually specify the area and corresponding object description during implementation. Secondly, spatial guidance can cause instability in the latent space in certain situations, leading to the generated images degrading and deviating from the existing control signals. Therefore, finding a stable anchor point as a reference during the generation process is also worth exploring.
Broader Impacts. The integration of multimodal control signals as inputs for synthesis greatly enhances user interaction flexibility and streamlines the utilization of text-conditional diffusion models. However, the process of fine-tuning large-scale generative models to accommodate diverse modalities requires a significant amount of energy, and we anticipate that the development of a universal model capable of accepting multiple modalities will help mitigate this impact. However, the growing capabilities in image generation also facilitate the production of manipulated images with malicious intent, such as the creation of counterfeit or deceitful information.
[54] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[55] Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu. Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations. arXiv preprint arXiv:2207.06635, 2022.
[56] Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412-23425, 2022.
[57] Jianbin Zheng, Daqing Liu, Chaoyue Wang, Minghui Hu, Zuopeng Yang, Changxing Ding, and Dacheng Tao. Mmot: Mixture-of-modality-tokens transformer for composed multimodal conditional image synthesis. arXiv preprint arXiv:2305.05992, 2023.
A Experimental settings
A.1 Datasets and Annotations.
All of our experiments are performed on LAION-AESTHETICS-6.5 dataset, which contains about 600K image-text pairs with predicted aesthetics scores of higher than 6.5.
Our experiment includes three types of commonly used additional control signals, and we use different methods to obtain pseudo-labeling annotations:
- Sketch map: The delineation of texture details within the structured control signal is accomplished using the HED boundary detector [50]
- Semantic segmentation map: To derive segmentation maps from images, we leverage SAN [52], a high-performing, open-vocabulary segmentator.
- Human pose map: The generation of a comprehensive human pose map, including body, face, and hand positions, is facilitated by OpenPose [5].
It is crucial to note that our proposed Generalized ControlNet exhibits broad applicability and can be effortlessly adapted to accommodate a multitude of other modality inputs.
A.2 Implementation Details.
gControlNet is adapted from the pretrained Stable Diffusion v2.1 in this paper and trained for 20 epochs with a batch size of 64 on 4 NVIDIA 80G-A100 GPUs within 4 days. We use the AdamW optimizer with a learning rate of  $3.0\mathrm{e - }05$ . All the training images in the LAION-AESTHETICS-6.5 are first resized to 512 by the short side and then randomly cropped to  $512\times 512$ . During inference, the sampler is DDIM, the sampling steps are 50, and the classifier-free guidance scale is 9.0 by default.
A.3 Evaluation Metrics.
In order to compare the generation performance of existing methods with ours, we adopted six metrics to evaluate the quality, text-image alignment, aesthetics, and human preference of the generated images. They are Frechet Inception Distance (FID) [17], CLIP/BLIP Score [38, 28], Aesthetic Score [44], Human Preference Score (HPS) [49] and ImageRewarded [51]. Specifically, FID measures the distribution distances of real and generated image sets utilizing a pre-trained classification network (e.g., Inception V3 trained on ImageNet). Instead, CLIP/BLIP Score calculates cosine similarities between the corresponding image and text features extracted by the CLIP/BLIP image and text encoders, respectively. The Aesthetic Score is based on an additional MLP layer on top of a pretrained CLIP image encoder to measure the human aesthetic aspect of a single image. To consider the alignment with human values and preferences, HPS and ImageReward were proposed and are based on the reward model pretrained on CLIP or BLIP.
However, these metrics do not effectively measure the fidelity of our model to different modalities. Therefore, we additionally employ several other evaluation metrics, including the Learned Perceptual
Table 2: Additional quantitative comparison on the COCO validation set. The best result is highlighted.
(a) Text + Edge
Method	FID ↓	CLIP Score ↑	BLIP Score ↑	Aesthetic Score ↑	HPS ↑	ImageReward ↑
ControlNet	18.73	0.2588	0.5338	5.39	19.53	0.3726
T2I-Adapter	20.21	0.2638	0.5335	5.36	19.69	0.3344
Ours	16.66	0.2561	0.5321	5.35	19.53	0.2989
(b) Text + Segmentation Map
Method	FID ↓	CLIP Score ↑	BLIP Score ↑	Aesthetic Score ↑	HPS ↑	ImageReward ↑
ControlNet	20.53	0.2640	0.5262	5.43	19.85	0.2609
T2I-Adapter	21.81	0.2626	0.5225	5.25	19.57	0.0837
Ours	23.88	0.2613	0.5260	5.64	19.89	0.3735
(c) Text + Pose
Method	FID ↓	CLIP Score ↑	BLIP Score ↑	Aesthetic Score ↑	HPS ↑	ImageReward ↑
ControlNet	35.13	0.2656	0.5171	5.49	20.17	0.3683
T2I-Adapter	35.53	0.2697	0.5211	5.45	20.19	0.3810
Ours	39.64	0.2641	0.5210	5.70	20.22	0.4488
(d) Text + Sketch + Segmentation + Keypoints
Method	FID ↓	CLIP Score ↑	BLIP Score ↑	Aesthetic Score ↑	HPS ↑	ImageReward ↑
Ours	16.70	0.2562	0.5326	5.36	19.56	0.3118
Image Patch Similarity (LPIPS) [54] for overall image quality, the L2 distance for Holistically-Nested Edge Detection (HED), the mean Pixel Accuracy (mPA) [31] and mean Intersection over Union (mIoU) [31] for segmentation maps, and the mean Average Precision (mAP) over 10 object keypoint similarity (OKS) thresholds [26] for human pose map. Specifically, LPIPS is utilized to assess the dissimilarity between the generated image and the ground-truth image. Furthermore, we extract the conditions, namely the sketch map, segmentation map, and pose map, from both the generated image and the ground-truth image. By computing the distance, specifically employing L2 Distance, mPA, mIoU, and mAP, between these extracted conditions, we can gain a deeper understanding of the fidelity with respect to various modalities.
Through these metrics, we can more clearly ascertain whether the generated images can follow the guidance of different modal conditions.
B Additional quantitative analysis
We compare Cocktail with two state-of-the-art methods: ControlNet [53] and T2I-Adapter [33], in the context of text-guided image-to-image translation within a single modality. As demonstrated in Table 2, our model not only acquires multi-modal capabilities but also demonstrates performance on par with the comparative methods in single-modal tasks. It is worth noting that FID was measured on the basis of 5000 images from zero-shot generation.
It worth noting that FID may not provide a precise evaluation as they are substantially contingent on the sample size and the characteristics of the training dataset. Our methodology prioritizes the fidelity of the generated images, a perspective that is at odds with the principles of CLIP/BLIP.
C Additional qualitative results
We provide additional generated images under various multi-conditional scenarios and compare them with Multi-ControlNet and Multi-Adapter. In Fig. 6, we have demonstrated some samples within less complexity, we further show the comparative performance of our model against other
A man with a hat.
w/ Spatial Guidance Sampling
Figure 7: With spatial guidance, our method is capable of maintaining constancy in certain regions while modifying specific objects.
w/o Spatial Guidance Sampling
state-of-the-art models under challenging disjoint scenarios in Fig. 8. In Fig. 10, we also conduct an ablation study on the effectiveness of our ControlNorm module. It can be observed that our method outperforms other state-of-the-art models in terms of balancing and expressing multi-modal signals, and ControlNorm provide a more stable integration performance. Besides, our model is able to accept arbitrary combiations of the given modalities, as shown in Fig. 9.
We also provide further samples under single modality signals, e.g., Human Pose in Figs. 11&12, Sketch Map as shown in Figs. 13&14 and Segmentation Map in Fig. 15. It can be seen that our method is more faithful to the given conditional signals.
Ground Truth	Conditions	Multi-ControlNet	Multi-Adapter	Ours
A boy with a scarf	√Key Points
×Segmentation
×Sketch	√Key Points
×Segmentation
×Sketch	√Key Points
√Segmentation
√Sketch
A girl in a beautiful dress	√Key Points
×Segmentation
×Sketch	√Key Points
×Segmentation
×Sketch	√Key Points
√Segmentation
√Sketch
A man in a tie leans against a chair	√Key Points
×Segmentation
√Sketch	√Key Points
×Segmentation
×Sketch	√Key Points
√Segmentation
√Sketch
Figure 8: Additional results for Disjoint Multi-Modality Control.
Figure 9: Our model accepts arbitrary combinations of the given modalities.
gControlNet w/ ControlNorm
gControlNet w/o ControlNorm
Multi-ControlNet
Multi-Adapter
A girl with eye-glasses.
Figure 10: ControlNorm can address the imbalance among various modalities. Note that the framed areas in magenta and cyan do not provide a high-fidelity interpretation.
Multi-Adapter1
Figure 11: Additional results for Single Modality Control: Key Points
Figure 12: Additional results for Single Modality Control: Key Points
An old wooden clock attached to the wall
Two men
smiling and
taking a
picture
together.
Control Signal
T2I Adapter
ControlNet
Ours
Figure 13: Additional results for Single Modality Control: Sketch
Figure 14: Additional results for Single Modality Control: Sketch
Figure 15: Additional results for Single Modality Control: Segmentation Map