SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Text-alignment and image-alignment vary between the single-client tuned model and the center-node fused model. The upper bound of text-alignment and the lower bound of image-alignment are computed by replacing the concept's token (e.g.,  $V^{dogA}$ ) with its class token (e.g., dog) and sampling using the pretrained model.
and attribute through a text prompt can lead to attribute binding problems, as in Fig. 5(a), where the identities of three people are mixed, and the "red dress" is incorrectly assigned to other concepts.
To address these challenges, we propose a method called regionally controllable sampling. This approach utilizes both a global prompt and multiple regional prompts to describe an image based on spatial conditions. The global prompt provides the overall context, while the regional prompts specify subjects within specific regions, including their attributes and contextual information from the global prompt (e.g., "near a lake"). To achieve this, we introduce region-aware cross-attention. Given a global prompt  $P_{g}^{*}$  and  $n$  regional prompts  $P_{r_i}^*$ , we first incorporate the global prompt via cross-attention with the latent  $z$  by  $h = \text{softmax}\left(\frac{Q(z)K(P_g^*)}{\sqrt{d}}\right) \cdot V(P_g^*)$ . Then, we extract the regional latent feature by  $z_{i} = z \odot M_{i}$ , where  $M_{i}$  represents the binary mask associated with the region specified by  $P_{r_i}^*$ . We obtain regional features using  $h_i = \text{softmax}\left(\frac{Q(z_i)K(P_{r_i}^*)}{\sqrt{d}}\right) \cdot V(P_{r_i}^*)$ . Finally, we replace the features in the global output with the regional features:  $h[M_i] = h_i$ . As shown in Fig. 5(b), regionally controllable sampling allows for precise assignment of subjects and attributes, while maintaining a harmonious global context.
To conduct evaluation for Mix-of-Show, we collect a dataset containing characters, objects, and scenes. For ED-LoRA tuning, we incorporate LoRA layer into the linear layer in all attention
Reference Concepts
$V = V_{class}$
$V = V_{rand}^{+}V_{class}^{+}$
(a) Embedding Expressiveness
w/o Region Control
(c) Regionally Controllable Sampling
w/ Region Control (Ours)
(b) Gradient
Fusion (Ours)
w/ Weight Fusion
(b) Ablation of Fusion Type
Figure 7: Qualitative ablation study of Mix-of-Show.  $P^{*}$  means the text prompt.  $\Phi_0$  and  $\Delta \Phi$  denotes the pretrained model and LoRA weight, respectively.
w/ Gradient Fusion (Ours)
Methods	Real-Objects
Single→Fused	Real-Characters
Single→Fused	Real-Scenes
Single→Fused	Mean Change
Image-alignment	Lower Bound	0.721	0.471	0.595	0.596
LoRA + Weight Fusion	0.864→0.778 (-0.086)	0.761→0.555(-0.206)	0.824→0.769 (-0.055)	0.816→0.701 (-0.115)
ED-LoRA + Weight Fusion	0.868→0.798 (-0.070)	0.802→0.634 (-0.168)	0.858→0.816 (-0.042)	0.843→0.749 (-0.094)
ED-LoRA + Gradient Fusions	0.868→0.846 (-0.022)	0.802→0.770 (-0.032)	0.858→0.838 (-0.020)	0.843→0.818 (-0.025)
(a) Subject identity preservation (i.e., image-alignment) measured by CLIP score between LoRA+weight fusion, ED-LoRA+weight fusion, and ED-LoRA+gradient fusion. Our ED-LoRA+gradient fusion achieves the least loss in image-alignment after multi-concept fusion, preserving the best subject identity.
(b) Human preference study interface on Amazon Mechanical Turk.
(c) Human preference study between weight fusion and gradient fusion for fusing ED-LoRAs.
Human Evaluation	Image Alignment	Text Alignment
ED-LoRA + Weight Fusion	33.5%	47.5%
ED-LoRA + Gradient Fusion	66.5%	52.5%
Table 2: Quantitative ablation study of our main components. (a) Ablation study between the LoRA+weight fusion, ED-LoRA+weight fusion and our ED-LoRA+gradient fusion. (b, c) Human preference study to comparing weight fusion and gradient fusion for fusing our ED-LoRAs.
modules of the text encoder and Unet, with a rank of  $r = 4$  in all experiments. We use the Adam [48] optimizer with a learning rate of 1e-3, 1e-5 and 1e-4 for tuning text embedding, text encoder and Unet, respectively. For gradient fusion, we use the LBFGS optimizer [49] with 500 and 50 steps to optimize the text encoder and Unet, respectively. More details are provided in the supplementary.
Single-Concept Results. We compare our ED-LoRA with LoRA [3], Custom Diffusion [11] and  $\mathrm{P + }$  [4] for single-concept customization. The results are shown in Fig. 6 (a). Our ED-LoRA achieves comparable performance to previous methods on customizing objects, while maintaining better identity for character customization. More comparisons are provided in the supplementary.
Multi-Concept Results. We compare Mix-of-Show with LoRA [3], Custom Diffusion [11], and  $\mathrm{P + }$  [4] for decentralized multi-concept customization. For LoRA, we utilize weight fusion to combine the different concepts. In the case of  $\mathrm{P + }$ , we directly incorporate the tuned concept embedding into the pretrained model. And for Custom Diffusion, we follow their approach of constrained optimization to merge the key and value projections in cross-attention. To ensure fair evaluation, we employ the same regionally controllable sampling for multi-concept generation across all models and the results are summarized in Fig. 6 (b).
$\mathrm{P + }$  [4] and Custom Diffusion [11] only tunes text-related module (i.e., text embedding, or the key and value projection of cross-attention). In contrast, LoRA and Mix-of-Show add LoRA layers to the
entire model. The limited scope of tuned modules in  $\mathrm{P + }$  and Custom Diffusion leads to an excessive encoding of out-domain low-level details within the embedding. This leads to unnatural and less desirable outcomes when compared to LoRA and Mix-of-Show. In comparison to LoRA, which loses concept identity after weight fusion, Mix-of-Show effectively preserves the identity of each individual concept.
4.3 Quantitative Comparison
Following Custom Diffusion [11], we utilize the CLIP [47] text/image encoder to assess text alignment and image alignment. We evaluate on different category of concepts on both single-concept tuned model and multi-concept fused model. We include detailed evaluation setting in the supplementary.
Based on the results presented in Table. 1, both Mix-of-Show and LoRA exhibit superior image alignment compared to other methods, all the while maintaining comparable text alignment in the single-client tuned model. This achievement stems from their fine-tuning the spatial-related layer in Unet (e.g., linear projection layer in self-attention), a critical aspect for accurately capturing the complex concepts' identity, such as characters.
However, the main difference between LoRA and Mix-of-Show emerges in the context of multi-concept fusion. In the center-node fused model, LoRA experiences a significant decline in image alignment for each concept, progressively deteriorating towards the lower bound. In contrast, our Mix-of-Show method undergoes far less degradation in image alignment after multi-concept fusion.
**Embedding Expressiveness.** In Fig. 7(a), it is evident that our decomposed embeddings better preserve the identity of the specified concept compared to the standard text embeddings used in LoRA. This results in a more robust encoding of concept identity. As shown in the quantitative results in Table. 2(a), when LoRA is replaced with ED-LoRA, the identity loss from weight fusion (measured by mean change of image-alignment) is reduced from 0.115 to 0.094. This result verifies that expressive embeddings help reduce identity loss during multi-concept fusion.
Fusion Type. Built with the same ED-LoRAs, we conduct experiments to compare weight fusion and gradient fusion. As shown in Fig. 7(b), gradient fusion effectively preserves concept identity after concept fusion, resulting in superior results for multi-concept sampling. According to the quantitative results in Table. 2(a), gradient fusion significantly reduces the identity loss of weight fusion, decreasing it from 0.094 to 0.025. We also conduct a human evaluation and confirm a clear preference for gradient fusion, as indicated in Table. 2(c).
Regionally Controllable Sampling. As shown in Fig. 7(c), direct sampling leads to attribute binding issues, where the concept identities are mixed. However, our regionally controllable sampling overcomes this problem and achieves correct attribute binding in multi-concept generation.
In this work, we explore decentralized multi-concept customization and highlight the limitations of existing methods like LoRA tuning and weight fusion, which suffer from concept conflicts and identity loss in this scenario. To overcome these challenges, we propose Mix-of-Show, a framework that combines ED-LoRA for single-client concept tuning and gradient fusion for center-node concept fusion. ED-LoRA preserves individual concept essence in the embedding, avoiding conflicts, while gradient fusion minimizes identity loss during concept fusion. We also introduce regionally controllable sampling to handle attribute binding in multi-concept generation. Experiments demonstrate Mix-of-Show can successfully generate complex compositions of multiple customized concepts, including characters, objects and scenes.
[39] Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents. arXiv preprint arXiv:2201.00308, 2022. 4
[40] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022. 4
[41] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022. 4
[42] Chitwan Sahara, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022. 4
[43] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. 4
[44] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021. 4
[45] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022. 4
[46] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. edifi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022. 4
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763. PMLR, 2021. 4, 10
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 9
[49] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3):503-528, 1989. 9