Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Authors: Yuchao Gu
Xintao Wang
Jay Zhangjie Wu
Yujun Shi
Yunpeng Chen
Zihan Fan
WUYOU XIAO
Rui Zhao
Shuning Chang
Weijia Wu
Yixiao Ge
Ying Shan
Mike Zheng Shou
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-to-Image Diffusion Models
Concept Customization

Abstract: Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community. These models can be easily customized for new concepts using low-rank adaptations (LoRAs). However,  the utilization of multiple-concept LoRAs to jointly support multiple customized concepts presents a challenge. We refer to this scenario as decentralized multi-concept customization, which involves single-client concept tuning and center-node concept fusion. In this paper, we propose a new framework called Mix-of-Show that addresses the challenges of decentralized multi-concept customization, including concept conflicts resulting from existing single-client LoRA tuning and identity loss during model fusion. Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client tuning and gradient fusion for the center node to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion. Additionally, we introduce regionally controllable sampling, which extends spatially controllable sampling (e.g., ControlNet and T2I-Adapter) to address attribute binding and missing object problems in multi-concept sampling. Extensive experiments demonstrate that Mix-of-Show is capable of composing multiple customized concepts with high fidelity, including characters, objects, and scenes.

Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models
Yuchao Gu', Xintao Wang Jay Zhangjie Wu', Yujun Shi?, Yunpeng Chen', Zihan Fan", Wuyou Xiao", Rui Zhao', Shuning Chang', Weijia Wu', Yixiao Ge Ying Shan Mike Zheng Shou
Show Lab, National University of Singapore
3 ARC Lab, Tencent PCG
https://showlab wwthub.i..hiiiiie...o.
Abstract
Public large-scale text-to-image diffusion models, such as Stable Diffusion, have gained significant attention from the community. These models can easily customized for new concepts using low-rank adaptations (LoRAs). However, the utilization of multiple concept LoRAs to jointly support multiple customized concepts presents challenge. We refer this scenario decentralized multi- concept customization, which involves single-client concept tuning and center-node framework called Mix-of-Show that
concept fusion. In this paper, propose addresses the challenges of decentralized multi-concept customization, including concept conflicts resulting from existing single-client LoRA tuning and identity loss during model fusion. Mix-of-Show adopts an setbedding-decomposss LoRA (ED- LoRA) for single-client tuning and gradient fusion for the center node to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion. Additionally, we introduce regionally controllable sampling, which extends spatially controllable sampling (e.g., ControlNet and T2I-Adapter) to address attribute binding and missing object problems in multi-concept sampling. Extensive experiments demonstrate that Mix-of-Show is capable of composing multiple customized concepts with high fidelity, including characters, objects, and scenes.
Introduction
Open-source text-to-image diffusion models, such as Stable Diffusion [1], empower com- munity users to create customized models by collecting personalized concept images and fine-tuning them with low-rank adaptation (LoRA) [2, 3]. These tailored LoRA models achieve unparalleled quality for specific con- cepts through meticulous data selection, pre- processing, and hyperparameter tuning. While existing concept LoRAs serve as plug-and-play plugins for pretrained models, there are still chal- lenges in utilizing multiple concept LoRAs to extend the pretrained model and enable joint composition of those concepts. We refer this
Chiem-
Cherno
Cliens
Chemed
Chimi-
Shared
Gradient Fuslon Update
Pretrain Models v2
Sample
Figure 1: Illustraioon of decentralized multi- concept customization via fix-of-Show
"Corresooning Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Customized Characters
Customized
V potter
V hermione
V Vthanoe
batmann
V
ironman
Scenes V /igramid
Customized Objects
V/ ddeg8
V cat
dogA
V Cable
V chair
V ware
Vrock
Mix-of-Show
Meter harmione and V Chanae, castle
V Fhatman V Chanse and V /irmnss mountain.
(a) Multi-Characters
drgst Van and V Aajs the grass,
Two Volair Viace and Viabie, living
under the sunset.
(b) Multi-Objectss
pettr sit on Vihain, V dight V yoat and Cabble living room.
VV JOHNAAN
sit Vohnir J, CV bacmen sit V shair V Vasse and Vessle in living room.
(c) Perracters-Object
V potter V harnione V dogh and Vent near Vrack
V Drenande sit Vhhaie, CV batmon sit V thoo Vince and Viabls near Vrgranid
(d) Characters-Objects-SSeee
Figure 2: How to generate Harry Potter and Thanos, these two (or even more) concepts from different shows, in the same image? Our Mix-of-Show enables complex compositions of multiple customized concepts (e.g., characters, objects, scenes) with individually trained concept LoRAs.

---

scenario as decentralized multi-concept customization. As shown in Fig. 1, it involves two steps: single-client concept tuning and center-node concept fusion. Each client retains their private concept data while sharing the tuned LoRA models. The center node leverages these concept LoRAs to update the pretrained model, enabling joint sampling of these customized concepts. Decentralized multi-concept customization facilitates maximum community engagement in producing high-quality concept LoRAs and offers flexibility in reusing and combining different concept LoRAs.
However, the existing LoRA tuning and weight fusion techniques [3] fail to address the challenges of decentralized multi-concept customization. We have identified two main challenges: concept conflict and identity loss. Concept conflict arises because current LoRA tuning methods do not differentiate between the roles of embeddings and LoRA weights. Our research reveals that embeddings effectively capture concepts within the pretrained models' domain, while LoRA weight assist in capturing out- of-domain information (e.g., styles or fine details cannot be directly modeled by the pretrained model). However, existing LoRA tuning methods place excessive emphasis on LoRA weights while overlooking the importance of embeddings. Consequently, the LoRA weights encode a significant portion of the identity of given concept, resulting in semantically similar embeddings being projected onto concepts with different appearances. This, in turn, leads to conflicts during model fusion. Furthermore, existing weight fusion strategies compromise each concept's identity and introduce interference from other concepts by performing a weighted average of all concept LoRAs.
To overcome the challenges of decentralized LoRA (ED-LoRA) for single-client tuning and gradient
customization, we propose Mix-of- -howh
which involves boneeddinn-deoompoon fusion for center-node fusion. In single-client tuning, ED-LLRA is designed to address concept conflicts by preserving more in-domain essence within the embedding. To achieve this, we enhance the expressive ability of the concept embedding by decomposing it into layer-wise embeddings [4] and multi-word representations. At the central node, gradient fusion leverages multiple concept LoRAs to update the pretrained model. Since the diffusion model includes both forward and reverse diffusion processes, we can obtain the input/output features of each layer through sampling, even in the absence of data. Features from multiple concept LoRAs are combined to generate the fused gradient, which is used for layer-wise updating. Compared to weight fusion [3], gradient fusion aligns the inference behavior of each individual concept. significantly reducing identity loss.
To demonstrate the capabilities of Mix-of-Show we introduce regionally controllable sampling for multi-concept generation. Direct multi-concept generation often encounters issues such as missing objects and attribute binding [5, 6]. Recently, spatially controllable sampling (e.g., ControlNet [7], T21-Adapter [8] have been introduced to guide diffusion models using spatial hints (e.g., keypose or sketch), which resolve the problem of missing objects but still faces challenges of attribute binding in multi-concept generation. Considering that spatial layout is pre-defined when adopting spatial conditions, we propose injecting region prompts through regional-aware cross-attention. Powered by Mix-of-Show and regionally controllable sampling, we can achieve complex compositions of multiple customized concepts, including characters, objects, and scenes, as illustrated in Fig. 2. In summary, contributions are follows: 1) We analyze the challenges of decentralized multi- concept customization. 2) We propose the Mix-of-Show framework, consisting of an embedding- decomposed LoRA (ED-LoRA) and gradient fusion, address the concept conffict and identity loss in decentralized multi-concept customization. 3) We introduce regionally controllable sampling to demonstrate the potential of Mix-of-Show in composing multiple customized concepts.
2
Related Work
2.1 Concept Customization
Concept customization aims to extend pretrained diffusion models to support personalized concepts
using only a few images. There are two main types of concept tuning methods: embedding tuning
(e.g., Textual Inversion [9] and P+ [4]) and joint embedding- weight tuning (e.g., Dreambooth [10]
and Custom Diffusson [11]). Additionally, the community [3] adopts low-rank adapter (LoRA) [2] for concept tuning, which is lightweight and can achieve comparable fidelity to full weight tuning.
Although significant progress has been made in single-concept customization, multi-concept cus- tomization remains challenge. Custom Diffusion [11] proposes co-training of multiple concepts or
constrained optimization of several existing concept models. Following this, SVDiff [12] introduces data augmentation to prevent concept mixing in co-training multi-concepts, and Cones [13] discovers
3

---

concept neurons that can be added to support multiple concepts. However, their methods are typi- cally restricted to fuse 2-3 semantically distinct concepts. contrast, Mix-of-Shov can combine theoretically limitless customized concepts, including those within the same semantic category.
explored in studies by Instantbooth [14], ELITE [15], and Jia et al. [16]. focuses on achieving fast test-time customization. These meth- ods involve pretraining an encoder on a large-scale dataset specific the desired category. During inference, when provided with a few representative concept images from the trained category, the encoder extracts features that complement the pretrained diffusion models and support customized generation. However, these methods require training separate encoder for each category, typically limited common categories (e.g. person or cats). This limitation hinders their ability customize and compose more diverse and open-world subjects.
Another research line in concept customization,
2.2 Decentralized Learning
Decentralized or federated learning aims to train models collaboratively across different clients without sharing data. The de facto algorithm for federated learning, FedAvg, was proposed by [17]. This method simply averages the weights of each client's model obtain the final model. However, we find that directly applying this simple weight averaging is not ideal for fusing LoRAs of different concepts. To improve over FedAvg, previous works have either focused on local client training [18, 19, 20, 21, 22, 23, 24] or global server aggregation [25, 26, 27, 28, 29, 30]. Motivated by this, we explore the optimal design of single-client tuning and center-nod node fusion for decentralized multi-concept customization.
2.3 Controllable Multi-Concepp Generation
Direct multi-concept generation using text prompts alone faces challenges such as missing objects and attribute binding [6, 31, 32, 33, 34]. Previous approaches, like Attend-and-Excite [5] and Structure Diffusion [6], have attempted to address these issues, but the problem still persist, limiting the
effectiveness of multi-concept generation. Recent works, such as ControlNet [7] and T2I-Adapter [8], introduce spatial control (e.g., keypose and sketch) and enable more accurate compositions, resolving the problem of missing objects in multi-concept generation. However, attribute binding remains challenge. In our work, we tackle this challenge through regionally controllable sampling.
Methods
In this section, we provide a brief background on text-to-image diffusion models and concept customization in Sec. 3.1. We then introduce the task formulation of decentralized multi-concept customization in Sec. 3.2, followed by a detailed description of our method in Sec. 3.3 and Sec. 3.4.
3.1 Preliminary
Text-to-Image Diffuusion Models. Diffusion models [35, 36, 37, 38, 39, 40, 41] belong class of generative models that gradually introduce noise into an image during the forward diffusion process and learn to reverse this process synthesize images. When combined with pretrained text embeddings, text-to-image diffusion models [1, 42, 43, 44, 45, 46] are capable of generating high-fidelity images based on text prompts. this paper, we conduct experiments using Stable Diffusion [1]. which is a variant of the eoxt-to-imag diffusion model operating in the latent space. Given a condition
(P"), where p is the text prompt and &i the pretrained CLIP text encoder [47], the training objective for stable diffusion is to minimize the denoising objective by
== Es..,..
eø( 2t. f, (])],
(I)
where 2, is the latent feature at timestep and 6g is the denoising unet with learnable parameter f.
Embedding Tuning for Concept Customization. Textual Inversion [9] represents the input concept using a unique token V. When provided with a few images of the target concept, the embedding of V is tuned using Eq. 1. After tuning, the embedding for V encodes the essence of the target concept and functions like any other text in the pretrained model. To achieve greater disentanglement and control, P+ [4] introduces layer-wise embeddings for concept tokens, denoted V in this paper.

---

Single-Conces
Multi-Concept
Emb (P
Emb PP
Emb (P()"
Emb&LORA (PP
Emb (p()") +c+dd)(()))
EmbBLoRA
In In-Domain
Weight Weight FusionGradient Fusion
Out- nit-Domain
Concept
(a) (b) P+
(c) LoRA
Id) ED-LORA
(c) LoRA
61] ED-LORA
Figure 3: Single- and multi-concept customization between the embedding tuning (i.e., Textual Inversion (TI) [9] and P+ [4]), and joint embedding-wertd tuning (i.e., LoRA [3] and our ED-LoRA). p = "Photo of a V, near the beach". Po and A denotes the pretrained model and LoRA weight.
Low-Rank Adaptation. Low-rank adaptation (LoRA) [2] was initially proposed to adapt large- language models to downstream tasks. It operates under the assumption that weight changes during adaptation have low "intrinsic rank" and introduces low-rank factorization of the weight change to obtain the updated weight W, which is given by W = Wo + AW Wo + BA. Here, Wo E Raxk represents the original weight in the pretrained model, and B E Rddr and E Rrxk represent the low-rank factors, with < min(d, k). Recently, the community [3] has adopted LoRA for fine-tuning diffusion models, leading to promising results. LoRA is typically used as plug-and-play plugin in pretrained models, but the community also employs weight fusion techniques to combine multiple LoRAs:
W = Wo + W,W,,
s.t.
W
1,
(2)
where uo denotes the normalized importance of different LoRAs
3.2
Task Formulation: Decentralized Multi-Concept Customization
While custom diffusion [11] has attempted to merge two tuned concepts models into pretrained model, their findings suggest that co-training with multiple concepts yields better results. However, considering scalability and reusability, focus merging single-concept models to support multi-concept customization. We refer to this setting as decentralized multi-concept customization.
Formally, decentralized multi-concept customization involves a two-step process: single-clien concept tuning and center-node concept fusion. As shown in Fig. 1, each of the n clients possesses its own private concept data and tunes the concept model AW, Here, A AW; represents the changes in network weights, which specifically refers to LoRA weights in our work. We omit discussing the merging of text embeddings, the tuned embeddings can be seamlessly integrated into the pretrained model without conflicts.
After tuning, the center node gathers all LoRAs to obtain the updated pretrained weight W by:
W = f fW, AW1 AW,........ ,, where / represents the update rule that operates on the original pretrained model weight Wo and the concept LoRAs (AW,,i ......) n). One straightforward updating rule f is weight fusion, as illustrated in Eq. 2. Once updated, the new model should be capable of generating all the concepts introduced in the n LoRAs
3.3 Mix-of-Show
In this section, we introduce Mix-of-Show, containing ED-LORA (in Sec. 3.3.1) for single-client concept tuning, gradient fusion (in Sec. 3.3.2) for center-node concept fusion.
5

---

Prezan
Updatd
LoRA
Enteeling
Excodar
Dansang
|
VA. V3 walking newe lake
Plute of
Testref
Enceser
Layerittin
Prstriinn Model
Fusion Rule:
J0% AMG WX.1
(a) Single-Clientl ED-LoRA
(b) Center-Node: Gradient Fusion
(c) Mult-Cooncspt Generation
Figure 4: Pipeline of Mix-of-Show. In single-client concept tuning, the ED-LoRA adopts the layer- wise embedding and multi-word representation. center node, gradient fusion adopted to fuse multiple concept LoRAs and then support composing those customized concepts.
3.3.1 Single-Client Concept Tuning: ED-LoRA
Vanilla LoRA [3] is not suitable for decentralized multi-concept customization due to the issue of concept conflict. To better understand this limitation, we start by examining the distinct roles of embeddings and LoRA weights in concept tuning.
Single-Concepl Tuning Setting, We investigate embedding tuning (i.e., Textual Inversion [9] and P+ [4]) and the joint embedding-weit conduct experiments on both in-domain concept (i.e., directly sampled from the pretrained model), and out-domain concepts. The weights of the pretrained model, including the unet 0 and the text encoder v, are denoted 4 {0o, 3. Given text prompt P containing the concept V, visualize the tuned embedding of concept V using the pretrained weights Po and visualize the tuned embedding along with the LoRA weight using (Øo + 4 (P*).
tuning (i.e., LoRA [3]) on single concept customization, We
Analysis. Based on the experiment results in Fig. 3, we draw the following two observations regarding existing embedding tuning and joint ambedding-weighhe tuning approaches.
Observation 1: The embeddings are capable of capturing concepts within the domain pretrained models, while the LoRA helps capture out- out- domain information.
In Fig. 3(a, b), observe that embedding tuning approaches such as Textual Inversion and P+ struggle to capture out-domain concepts. This is because they attempt to encode all out-domain details (e.g., anime styles or details not modeled by the pretrained model &b) within the embedding, resulting in semantic collapse. However, for in-domain concepts sampled from the model, embedding tuning accurately encodes the concept identity within the embedding, benefiting from the accurate modeling of concept details by the pretrained model weights do. Furthermore, when jointly tuning the embedding with LoRA, the embedding no longer produces oversaturated outputs. This is because the out-domain information is captured by the pretrained model with LoRA weight shift (i.e. to + Ab).
Observation 2: Existing LoRA weights encode most the concept identity and project semantically similar embeddings to visually distinct concepts, leading to confficts during concept fusion.
In the joint embedding- LoRA tuning results shown in Fig. 3(c), we observe that directly visualizing the embedding with the pretrained model (P*) yields semantically similar results. However, when the LoRA weights are loaded (Øo + suggests that the majority the concept identity encoded within the LoRA weights rather than the embedding itself. However, when attempting to support multiple semantically similar concepts within a single model, it becomes problematic determine which concept to sample based on similar embeddings, resulting in concept conflicts. As shown in Fig. 3(e), when fused into model, the identity of each individual concept is lost.
the target concept can be accurately captured. This
Our Solution: ED-LORA. Based on the aforementioned observations, our ED-LoRA is designed to preserve more in-domain essence within the embedding while capturing the remaining details using LoRA weights. To achieve this, we enhance the expressiveness of the embedding through decomposed embedding. As illustrated in Fig. 4, we adopt layer-wise embedding similar [4] and create multi-world representation for the concept token (V V+ V (. Here, Vit.nd is randomly initialized to capture the variance of different concepts, while V Vlaae is initialized based on

---

Vt, in rod drese and V2, while rail, and wearing hat, are walling naur
Bakd Poceon Regor
tuo (P start werr antel 50 Dit stiek Bot
V1
Paggen Awers
[a] Spatbally Controllable Sampling
Region Awarr Attendion
(b) Regionally Controllable Sampling
(c) Workflow of Regionally Controllable Sampling
Figure 5: Regionally controllable sampling for multi-concept generation.
its semantic class to maintain semantic meaning. Both tokens are learnable during concept tuning. As shown in Fig. 3()), the learned embedding of ED-LoRA effectively preserves the essence of the given concept within the domain of the pretrained model, while LoRA helps capture the other details.
3.3.2 Center-Node Concept Fusion: Gradient Fusion
At the center node, we have access to all the concept LoRAs and can use these models update the pretrained model, enabling multi-concepp customization. However, the existing weight fusion strategy described in Eq. 2 insufficient to achieve this goal, as we will discuss further below.
Multi-Concept
Fusion Setting. In this experiment, we apply the weight fusion described in Eq. 2 to weighted average concept LoRAs ED-LoRAS (A i 1...nn into the pretrained model to. resulting in a new model Φb We then use the new model & (P,") to sample each concept and compare its identity with the corresponding single-concept sample (#o + (P;).
.n)
Analysis. Based on the results Fig. 3, we make the following observation about fusion strategy.
Observation 3: Weight fusion leads to identity loss of individual concepts in concept fusion.
As shown in Fig. (multi-concept), to significant loss of concept identity due to conflicts between concepts. Even when combined with ED-LORA, weight fusion still compromises the identity of each individual concept. In theory, a concept achieves its complete identity through LoRA weight shift A P*), fusing it with n-1 other concept LoRAs requires reducing its weight _ (P*) and introducing other concept LoRA weights, which ultimately diminishes the concept's identity.
observe that weight fusion in the case of LoRA leads
Gradient Fusion. Based the previous analysis, our objective is preserve the identity of each concept in the fused model by aligning their single-concept inference behavior. Unlike in federated learning [17, 19], where models typically single-direction classification models that cannot access gradients without data, text-to-image diffusion models have the inherent capability to decode concepts from text prompts. Leveraging this characteristic,
Our Solution:
first decode the individual
concepts using their respective LoRA weights, as depicted in Fig. 4(b). We then extract the input and output features associated with each LoRA layer. These input/output features from different concepts are concatenated, and fused gradients are used to update each layer W using the following objective: W arg minw Σi Il(Wo + AW,) W 1 where X; represents the input activation of the i-th concept, and | denotes the Frobenius norm. By adopting this approach, we can fuse different concept LoRAs without accessing the data and without considering their differences during training.
The results of our gradient fusion are shown in Fig. 3(f). demonstrating improved preservation of each concept's identity and consistent stylization across different concepts.
3.4 Regionally Controllable Sampling
Direct multi-concept sampling often encounters challenges missing objects and attribute binding [6. 31, 32, 33, 34]. While spatially controllable sampling methods (e.g., ControlNet [7] and T21- Adapter [8]) can address the issue of missing objects in multi-concept generation, they cannot accurately bind concepts specific keyposes or sketches. Merely indicating the desired concept
7

---

Reference Concepts
ED-LoRA (Ours)
Diffusion
V Veemione wearing headphone.
A Ven in the swimminng pool.
(a) Single-Concept Results
(b) Multi-Concept Results
Reference Concepts
Spatial Condition
$
Mix-of-Show (Ours)
LoRA
Custom Diffusion
Figure 6: Qualitative comparison on single- and multi-concept customization.
Methoes
Reil L'hpst Sirgle-Poood
Real Claracees Sinlo-Pooo
Red Steres Sirgle-Pased
Moon Change
Upper Bourd
00811
0.514
vllggment
Cammn Drinaize [11]
[[]
0371 10.773 63
06685 >0,586 (-)
0:340-m0 (dulte)
0.750 10.759 63
722-2001122 (4.00)))
0,239 30.730 L3
11716-66
Innge-aliguneo
Dithedoe
0.547 64J3))
0714-0055 (4.80m
7977 -0-711 (48BM0
LoRA
0.864-60 778 (4.186)
B 761 -ed 419)44
0.534-00 w0.0..44t (4ua5))
n 1156--007700 (4.111)
Mix Show (Ches)
0.86--d 546 (4.12)
0150 (483))
0.55--0 (4.2)))
0.44--8.115 (4.825)
Table 1: Text-alignment and image-alignnen
vary between the single-client tuned model and the center-node fused model. The upper bound text-alignn alignmettnnnt and the lower bound of image- aageealignment computed by replacing the concept's token (e.g., VdogA) with its class token (e.g., dog) and sampling using the pretrained model.
and attribute through text prompt can lead to attribute binding problems, in Fig. 5(a), where the identities three people are mixed, and the "red dress" is incorrectly assigned to other concepts.
To address these challenges, we propose a method called regionally controllable sampling. This approach utilizes both a global prompt and multiple regional prompts to describe an image based on spatial conditions. The global prompt provides the overall context, while the regional prompts specify subjects within specific regions, including their attributes and contextual information from the global prompt (e.g.. "near lake"). To achieve this, we introduce region-aware cross-attention. Given a global prompt pt and regional prompts P", we first incorporate the global prompt via
cross-attention with the latent z by h = softmax ((=))((P)) latent feature by 2 specified by P We obtain regional features using h; softmax (Q(Z)K(P)) V(P.,) Finally,
Vd VV()). Then, we extract the regional M,, where M; represents the binary mask associated with the region
Vd
replace the features in the global output with the regional features: h[M,] h,. As shown in Fig. 5(b), regionally controllable sampling allows for precise assignment of subjects and attributes, while maintaining harmonious global context.
4 Experiments
4.1 Datasets and Implementation Details
To conduct evaluation for Mix-of-Show, we collect dataset containing characters, objects, and scenes. For ED-LoRA tuning, incorporate LoRA layer into the linear layer in all attention

---

Embedding
LaRA Migt alM("))
Embedding Pe(P")
ED-LoRA Oue 50(P)
Before Multi-Concept Fusion
Wh(P")
ED-LORA
After Muub-CConeee Fusion
(e) Weight Fusion
Reference
Yeinss
Vinnt Veluss
[b] Gradient Fusion [Ours]
Concepts
(a) Embedding Expressiveness
w/o Region Control w/ Region Control (Ours)
w/ Weight Fusion
w/ Gradient Fusion (Ours)
(c) Regionally Controllable Sampling
(b) Ablation of Fusion Type
Figure 7: Qualitative ablation study of Mix-of Mfx-of-Show. P means the text prompt. Py and A& denotes the pretrained model and LoRA weight, respectively.
Real Objects Single- Single-Feused
Real-Charscters Single-Fused
Real-Strnes Single- Sangle-Pused
Methods
Mean Change
Lower Bound
0.721
0.471
0,905
0.596
Image- moage-alignment
LoRA + Weight Fusinn
0.864-00.77 (-8.086)
0.761 -00 10.1--0.555---- -5..555-00.000
0.825-0.776 (-0.655)
0.816-00771 (-L1115)
ED-LoRA Weight Fusion
1. (-0.070)
0.802--0.634 (-8.665)
1.55--88116 (-0.442)
0.843-90.744 (-0.194)
FDJLaRA + Graiient Fosions
0,868-0 846 (-0,022)
0802-00.77 770 (-88332)
0.858-00886 (-0,620)
0,843 0..818 (-U0125)
(a) Subject identity preservation (i.e,, image-alignment)) measured by CLIP A +gradient fusion. Our ED-LoR A+gradiinn fusion achieves after multi-concept fusion, preserving the best subject identity.
hetween Aweght
fusion, ED-LoRA weiigtt fusion, and ED- the least loss in
Image Alignment
Text
Human Evaluation
Alignment
ED-LoRA + Weight Fusion
33.5%
47.5% 52.5 %
ED-LoRA +
Grudient Fusion
66.5%
(b) Human preference study interface on Ama- zon Mechanical Turk.
(c) Human preference study between weight fusion and gradient fusion for fusing ED-LoRAs.
Table 2: Quantitative ablation study of our main components. (a) Ablation study between the weight fusion, ED-LoRA+weighh fusion and our EDD-LRRAggadde fusion. (b, c) Human preference study to comparing weight fusion and gradient fusion for fusing our ED-LORAS.
modules of the text encoder and Unet, with a rank of r = 4 in all experiments. We use the Adam [48] optimizer with learning rate of 1e-3, le-5 and le-4 for tuning text embedding, text encoder and Unet, respectively. For gradient fusion, we use the LBFGS optimizer [49] with 500 and 50 steps to optimize the text encoder and Unet, respectively. More details are provided in the supplementary.
4.2 Qualitative Comparison
Single-Concept Results. We compare ED-LoRA with LoRA [3], Custom Diffusion [11] and P+ [4] for single-concept customization. The results are shown in Fig. 6 (a). Our ED-LoRA achieves comparable performance to previous methods customizing objects, while maintaining better identity for character customization. More comparisons are provided in the supplementary.
Multi-Concept Results. We compare Mix-of-Show with LoRA [3], Custom Diffusson [11], and P+ [4] for decentralized multi-concept customization. For LoRA, we utilize weight fusion to combine the differrnt concepts. In the case of P+, directly incorporate the tuned concept embedding into the pretrained model. And for Custom Diffusion, we follow their approach of constrained optimization to merge the key and value projections in cross-attention. To ensure fair evaluation, employ the same regionally controllable sampling for multi-concept generation across all models and the results summarized in Fig. 6 (b).
P+ [4] and Custom Diffusion [11] only tunes text-related module (i.e., text embedding, or the key and value projection of cross-attention). In contrast, LoRA and Mix-of- Showwwwwwww--oo-- add LoRA layers the

---

entire model. The limited scope of tuned modules in P+ and Custom Diffusion leads to an excessive encoding of out-domain low-level details within the embedding. This leads unnatural and less desirable outcomes when compared to LoRA and Mix-of- Show. In comparison to LoRA, which loses concept identity after weight fusion, Mix-of- Show effectively preserves the identity of each individual concept.
4.3 Quantitative Comparison
Following Custom Diffusion [11], utilize the CLIP [47] text/image encoder assess text alignment and image aligmment. We evaluate on different category of concepts on both single-concept tuned model and multi-coneept fused model. We include detailed evaluation setting in the supplementary.
Based on the results presented in Table. 1, both Mix-of-Show and LoRA exhibit superior image alignment compared other methods, all the while maintaining comparable text alignment in the single-client tuned model. This achievement stems from their fine-tuning the spatial-related layer in Unet (e.g., linear projection layer in self-attention), a critical aspect for accurately capturing the complex concepts' identity, such as characters.
However, the main difference between LoRA and Mix-of-Show emerges in the context of multi- concept fusion. In the center-node fused model, LoRA experiences a significant decline in image alignment for each concept, progressively deteriorating towards the lower bound. In contrast, Mix-of-Show method undergoes far less degradation in image alignment after multi-concept fusion.
4.4 Ablation Study
Embedding Expressiveness. In Fig. 7(a), it is evident that our decomposed embeddings better preserve the identity of the specified concept compared the standard text embeddings used in LoRA. This results in more robust encoding of concept identity. As shown in the quantitative results in Table. 2(a), when LoRA replaced with ED-LoRA, the identity loss from weight fusion (measured by mean change of image-alignment) is reduced from 0) 15 to 0.094. This result verifies that expressive embeddings help reduce identity loss during multi-concept fusion.
Fusion Type. Built with the same ED-LoRAs, we conduct experiments to compare weight fusion and gradient fusion. As shown in Fig. 7(b), gradient fusion effectively preserves concept identity after concept fusion, resulting in superior results for multi-concept sampling. According to the quantitative results in Table. 2(a). gradient fusion significantly reduces the identity loss of weight fusion, decreasing it from 0.094 to 0.025. We also conduct a human evaluation and confirm clear preference for gradient fusion, as indicated in Table. 2(c).
Regionally Controllable Sampling, As shown in Fig. 7(c), direct sampling lead to attribute binding issues, where the concept identities mixed. However, regionally controllable sampling overcomes this problem and achieves correct attribute binding in multi-concept generation.
Conclusion
In this work, we explore decentralized multi-concept customization and highlight the limitations of existing methods like LoRA tuning and weight fusion, which suffer from concept conflicts and identity loss in this scenario. To overcome these challenges, framework that combines ED-LoRA for single-client concept tuning and gradient fusion for center- node concept fusion. ED-LoRA preserves individual concept essence in the embedding, avoiding conflicts, while gradient fusion minimizes identity loss during concept fusion. We also introduce regionally controllable sampling to handle attribute binding in multi-concept generation. Experiments demonstrate Mix-of-Show can successfully generate complex compositions of multiple customized concepts, including characters, objects and scenes.
propose Mix-of-Show,
Acknowledgements
This project is supported by the National Research Foundation, Singapore under its NRFF Award NRF-NRFF11-202) 3-2021-0000. and the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2022).
10

---

References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR. pages 10684-10695, 2022.
1,4
[2] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900
2021. 1,
[3] Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning. https://githh:: com/cla com/cloneofen http://ooofssooo//a///o
1, 3,5, 6, 8,
[4] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.09522. 2023. 4, 6,
[5] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and- excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv preprint WiXv:2301.13826 2023. 3,
[6] Weixi Feng, Xuchai Hc, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free frreenngg-eeeeeeee structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032. 2022. 3, 4,
[7] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, arXiv preprint arXiv:2302.05543, 2023. 7,
[8] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprinl arXiv:2302.08453, 2023. 3.
[9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0166
2022. 3, 4, 5,
[10] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning to-ima diffusion models for subject-driven generation, arXiv preprint arXiv:2208. / 12242, 2022. 3
[11] Nupur Kumari. Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of text-to-image diffusion. arXiv preprint XXXx:2212.00488, 2022.
8, 9, 10
[12] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. arXiv preprint arXiv:2303.11300 2023.
[13] Zhiheng Liu, Ruili Feng, Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.05122 2023. 3
[14] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint wxXiv:2304.03411 2023. 4
[15] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint 232. 2023. 4
[16] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304..0 02642, 2023. 4
[17] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas Communication-efiice efficcient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017. 4,7
[18] Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10713-10722
2021.
11

---

[19] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 450, 2020. 7,7
[20] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020. 4
[21] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2 1.04663, 2021. 4
[22] Maruan Al-Shedivat, Jennifer Gillenwater, Eric Xing, and Afshin Rostamizadeh. Federated learning via posterior averaging: A new perspective and practical algorithms. arXiv preprint
arXiv:2010.05273. 2020. 4
[23] Lixu Wang. Shichao Xu, Xiao Wang, and Qi Zhu. Addressing class imbalance in federated learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
10165-10173, 2021. 4
[24] Yujun Shi, Jian Liang, Wenqing Zhang, Vincent YF Tan, and Song Bai. Towards understanding and mitigating dimensional collapse in heterogeneous federated learning. arXiv preprint
arXiv:2210.00226, 2022. 4
[25] Jianyu Wang. Qinghua Liu, Hao Liang. Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:761 1-7623, 2020. 4
[26] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of -dniiiettaall data distribution for federated visual classification. arXiv preprint arXiv:1909.0633 2019. 4
[27] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of hetero- geneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34, 2021. 4
[28] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint
2020. 4
[29] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems,
33:2351-2363, 2020. 4
[30] Sashank Reddi, Zachary Charles. Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.002: 00295, 2020. 4
[31] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022. 4, 7
[32] Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.1 /2/92, 2023. 4,7
[33] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao and Hongsheng Li. Better aligning text-to- image models with human preference. arXiv preprint arxiv:2303. /4420, 2023. 4,7
[34] Wan-Duo Kurt Ma, JP Lewis, W Bastiaan Kleijn, and Thomas Leung. Directed diffusion: Direct control of object placement through attention guidance. arXiv preprint MxXiv:3302.13555 2023.
4, 7
[35] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020. 4
[36] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 4
[37] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021. 4
[38] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint
arXiv:2207.12598. 2022. 4
12

---

[39] Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevae: Effi- cient, controllable and high-fidelity generation from low-dimensional latents, arXiv preprint Stt1:2201.003000 2022. 4
[40] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprini WwXiv:2206.00927. 2022. 4
[41] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint
arX1v:2211.01095, 2022.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
[43] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204. 06/25, 2022.
[44] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint ar 2021. 4
[45] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
[46] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint vrXiv:2211.01324 2022.
[47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages
PMLR, 2021. 10
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014. 9
[49] Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3)5003-528 1989. 9
13


Reviews:
[{"review_id": "sU9krJrKvn", "reviewer": "Reviewer_B3ui", "summary": "The paper proposes a framework called Mix-of-Show for decentralized multi-concept customization of diffusion models. The framework uses embedding-decomposed LoRA for single-client tuning and gradient fusion for center-node concept fusion to preserve the in-domain essence of single concepts and support theoretically limitless concept fusion. Additionally, it introduces regionally controllable sampling to extend spatially controllable sampling to address attribute binding and missing object problems in multi-concept sampling.", "questions": "Question:\n1. How do you initialize V+_{class} in Line 194? Can you provide a more detailed explanation?\n2. In Figure 2, what does @ mean?\n3. In Figure 2, what is - [x1, y1, x2, y2]? Are they the spatial coordinates of different concepts? \n4. Did you consider conducting human evaluation to make the results more convincing? \n\nSuggestions:\n1. The related work content of the paper is less. It would be beneficial to expand a more detailed explanation of the relevant content.\n2. A visual explanation of the overall architecture that includes the individual modules is useful, as it will help the readers to better understand the structure of the new framework.\n3. The experiment section needs to introduce more details about the dataset, such as data quantity, source, input. This would provide readers with a better understanding of experiment details.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a framework called Mix-of-Show for decentralized multi-concept customization of diffusion models. It addresses the challenges of concept conflicts and identity loss during model fusion, and demonstrates the capability of composing multiple customized concepts with high fidelity.\n2. The logical flow of the paper makes it easy for readers to understand and follow the research process.\n3. The paper provides detailed analyses of the experimental results.", "weaknesses": "1. In Figure2(b), in the generated picture with text \" V^{dogA}, a V^{cat} and a V^{dogB}, in a living room.\", is the text wrong here? Or is this picture exactly the output of the model? I'm a little confused because the background of the picture is suburb rather than living room.\n2. In line 230 to 231, you mentioned that Figure3(f) demonstrates that the gradient fusion preserves the concept identity, but I think the effect of this is not obvious. In other words, in the method that only uses ED-LoRA with weight fusion, the problem of identity loss basically does not exist. I can't see much difference between these two pictures. Maybe Thanos in Figure7(c) is a better example. It's a good way to find some special examples to show the effect, but I'd like to know if there is a quantitative way to accurately evaluate the effectiveness of this fusion approach.\n3. In section 4, you made a qualitative comparison. Have you thought about using a quantitative approach such as CLIP score to compare your model with others, or have you ever thought about developing an new evaluation metric which can accurately tests your own approach?\n4. When you use the method you proposed, have you encountered any poor-quality images? Have you done an error analysis? And have you analysed the limitations and shortcomings of these methods?\n5. Some equations lacks essential explanatory notes for some of the mathematical symbols used:\nEquation 1, the notation of $\\epsilon$ is missing\n6. Some figure misses detailed explanation which causes confusion (See question 2 and 3)\n7. Lack of details concerning the collected dataset and the text encoder used in the study.\n8. The dataset contains different characters, objects, and scenes, however, I only see the demonstration of characters. More examples about different objects and scenes can be provided to make the results more solid. \n9. There is a lack of quantitative analysis in this work. The effect of the method proposed in this paper is represented by the sample pictures, which is a qualitative analysis and can not well reflect the generalizability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing many detailed comments. We will address your concerns by referring to the relevant contents in the supplementary material and the attached PDF (in the global response).\n\n**Notations**: W: Weakness, Q: Question, S: Suggestion\n\n---\n\n**W.1**: Thank you for pointing that out. It was a typo. The correct text prompt should be \"a $V^{dogA}$, a $V^{cat}$, and a $V^{dogB}$, on the grass, under the sunset.\" We will correct it in the revised version.\n\n**W.2**: Thanks for raising this concern. We admit that it might be challenging for readers who are not familiar with those anime characters to see the difference. To facilitate a better comparison between weight fusion and gradient fusion for fusing our ED-LoRAs, we provide **more qualitative examples of real-world characters and objects** in Figure 1(a) of the attached PDF. Additionally, we conduct **two quantitative evaluations** to compare weight fusion and gradient fusion:\n\n- We evaluate the **CLIP image alignment** before and after multi-concept fusion to measure the subject identity loss, in the Table 1(a) of the attached PDF. From the result, we can find gradient fusion achieves less image-alignment loss compared to weight fusion for fusing ED-LoRAs.\n\n- We perform a **human preference study** on Amazon Mechanical Turk. We collected 400 questionnaires to assess human preference for these two fusion methods. The evaluation interface and results are summarized in Table 2(b, c) in the attached PDF. Based on the results, human raters believe that gradient fusion better preserves subject identity compared to weight fusion (`66.5%` vs. `33.5%`) when fusing ED-LoRAs.\n\n**W.3**: We have conducted a quantitative evaluation following Custom Diffusion [1], which involves utilizing the CLIP text model and CLIP image model to assess text alignment and image alignment. For detailed settings and results of our quantitative evaluation, please refer to Section 3.1.1 and Section 3.1.2 in the supplementary material.\n\n**W.4**: Limitations of gradient fusion and regionally controllable sampling have been discussed in Section 4.1 of the supplementary material. Additionally, the limitation of ED-LoRA lies in the inherent bottleneck of Stable-Diffusion, resulting in poor-quality full-body character generation, as shown in Figure 1(b) in the attached PDF. However, this limitation can be overcome by adjusting the aspect ratio to increase the size of the facial region.\n\n**W.5**: Thank you for pointing that out. We will thoroughly proofread the paper and ensure that the missing explanatory notes are included. In Equation 1, the symbol $\\epsilon$ represents the noised feature map at timestep $t$, which serves as the ground truth for the denoiser UNet prediction.\n\n**W.6**: Thank you for pointing that out. We will revise the related figure captions in this paper to add detailed instructions to guide the comparison of our results.\n\n**W.7**: The description of evaluation datasets and data quantity of each concept are included in Section 2.1 and Table 2 of the supplementary, and the representative image of each concept is shown in Figure 1 and Figure 2 in the main paper. Our experiment is based on the Stable Diffusion, which utilizes the pretrained CLIP text encoder.\n\n**W.8**: Thank you for the suggestions. Given that characters are **the most challenging** in concept customization due to their intricate facial details, most of our results are showcased using characters. We also include more examples for objects and scenes in the Figure 4 of the supplementary and Figure 1(c) of the attached PDF.\n\n**W.9**: Thanks for pointing that out. We include the quantitative analysis in Table 1(a) of the attached PDF, comparing LoRA+weight fusion, ED-LoRA+weight fusion, and ED-LoRA+gradient fusion. From the results, we clearly find:\n- With the weight fusion strategy, replacing vanilla LoRA with ED-LoRA reduces the image-alignment loss after multi-concept fusion.\n- ED-LoRA+gradient fusion achieves **the least image-alignment loss** after multi-concept fusion.\n\nAdditionally, we conducted a human preference study, as discussed in **W.2**, to verify the effectiveness of gradient fusion.\n\nThese quantitative analyses **strongly support our observations and methods**. We will add these quantitative analyses to the main paper in the revised version.\n\n---\n\n**Q.1**: The $V^{+}_{class}$ is initialized by the class token, representing the super-class of the given concept. For example, for the concept \"dogA,\" it is initialized by \"dog,\" and for \"Hermione,\" it is initialized by \"woman.\"\n\n**Q.2-Q.3**: Do you mean Figure 5 (not Figure 2)? In Figure 5, the \"@\" symbol represents matrix multiplication. The values [x1, y1, x2, y2] correspond to the spatial coordinates of each concept. In the revised version, we will add an illustration of these notations in the caption.\n\n**Q.4**: Sure, as discussed in **W.2**, we conduct a **human preference study** on Amazon Mechanical Turk to compare the gradient fusion and weight fusion for fusing our ED-LoRAs. From the results, human raters believe that gradient fusion better preserves subject identity compared to weight fusion (`66.5%` vs. `33.5%`).\n\n---\n\n**S.1**: Thanks for the suggestion. Due to the page limit, we only discuss the tuning-based customization works, which most related to this work. In the revised version, we will expand the discussion of existing related works and also add the discussion of tuning-free customization works.\n\n**S.2**: Thanks for the suggestion. In the revised version, we will further expand the details of the framework shown in the main paper (Figure 4).\n\n**S.3**: Please refer to **W.7** and Section 2.1 in the supplementary material for our dataset and implementation details. We will move some important contents into the main paper in the revised version.\n\n---\n**Reference**:\n\n[1] Multi-concept customization of text-to-image diffusion. CVPR 2023."}, {"review_id": "VgWJecnTo4", "reviewer": "Reviewer_Khyz", "summary": "This paper addresses a problem of significant practical importance, which is how to employ a number of individually fine-tuned concept plug-ins to produce a model, which can then be used to generate an image that contains any combinations of the concepts. The key method is center node fusion, which trains a fused model instead of simple arithmetic averages over tuned Lora weights.  ", "questions": "This reviewer does not have questions to the authors.", "limitations": "This work is neutral in terms of potential negative societal impact. ", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 3, "strengths": "The paper tackles a problem of significant practical import, and it appears that the proposed method could work well. The center node fusion method is intuitive and can be implemented as is or tweaked to suit particular situations. ", "weaknesses": "There are many. In no particular order,\n- the term \"decentralized\" is quite misleading. What the paper meant was that the inverted embedding and weights for concept n must be trained by only knowing the data of concept n. That is not decentralized.\n- observations from empirical observations do not add to the paper's technical contribution; in this reviewer's opinion, they are not convincing.\n- fine details can influence the outcome a lot, for instance the split between the fine-tuning between embedding inversion (V is 1 token? 10 tokens?) and LoRA (dimensions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging the contribution of our work. We address the concerns as follows:\n\n**Notations**: W: Weakness\n\n---\n\n**W.1**: Sorry. We might not fully grasp the meaning behind this statement, \"What the paper meant was that the inverted embedding and weights for concept n must be trained by only knowing the data of concept n. That is not decentralized.\" **Are you suggesting that  decentralized learning is related to obtain individual concept models (embedding+weight) without utilizing data?**\n\nIf it is the case, your understanding of the definition of decentralized learning could be incorrect. Decentralized learning does not imply that we cannot train the model of each client using data. Instead, it means the data from each client **cannot be shared** to obtain the fused model. For example, in the experiment setting in FedAVG [1], they split the CIFAR dataset into 100 clients, where each client hosts 600 samples. The data split of different clients will not be shared, but each client can train the model using their own data split.\n\nIn the context of our multi-concept customization, we can use data to tune the ED-LoRA for each concept individually. The decentralization in our setting means that to obtain a multi-concept fused model, we do not need to access the source data of all concepts (1 to N). Instead, we can directly use gradient fusion to combine the ED-LoRA models of concepts (1 to N). This precisely captures the essence of decentralized learning.\n\n**If we have misunderstood your meaning in Weakness.1, please correct me during the discussion phase**.\n\n**W.2**: We respectfully disagree with this point. In academic research, empirical observations play a crucial role in motivating the development of reasonable methods and facilitating future research. For example, our Observation 1 about the role of embedding and weight in concept tuning motivated us to design ED-LoRA, and Observation 2 about identity loss in weight fusion motivated us to design gradient fusion. Without empirical observations, we could not have arrived at our final solution.\n\nHowever, to make our observations more convincing, we conducted **quantitative evaluations** to compare LoRA+weight fusion, ED-LoRA+weight fusion, and ED-LoRA+gradient fusion, as shown in Table 1(b, c) in the attached PDF. From the results, we can find:\n\n- With the weight fusion strategy, replacing vanilla LoRA with ED-LoRA improves image-alignment after multi-concept fusion.\n- ED-LoRA+gradient fusion achieves the best image-alignment after multi-concept fusion.\n\nThese results also **strongly support** our observations and methods.\n\n**W.3**: The contribution of this work in the single-concept tuning part is to **unveil the distinct roles between embedding and weight, and their impact on concept fusion**. The conclusion is that we should transfer the in-domain essence of a given concept to the embedding and utilize the LoRA weight to capture out-domain information to avoid concept conflict, rather than relying solely on LoRA weight to encode all identity.\n\nThe proposed ED-LoRA serves as a demonstration of a simple yet robust approach to achieve that goal. Using more tokens (e.g., 10) is probably not reasonable as it disrupts the grammar structure (adjective + noun) used to represent a concept, making it problematic to interact with other text prompts. The LoRA rank is indeed important, but we find that keeping a default low rank (r=4) is sufficient. A higher rank conflicts with our goal to shift in-domain essence to embedding. Consequently, the proposed ED-LoRA does not require extensive hyperparameter tuning for each concept, maintaining the same configuration for all experiments. Therefore, the performance does not come from changing any fine details.\n\n---\n**Reference**:\n\n[1] Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS 2017."}, {"review_id": "aSuKWpWLTl", "reviewer": "Reviewer_acdR", "summary": "This paper focuses on the problem of multi-concept customization of diffusion model. Compared to the previous methods, this work focuses on fusing multiple customized objects into a single image with different extra conditions such as poses contours. The paper did an extensive study to analyze the roles of tuning the embedding and weights and based on this study it proposes a group of improvement to the multi-concepts customization. Based on the qualification results, the proposed methods are quite effective. ", "questions": "N/A", "limitations": "The author has address the limitations. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Compared to previous methods, this paper gives a more flexible concept customization framework where multiple concepts and different types of conditions are supported. \n2. This paper provide some insights on how embedding and model weights affects the concept finetuning differently and propose effective  several modifications of the method accordingly. \n3. Extensive qualification comparison is provided which proves the effectiveness of the proposed method. \n4. The distributed training framework brough some insight on privacy preserving and federated learning related research. ", "weaknesses": "This paper only provide qualification results with limited use cases. It would be better to provide some quantification analysis on more data, following some previous concept customization works. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging our work. Regarding the weakness concerning quantitative experiments, we have included **extensive quantitative evaluation** in Section 3.1 of the supplementary material and added **quantitative analysis of our main components** (i.e., ED-LoRA, gradient fusion) in Table 1(a) in the attached PDF. Additionally, we conducted a **human preference study** on *Amazon Mechanical Turk* to further verify our gradient fusion, and the results are presented in Table 1(b, c) in the attached PDF. Here are the details about our quantitative experiments.\n\n1) **Quantitative Evaluation**: In the supplementary material (Section 3.1.1 and Section 3.1.2), we have included our quantitative evaluation setting and results. Our evaluation is **spread across different methods** (i.e., LoRA [1], Custom Diffusion [2], and P+ [3]),  **covering all concept data**, including characters, objects, and scenes, and **benchmarking both the single-concept tuned and multi-concept fused model**. Through this extensive quantitative evaluation, we arrive at our conclusion: Mix-of-Show achieves similar single-concept tuning performance to LoRA but largely preserves the customized concept identity (measured by image-alignment) after multi-concept fusion. \n\n2) **Quantitative Analysis of Main Components**: In the Table 1 of the attached PDF, we quantitative analyze the LoRA+weight fusion, ED-LoRA+weight fusion and ED-LoRA+gradient fusion across all concept data from different categories. From the results, we can find:\n\n   - With the weight fusion strategy, replacing vanilla LoRA with ED-LoRA improves image-alignment after multi-concept fusion.\n\n   - ED-LoRA+gradient fusion achieves the best image-alignment after multi-concept fusion.\n   \n   These quantitative results strongly support our observation and methods.\n\n3) **Human Preference Study**: In order to further verify the effectiveness of gradient fusion in multi-concept fusion, we conducted a human preference study on Amazon Mechanical Turk. We collected 400 questionnaires to assess human preference between gradient fusion and weight fusion for fusing our ED-LoRAs. The evaluation interface and results are summarized in Table 2(b, c) in the attached PDF. From the results, human raters significantly prefer gradient fusion over weight fusion (``66.5%`` vs. ``33.5%``) in fusing ED-LoRAs.\n\nBesides, we also provide more qualitative comparisons in Figure 3 and Figure 4 in the supplementary material, as well as Figure 1(c) in the attached PDF. Based on the above detailed qualitative and quantitative evaluations, we can see that Mix-of-Show **not only performs well in the limited use cases demonstrated in the main paper** but is also **generalizable across different data categories** (i.e., characters, objects, and scenes) and **different concept domains** (i.e., real-world concepts or anime concepts).\n\n---\n\n**Reference**:\n\n[1] https://github.com/kohya-ss/sd-scripts\n\n[2] Multi-Concept Customization of Text-to-Image Diffusion. CVPR 2023.\n\n[3] P+: Extended Textual Conditioning in Text-to-Image Generation. Arxiv 2023."}, {"review_id": "VPD3XrNjCT", "reviewer": "Reviewer_TCkG", "summary": "The paper presents Mix-of-Show, a framework that tackles the challenges of decentralized multi-concept customization in public large-scale text-to-image diffusion models. It introduces an embedding-decomposed LoRA (ED-LoRA) for single-client tuning and gradient fusion for the center node, enabling limitless concept fusion. The framework also incorporates regionally controllable sampling to address attribute binding and missing object issues. Experimental results demonstrate Mix-of-Show's ability to compose multiple customized concepts, including characters, objects, and scenes, with high fidelity. Overall, the framework offers a promising solution for decentralized multi-concept customization in text-to-image diffusion models.", "questions": "Please refer to the above weakness.", "limitations": "Authors should demonstrate some failure cases in order to get a full grasp on the limitation of the proposed framework.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strength of Mix-of-Show lies in its comprehensive approach to decentralized multi-concept customization. It addresses the challenges of concept conflicts, identity loss, and missing objects through innovative techniques such as ED-LoRA, gradient fusion, and regionally controllable sampling. The framework enables the composition of complex and diverse concepts with high fidelity. It demonstrates promising results in generating multi-concept compositions, surpassing previous methods in terms of performance and flexibility.", "weaknesses": "The paper does not extensively discuss the computational complexity and resource requirements associated with the proposed framework. \n\nWhile I'm not an expert in related area, the paper is generally well-written and easy to follow. At current stage, I have no further concerns regarding the technical details of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our work. Part of the discussion on time complexity and limitations is included in Section 2.2 and Section 4.1 **in the supplementary material**, respectively. Here, we provide a more detailed discussion.\n\n**Notations**: W: Weakness, L: Limitation\n\n---\n\n**W.1**: There are three parts in our methods, i.e., ED-LoRA, gradient fusion, and regionally-controllable sampling. We will separately discuss their time complexity and resource constraints.\n\n- ED-LoRA takes about 10 minutes for character concepts (about 15 images) and about 5 minutes for object concepts (about 5 images) on 2 A100 GPUs. With optimized techniques such as gradient checkpoint, xformer attention, and gradient accumulation provided by the open-source Diffusers repository [1], the training can fit into a single 8 GB GPU. The time complexity and memory cost are similar to LoRA [2]. It is worth noting that our methods do not require any regularization datasets like custom diffusion [3] and dreambooth [4], which helps reduce computation and data storage costs. For storing each concept, one ED-LoRA for each concept only takes 4.5 M storage, making it quite lightweight to share.\n\n- Gradient fusion requires a 12 GB memory GPU and about 24 and 50 minutes for composing 5 and 14 concepts on an A100 GPU, respectively. However, after further investigation, we found the bottleneck is in merging spatial layers (e.g., QKV in self-attention, Q in cross-attention) in Unet, where we should consider all sample timesteps. Our previous experiment was based on the DDIM scheduler, which required 50 timesteps. In our improved version, we switched to the more accurate DPM-Solver, which only takes 20 timesteps. Therefore, in this improved version, we only take 12 and 26 minutes to merge 5 and 14 concepts on an A100 GPU.\n\n- Regionally controllable sampling generates multiple concepts at once, and thus it requires nearly the same speed and memory cost to generate a 512 * 1024 image with T2I-Adapter, which takes around 4 seconds with a 10GB memory cost.\n\n---\n\n**L.1**: We will discuss the limitations for ED-LoRA, gradient fusion, and regionally-controllable sampling, respectively.\n\n- In single concept tuning with ED-LoRA, it is challenging to generate full-body characters with detailed face texture in 512 * 512 resolution, as shown in Figure 1(b) in the attached PDF. This limitation is inherent to Stable Diffusion since the pretrained VAE compresses the information of the dedicated facial details. To overcome this limitation, when generating a full-body character, we need to adjust the aspect ratio (from 512 * 512 to 1024 * 512) to enlarge its facial region, as shown in the example in Figure 1 (b) in the attached PDF.\n\n- The limitation of gradient fusion lies in taking a relatively longer time compared to simple weight fusion. Even with our improvement discussed above, it still requires 12 and 26 minutes to merge 5 and 14 concepts. Further analysis on its redundancy to improve the time complexity will be left as future work.\n\n- In regionally controllable sampling, the limitation lies in the generated contents with structured backgrounds. For example, in Figure 7 of the supplementary material, the tree (left) and the railing (right) in the background lose quality and consistency when adopting regionally controllable sampling. This is because regionally controllable sampling breaks the interaction between regions, which should be further investigated to improve.\n\n---\n**Reference**:\n\n[1] https://github.com/huggingface/diffusers\n\n[2] https://github.com/kohya-ss/sd-scripts\n\n[3] Multi-Concept Customization of Text-to-Image Diffusion. CVPR 2023.\n\n[4] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. CVPR 2023.\n"}, {"review_id": "mB7ZnNUICV", "reviewer": "Reviewer_NQJc", "summary": "This paper proposes Mix-of-Show for decentralized multi-concept customization using diffusion models. To tackle the concept conflict and identity loss problem, the authors propose the embedding-decomposed LoRA and the gradient fusion. The regionally controllable sampling is proposed to address the attribute binding and missing object problems. Experiments show that the proposed method outperforms existing methods in multi-concept customization.", "questions": "Please refer to the weakness part. It is better to emphasize the necessity of the decentralized design.", "limitations": "The authors discussed the limitations in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Most part of the paper is well-written and easy to understand.\n\n2. Each component of the proposed method is reasonable and well-motivated. \n\n3. The three observations are quite important and interesting to finetuning pretrained diffusion models. \n\n4. Experiments show that the proposed Mix-of-Show outperforms existing methods in multi-concept customization.", "weaknesses": "1. The gradient fusion is also adopted in CustomFusion, it is better to clarify it in the paper.\n\n2. The biggest concern is the necessity of the decentralized design, since in current customization scenario, we can always obtain all the subjects together and we can fine-tune them all, and also the CustomFusion indicates this leads to better results. If the decentralization is not necessary, the contribution of this paper will become smaller.\n\n3. Some minor typos, e.g., the caption for Figure 2(b) seems to be incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for acknowledging our work. The concerns are addressed as follows:\n\n**Notation**: W: Weakness\n\n---\n\n**W.1**: Thanks for raising this concern. Both the Custom Diffusion [1] and Mix-of-Show formulate multi-concept fusion as an optimization problem, and thus both methods are indeed related to the gradient. However, the difference lies in how they handle the optimization problem. Custom Diffusion formulates it as a **constrained optimization** problem, where the gradient needs to be considered from both the new concepts (i.e., optimization target) and old concepts (i.e., constraints). On the other hand, Mix-of-Show formulates it as a **non-constrained optimization** problem, which only requires fusing the gradient from new concepts.\n\nOur non-constrained optimization formulation **eases the difficulties in handling constraints** compared to Custom Diffusion. Custom Diffusion derived a closed-form solution for the constrained optimization using Lagrange multipliers, but it only supports merging key and value projection layers of cross-attention, where the input feature is the text embedding. However, for layers where the input contains spatial features, such as query, key, and value projections of self-attention, it does not work because:\n\n- The spatial feature is significantly larger than text embedding, which makes the problem scale too large to compute the closed-form solution.\n- Unlike text embedding, spatial features are usually redundant, leading to redundant constraints, which hinder the computation/accuracy of the Lagrange multiplier.\n\nHowever, in mix-of-show, we don't need to handle constraints because, in our ED-LoRA, we have learned **concept embeddings that already encode the essence of the concept**, which will not conflict with existing embeddings. Therefore, in merging multiple concepts, the changed weight only relates to the new embedding, and thus we can remove the constraints and formulate the problem as a non-constrained optimization. Without the restriction of constraints, we can **flexibly merge enlarged concepts without concept conflicts** and **support the merging of different modules**, including cross-attention, self-attention, and text encoders, making it highly compatible with LoRAs.\n\nWe will incorporate these discussions to clarify the similarity and difference between Custom Diffusion and Mix-of-Show in the revised version of the paper.\n\n---\n\n**W.2**: Thanks for raising this concern. The decentralized setting is necessary for achieving **reusability and scalability**, **avoiding feature mixing**, and **facilitating tuning and model selection**.\n\n- **Reusability and Scalability**: Considering a real scenario that users share tuned models instead of sharing their collected source data. For instance, they may share LoRA models based on Stable Diffusion on platforms like Civitai [1] or Huggingface [2]. In such a scenario, if we want to make our **own concept interact with the shared concept**, joint tuning doesn't work since it cannot access to source data of shared concepts. Although users can fine-tune their customized concept data on top of the shared model, it probably leads to gradual degradation of the performance of previously learned concepts. This phenomenon is known as catastrophic forgetting in continuous learning. However, in our decentralized setting, these issues are naturally overcome. We can direct reuse the share concept models without source data.\n\nEven in a setting where all customized concept data is available, decentralized multi-concept customization (i.e., separate tuning and then model fusion) offers a better solution because:\n\n- **Mitigating the feature mixing issue**: As mentioned in SVDiff [3], when Custom Diffusion [4] fine-tunes multiple concepts together, features from the similar category can become mixed. This was also the issue encountered in our preliminary exploration. On the other hand, Mix-of-Show performs separate tuning, which first gets the embeddings describe the essence of given concept. The learned LoRA weight only related to those embeddings, mitigating feature mixing with other concepts in joint tuning.\n\n- **Reduce difficulties in evaluating all concepts together**. Users will need to tweak their dataset repeat number or tuning epochs on each concept to select the best performing checkpoints. Balancing the training epochs of different concepts and achieving a good checkpoint considering the quality of all concepts in joint training becomes problematic when the concept numbers increasing and the data number of each concept varies. This problem is not mentioned in Custom Diffusion [4] or SVDiff [3] since they focus on customize 2 object concepts. But it is problematic in our case, i.e., customizing 14 concepts with varying number of different concepts (about 15 images for character, 5 for objects and 20 for scenes).\n\nIn summary, decentralized multi-concept customization mitigating feature mixing, reduces the workload of jointly evaluating all new concept performance in concept tuning, and becomes the most flexible and reusable way to scale up concept customizations for production.\n\n---\n\n**W.3**: Thanks for pointing out, in the Figure 2(b), the caption should be “a $V^{dogA}$, a $V^{cat}$ and a $V^{dogB}$, on the grass, under the sunset”.\n\n---\n\n**Reference**:\n\n[1] https://civitai.com/\n\n[2] https://huggingface.co/\n\n[3] SVDiff: Compact Parameter Space for Diffusion Fine-Tuning. ICCV 2023.\n\n[4] Multi-Concept Customization of Text-to-Image Diffusion. CVPR 2023.\n\n"}]

Author rebuttal:
Thank you to all the reviewers for providing valuable comments to help us improve our submission. We would like to clarify that, due to the page limit, the **quantitative evaluation**, **dataset details**, **limitation discussion** and **potential negative society impact discussion** are provided **in the supplementary material**. In the following response, we will refer you to the relevant section. We plan to move parts of important quantitative results and discussions to the main paper in the revised version. Additionally, we have included **supporting figures and tables in the attached PDF for this response**.

Meta-review: This paper received a range of scores, with four reviewers recommending acceptance and one reviewer suggesting rejection. The authors effectively addressed the reviewers' comments during the rebuttal process. The paper introduces a novel framework called "Mix-of-Show" to tackle the challenge of decentralized multi-concept customization of diffusion models. The method shows promising results and outperforms existing approaches in the domain of multi-concept customization. However, it is suggested that more practical results should be included. In light of the overall assessment and improvements made by the authors, the AC has decided to accept this paper.
Decision: Accept (poster)