Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept
  Customization of Diffusion Models
Authors: name: Yuchao Gu, name: Xintao Wang, name: Jay Zhangjie Wu, name: Yujun Shi, name: Yunpeng Chen, name: Zihan Fan, name: Wuyou Xiao, name: Rui Zhao, name: Shuning Chang, name: Weijia Wu, name: Yixiao Ge, name: Ying Shan, name: Mike Zheng Shou

Public large-scale text-to-image diffusion models, such as Stable Diffusion,
have gained significant attention from the community. These models can be
easily customized for new concepts using low-rank adaptations (LoRAs). However,
the utilization of multiple concept LoRAs to jointly support multiple
customized concepts presents a challenge. We refer to this scenario as
decentralized multi-concept customization, which involves single-client concept
tuning and center-node concept fusion. In this paper, we propose a new
framework called Mix-of-Show that addresses the challenges of decentralized
multi-concept customization, including concept conflicts resulting from
existing single-client LoRA tuning and identity loss during model fusion.
Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client
tuning and gradient fusion for the center node to preserve the in-domain
essence of single concepts and support theoretically limitless concept fusion.
Additionally, we introduce regionally controllable sampling, which extends
spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address
attribute binding and missing object problems in multi-concept sampling.
Extensive experiments demonstrate that Mix-of-Show is capable of composing
multiple customized concepts with high fidelity, including characters, objects,
and scenes.