Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept
  Customization of Diffusion Models
Authors: name: Yuchao Gu; name: Xintao Wang; name: Jay Zhangjie Wu; name: Yujun Shi; name: Yunpeng Chen; name: Zihan Fan; name: Wuyou Xiao; name: Rui Zhao; name: Shuning Chang; name: Weijia Wu; name: Yixiao Ge; name: Ying Shan; name: Mike Zheng Shou
Published: 2023-05-29 17:58:16

Abstract: Public large-scale text-to-image diffusion models, such as Stable Diffusion,
have gained significant attention from the community. These models can be
easily customized for new concepts using low-rank adaptations (LoRAs). However,
the utilization of multiple concept LoRAs to jointly support multiple
customized concepts presents a challenge. We refer to this scenario as
decentralized multi-concept customization, which involves single-client concept
tuning and center-node concept fusion. In this paper, we propose a new
framework called Mix-of-Show that addresses the challenges of decentralized
multi-concept customization, including concept conflicts resulting from
existing single-client LoRA tuning and identity loss during model fusion.
Mix-of-Show adopts an embedding-decomposed LoRA (ED-LoRA) for single-client
tuning and gradient fusion for the center node to preserve the in-domain
essence of single concepts and support theoretically limitless concept fusion.
Additionally, we introduce regionally controllable sampling, which extends
spatially controllable sampling (e.g., ControlNet and T2I-Adaptor) to address
attribute binding and missing object problems in multi-concept sampling.
Extensive experiments demonstrate that Mix-of-Show is capable of composing
multiple customized concepts with high fidelity, including characters, objects,
and scenes.