Customizable Image Synthesis with Multiple Subjects
Authors: Zhiheng Liu
Yifei Zhang
Yujun Shen
Kecheng Zheng
Kai Zhu
Ruili Feng
Yu Liu
Deli Zhao
Jingren Zhou
Yang Cao
Venue: NeurIPS 2023 poster (2023)
Keywords: text2img
diffusion model
customization

Abstract: Synthesizing images with user-specified subjects has received growing attention due to its practical applications. Despite the recent success in single subject customization, existing algorithms suffer from high training cost and low success rate along with increased number of subjects. Towards controllable image synthesis with multiple subjects as the constraints, this work studies how to efficiently represent a particular subject as well as how to appropriately compose different subjects. We find that the text embedding regarding the subject token already serves as a simple yet effective representation that supports arbitrary combinations without any model tuning. Through learning a residual on top of the base embedding, we manage to robustly shift the raw subject to the customized subject given various text conditions. We then propose to employ layout, a very abstract and easy-to-obtain prior, as the spatial guidance for subject arrangement. By rectifying the activations in the cross-attention map, the layout appoints and separates the location of different subjects in the image, significantly alleviating the interference across them. Using cross-attention map as the intermediary, we could strengthen the signal of target subjects and weaken the signal of irrelevant subjects within a certain region, significantly alleviating the interference across subjects. Both qualitative and quantitative experimental results demonstrate our superiority over state-of-the-art alternatives under a variety of settings for multi-subject customization.

Cones 2: Customizable Image Synthesis with Multiple Subjects
Zhiheng Liu
Yifei Zhang2+t
Yujun Shen
Kecheng Zheng²
Kai Zhu 4.4
Ruili Feng 1.4
Yu Liu
Deli Zhao
Jingren Zhou*
Yang Cao
USTC
SSTU
3 Ant Group
Alibaba Group
Abstract
Synthesizing images with user-specified subjects has received growing attention due to its practical applications. Despite the recent success in single subject customization, existing algorithms suffer from high training cost and low success rate along with increased number of subjects. Towards controllable image synthesis the constraints, this work studies how to efficiently
with multiple subjects represent a particular subject as well as how to appropriately compose different subjects. We find that the text embedding regarding the subject token already serves as a simple yet effective representation that supports arbitrary combinations without any model tuning. Through learning residual top of the base embedding, we manage to robustly shift the raw subject to the customized subject given various text conditions. We then propose to employ layout, very abstract and easy- to-obtain prior, as the spatial guidance for subject arrangement. By rectifying the activations in the cross-attention map, the layout appoints and separates the location of different subjects in the image, significantly alleviating the interference across them. Both qualitative and quantitative experimental results demonstrate our superiority over state-of-the-ar alternatives under a variety of settings for multi-subject customization. Project page can be found here.
Introduction
The remarkable achievements of text-to-image generation models [1-14], have garnered widespread attention due to their ability to generate high-quality and diverse images. To allow synthesizing images with user-specified subjects, customized generation techniques [15-17] propose to fine-tune the pre-trained models on few subject-specific images. Despite the notable success in single subject customization [15-20], multi-subject customization remains seldom explored but better aligns with the practical demands in real life.
Recent studies [16, 17] have investigated multi-subject customization through joint training, which tunes the model with all subjects of interest simultancously. Such a strategy has two drawbacks. First, they require learning separate models for each subject combination, which may suffer from exponential growth when the number of subjects increase. For example, the customization of objects BA, c} fails to inherit the knowledge obtained from the customization of objects (A, B}. Second, different subjects may interfere with each other, causing the issues that some subjects fail to show up the final synthesis or the subject attribute gets confused among subjects (e.g., cat with the features of another dog). This phenomenon is particularly evident when the semantic similarity between subjects high (see Fig. 4).
"Equal contribution. Work performed during internship at Alibaba DAMO Academy. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

dog
dog
dog
hat
barn lake
A dog wearing sunglasses, hat, sitting next to dog wearing sunglasses, with barn in the background.
A dog wearing sunglasses, hat, sitting next to dog wearing sunglasses, with lake the background.
sunglasses sunglasses
Figure 1: Customizable image generation with the subjects listed on the left. Cones 2 is highlighted from three aspects.
(1) Using a simple yet effective representation to register a subject, we can compose various subjects arbitrarily withoul any model luning. (2) Employing spatial layout, which is very easy to obtain in practice, as a guidance, we can control the specific location of each subject and meanwhile alleviate the interference across subjects. (3) Our method achieves appealing performance even under some challenging settings, such as customizing the synthesis with six or more subjects and exchanging the sunglasses the two dogs.
In this work, we present Cones 2, novel approach for multi-subject customization using pre-trained text-to-image diffusion model. Our method utilizes simple yet effective representation to register subject and enables the arbitrary composition of various subjects without requiring any model retraining. To that end, we decompose the challenging task of multi-subject customization into two components: how to efficiently represent a subject and how to effectively combine different subjects. Given set of subjects and their photos (3-5 for each), our goal first to bind the characteristic of each specific subject a "plugin" that be used flexibly. Driven by this, we fine-tune the text encoder part of a pre-trained text-to-image diffuusin model with images of specific subject, making the tuned model can customize this specific subject. Moreover, we propose which limits the output of the tuned text encoder to only differ from the original text encoder in token embedding regarding the specific subject. Then we calculate the mean difference between the tuned text encoder with the original text encoder to derive the residual token embedding which can robustly shift the raw category the customized subject (e.g., dog - customized dog).
loss, ceanbbddding-rrseervade
To effectively combine different subjects, we propose a layout guidance method to control the generation process. More formally, we employ pre-defined layout, very abstract and easy-to-obtain prior, to guide different subjects to show up in different positions by rectifying the activation in the cross-attention maps. We encourage all subjects show up in the final synthesis by strengthening the activations of the target subject. Simuitancously, to prevent the subject attribute gets confused, we weaken the activations of the irrelevant subjects. In addition, in practice, we define the layout as a set of subject bounding boxes with subject annotation, which describes the spatial composition of the customized subjects and easy for users to specify in advance. Through our method, users can compose various subjects arbitrarily with pre-defined layout (see Fig. l).
make this easy to implement
Our method is evaluated under a variety of settings for multi-subject customization involving extensive subject categories such pets, scenes, decorations, etc. Qualitative and quantitative results demonstrate that, compared to existing baselines, our method exhibits competitive performance in terms of both text alignment input prompt and visual similarity to the target images. It is noteworthy that our method even facilitates the customization of larger number of subjects (e.g. six subjects in Fig. 1), which a far more challenging setting in practice.
2 Related work
Large-scale text-conditioned image synthesis. Synthesizing images from the language description has received growing attention due to its ability to generate high-quality and diverse images. Earlier

---

works [21] explored the utilization of language deseription into GAN as a condition on specific domains under the closed-workd assumption. With the development of diffusion models [22, 23]
and large-scale multi-modality models [24], lext-conditioned image synthesis has shown remarkable
improvement in an open-vocabulary text description. Specifically, GLIDE [3], DAILLE2 [4], StableDiffusion [5] and Imagen [6] are representative diffusion models that can produce photorealistic outputs, Autoregressive models such as DALLE [1], Make-A-Scene [2], Cog View [25] and Parti [26] have also shown exciting results. Although these models demonstrate
unparalleled ability to synthesize images, they require time-consuming iterative processes to achieve high-quality image sampling. Recent large text-to-image GAN models such as StyleGAN-T [7], GALIP [27], and GigaGAN [8] also demonstrated unprecedented semantic generation, which is orders of magnitude faster when sampling.
Customized image generation. Thanks to the significant progress of large-scale text-to-image models, users can adopt these well-trained models to generate customized images with user-specified subjects. There are two earliest attempts to solve the customized generation through few-shot images of one specific subject, i.e. Text Inversion [18] and DreamBooth [15]. Concretely, Text Inversion [18] represents a new subject by learning an extra identifier word and adding this word to the dictionary of the text encoder. DreamBooth [15] binds rare new words with specific subjects through few-shot finettning the whole Imagen [6] model. To compose multiple new concepts together, Custom [16] chooses to only optimize the parameters of the cross- attentiottnnion in the StableDiffusion [5] model to represent new concepts and then joint trains for the combination of multiple concepts. In addition, Cones [17] associates customized subjects with activating a small cluster of neurons in the diffusion model. Although both Custom [16] and Cones [17] have explored combination multi-subject customization, they suffer from high training costs and low success rates along with the increased number of subjects. In this work, we study how to efficiently represent a particular subject as well how to appropriately compose different subjects. Specifically, learning residual on top of the base embedding can represent new concept, and the introduction of layout into the attention map can help the model generate more accurate user-specified subjects. We find these design choices lead to better results in directly composing different subjects than joint training.
Spatial guidance in diffusion models. To further enhance the controllability of synthesizing images, some works [28-32] have tried to explore how to guide the generation process by more spatial information. Composer [28] directly adds spatial information
condition input during the training phase. ControlNet [29] and 121-Adapters [30] add spatial information to the pre-trained model by training new adapter. Prompt-to-promp [31] presents a training- free edit method by editing the cross-attention. In addition, a diffusion-based image translation [32] keeps the generated spatial structure by limiting the cross-attention map. Inspired by these works, we also adopt the layout the spatial guidance for subject arrangement that can well appoint and separate the location of different subjects in the image, significantly alleviating the interference across them.
3 Method
Given a set of subjects and their photos (3-5 for each) from different views, we aim to generate new images of any combination containing those subjects vividly and precisely. We accomplish this by combining subject-specific residual token emheddings with a pre-trained diffusion model and guiding the generation process with layout. The overall framework is presented in Fig. 2. Specifically, we represent each subject as a residual token embedding shifted from its base category. Adding the residual token embedding the base category embedding can yield the corresponding subject the generated images. We present how to get this residual token embedding in Sec. 3.2. At inference time, subjects failing to show up and the subject attribute getting confused among subjects are two key problems in multi-subject customized generation. To address these issues, we present a method of composing subjects by leveraging layout guidance in Sec. 3.3.
3.1 Text-condittoned diffusion model
Diffusion models learn a data distribution by the gradual denoising of a variable sampled from a Gaussian distribution. This corresponds learning the reverse process of a fixed-length Markov chain. In text-to-image tasks, the training objective of a conditional diffusion model eq can be simplified as reconstruction loss
Lice Excee -E1))
(I)
3

---

Subject
"a cal and dog"
Text Encoder (Frozen)
dog
U-Net (Frozen)
custom cat
Cross-atiention
(Taning)
"a photo dog"
Text Limbedding
A
custom dog
Cross-atrention
Edited
Text Encoder (Frienn)
a
cat and
dog
(b)
Figure 2: Illustration of the proposed approach. (a) We first learn a residual token embedding (e.g.. A custom) on top of the base embedding register user-specified subject, which allows composing various subjects arbitrarily without further model tuning. (b) Given layout the spatial guidance, we then arrange the subjects by rectifying the activations in cross-attention maps, which enables the control of the location of each subject and reduces the interference between them.
where ([0,, 1]) is the time variable, image from the ground-truth image The parameters oą and are coefficients formulating the forward diffusion process. The model Eg conditioned on the text embedding F(c) and The text embedding E(c) is injected into the model еş through the cross-attention mechanism. At inference time, the network Eø is sampled by iteratively denoising xy ~ NO I) using either deterministic samplers [33-35] or stochastic sampler [23].
pre-trained text encoder and x; = a+x a,E a noised
3.2 Representing subjects with residual token embedding
Representing subjects with residual text embedding. Our goal is first to represent each subject with a residual text embedding among the output domain of the text encoder. An ideal residual text embedding A custom that can robustly shift the category to specific subject. For example, the model Eg with embedding input (E("a photo of dog") + A dog can truly generate a photo of specific "dog". One way to get this objective is to calculate an embedding direction vector [32] from source (original) text encoder to the target (fine-tuned) text encoder Eesiion The fine-tuned text encoder Exustom needs to able to customize subject & combined with the original diffusion model loss, (x,, Excutto (c), t) <l2).
cg. Similarly DreamBooth [15], Ecustom can be trained with the
L (eeutoomm
where D.= [(x), "a photo of xx]] E X*} is the reference few-shot data of subject
Regularization with text-embedding prreddiatppeeeetttionnonn loss. The residual text embedding obtained according to the previous section can only perform single-subject customized generation. Since those residual text embeddings conflicts so that they cannot be combined together directly while carrying out inference. Therefore, we propose texb-enmg preservation loss to make the residual text embedding mainly act on the text embedding regarding the subject token. The core idea is to minimize the difference between Fyeustom and for tokens apart from the subject token s. Take the "dog" case above as an example, 1000 containing the word "dog" using ChatGPT [36], like "a dog on the beach".. and then minimize the difference between Fecustom and F. for all the token besides "dog". In detail, given any caption (e.g. c "a dog on the beach"), we split its text embedding into sequence (E(c) (E(c)a. E(c) dog ... E(() beach)) )) Then we wish E[)p) Ep (c)) [|3 0 for any that is not equal to "dog". Namely, the text- sostt--mbedding-ressssse preservation loss is regularization, as
applied to the entire text, any of them can admit significant
we sample 1,000 sentences Cang {c' ]i=
Lieg (Lesstom) EowCinl
Σ
Lesstom (c), E1))
(3)
PEc.pss

---

Algorithm 1 N-Subject Customization with Layout Guidance
Require: Prompt с, customized set 5 = { C c, pre-trained residual token embeddings
(A [
I. Yi=1. guidance layout M {M, : SJ.
1: Edit the text embedding: Effiaa = ((c) @ (Aststom)) N
2: Define guidance layout for each S from M: Ti=li
(1,3) Rshow
M.(i,))
= ?
(i,j) E Rirrlevann
Otherwise
3: Sample XT (O,
4: for = T,T 1, I..., do
5: CA c0(,,, Ethnaal (c), f);
6: CAadieed f EditedCA (CA, (CA, M. c);
7: X;-1 cx( Efinal (c), {) CAedited)
8: end for
where p traverses all tokens inside sentence с except the subject token 8. Our complete training objective then comes
Luub + XLreg
(4)
where À controls for the relative weight of the text-embedding preserrvtionn term. As shown in Fig. 2a, after the customized text encoder is obtained, we derive the residual token embedding of "dog" via computing the average shift of Exsustom dug over these 1,000 sentences from E ((C))og+>
1
custom
E
(c)asg E(()dog)-
(5)
dog
|Cdogl
CECdsa
Inference with residual token embedding. The residual token embedding we get aforementioned be used directly in any subject combinations involving them without further tuning. As shown in Fig. 2b, when we do customized generation with N specific subjects $1, 82, ... &N, all we need is to fetch the pre-computed A gustom A
custom
custom and add them to the token embedding, A
Ethnal (c)s: E(()., + Agustom 1 ...
(6)
In fact, the operation in Eq. (6) is all in the token dimension. This characteristic endows our method with significant convenience and high efficiency for large-scale applications. On the one hand, any pre-trained residual token embedding A cussom can be used repeatedly and combined with another
A castom, On the other hand, for each subject, we merely need store float32 vector, getting rid of storing large parameters as in previous methods [15-17].
3.3 Composing subjects with layout guidance
The text-to-imane image diffusion models [4-6] commonly inject the text embedding E(c) to its diffusion model Eg via the cross-attention mechanism. The attention map among a cross-attention layer is CA (Wo .(xx)) (Wx E(c)), where sp(x)) denotes the transformed image feature and We. W denotes the parameters for computing query and key. The cross-attention map directly affects the spatial layout of the final generation [31]. Below we will discuss how improve the quality of customized generation by rectifying the activations in the cross-attention map.
Strengthening the signal of target subject. One issue in multi-subject customization is that some subjects may fail to show up. We argue that this is caused by insufficient activations in the cross- attention map of these subjects. To avoid this, we choose to strengthen the signal of the target subject in the region where we want it to show up.
Weakening the signal of irrelevant subject. Another issue in subject attribute gets confused among subjects, i.e. the subjects in generated images may contain characteristics from the other subjects. We argue that this is due to the overlapping activation regions of different subjects the cross-attention map. To avoid this, we choose to weaken the signal of each subject appearing in the region of the other subjects.
customization is that the
5

---

Layout-guided iterative generation process. Combining the above two ideas, we present method guide the generation process according to pre-defined layout M. In practice, we define the layout M as set of subject bounding boxes and then get the guidance layout M, for each subject 8. In detail, as shown in Fig. 2b we divide M, into different regions: we set the value of M, to a positive value y E IR+ in the region where we want the subject to show up (denote as Reshow) and set the value M2 to a negative valueyy E RT in the region that irrelevant to the subject (denote as Rg I At the inference time, we replace all the output of cross-attention with edited results at every generation step, as
EditedCA(CA: M, = Softmax CCA @ {n(()
- -,,..- ..).,,N)) (Wv- E(c)), (7)
where @ denotes the operation that adds the corresponding dimension of CA and M. which is also visualized Fig. 2b and r7(1) a concave function controlling the edit intensity different time l. The implementation details refer to Algorithm 1.
4 Experiments
4.1 Experimental setups
Datasets. For fair and unbiased evaluation, we select subjects from previous papers [15, 18, 16, 17] spanning various categories for a total of 15 customized subjects. It consists of two scenes, five pets and eight objects. We perform extensive experiments on various combinations of subjects, explaining the superiority of our approach.
Evaluation metrics. We evaluate our approach with two following metrics for customized generation proposed in Textual Inversion [18]. (1) Image similarity, which measures the visual similarity between the generated images and the target subjects. For multi-subject generation, we calculate the image similarity of the generated images and each target subject separately and finally calculate the mean value. (2) Textual similarity, which evaluates the average CLIP [24] similarity between all generated images and their textual prompts. To this end, we use a variety of prompts with different settings to generate images, including modifying the scenes, attributes, and relation between subjects.
Baselines. To evaluate our generation quality, compare our approach with three state-of-art competitors, Le., DreamBooth [15] that fine-tunes all parameters in diffusion model; Custom diffiusion [16] that optimizes the newly added word embedding in text encoder and few parameters in diffusion model, namely the key and value mapping from text to latent features in the cross-attention; and Cones [17] that finds a small cluster of neurons diffusion model corresponding each customized subject. As Custom diffusion and Cones demonstrated, we omit Textual Inversion [18] as it performs much less competitively. And the implementation details of our approach and those of baselines are also reported in the Appendix A.
4.2
Main results
In this section, to demonstrate the superiority of our approach, conduct experiments authentic images from diverse categories, including objects, pets, backgrounds, etc.. We not only present the qualitative results between approach and other baselines but also showcase quantitative comparison. This further substantiates the effectiveness of our approach.
Qualitative comparison. As depicted in Fig. 3, we present collection of generated images featuring two to four subjects. For single-subject generation, as shown in Appendix B.I, our approach achieves comparable results to competing methods while requiring significantly less storage space. However, as the number of subjects increases, the other three methods fail to include certain subjects and exhibit attribute confusion, resulting in generated images that deviate from the reference images. In contrast, our approach consistently produces highly visually accurate images for all subjects. is important to note that our approach utilizes learned single-subject residual token embeddings for seamless combinations without retraining, thereby avoiding exponential training costs associated with the other methods. The next section will discuss this in detail.
Quantitative comparison. In the context of generating customized subjects with varying numbers, we have carefully selected four evaluation metries: textual similarity, visual similarity, required storage space, and computational complexity. As shown in Tab. 1, for single-subject generation, approach exhibits slightly lower visual and textual similarity compared to DreamBooth while

---

Customized subjects
DreamBooth
Custom Diffusion
Cones
Ours
candle and a teapot on the table.
A monster toy next to a duck toy, with barn in the background.
A cat is wearing sunglasses and sitting chair, with lake in the background.
Figure 3: Qualitative comparison of multi-subject generation ability between approach and baselines. Our approach surpasses existing alternatives with higher success rate in generating these subjects and less attribute confusion among different subjects.
remaining comparable to the other two methods. However, as the number of subjects increases, our approach consistently outperforms the other methods across four evaluation metrics. This demonstrates its effectiveness in capturing subject characteristics, maintaining fidelity the given prompt, and achieving higher efficiency in practical applications. Please note that the storage space and computational complexity presented in Tab. 1 for approach assume that there no existing learned single-subject residual token embeddings. However, in practice, we already have required subject-specific residual token embeddings for multi-subject generation, additional storage space or computational complexity is needed. This ability to seamlessly combine existing models without retraining is unique advantage of approach. In contrast, other methods require storage space and training time for generating multiple new subjects.
User study. We conduct user study further evaluate approach. The study investigates the performance of the four methods on the multi-object customization task. For each task and each method, generated 80 images from 4 subjects combination, 4 conditional prompts, and 5 random seeds, resulting in 1,280 generated images for the whole user study. We presented two sets of questions participants to evaluate image similarity and textual similarity. Taking the prompt "A cat and dog on the beach" as an example, where "cat" and "dog" are customized subjects, we provided reference images of the customized subjects and asked participants questions like: "Does the image contain the customized cat?" to evaluate visual similarity. For textual similarity, based on the textual description, we presented questions like "Does the image contain a cat and dog?". As shown in Tab. 2, our approach is most preferred by users regarding both image and text alignment.
4.3 Towards challenging cases
In this section, we further illustrate our superiority by showcasing two scenarios: generating larger number of customized subjects and generating subjects with high semantic similarity that other methods fail to achieve.
7

---

Table 1: Quantitative comparisons. Our approach outperforms other methods in all aspects of multi-subject customization, particularly three-subject and four-subject generation. The complexity metric determined by calculating the number of fine-tuning iterations required for each method to generate a certain combination of n subjects.
Method
Text Aligument
Image Alignment
Storage
Complexity
DreamBooth [15]
0.314
0.727
3.3 GB
O(n)
Single Subject
Custom Diffusion [16]
0.327
0.721
MR
Q(n)
[17]
(1.431: 0.34)
O(n)
Ours
0.330
0.725
4.8 KB
O(n)
DreamBooth [15]
0.278
O(n²)
Two
Custem Diffusion
0.284
0.676
MB (3.41 0.56) MB
O(n²)
Subjects
Cones [17]
0.292
0.685
O(n?)
Ours
0.309
O(n)
DreamBooth [15]
0.252
0.649
3.3 GB
O(n²)
Three Subjects
Custom Diffusion [16]
0.270
0.658
72 MB
O(n
Cones [17]
0.281
(4.96 0.70) MB 14.4 KB
O(n
Ours
0.304
0.689
O(n)
DreamBooth [15]
0.241
0.604
3.3 GB
O(n
Four Subjects
Custom Diffusion [16]
0.254
O(n))
Cones [17]
0.271
0.638
(7.75tt: 0.56) MB
O(n')
Ours
0.299
0.673
19.2 KB
0(n)
Table User study. The value represents the percentage of users that score positive for the image generated corresponding to the given questions. The results show that approach is the most preferred by users for multi-subject customization, on both image and text alignment.
DreamBooth [15]
Custom Diffusion [16]
Cones [17]
Ours
Text
Image
Text Alignment
Inuge Alignment
Text Alignment
Image
Text Alignment
Image
Alignment
Alignment
Alignment
Alignment
Single Subject
71.35%
71.50%
76.85%
67.60%
76.85%
69.60%
75.05%
69.05%
Two Subjects
52.58%
43.13%
59.88%
46.83%
62.55%
57.50%
77.87%
69.75%
Subjects
57.78%
31.83%
58.20%
34.28%
64.87%
37.94%
79.20%
64.42%
Four Subjects
36.42%
25.63%
40.73%
25.44%
42.10%
28.75%
77.35%
$9.08%
Customization with similar subjects. To illustrate the superiority of our approach in mitigating attribute confusion among multiple customized subjects, we select Cones, one of the methods that perform well in multi-subject generation among the three baselines. and compare the generated images with our approach in challenging cases shown in Fig. 4. We observe that when the raw categories of the customized subjects have high semantic similarities, especially in the case of two customized dogs, Cones exhibits notable propensity for attribute confusion arise. contrast, our approach demonstrates excellent performance in both visual and textual similarities.
Customization with large number of subjects. As shown in Fig. 3, observe significant decrease in the quality of generated images by other methods as the number of customized subjects increases. However, our approach shows relatively smaller impact. Therefore, in Fig. 5, present the generated images with an increased number of customized subjects, further demonstrating the effectiveness of our approach.
4.4
Ablation studies
Verify the effect of strength and weaken cross-attention map. We conduct ablation experiments to examine the individual effects of strengthening the target subject region and weakening irrelevant subject regions shown in Fig. 6. We observe that strengthening the target subject alone can lead to attribute confusion between subjects. For instance, it may result in a customized subject exhibiting attributes of another subject. However, when only irrelevant subjects are weakened, certain subjects may fail to show up or exhibit a lack of specific attributes. Furthermore, for simple combinations like "mug + teapot," satisfactory results could be achieved with 30 steps of guidance. However, for more challenging combinations such as "cat dog." 50 steps of guidance were required achieve better attribute binding results.

---

Customized subjects
Cones
Ours
dog and in the forest.
A dog and cat in the forest.
watercolor painting of dog and a real cat the grass.
A dog the floor and a cat the table.
A dog and dog in the forest.
A dog and a dog in the forest.
dog and dog on the beach.
dog and dog under pyramid.
Figure 4: Visualizations of customization with challenging cases. When the subjects that need to be customized belong to the category with high semantic similarity (shown the first row) or even the same category (shown in the second row), the baseline using joint training has serious attribute confusion problem, while our approach circumvents this problem.
Customized subjects
A dog with sunglasses on face and a hat on next a flower with barn in the background.
A monster toy, teapot, and a mug with a flower it, on the label, with a barn background.
A duck toy, pair of sunglasses, mug with flower it, and teapot,on the tabel.
A with sunglasses face and hat head sitting on a chair with a lake in the background.
cat with sunglasses face and hat head sitting next to chair with a barn in the background.
A dog with sunglasses face and hat head, sitting next to dog, with sunglasses on face, with lake the background.
Figure 5: Visualizations of customization with large number of subjects. Here show diverse generation results customizing 5 and subjects.
Verify the effect of guidance in generation process. Recent studies [37, 38] have also demonstrated that incorporating guidance during the sampling process leads superior generation results. We select Cones [17], which exhibits relatively better performance among competing methods, as our baseline, and compare with the ofate-of-the-ee semantic guidance approach as well as our guidance approach. As shown in Fig. 7, combining the Attend and Excite [38] improves the generation quality compared to cones, and further improvement achieved when combine with our guidance approach. However, overall, our approach outperforms others, showcasing the best performance. This proves that the problem attribute confusion in other methods can't be solved by simply adding guidance algorithm while combining residual token embedding with our guidance algorithm can solve it.

---

Strengthen only
Weaken only
T=30
T=50
D
Dodl
A mug and a teapot on the table.
A dog and a cat under mount fuji.
Figure 6: Ablation study on the effectiveness of strengthening and weakening cross-attention map. The first column shows that only strengthening leads to attribute confusion. The second column shows only weakening leads to some subjects failing to show up. The last two column shows that those challenging cases require longer guidance.
Customized subjects
Cones
Cones w/ Att
Cones w/ Layout
Ours
A dog and a dog in the garden on a sunny day.
Figure 7: Qualitative comparison for ablations on grond-and-Excite [38] and our layout guidance. "Att" refers to Attend-and Excite and "Layout" refers to our guidance method. These results show that the baseline method obtained through joint training cannot avoid attribute confusion by simply combining a certain guidance method. Correspondingly, our method can solve this problem.
5
Conclusion
This paper proposes novel approach for multi-subject customization. Our method combines subject- specific residual token embeddings with a pre-trained diffusion model and utilizes casy-to-obtain layout prior to guiding the generation process. This allows us to combine individually learned subject-speciffc residual token embeddings for multi-subject customization without retraining. Our method consistently delivers exceptional performance even in challenging scenarios, including the customization of image synthesis with six subjects and the customization of semantically similar subjects. Through qualitative and quantitative experiments, we demonstrate our superiority over existing state-of-the-art methods in various settings of multi-subject customization. These results highlight the effectiveness and robustness of our method.
Acknowledgments
This work is supported by National Key R&D Program China under Grant 2020A A01005000 and Alibaba Group through Alibaba Research Intern Program.
10

---

References
[1] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-lo-image generation. Int. Conf. Much. Learn., pages 8821-8831, 2021.
[2] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-A Mlke-A-Scene: Scene-based text-to-image generation with human priors. In Eur. Conf. Comput. Vis., pages 89-106, 2022.
[3] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffuusion models. arXiv preprint arXiv:2112. 2021.
[4] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with CLIP latents. arXiv preprint 2rXiv:2204.06122 2022.
[5] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 10684- 10695, 2022.
[6] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1187,, 2022.
[7] Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. StyleGAN-T: Unlocking the power of GANs for fast large-scale text-to-iman to-image synthesis. arXiv preprint arXiv:2301.09515 2023.
[8] Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up GANs for text-to-image synthesis. arXiv preprint arXiv:2303.05511. 2023.
[9] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong. Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diftusion paths. arXiv preprint 2xxx:23355.1899 2023.
[10] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances Neural Information Processing Systems, 2023.
[11] Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan- Yee K Wong. Vico: Detail- preiilppessrring visual condition for personalized text-to-image generation. arXiv preprint arXiv:2306.0000 00977, 2023.
[12] Xiang Wang. Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang. Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint arXiv:2306.02018, 2023.
[13] Pandeng Li, Chen Wei Xie, Hongtao Xie, Liming Zhao, Lei Zhang, Yun Zheng, Deli Zhao, and Yongdong Zhang. Momentdiff: Generative video moment retrieval from random to real. In Advances in neurul information processing systems, 2023.
[14] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: shot object-level image customization. arXiv preprint arXiv:2307.0944) 2023.
[15] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
[16] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi-concept customization of text-lo-image diffusion. arXiv preprint arXiv:2212.04488, 2022.
[17] Zhiheng Liu, Ruili Feng. Kai Zhu, Yifei Zhang, Kecheng Zheng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao. Cones: Concept neurons in diffusion models for customized generation. arXiv preprint arXiv:2303.0522 2023.
[18] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint wxXiv:2088.01613, 2022.
[19] Ziyi Dong, Pengxu Wei, and Liang Lin. DreamArtist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning arXiv preprint arXiv:221..11337, 2022.
11

---

[20] Wenhu Chen, Hexiang Hu. Yandong Li, Nataniel Rui. Xuhui Jia, Ming-Wer Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. arXiv preprint urXiv:2304.00188 2023.
[21] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention, arXiv preprint arXiv: 15110029, 2015.
[22] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Adv. Neural Inform. Process. Syst., pages 8780-8794, 2021.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Adv. Neural Inform. Process. Syst., pages 6840-68 6851,8511100 2020.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn., pages 8748-8763, 2021. 2021.
[25] Ming Ding. Zhuoyi Yang. Wenyi Hong. Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. View: Mastering lext-to-image generation via transformers. Adv. Neural Inform. Process. Syst. 34:19822-19835, 2021.
[26] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[27] Ming Tao, Bing-Kun Kun Bao, Hao Tang, and Changsheng Xu. GALIP: Generative adversarial CLIPs for text-to-image synthesis. arXiv preprint arXiv:2301.12959 2023.
[28] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint 230.022302.09778, 2023.
[29] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffussio models. arXiv preprint arXiv:2302.05543 2023.
[30] Chong Mou, Xintao Wang. Liaggin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T21- Adapter: Learning adapters to dig out more controllable ability for text-to- toa-te---meeeeee diffusion models. arXiv preprint arXiv:2302.08455 2023.
[31] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- Prompt image editing with cross attention control. arXiv preprint arXiv:2208.012: 2108.0066, 2022.
[32] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. arXiv preprint 232. 2023.
[33] Jiaming Song, Chenlin Meng, and Stefano Emonn. Denoising diffusion implicit models. In Int, Conf. Learn. Represent. 2021.
[34] Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-DPM: analytic estimate of the optimal reverse variance diffusion probabilistic models. In Int. Conf. Learn. Represent. 2022.
[35] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: fast ODE solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0099 2022.
[36] Long Ouyang, Jeffrey Wa, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[37] Weixi Feng. Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana. Sugato Basu, Xin Eric Wang, and William Yang Wang. Training free structured diffusion guidance for compositional text-to-image synthesis, arXiv preprint 2rxiv:22/2.050 2.00032 2022.
[38] Hila Chefer. Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Autend-and-Excit: Attention- based semantic guidance for text-to-image diffusion models. arXiv preprint arXiv:2301.13822 2023.
[39] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. and Thomas Wolf. Diffusers: of diffusion models. bttpp://githhe com/ huggingface/ wtt.ceeeeefffuerrsse.. 2022.
12

---

[40] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang. Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusson models. arXiv preprint arXiv:2211.32227, 2022.
[41] Yuxiang Wei, Yabo Zhang. Zhilong Ji. Jinfeng Bai. Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings customized text-to-image generation. arXiv preprint arXiv:2302.13848, 2023.
[42] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, Daniel Cohen Or. Encoder- based domain tuning for fast personalization of text-to to-image models. arXiv preprint ar ar2i:2200..22228, 42: 15:1 150113, 2023.
13

---

Appendix
A Experimental details
We supplement the experimental details of each baseline method and our method in this section. For better generation quality, we use Stable Diffusion v2-1 12-1-base as the pre-trained model. For a fair comparison, use 50 steps of DDIM [33] sampler with scale of 7.5 for all the above methods. All experiments are conducted using one A-100 GPU.
Textual Inversion [18]. We use the third-party implementation of huggingface [39] for Textual Inversion. We train each subject-specific token with the recommended batch size of 4 and learning rate of 0.002 for 3000 steps. In particular, we initialize the subject-specific token with the corresponding class token. For example, to customize specific cat, we initialize the subject-specific token "<cat>" with the original "cat" token.
DrcamBooth [15]. We the third-party implementation of huggingface [39] for DreamBooth. Training is with batch size of 2, learning rate 5x 10-5 and training steps of 800 number of subjects. Custom Diffusion [16]. We use the official implementation" for Custom Diffusion. Training is with a batch size of 2, learning rate 1 x 10 and training steps of 250 x number of subjects.
Cones [17]. We use the official implementation for Cones. The batch size of the training stage is set to 2, the learning rate is x 10-2 and training steps is 1200 x number subjects.
Ours. For our approach, We train each subject-specific residual token embedding with a batch size of and learning rate of x 10-6 for 3,000 steps. At inference time, the layouts are appointed by bounding boxes given by the users to indicate the location of each subject. We use a positive value of +2.5 to strengthen the signal of the target subject and we use a negative value of -1 10 to weaken the signal of irrelevant subjects. Furthermore, guide all 50 steps with the layout guidance in the whole generation process to get good customized generation results.
User study. For two- to four-subject generation tasks, we design four different subject combinations for each task. This will yield subject combinations in total. For each subject combination, we design four different text prompts generate images with 5 random seeds. We conduct this procedure to all four methods. With such settings, each method generates 80 different images for each task. We give each generated image 4-8 questions for testing image alignment (2-4 questions) and text alignment (2-4 questions). The number of questions is proportional to the number of subjects used to customize the image (average 6 questions per generated image). Finally, we shuffle the order of all the amage-question pairs and assigned them to 25 different users for scoring, and finally summarized the results. In detail, every user needs to score 4 x 4 x 5 6 480 questions for each task and for each method.
B More results
In this section, present additional visualization results that provide comprehensive demonstration of the superiority of our proposed method.
B.1 More comparisons with other baselines
We conduct further comparison between our approach and three other baselines. As shown in Fig. A1, regarding the generation of single subjects, the four methods exhibited similar performance. However, when dealing with semantically similar subjects, such as a dog and a cat, as well as scenarios involving three more subjects, approach clearly exhibit superior performance. Moreover, as shown in Fig. A2, we further showcase additional generated results, providing further evidence of the robustness of method.
http://hhh..cccmmmmoooooatcttuuu iingenenaeeetttttttuessssinnnnnn inversion
14

---

Customized subjects
DreamBooth
Custom Diffusion
Cones
Ours
A monster toy in Times Square.
A dog on the table and a cat on the ground.
dog with sunglasses and mug the table.
A dog with sunglasses, watching flower a mug, on Mars.
Figure A1: More comparisons of our approach and existing baselines.
B.2 More challenging cases
As shown in Fig. A3, we present larger number of images generated by our approach, featuring greater diversity of customized subjects. In comparison with other methods, we observe that when the number of customized subjects reaches four, the performance of other methods significantly deteriorates. contrast, our approach can generate larger number customized subjects, exemplifying the superiority.
B.3 Comparison with image composition method
we select Paint by example [40] as a baseline of image composition methods. In order to compare the effect of the baseline and our method more intuitively, we use Paint by example inpaint the image generated by our method with the reference image as input. We conduct visualization results Fig. A4. Generated results of method have better visual similarity, and will provide more comparisons in the revision. Besides, We consider multi-subject customized generation is different with image composition two a aspects. The purpose of subject customized generation is to implant all user-provy user-provided subjects into the diffusion model; so that the model can generate various images of subjects vividly guided by prompts. However, the purpose of image compositing is to insert an object into another image in realistic way, without guidance from prompts. In addition, to compose multiple subjects by the image composition methods, multiple iterations inference are required, especially when there is an overlapping relationship among subjects.
15

---

Customized subjects
Ours
monster toy under the water.
chair under mount fuji.
cat and dog on the beach.
A teapot and mug on the grass.
teapot and mug with flower on the table.
P
dog wearing sunglasses, sitting mug, the table.
Figure A2: More results of our approach.
B.4 More fine-grained control
Actually, users have the flexibility to input more specific mask, enabling them to achieve fine- grained control. Experimental results Fig. A5 demonstrate that our method can use specific mask for sampling.
B.5
About generated results with more subjects
As for more subjects, it is limited by the capabilities of the pre-trained model itself. As shown Fig. A6, more involved subjects usually decrease the final visual quality. we observe significantly more failure cases when generating five or more subjects. Several recent works [17, 16, 38] point out that stable diffusion is struggle generating multiple subjects. potential way to ease this issue may be to apply our method to better text-conditioned diffusion model.
16

---

Customized subjects
A cat with sunglasses, sitting on chair, with barn in the background.
A dog with sunglasses, sitting mug, lake in the background.
dog sitting next to a chair sunglasses it, a lake the background.
toy, mug, teapot, and 8 sunglasses on
dog sunglasses, watching a flower in mug, Mars.
dog with sunglasses and hat, sitting next to monster toy, with lake the background.
A monster toy, a dog, teapot, with barn in the background.
A dog with sunglasses with sunglasses on the grass,
dog with sunglasses and a sitting duck toy, with a lake in the background.
dog sunglasses hat, sitting next to monster toy, background.
toy, teapot, and mug, with a barn in background.
monster toy, teapot, and mug with flower in it, with a barn in the background.
A monster toy wearing sunglasses, hat, sitting next dog, with barn background.
dog wearing sunglasses, a hat, sitting dog wearing sunglasses, with barn the background.
dog wearing sunglasses, hat, sitting dog wearing sunglasses, with lake the background.
Figure A3; More results of multi-subject generation.
B.6 Comparison with fast customized methods
Currently, some works [41, 42] focus achieving faster customized generation by pretraining an image encoder and using it to encode reference image during generation. However, these methods share some common issues. They require collecting data in advance to train the encoder, which can limit their generalization. We compare our method with Elite in Fig. A7. Compared to methods that fine-tune for each individual subject, these approaches may exhibit difficulty in preserving fine details. In addition, these methods can only encode one image at time, they cannot eustomize multiple subjects simultancously.
17

---

Paint by Example
Ours
Paint by Example
Ours
Paint by Example
Ours
Figure A4: Comparisons with Paint by Example.
Cat
Dog
Cat
A dog and a cat on the beach.
A dog and cat on the grass.
Figure A5: Generated results using specific mask. We use a more specific mask as a prior to get more fine-grained generated results.
C Importance of residual token embedding
To demonstrate the superior generalization of the residuals, we conduct comparative experiments. As shown in Tab. AI, compared directly updating the class embedding parameters in single text embedding, our approach, which involves updating the text encoder and calculating the average shift from the class to the specific subject based certain number of text templates, outperforms in both textual and visual similarity.
C.1 Generated results of textual inversion
We observe from Fig. A8 that Textual Inversion [18] struggles with the generation of complex single subject and multiple subjects.
C.2 About difference between residual token embedding and Textual Inversion
Textual inversion find single word embedding (input of the text encoder) represent user- srovipdovided subject. Different from textual inversion, we find the residual token embedding for each subject, and add these residuals to their corresponding token embedding (output of the text encoder). The "residual" actually refers to the ability to transform subject of one category into a customized subject that need, for example "random dog" to the specific "customized dog". As shown in Fig. A9, train model using method in textual inversion but learn a text embedding (output of the text encoder) and apply our layout guidance approach. We can that simply employing Textual Inversion to learn customized text embedding does not adequately fulfill the customized requirements. This approach fails fully capture all the features the reference subject, specially when dealing with multi-subject customization. Even with the utilization of our layout guidance method, better results cannot be achieved.
D
Social impact and limitations
Social impact. While training individual large-scale scale diffusion models remains prohibitively expensive, advancements in fine-tuning techniques have enabled individual users to customize their own models. our approach empowers users linearly combine their personalized single-subject models, generating high-quality images with multiple customized subjects while maintaining significant advantages in terms of computation and storage efficiency. Furthermore, there a growing need for more reliable detection techniques to identify and mitigate the presence of fake data.
18

---

Pretrained model
Ours 17 subjects)
Pretrained modet-Culddnce
Ours subjects)
sunglasses sunglasses
dog wearing sunglasses and sitting safo, next to wearing sunglasses, with born in the background.
A wearing sunglasses and sitting choir, next to dog wearing sunglasses and flower, sitting on sofo, with a lake in the background.
Figure A6: Generated results with more customized subjects. "Pre-trained model + Guidance" refers to apply our guidance approach to pre-trained model.
ELITE
Ours
ELITE
Ours
dog on the beach.
A robot on the beach.
Figure A7: Comparisons with ELITE and our approach. Our method better preserves the identity of all subjects.
Limitations. approach is limited by the inherent capabilities of the base model. Specifically, when it comes to combining more than six subjects, our approach may not be able to consistently generate satisfactory results. In order achieve the desired generation results, the provided layout by the user needs to be roughly consistent with the textual description.
19

---

Customized subjects
Textual Inversion
monster toy in Times Square.
A candle and teapot the table.
monster toy next to a duck toy, with a barn in the background.
cat wearing sunglasses and sitting chair, with lake in the background.
Figure A8: Generated results of Textual Inversion [18].
Table A1: Quantitative comparisons between our approach (learning residual token embedding) and learning a token embedding directly.
Single Subject
Two Suhjects
Three Subjects
Four Subjects
Text Alignment
Image Alignment
Text Alignment
Image
Text Alignment
Image Alignment
Text Alignment
Image
Alignmeut
Aligament
Our approach
0.330
0.725
0.309
0.708
0.304
0.689
0.299
0.673
Token embedding
0.324
0.720
0.291
0.686
0.292
0.669
0.281
0.651
Token embedding w/ Textual inversion
+Guidance
A dog...
A cat...
A dog and a cat on the beach.
Figure A9: Visualization of learning embedding directly using Textual Inversion. "+Guidance" refers to applying our layout guidance.
20


Reviews:
[{"review_id": "mLeh3wuugu", "reviewer": "Reviewer_X4Jp", "summary": "This paper presents a new method for generating new images of any combination containing given objects. It combines individually learned single-subject residuals for multi-subject generation without retraining. The authors proposed to use text-based embedding to represent individual objects. Once a residual token was learned, they can then add these residuals to the embedding and adjust the attention maps based on a given layout. The proposed method outperforms other existing methods in all multi-subject customized tasks, especially in a three-subject and four-subject generation.", "questions": "1. Would the authors consider other image composition methods as baselines to this method? For example, CVPR 2023 paper titled \"ObjectStitch: Object Compositing with Diffusion Model\"?\n2. Is there a way to apply more fine-grained control on the composition of multiple objects, for example, by applying a specific mask or changing the pose of each object in the new composition? does the proposed require non-overlapping boxes? It would be more informative to have examples of bounding box layouts on the side of each result.  \n2. What happened when the user-provided layouts are not consistent with the text description? Are there other potential failure cases for this method?\n\nTypo: Line 245 Verift -> Verify \n", "limitations": "The authors address the limitations in supplementary material in that their method is limited by the inherent capabilities of the base model. The paper also discussed the potential societal impact of user-specific image generation. I think they are valid. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This is a timely paper that works on an interesting and challenging problem of controllable image synthesis based on the diffusion model. The authors proposed to compose multiple subjects by leveraging layout guidance. Such prior is simple and intuitive to use. The paper is well written and easy to follow. They performed extensive evaluation/ablation studies and outperforms other existing methods both quantitatively and qualitatively. They also conducted a user study to further evaluate the performance of their method. ", "weaknesses": "This paper is a combination of many existing ideas, such as text embedding vector from zero-shot img2img paper, text-embedding-preservation loss, and cross-attention map. While the execution and presentation of these ideas are well done, I wes hoping for a more original contribution and unique solution to the problem. Overall, despite this, the paper still demonstrates impressive results and meets the standard for acceptance. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author Response to Reviewer X4Jp**\nThank you for your positive comments and valuable feedback on our work! We are excited and encouraged by your support!  Bellow we address your concern separately.\n#### **Q1: About the main contributions.**\n**A1:** Thanks. As you have pointed out, using diffusion models for customizable image synthesis is currently a crowded field and hence many similar techniques have been proposed. Compared to existing studies, our approach enjoys two original designs.\n- Regarding **subject representation**, we propose to learn a **residual** on top of the text embedding of a \"base word\" (*e.g.*, from \"dog\" to the \"customized dog\"). Unlike prior works that directly optimize the word embedding for the customized subject, our residual design helps the customization well blends with various context. Our motivation is that, assuming the text encoder could already adequately encode \"dog\" with different surroundings, the learned embedding shift could harmoniously blend \"customized dog\" with the same surrounding as well. The table below suggests that, without the residual design, the overall performance degrades given varying context.\n| Average CLIP Image Similarity | Ours | Learn Directly |\n| :-: | :-: |:-: |\n| Single-subject | 0.7949 | 0.6953 |\n| Two-subject    | 0.7075 | 0.6092 |\n- Regarding **subject arrangment**, we introduce layout as a very abstract and easy-to-obtain prior to guide the generation process. Different from existing methods that strengthen the signal of the target subject within the layout-indicated area of the cross-attention map, we also propose to **weaken** the signal of irrelevant subjects in the same area. Such a design helps alleviate the issue of attribute mixing across different subjects, especially along with the number of subjects increasing. Fig. 6 of the submitted manuscript and the table below demonstrate the effectiveness of our weakening design both qualitatively and quantitatively.\n| Average CLIP Image Similarity | Ours | Only Strength | Only Weaken |\n| :-: | :-: |:-: |:-: |\n| Single-subject | 0.7949 | 0.8081 | 0.7988 |\n| Two-subject    | 0.7075 | 0.6736 | 0.6568 |\n\nWe will clearly explain our core contributions in the revision.\n#### **Q2: Comparison with image composition methods.**\n**A2:** Thanks. Since [1] mentioned by reviewer has no official open source, we select [2] as a baseline of image composition methods. In order to compare the effect of the baseline and our method more intuitively,  we use Paint by example to inpaint the image generated by our method with the reference image as input. We conduct visualization results in Fig. R1 **in the newly added PDF file**. Generated results of our method have better visual similarity, and we will provide more comparisons in the revision. Besides, We consider multi-subject customized generation is different with image composition in two a aspects.\n- The  purpose of multi-subject customized generation is to implant all user-provided subjects into the diffusion model; so that the model can generate various images of all subjects vividly guided by prompts. However, the purpose of image compositing is to insert an object into another image in a realistic way, without guidance from prompts. \n- Specifically, existing effective image composition methods require collecting data pairs and training an image encoder. During inference, they utilize the reference image as a condition to generate the image. In addition, to compose multiple subjects by the image composition methods, multiple iterations of inference are required, especially when there is an overlapping relationship among subjects. For example, in case \"a cat wearing sunglasses sitting on a chair\", inpainting \"chair\", \"cat\", and \"sunglasses\" in turn is necessary.\n#### **Q3: More fine-grained control.**\n**A3:** Actually, users have the flexibility to input a more specific mask, enabling them to achieve fine-grained control. In the paper, we choose bounding boxes due to their ease of acquisition. Experimental results in Fig. R2 **in the newly added PDF file** demonstrated that our method can use specific mask for guidance sampling. Our method supports overlapping boxes, and we demonstrate the bounding boxes we adopted in Fig. R3 **in the newly added PDF file**. Specifically, the bounding boxes for glasses and hat overlap with the bounding box for the dog. We will add more examples of the bounding boxes we used in the revision.\n#### **Q4: More failure cases.**\n**A4:** We add more failure cases **in the newly added PDF file**. Specifically, in Fig. R4 we show a case that user-provided layouts is contradict to the text description. In Fig. R5, we show that our method is limited by the performance of the pretrained model when it comes to customizing the generation of more than 6 subjects. When dealing with 7 concepts, the performance of the pretrained model deteriorates. In our generated results, details of certain subjects are not fully preserved. With the generation of 9 subjects, the combination of the pretrained model and our guidance method contributes to some improvement in the generation quality. However, at this point, both the pretrained model and our method exhibit some subject disappearance, and our method fails to maintain the identity of all subjects under such circumstances.\n\n[1] ObjectStitch: Object compositing with diffusion model. Song *et al.* CVPR'23.\n\n[2] Paint by example: Exemplar-based image editing with diffusion models. Yang *et al.* CVPR'23."}, {"review_id": "HAkcxrEYtW", "reviewer": "Reviewer_QhyH", "summary": "This paper studies how to efficiently represent a particular subject as well as how to appropriately compose different subjects. The author finds that the text embedding regarding the subject token can serve as a simple yet effective representation. To capture features of a specific subject, the author propose a text-embedding-preservation loss to learn a residual token embedding. Based on the residual embeddings, the author employ layout as the spatial guidance for subject arrangement into the attention maps. Both qualitative and quantitative experimental results demonstrate the superiority of the proposed method", "questions": "a) What is the difference between residual token embedding with textual inversion? If a textual inversion is trained for each concept to obtain a token embedding, and then the layout is used for guidance, what are the results like?\n\nb) In Table 2, the author calculates the complexity of different methods. Regarding multiple subjects, what are the training time results between different methods?\n\nc) From the second row of figure 4, the color of the white puppy does not seem to have been well maintained.", "limitations": "Please refer to the weaknesses part.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "a)\tThe paper is clear and easy to read.\n\nb)\tThe proposed method shows superiority towards challenging cases, compared with existing methods. \n\nc)\tFrom both quantitative and qualitative results, the author conducted detailed experiments to analyze and demonstrate the effectiveness of the proposed method. \n", "weaknesses": "a)\tWhat is the difference between residual token embedding with textual inversion? If a textual inversion is trained for each concept to obtain a token embedding, and then the layout is used for guidance, what are the results like?\n\nb)\tIn Table 2, the author calculates the complexity of different methods. Regarding multiple subjects, what are the training time results between different methods?\n\nc)\tFrom the second row of figure 4, the color of the white puppy does not seem to have been well maintained.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author Response to Reviewer QhyH**\nWe sincerely appreciate the affirmation from the reviewer for our work. It serves as a strong motivation for us! Bellow we address your concerns separately.\n#### **Q1: About difference between residual token embedding and Textual Inversion.**\n**A1:** Textual inversion find a single word embedding (input of the text encoder) to represent user-provided subject. Different from textual inversion，we find the residual token embedding for each subject, and add these residuals to their corresponding token/text embeddding (output of the text encoder). The \"residual\" actually refers to the ability to transform a subject of one category into a customized subject that we need, for example a \"random dog\" to the specific \"customized dog\". As shown in Fig. R6 **in the newly added PDF file**，we train model using method in textual inversion but learn a text embedding (output of the text encoder) and apply our layout guidance approach. We can see that simply employing Textual Inversion to learn a customized text embedding does not adequately fulfill the customized requirements. This approach fails to fully capture all the features of the reference subject, sepecially when dealing with multi-subject customization. Even with the utilization of our layout guidance method, better results cannot be achieved.\n\n#### **Q2: Training time results between different methods.**\n**A2:** We present the training time between different methods for single-subject generation in the table. All experiments are completed on a single 80G-A100.\n\n| Method | Textual inversion |DreamBooth |Custom diffusion|Cones|Ours|\n| :-: | :-: |:-: |:-: |:-: |:-: |\n| Training time| 30 minutes |15 minutes |10 minutes |10 minutes |20 minutes |\n\nIt is important to note that our approach utilizes learned single-subject residual token embeddings, enabling seamless combinations without the need for retraining. This helps us avoid the exponential training costs associated with other methods. In contrast, other methods require additional storage space and training time for each new combination of subjects, and their training time increases linearly with the number of customized subjects.\n\n#### **Q3: Color of white puppy.**\n**A3:** Our training data is sourced from the official DreamBooth dataset. In Fig. R7 **in the newly added PDF file**, We present the training dataset for the \"white puppy\". This dataset contains instances of blurred and poorly lit images, which to some extent, influence the generated results."}, {"review_id": "zMuLh1uMpf", "reviewer": "Reviewer_Nfe2", "summary": "This paper introduces a novel method for multi-subject, subject-driven text-to-image generation. The authors develop an efficient system that embeds single subject information effectively and seamlessly combines these separate subject embeddings to generate the final multi-subject image. The key concept is to learn a residual on top of the clip text-embedding for the subject token, thereby enriching it with subject-specific information. This residual is optimized so that it encapsulates specific information about the single subject without imposing further restrictions on the images. Another innovation is a test-time cross attention manipulation technique. Notably, most previous personalization approaches experience object neglect or attribute confusion as the number of subjects increases. This paper addresses these issues by strengthening and weakening certain regions of the cross-attention maps, guided by user-provided layouts. Both quantitative and qualitative results demonstrate superior performance compared to the DreamBooth, Custom-Diffusion, and Cone baselines", "questions": "Here are a few minor comments and suggestions.\n\n1. line 64 over -> out \n2. EDIFF-I [1] uses similar cross-attention manipulation formulation for layout-to-image generation. See Section 4.3 of the paper. I suggest the authors to add some discussions.\n3. figure 4 mismatch with the text description. Which baseline is used here? DreamBooth or Cone ? \n\n[1] Balaji, Yogesh, et al. \"ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.\" arXiv preprint arXiv:2211.01324 (2022).", "limitations": "The authors partially address the limitations. Please find other suggestions in the weakness and question section. ", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1: The paper innovatively designs a preservation loss to optimize text embedding offset, enhancing class information preservation and reducing overfitting. \n\nS2: The proposed layout manipulation effectively addresses object neglect and attribute confusion, enabling better scalability for larger numbers of subjects. \n\nS3: The method significantly reduces storage and computational costs from exponential to linear, enhancing accessibility for multi-subject image generation.\n", "weaknesses": "W1: Despite the novel use of text embedding, the paper exhibits a limitation in this area. Without model fine-tuning, the images don't preserve subject detail as effectively as one might hope (e.g. see figure 1 white dog), which would be problematic when the subject details are important, e.g. when the subject is a human (which is not tested in this paper).\n\nW2: The use of layout as guidance may also present some constraints. It relies heavily on user inputs, potentially limiting the system's flexibility and automation. Furthermore, it appears to be primarily suited to group photos, and may struggle with more diverse actions that involve intricate interactions.\n\nW3: Although the paper presents an effective approach to reduce computation and storage costs, the process is still relatively expensive, particularly when compared to methods such as tuning an encoder [1] or ELLITE [2]. The requirement for fine-tuning limits the method's accessibility or widespread adoption. \n\nW4: The paper's approach to the evaluation of multi-subject generation raises some concerns. The authors mentioned that \"For multi-subject generation, we calculate the image similarity of the generated images and each target subject separately and finally calculate the mean value.\" A potentially more meaningful approach could be to perform object detection first, matching the resulting detections with the reference subjects. Without this, the similarity between single subject and multi subject image doesn't seem very meaningful. \n\n[1] Gal, Rinon, et al. \"Designing an encoder for fast personalization of text-to-image models.\" Siggraph 2023 \n[2] Wei, Yuxiang, et al. \"Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.\" arXiv preprint arXiv:2302.13848 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author Response to Reviewer Nfe2**\nWe greatly appreciate all of your valuable suggestions, which play a pivotal role in enhancing the quality of our paper. Bellow we address all your concerns.\n#### **Q1: Ability to maintain subject detail.**\n**A1:** Thank you for the questions raised by the reviewer. As shown in Fig. R8 **in the newly added PDF file**, we show the generated results of the subjects robot and human, both of which attain comparable results. However, for subjects with a large number of intricate details, there is indeed a gap compared to model tuning methods. Finding better conditions to enhance the preservation of subject details is a promising direction for future improvement. However, for multi-subject generation, existing model tuning methods suffer from high training costs, subject disappearance, and attribute confusion. Our method exhibits superior performance.\n#### **Q2: Constraints of layout guidance.**\n**A2:** Actually, it is very easy to obtain a bounding box. Users can simply select the customized subjects in text prompt by clicking on them and then place and resize the bounding boxes accordingly. We do not make any corrections to the attention maps corresponding to relational words; thus, our ability to represent the interactions between subjects relies entirely on the pre-trained model. As shown in Fig. R9 **in the newly added PDF file**, in the case of straightforward interaction relationships, such as \"sit\" and \"wear,\" both our approach and the pre-trained model achieve satisfactory generated results. However, for more intricate interaction relationships, such as \"handshake,\" the performance of both our approach and the pretrained model falls short of expectations.\n Improving the representation of interaction relationships between subjects is a common challenge faced by all existing customized generation methods. Currently, there are also some recent works [1, 2] exploring better ways to represent interaction relationships. \n#### **Q3: Computation comparisons with fast customized methods.**\n**A3:** We are very grateful for the reviewer's suggestion, and we will add discussion in the revision. Currently, some works [3, 4] focuses on achieving faster customized generation by pretraining an image encoder and using it to encode a reference image during generation. However, these methods share some common issues：\n- They require collecting data in advance to train the encoder, which can limit their generalization [1].\n- We compare our method with Elite in Fig. R10 **in the newly added PDF file**. Compared to methods that fine-tune for each individual subject, these approaches may exhibit difficulty in preserving fine details.\n- In addition, these methods can only encode one image at a time, they cannot customize multiple subjects simultaneously.\n#### **Q4: Quantitative evaluation with a detection-based metric.**\n\n**A4:** Great point! We sincerely appreciate the valuable suggestion. Our evaluation metric of multi-subject generation is inherited from Custom Diffusion and Cones. However, we firmly believe that the evaluation metric you mentioned is more reliable. Therefore, we conduct an evaluation and show the results in the table below. \n\n|Subjects | DreambBooth |Custom diffusion|Cones|Ours|\n| :-: |:-: |:-: |:-: |:-: |\n| 2| 0.7301 $\\pm$ 0.0054 |0.7238 $\\pm$ 0.0023 |0.7591 $\\pm$ 0.0013 | **0.8107 $\\pm$ 0.0004** |\n| 3| 0.6981 $\\pm$ 0.0101 |0.7150 $\\pm$ 0.0086 |0.7276 $\\pm$ 0.0072 | **0.7752 $\\pm$ 0.0031** |\n| 4| 0.6312 $\\pm$ 0.0183 |0.6387 $\\pm$ 0.0109 |0.6771 $\\pm$ 0.0089 | **0.6987 $\\pm$ 0.0042** |\n\nDuring the evaluation process, we apply GLIP [5] to detect the corresponding subjects in the generated images and crop the images accordingly. Then we calculate the CLIP image similarity for each customized concept and take the average. The results indicate that our method outperforms the baseline in terms of CLIP image similarity. Additionally, the evaluation shows that our method has a lower variance, indicating that it can generate all the required subjects in a more stable and consistent manner.\n#### **Q5: Discussion about difference from EDIFF-I.**\n**A5:** Different from existing methods [6] that strengthen the signal of the target subject within the layout-indicated area of the cross-attention map, we also propose to **weaken** the signal of irrelevant subjects in the same area. Such a design helps alleviate the issue of attribute mixing across different subjects, especially along with the number of subjects increasing. Fig. 6 in the submitted manuscript and the table below demonstrate the effectiveness of our weakening design both qualitatively and quantitatively.\n\n| CLIP Image Similarity | Ours | Only Strength | Only Weaken |\n| :-: | :-: |:-: |:-: |\n| Single-subject | 0.7949 | **0.8081** | 0.7988 |\n| Two-subject    | **0.7075** | 0.6736 | 0.6568 |\n| Two-subject (w/ Detection) | **0.8107 $\\pm$ 0.0004** | 0.7508 $\\pm$ 0.004 | 0.7993 $\\pm$ 0.0006 |\n#### **Q6: The baseline used in Figure 4.**\n**A6:** We sincerely apologize for our oversight. Actually, we choose Cones as the baseline. In Fig. R11 **in the newly added PDF file**, we conduct a new experiment to compare the generation capabilities of DreamBooth, Cones, and our method in challenging cases.\n\n[1] ReVersion: Diffusion-Based Relation Inversion from Images. Huang *et al.* arXiv preprint arXiv:2303.13495.\n\n[2] ProSpect: Expanded Conditioning for the Personalization of Attribute-aware Image Generation. Zhang *et al.* arXiv preprint arXiv:2305.16225.\n\n[3] Designing an encoder for fast personalization of text-to-image models. Gal *et al.* SIGGRAPH'23.\n\n[4] Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. Wei *et al.* arXiv preprint arXiv:2302.13848.\n\n[5] Grounded language-image pre-training. Li *et al.* CVPR'22.\n\n[6] eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers. Balaji *et al.* arXiv preprint arXiv:2211.01324.\n"}, {"review_id": "Tbrj1LI6j7", "reviewer": "Reviewer_msmN", "summary": "This paper proposes a method to achieve customizable image generation with multiple subjects. To achieve the subject generation, it is done by learning a residual prediction for the subject tokens. To make multiple subject tokens combinable, it proposes a text-embedding preservation loss to make the embedding of the category from fine-tunes E and the freezed E to be the same expect for the subject. To ensure the subjects does not conflict and the spatial layouts, it additionally use bounding boxes to modulate the cross-attention when generate the outputs. The proposed method is compared with baselines such as DreamBooth, CustomDiffusion, Cones on the subjects previous works used. The results shows promising results on custimizable image generation with up to four or even more subjects.", "questions": "- Ablation study for the proposed components such as text-embedding-preservation loss and the effects of the layout-guidance generation are missing.\n\n- What's the limitations of the proposed methods beside scaling up to even more subjects? \n\n- Typo: Line 245: Verift -> Verify.", "limitations": "The authors did not address the limitation and potential negative societal impact of their work. Subject generations can produce fake contents and might be dangerous if not used carefully. The authors should discuss this aspect of their work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Proposed a efficient method to handle multiple instance of customization by learning a residual token from the base-category. This makes the customization succeeds without fine-tuning a large number of parameters. \n\n- The text-embedding preservation loss is proposed to make multiple subject tokens combinable. I really like this simple yet effective design of the loss.\n\n- Using layout to guide the generation process makes sense: it provides fine-grained control and avoid the conflicts between subjects.\n\n- The paper is easy to follow and the presentation of the results is clear.", "weaknesses": "\n- Some important aspects of the method can be elaborated more clearly. For instance, how is the layout be taken for the model as the inputs? Is it also the input in the denoising steps or they are just used to find the corresponding area in the attention maps? Why do we need \\ita(t) in Eq. 5? Are layouts used during the pretraining of the subject tokens?\n\n- Although the method has been compared with previous works, some ablation studies and discussion of the limitation are missing. For instance, the quantitative evaluation of the ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author Response to Reviewer msmN**\nWe greatly appreciate all of your valuable suggestions, which play a pivotal role in enhancing the quality of our paper. Bellow we address all your concerns.\n#### **Q1: More details about the layout guidance method.**\n**A1:** During inference, the layout provided by the user is not directly input into the model. The core of our proposed layout guidance method lies in editing the activation matrices of the cross-attention layers in each iteration of denoising to guide the model in generating the desired image. The layout serves as a reference for this editing operation.\n\nIn Equation 5, $\\eta(t)$ is a concave function designed as a noise scheduler referencing the pre-train model. It gradually decreases as $t$ decreases（from $T$ to 1). Its role is to weaken the intensity of layout guidance during the sampling process as time step $t$ decreases. In the sampling process of the diffusion model, as t decreases from $T$ to 1, the inference process can be understood as transitioning from determining high-level semantics to determining low-level semantics. The purpose of gradually decreasing $\\eta$ with $t$ is to prevent excessive guidance from the layout when t approaches 0, which could result in noticeable disharmonious artifacts in the generated images.\n\nWhen training the residual token embedding for a specific subject, layout is not required. Only a set of multi-view reference images of this subject is needed.\n\nWe will clearly add these details discription in the revision.\n\n#### **Q2: About supplementing ablation study.**\n**A2:** Thanks. Here, we present the quantitative results of three ablation studies. Firstly, we compare the approach of directly learning a text embedding like Textual Inversion with our design of learning residuals. The table below suggests that, without the residual design, the overall performance degrades given varying contexts.\n\n| Average CLIP Image Similarity | Ours | Learn Directly |\n| :-: | :-: |:-: |\n| Single-subject | **0.7949** | 0.6953 |\n| Two-subject    | **0.7075** | 0.6092 |\n\nSecondly, we present the quantitative results of the ablation experiments for the two operations in our layout guidance: strength and weaken attention activations. The visualizations of these results correspond to Fig. 6 in our paper.\n\n| Average CLIP Image Similarity | Ours | Only Strength | Only Weaken |\n| :-: | :-: |:-: |:-: |\n| Single-subject | 0.7949 | **0.8081** | 0.7988 |\n| Two-subject    | **0.7075** | 0.6736 | 0.6568 |\n\nFinally, we conducted additional ablation experiments on the text preservation loss weight $\\lambda$ we designed, and the quantitative results are as follows:\n\n| Average CLIP Image Similarity | $\\lambda=0$ | $\\lambda=0.5$ | $\\lambda=1.0$ | $\\lambda=2.0$ |\n| :-: | :-: |:-: |:-: | :-: |\n| Single-subject | 0.7900 | 0.7881 | **0.7949** | 0.7886 |\n| Two-subject    | 0.6196 | 0.6719 | **0.7075** | 0.6631 |\n\n#### **Q3: About limitations of the proposed method.**\n**A3:** Besides scaling up to more subjects, our method cannot strengthen the interaction relationships between multiple subjects. The difficulty in generating complex interaction relationships of the pretrained model is inherited in our method. This issue is shared by all existing customized methods. As shown in Fig. R9 **in the newly added PDF file**, in the case of straightforward interaction relationships, such as \"sit\" and \"wear,\" both our approach and the pretrained model achieve satisfactory generated results. However, for more intricate interaction relationships, such as \"handshake,\" the performance of both our approach and the pretrained model falls short of expectations. "}, {"review_id": "WPbkndO65i", "reviewer": "Reviewer_oYd1", "summary": "This paper aims to generate controllable images with multiple subjects as constraints. A residual token embedding is learned to shift the raw subject to the customized subject. A layout prior is further provided as the spatial guidance for subject arrangement. The experimental results demonstrate the effectiveness of the proposed method under a variety of settings.", "questions": "1. For an image with 6 or more subjects (e.g., Fig. 5), will the layout prior still be easy-to-obtain? It would be better to show all the predefined layouts in the figures.\n2. How to determine the number of steps of guidance to generate satisfactory results?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using residual token embedding for a specific subject is a simple and effective way to generate customized subjects. The residuals and layout priors could be further utilized to adjust the attention for multi-subject generation without retraining. Compared to existing works, the proposed method enables the generation of a greater number of subjects for multi-subject generation.", "weaknesses": "1. The key contribution of this paper is the ability to generate a greater number of subjects compared to existing works. However, in general, the maximum number of subjects that this paper can deal with is 6. I was wondering about the comparison results that contain more than 6 subjects.\n2. Different from existing works, this paper utilizes a predefined layout prior as the spatial guidance for multi-subject generation. Such result comparisons may be unfair, as the inputs of existing works do not contain the layout. Existing works that generate layouts from textual descriptions could be applied here to avoid predefining layouts.\n3. I cannot find any quantitative results for the ablation study. In addition, how to determine the term that controls the relative weight of the text-embedding-preservation loss?\n4. The authors state that they select cones as the baseline for customization with similar subjects, while Fig.4 shows the results of Dreambooth. I was wondering which baseline is actually used in this experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Author Response to Reviewer oYd1**\nWe greatly appreciate all of your valuable suggestions, which play a pivotal role in enhancing the quality of our paper. Bellow we address all your concerns.\n#### **Q1: About generated results with more than 6 subjects.**\n\n**A1:** Thanks. As shown in Fig. 3 in our paper, When dealing with three subjects, our method has already demonstrated significant superiority over existing methods. As for more subjects, it is limited by the capabilities of the pre-trained model itself. As shown in Fig. R5 **in  the newly added PDF file**, more involved subjects usually decrease the final visual quality. we observe significantly more failure cases when generating five or more subjects. We will include this in the revision. Several recent works [1, 2, 3] point out that stable diffusion is a struggle in generating multiple subjects. A potential way to ease this issue may be to apply our method to a better text-conditioned diffusion model.\n\n\n#### **Q2: About avoiding predefining layout.**\n\n**A2:** Thanks. Generating layouts from textual descriptions automatically [4,5,6,7] makes the entire process more convenient, and we will also include relevant discussions in the revision. \n\n#### **Q3: About showing quantitative results for the ablation study.**\n\n**A3:** Thanks. Here we present the quantitative results of the ablation experiments for the two operations in our layout guidance: strength and weaken attention activations. The visualizations of these results correspond to Fig. 6 in our paper.\n\n| Average CLIP Image Similarity | Ours | Only Strength | Only Weaken |\n| :-: | :-: |:-: |:-: |\n| Single-subject | 0.7949 | 0.8081 | 0.7988 |\n| Two-subject    | 0.7075 | 0.6736 | 0.6568 |\n\nIn addition, we conducted additional ablation experiments on the text preservation loss weight $\\lambda$ we designed, and the quantitative results are as follows:\n\n| Average CLIP Image Similarity | $\\lambda=0$ | $\\lambda=0.5$ | $\\lambda=1.0$ | $\\lambda=2.0$ |\n| :-: | :-: |:-: |:-: | :-: |\n| Single-subject | 0.7900 | 0.7881 | **0.7949** | 0.7886 |\n| Two-subject    | 0.6196 | 0.6719 | **0.7075** | 0.6631 |\n\nAs shown in the table, completely omitting the text-embedding preservation loss leads to a collapse in the performance of multi-concept customization. In practical implementation, we selected a loss weight of 1.0 for this regularization term.\n\n#### **Q4: About the baseline used in Figure 4.**\n\n**A4:** Thanks. Actually, we choose Cones as a baseline. In Fig. R11 **in the newly added PDF file**, we conduct a new experiment to compare the generation capabilities of DreamBooth, Cones, and our method in challenging cases.\n\n#### **Q5: Discussion about layout prior.**\n\n**A5:** In practice, users can simply select the customized subjects in the text prompt by clicking on them and then place and resize the bounding boxes accordingly. We show some examples of the bounding boxes we used in Fig. R3 **in the newly added PDF file** and will add all the bounding boxes we used in the revision.\n\n#### **Q6: About how to determine guidance steps.**\n\n**A6:** As shown in the two columns on the right in Figure 6 in our paper, for simple combinations like \"mug + teapot,\" satisfactory results could be achieved with 30 steps of guidance. However, for more challenging combinations such as \"cat + dog,\" 50 steps of guidance were required to achieve better attribute binding results.\n\n[1] Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis. Feng *et al.* ICLR'23.\n\n[2] Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. Chefer *et al.* SIGGRAPH'23.\n\n[3] MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation. Bar-Tal *et al.* ICML'23.\n\n[4] LayoutGPT: Compositional Visual Planning and Generation with Large Language Models. Feng *et al.* arXiv preprint arXiv:2305.15393.\n\n[5] Grounded Text-to-Image Synthesis with Attention Refocusing. Phung *et al.* arXiv preprint arXiv:2306.05427.\n\n[6] VisorGPT: Learning Visual Prior via Generative Pre-Training. Xie *et al.* arXiv preprint arXiv:2305.13777.\n\n[7] LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models. Lian *et al.* arXiv preprint arXiv:2305.13655.\n"}]

Author rebuttal:
## Author Response to All: 
Dear reviewers,

We thank all reviewers for their time and efforts in reviewing our paper. These constructive reviews can bring multiple improvements for our manuscript. We are encouraged that the reviewers appreciate our method, including
* innovatively designs [Reviewer Nfe2]
* a simple and effective method [Reviewer oYd1]
* outperform prior methods both quantitatively and qualitatively [Reviewer X4Jp,QhyH]
* well written and easy to follow [Reviewer X4Jp,QhyH,msmN]
* efficient storage and computational cost [Reviewer Nfe2,msmN]


We also have made diligent efforts to address all the concerns raised point by point. In this rebuttal, we have incorporated some new figures to more effectively address the concerns. Kindly review the newly uploaded one-page PDF.
* Figure R1 compares our method with Paint by Example. Our method can better preserve the identity of all customizable objects. [Reviewer X4Jp, Q2]
* Figure R2 gives generated images using specific semantic mask. We use a more specific semantic mask as a prior to get more fine-grained customizable objects. [Reviewer X4Jp, Q3]
* Figure R3 gives generated results using overlapping boxes. [Reviewer X4Jp, Q3; oYd1, Q5]
* Figure R4 gives generated results with inconsistent layouts and prompts. [Reviewer X4Jp, Q4]
* Figure R5 gives generated results with more customized subjects. [Reviewer X4Jp, Q5; oYd1, Q1]
* Figure R6 shows the effect of combining residual token embedding with textual inversion. [Reviewer QhyH, Q1]
* Figure R7 includes the examples of white puppy. [Reviewer QhyH, Q3]
* Figure R8 shows generated results of robot and human. [Reviewer Nfe2, Q1]
* Figure R9 shows generated results with interactions. [Reviewer Nfe2, Q2; msmN, Q3]
* Figure R10 compares our method with ELITE. [Reviewer Nfe2, Q3]
* Figure R11 compares our method with Dreambooth and Cones on some challenging cases. [Reviewer Nfe2, Q6]

We are open to discussions and addressing any issues from reviewers. Your constructive comments can further help us to improve our method. 

Sincerely yours,

Authors

Meta-review: This paper got positive recommendations: one "accept", three "weak accept", and one "borderline accept".

The reviews pointed out issues and concerns about the main contribution and possible comparisons with other methods. The authors' rebuttal emphasized the advantages of adopting the residual design and weakening the signal of irrelevant subjects. The rebuttal also provided additional results, including training time, quantitative evaluation with another metric, and ablation studies. 

The rebuttal seemed to successfully address the reviewers' questions and main concerns. After the rebuttal, both **Reviewer Nfe2** and **Reviewer oYd1** increased their rating to "weak accept". **Reviewer msmN** also mentioned increasing the score to "weak accept" in the discussion but somehow did not make it in the final rating.

The AC agrees with the reviewers' unanimous recommendations for accepting the paper. The AC recommends that the authors incorporate the reviewers' feedback and the additional results into the final version.
Decision: Accept (poster)