SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, to demonstrate the superiority of our approach, we conduct experiments on authentic images from diverse categories, including objects, pets, backgrounds, etc.. We not only present the qualitative results between our approach and other baselines but also showcase quantitative comparison. This further substantiates the effectiveness of our approach.
Qualitative comparison. As depicted in Fig. 3, we present a collection of generated images featuring two to four subjects. For single-subject generation, as shown in Appendix B.1, our approach achieves comparable results to competing methods while requiring significantly less storage space. However, as the number of subjects increases, the other three methods fail to include certain subjects and exhibit attribute confusion, resulting in generated images that deviate from the reference images. In contrast, our approach consistently produces highly visually accurate images for all subjects. It is important to note that our approach utilizes learned single-subject residual token embeddings for seamless combinations without retraining, thereby avoiding exponential training costs associated with the other methods. The next section will discuss this in detail.
Quantitative comparison. In the context of generating customized subjects with varying numbers, we have carefully selected four evaluation metrics: textual similarity, visual similarity, required storage space, and computational complexity. As shown in Tab. 1, for single-subject generation, our approach exhibits slightly lower visual and textual similarity compared to DreamBooth while
Figure 3: Qualitative comparison of multi-subject generation ability between our approach and baselines. Our approach surpasses existing alternatives with higher success rate in generating these subjects and less attribute confusion among different subjects.
remaining comparable to the other two methods. However, as the number of subjects increases, our approach consistently outperforms the other methods across all four evaluation metrics. This demonstrates its effectiveness in capturing subject characteristics, maintaining fidelity to the given prompt, and achieving higher efficiency in practical applications. Please note that the storage space and computational complexity presented in Tab. 1 for our approach assume that there are no existing learned single-subject residual token embeddings. However, in practice, if we already have required subject-specific residual token embeddings for multi-subject generation, no additional storage space or computational complexity is needed. This ability to seamlessly combine existing models without retraining is a unique advantage of our approach. In contrast, other methods require new storage space and training time for generating multiple new subjects.
User study. We conduct a user study to further evaluate our approach. The study investigates the performance of the four methods on the multi-object customization task. For each task and each method, we generated 80 images from 4 subjects combination, 4 conditional prompts, and 5 random seeds, resulting in 1,280 generated images for the whole user study. We presented two sets of questions to participants to evaluate image similarity and textual similarity. Taking the prompt "A cat and a dog on the beach" as an example, where "cat" and "dog" are customized subjects, we provided reference images of the customized subjects and asked participants questions like: "Does the image contain the customized cat?" to evaluate visual similarity. For textual similarity, based on the textual description, we presented questions like "Does the image contain a cat and a dog?" As shown in Tab. 2, our approach is most preferred by users regarding both image and text alignment.
4.3 Towards challenging cases
In this section, we further illustrate our superiority by showcasing two scenarios: generating a larger number of customized subjects and generating subjects with high semantic similarity that other methods fail to achieve.
Table 1: Quantitative comparisons. Our approach outperforms other methods in all aspects of multi-subject customization, particularly in three-subject and four-subject generation. The complexity metric is determined by calculating the number of fine-tuning iterations required for each method to generate a certain combination of  $n$  subjects.
Method	Text Alignment	Image Alignment	Storage	Complexity
Single Subject	DreamBooth [15]	0.314	0.727	3.3 GB	O(n)
Custom Diffusion [16]	0.327	0.721	72 MB	O(n)
Cones [17]	0.331	0.722	(1.43±0.34) MB	O(n)
Ours	0.330	0.725	4.8 KB	O(n)
Two Subjects	DreamBooth [15]	0.278	0.664	3.3 GB	O(n2)
Custom Diffusion [16]	0.284	0.676	72 MB	O(n2)
Cones [17]	0.292	0.685	(3.41±0.56) MB	O(n2)
Ours	0.309	0.708	9.6 KB	O(n)
Three Subjects	DreamBooth [15]	0.252	0.649	3.3 GB	O(n3)
Custom Diffusion [16]	0.270	0.658	72 MB	O(n3)
Cones [17]	0.281	0.663	(4.96±0.70) MB	O(n3)
Ours	0.304	0.689	14.4 KB	O(n)
Four Subjects	DreamBooth [15]	0.241	0.604	3.3 GB	O(n4)
Custom Diffusion [16]	0.254	0.623	72 MB	O(n4)
Cones [17]	0.271	0.638	(7.75±0.56) MB	O(n4)
Ours	0.299	0.673	19.2 KB	O(n)
Table 2: User study. The value represents the percentage of users that score positive for the image generated corresponding to the given questions. The results show that our approach is the most preferred by users for multi-subject customization, on both image and text alignment.
DreamBooth [15]	Custom Diffusion [16]	Cones [17]	Ours
Text Alignment	Image Alignment	Text Alignment	Image Alignment	Text Alignment	Image Alignment	Text Alignment	Image Alignment
Single Subject	71.35%	71.50%	76.85%	67.60%	76.85%	69.60%	75.05%	69.05%
Two Subjects	52.58%	43.13%	59.88%	46.83%	62.55%	57.50%	77.87%	69.75%
Three Subjects	57.78%	31.83%	58.20%	34.28%	64.87%	37.94%	79.20%	64.42%
Four Subjects	36.42%	25.63%	40.73%	25.44%	42.10%	28.75%	77.35%	59.08%
Customization with similar subjects. To illustrate the superiority of our approach in mitigating attribute confusion among multiple customized subjects, we select Cones, one of the methods that perform well in multi-subject generation among the three baselines, and compare the generated images with our approach in challenging cases shown in Fig. 4. We observe that when the raw categories of the customized subjects have high semantic similarities, especially in the case of two customized dogs, Cones exhibits a notable propensity for attribute confusion to arise. In contrast, our approach demonstrates excellent performance in both visual and textual similarities.
Customization with a large number of subjects. As shown in Fig. 3, we observe a significant decrease in the quality of generated images by other methods as the number of customized subjects increases. However, our approach shows a relatively smaller impact. Therefore, in Fig. 5, we present the generated images with an increased number of customized subjects, further demonstrating the effectiveness of our approach.
Verify the effect of strength and weaken cross-attention map. We conduct ablation experiments to examine the individual effects of strengthening the target subject region and weakening irrelevant subject regions as shown in Fig. 6. We observe that strengthening the target subject alone can lead to attribute confusion between subjects. For instance, it may result in a customized subject exhibiting attributes of another subject. However, when only irrelevant subjects are weakened, certain subjects may fail to show up or exhibit a lack of specific attributes. Furthermore, for simple combinations like "mug + teapot," satisfactory results could be achieved with 30 steps of guidance. However, for more challenging combinations such as "cat + dog," 50 steps of guidance were required to achieve better attribute binding results.
Customized subjects
Cones
A dog and a cat in the forest.
A dog and a dog in the forest.
A dog and a cat in the forest.
Ours
A watercolor painting of dog and a real cat on the grass.
A dog on the floor and a cat on the table.
A dog and a dog on the beach.
A dog and a dog under pyramid.
Customized subjects
Figure 5: Visualizations of customization with a large number of subjects. Here we show diverse generation results of customizing 5 and 6 subjects.
Figure 4: Visualizations of customization with challenging cases. When the subjects that need to be customized belong to the category with high semantic similarity (shown in the first row) or even the same category (shown in the second row), the baseline using joint training has a serious attribute confusion problem, while our approach circumvents this problem.
A monster toy, a teapot, and a mug with a flower in it, on the tabel, with a barn in the background.
A duck toy, a pair of sunglasses, a mug with a flower in it, and a teapot, on the tabel.
A dog with sunglasses on face and a hat on head next to a flower with a barn in the background.
A cat with sunglasses on face and a hat on head sitting on a chair with a lake in the background.
A cat with sunglasses on face and a hat on head sitting next to a chair with a barn in the background.
A dog with sunglasses on face and a hat on head, sitting next to a dog, with sunglasses on face, with a lake in the background.
Verify the effect of guidance in generation process. Recent studies [37, 38] have also demonstrated that incorporating guidance during the sampling process leads to superior generation results. We select Cones [17], which exhibits relatively better performance among competing methods, as our baseline, and compare it with the state-of-the-art semantic guidance approach as well as our guidance approach. As shown in Fig. 7, combining the Attend and Excite [38] improves the generation quality compared to cones, and further improvement is achieved when combine with our guidance approach. However, overall, our approach outperforms others, showcasing the best performance. This proves that the problem of attribute confusion in other methods can't be solved by simply adding a guidance algorithm while combining our residual token embedding with our guidance algorithm can solve it.
Strengthen only
Weaken only
T=30
T=50
Figure 6: Ablation study on the effectiveness of strengthening and weakening cross-attention map. The first column shows that only strengthening leads to attribute confusion. The second column shows only weakening leads to some subjects failing to show up. The last two columns show that those challenging cases require longer guidance.
A mug and a teapot on the table.
A dog and a cat under mount fuji.
Cones w/ Att
Cones w/ Layout
Figure 7: Qualitative comparison for ablations on Attend-and-Excite [38] and our layout guidance. "Att" refers to Attend-and Excite and "Layout" refers to our guidance method. These results show that the baseline method obtained through joint training cannot avoid attribute confusion by simply combining a certain guidance method. Correspondingly, our method can solve this problem.
Ours
A dog and a dog in the garden on a sunny day.
This paper proposes a novel approach for multi-subject customization. Our method combines subject-specific residual token embeddings with a pre-trained diffusion model and utilizes easy-to-obtain layout prior to guiding the generation process. This allows us to combine individually learned subject-specific residual token embeddings for multi-subject customization without retraining. Our method consistently delivers exceptional performance even in challenging scenarios, including the customization of image synthesis with six subjects and the customization of semantically similar subjects. Through qualitative and quantitative experiments, we demonstrate our superiority over existing state-of-the-art methods in various settings of multi-subject customization. These results highlight the effectiveness and robustness of our method.
This work is supported by National Key R&D Program of China under Grant 2020AAA0105701 and Alibaba Group through Alibaba Research Intern Program.
[40] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. arXiv preprint arXiv:2211.13227, 2022.
[41] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint arXiv:2302.13848, 2023.
[42] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models. arXiv preprint arXiv:2302.12228, 42:150:1–150:13, 2023.
We supplement the experimental details of each baseline method and our method in this section. For better generation quality, we use Stable Diffusion v2-1-base as the pre-trained model. For a fair comparison, we use 50 steps of DDIM [33] sampler with a scale of 7.5 for all the above methods. All experiments are conducted using one A-100 GPU.
Textual Inversion [18]. We use the third-party implementation of huggingface [39] for Textual Inversion. We train each subject-specific token with the recommended $^2$  batch size of 4 and a learning rate of 0.002 for 3000 steps. In particular, we initialize the subject-specific token with the corresponding class token. For example, to customize a specific cat, we initialize the subject-specific token "<cat>" with the original "cat" token.
DreamBooth [15]. We use the third-party implementation of huggingface [39] for DreamBooth. Training is with a batch size of 2, learning rate  $5 \times 10^{-5}$ , and training steps of  $800 \times$  number of subjects. Custom Diffusion [16]. We use the official implementation<sup>3</sup> for Custom Diffusion. Training is with a batch size of 2, learning rate  $1 \times 10^{-5}$  and training steps of  $250 \times$  number of subjects.
Cones [17]. We use the official implementation for Cones. The batch size of the training stage is set to 2, the learning rate is  $4 \times 10^{-5}$  and training steps is  $1200 \times$  number of subjects.
Ours. For our approach, We train each subject-specific residual token embedding with a batch size of 1 and a learning rate of  $1 \times 10^{-6}$  for 3,000 steps. At inference time, the layouts are appointed by bounding boxes given by the users to indicate the location of each subject. We use a positive value of  $+2.5$  to strengthen the signal of the target subject and we use a negative value of  $-1 \times 10^{-5}$  to weaken the signal of irrelevant subjects. Furthermore, we guide all 50 steps with the layout guidance in the whole generation process to get good customized generation results.
User study. For two- to four-subject generation tasks, we design four different subject combinations for each task. This will yield 12 subject combinations in total. For each subject combination, we design four different text prompts to generate images with 5 random seeds. We conduct this procedure to all four methods. With such settings, each method generates 80 different images for each task. We give each generated image 4-8 questions for testing image alignment (2-4 questions) and text alignment (2-4 questions). The number of questions is proportional to the number of subjects used to customize the image (average 6 questions per generated image). Finally, we shuffle the order of all the image-question pairs and assigned them to 25 different users for scoring, and finally summarized the results. In detail, every user needs to score  $4 \times 4 \times 5 \times 6 = 480$  questions for each task and for each method.
B More results
In this section, we present additional visualization results that provide a comprehensive demonstration of the superiority of our proposed method.
B.1 More comparisons with other baselines
We conduct further comparison between our approach and three other baselines. As shown in Fig. A1, regarding the generation of single subjects, the four methods exhibited similar performance. However, when dealing with semantically similar subjects, such as a dog and a cat, as well as scenarios involving three or more subjects, our approach clearly exhibit superior performance. Moreover, as shown in Fig. A2, we further showcase additional generated results, providing further evidence of the robustness of our method.
<sup>1</sup>https://huggingface.co/stabilityai/stable-diffusion-2-1
$^{2}$ https://github.com/rinongal/textual_inversion
<sup>3</sup>https://github.com/adobe-research/custom-diffusion
Customized subjects
DreamBooth
Custom Diffusion
A monster toy in Times Square.
A dog on the table and a cat on the ground.
A dog with sunglasses and a mug on the table.
Figure A1: More comparisons of our approach and existing baselines.
A dog with sunglasses, watching a flower in a mug, on Mars.
B.2 More challenging cases
As shown in Fig. A3, we present a larger number of images generated by our approach, featuring a greater diversity of customized subjects. In comparison with other methods, we observe that when the number of customized subjects reaches four, the performance of other methods significantly deteriorates. In contrast, our approach can generate a larger number of customized subjects, exemplifying the superiority.
B.3 Comparison with image composition method
we select Paint by example [40] as a baseline of image composition methods. In order to compare the effect of the baseline and our method more intuitively, we use Paint by example to inpaint the image generated by our method with the reference image as input. We conduct visualization results in Fig. A4. Generated results of our method have better visual similarity, and we will provide more comparisons in the revision. Besides, We consider multi-subject customized generation is different with image composition in two aspects. The purpose of multi-subject customized generation is to implant all user-provided subjects into the diffusion model; so that the model can generate various images of all subjects vividly guided by prompts. However, the purpose of image compositing is to insert an object into another image in a realistic way, without guidance from prompts. In addition, to compose multiple subjects by the image composition methods, multiple iterations of inference are required, especially when there is an overlapping relationship among subjects.
A monster toy under the water.
A chair under the mount fuji.
A cat and a dog on the beach.
A teapot and a mug on the grass.
A teapot and a mug with a flower on the table.
Figure A2: More results of our approach.
A dog wearing sunglasses, sitting in a mug, on the table.
B.4 More fine-grained control
Actually, users have the flexibility to input a more specific mask, enabling them to achieve fine-grained control. Experimental results in Fig. A5 demonstrate that our method can use specific mask for sampling.
B.5 About generated results with more subjects
As for more subjects, it is limited by the capabilities of the pre-trained model itself. As shown in Fig. A6, more involved subjects usually decrease the final visual quality. We observe significantly more failure cases when generating five or more subjects. Several recent works [17, 16, 38] point out that stable diffusion is a struggle in generating multiple subjects. A potential way to ease this issue may be to apply our method to a better text-conditioned diffusion model.
Figure A3: More results of multi-subject generation.
A cat with sunglasses, sitting on a chair, with a barn in the background.
A duck toy, a mug, a teapot, and a sunglasses on the table.
A monster toy, a dog, and a teapot, with a barn in the background.
A dog with sunglasses and a hat, sitting next to a monster toy, with a lake in the background.
A monster toy wearing sunglasses, a hat, sitting next to a dog, with a barn in the background.
A dog with sunglasses, sitting in a mug, with a lake in the background.
A dog with sunglasses and a dog with sunglasses on the grass.
A duck toy, a teapot, and a mug, with a barn in the background.
A dog sitting next to a chair with sunglasses on it, with a lake in the background.
A dog with sunglasses and a hat, sitting next to a duck toy, with a lake in the background.
A monster toy, a teapot, and a mug with a flower in it, with a barn in the background.
A dog wearing sunglasses, a hat, sitting next to a dog wearing sunglasses, with a lake in the background.
B.6 Comparison with fast customized methods
Currently, some works [41, 42] focus on achieving faster customized generation by pretraining an image encoder and using it to encode a reference image during generation. However, these methods share some common issues. They require collecting data in advance to train the encoder, which can limit their generalization. We compare our method with Elite in Fig. A7. Compared to methods that fine-tune for each individual subject, these approaches may exhibit difficulty in preserving fine details. In addition, these methods can only encode one image at a time, they cannot customize multiple subjects simultaneously.
Figure A4: Comparisons with Paint by Example.
A dog and a cat on the beach.
Figure A5: Generated results using specific mask. We use a more specific mask as a prior to get more fine-grained generated results.
A dog and a cat on the grass.
C Importance of residual token embedding
To demonstrate the superior generalization of the residuals, we conduct comparative experiments. As shown in Tab. A1, compared to directly updating the class embedding parameters in a single text embedding, our approach, which involves updating the text encoder and calculating the average shift from the class to the specific subject based on a certain number of text templates, outperforms in both textual and visual similarity.
C.1 Generated results of textual inversion
We observe from Fig. A8 that Textual Inversion [18] struggles with the generation of complex single subject and multiple subjects.
C.2 About difference between residual token embedding and Textual Inversion
Textual inversion find a single word embedding (input of the text encoder) to represent user-provided subject. Different from textual inversion, we find the residual token embedding for each subject, and add these residuals to their corresponding token embedding (output of the text encoder). The "residual" actually refers to the ability to transform a subject of one category into a customized subject that we need, for example a "random dog" to the specific "customized dog". As shown in Fig. A9, we train model using method in textual inversion but learn a text embedding (output of the text encoder) and apply our layout guidance approach. We can see that simply employing Textual Inversion to learn a customized text embedding does not adequately fulfill the customized requirements. This approach fails to fully capture all the features of the reference subject, specially when dealing with multi-subject customization. Even with the utilization of our layout guidance method, better results cannot be achieved.
D Social impact and limitations
Social impact. While training individual large-scale diffusion models remains prohibitively expensive, advancements in fine-tuning techniques have enabled individual users to customize their own models. our approach empowers users to linearly combine their personalized single-subject models, generating high-quality images with multiple customized subjects while maintaining significant advantages in terms of computation and storage efficiency. Furthermore, there is a growing need for more reliable detection techniques to identify and mitigate the presence of fake data.
Figure A6: Generated results with more customized subjects. "Pre-trained model + Guidance" refers to apply our guidance approach to pre-trained model.
Figure A7: Comparisons with ELITE and our approach. Our method better preserves the identity of all subjects.
Limitations. our approach is limited by the inherent capabilities of the base model. Specifically, when it comes to combining more than six subjects, our approach may not be able to consistently generate satisfactory results. In order to achieve the desired generation results, the provided layout by the user needs to be roughly consistent with the textual description.
A monster toy in Times Square.
A candle and a teapot on the table.
A monster toy next to a duck toy, with a barn in the background.