SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiheng Liu $^{1*†}$  Yifei Zhang $^{2*†}$  Yujun Shen $^{3}$  Kecheng Zheng $^{3}$  Kai Zhu $^{1,4}$  Ruili Feng $^{1,4}$  Yu Liu $^{4}$  Deli Zhao $^{4}$  Jingren Zhou $^{4}$  Yang Cao $^{1‡}$  USTC  $^{2}$ SJTU  $^{3}$ Ant Group  $^{4}$ Alibaba Group
Synthesizing images with user-specified subjects has received growing attention due to its practical applications. Despite the recent success in single subject customization, existing algorithms suffer from high training cost and low success rate along with increased number of subjects. Towards controllable image synthesis with multiple subjects as the constraints, this work studies how to efficiently represent a particular subject as well as how to appropriately compose different subjects. We find that the text embedding regarding the subject token already serves as a simple yet effective representation that supports arbitrary combinations without any model tuning. Through learning a residual on top of the base embedding, we manage to robustly shift the raw subject to the customized subject given various text conditions. We then propose to employ layout, a very abstract and easy-to-obtain prior, as the spatial guidance for subject arrangement. By rectifying the activations in the cross-attention map, the layout appoints and separates the location of different subjects in the image, significantly alleviating the interference across them. Both qualitative and quantitative experimental results demonstrate our superiority over state-of-the-art alternatives under a variety of settings for multi-subject customization. Project page can be found here.
The remarkable achievements of text-to-image generation models [1-14], have garnered widespread attention due to their ability to generate high-quality and diverse images. To allow synthesizing images with user-specified subjects, customized generation techniques [15-17] propose to fine-tune the pre-trained models on a few subject-specific images. Despite the notable success in single subject customization [15-20], multi-subject customization remains seldom explored but better aligns with the practical demands in real life.
Recent studies [16, 17] have investigated multi-subject customization through joint training, which tunes the model with all subjects of interest simultaneously. Such a strategy has two drawbacks. First, they require learning separate models for each subject combination, which may suffer from exponential growth when the number of subjects increase. For example, the customization of objects  $\{\mathsf{A},\mathsf{B},\mathsf{C}\}$  fails to inherit the knowledge obtained from the customization of objects  $\{\mathsf{A},\mathsf{B}\}$ . Second, different subjects may interfere with each other, causing the issues that some subjects fail to show up in the final synthesis or the subject attribute gets confused among subjects (e.g., a cat with the features of another dog). This phenomenon is particularly evident when the semantic similarity between subjects is high (see Fig. 4).
†Work performed during internship at Alibaba DAMO Academy.
$\ddagger$  Corresponding author.
sunglasses sunglasses
A dog wearing sunglasses, a hat, sitting next to a dog wearing sunglasses, with a barn in the background.
A dog wearing sunglasses, a hat, sitting next to a dog wearing sunglasses, with a lake in the background.
Figure 1: Customizable image generation with the subjects listed on the left. Cones 2 is highlighted from three aspects. (1) Using a simple yet effective representation to register a subject, we can compose various subjects arbitrarily without any model tuning. (2) Employing spatial layout, which is very easy to obtain in practice, as a guidance, we can control the specific location of each subject and meanwhile alleviate the interference across subjects. (3) Our method achieves appealing performance even under some challenging settings, such as customizing the synthesis with six or more subjects and exchanging the sunglasses on the two dogs.
In this work, we present Cones 2, a novel approach for multi-subject customization using a pre-trained text-to-image diffusion model. Our method utilizes a simple yet effective representation to register a subject and enables the arbitrary composition of various subjects without requiring any model retraining. To that end, we decompose the challenging task of multi-subject customization into two components: how to efficiently represent a subject and how to effectively combine different subjects. Given a set of subjects and their photos (3-5 for each), our goal is first to bind the characteristic of each specific subject to a "plugin" that can be used flexibly. Driven by this, we fine-tune the text encoder part of a pre-trained text-to-image diffusion model with images of a specific subject, making the tuned model can customize this specific subject. Moreover, we propose a text-embedding-preservation loss, which limits the output of the tuned text encoder to only differ from the original text encoder in token embedding regarding the specific subject. Then we calculate the mean difference between the tuned text encoder with the original text encoder to derive the residual token embedding which can robustly shift the raw category to the customized subject (e.g.,  $\mathrm{dog} \to$  customized dog).
To effectively combine different subjects, we propose a layout guidance method to control the generation process. More formally, we employ pre-defined layout, a very abstract and easy-to-obtain prior, to guide different subjects to show up in different positions by rectifying the activation in the cross-attention maps. We encourage all subjects to show up in the final synthesis by strengthening the activations of the target subject. Simultaneously, to prevent the subject attribute gets confused, we weaken the activations of the irrelevant subjects. In addition, to make this easy to implement in practice, we define the layout as a set of subject bounding boxes with subject annotation, which describes the spatial composition of the customized subjects and is easy for users to specify in advance. Through our method, users can compose various subjects arbitrarily with a pre-defined layout (see Fig. 1).
Our method is evaluated under a variety of settings for multi-subject customization involving extensive subject categories such as pets, scenes, decorations, etc. Qualitative and quantitative results demonstrate that, compared to existing baselines, our method exhibits competitive performance in terms of both text alignment to input prompt and visual similarity to the target images. It is noteworthy that our method even facilitates the customization of a larger number of subjects (e.g., six subjects in Fig. 1), which is a far more challenging setting in practice.
Large-scale text-conditioned image synthesis. Synthesizing images from the language description has received growing attention due to its ability to generate high-quality and diverse images. Earlier
works [21] explored the utilization of language description into GAN as a condition on specific domains under the closed-world assumption. With the development of diffusion models [22, 23] and large-scale multi-modality models [24], text-conditioned image synthesis has shown remarkable improvement in an open-vocabulary text description. Specifically, GLIDE [3], DALLE2 [4], StableDiffusion [5] and Imagen [6] are representative diffusion models that can produce photorealistic outputs. Autoregressive models such as DALLE [1], Make-A-Scene [2], CogView [25] and Parti [26] have also shown exciting results. Although these models demonstrate an unparalleled ability to synthesize images, they require time-consuming iterative processes to achieve high-quality image sampling. Recent large text-to-image GAN models such as StyleGAN-T [7], GALIP [27], and GigaGAN [8] also demonstrated unprecedented semantic generation, which is orders of magnitude faster when sampling.
Customized image generation. Thanks to the significant progress of large-scale text-to-image models, users can adopt these well-trained models to generate customized images with user-specified subjects. There are two earliest attempts to solve the customized generation through few-shot images of one specific subject, i.e. Text Inversion [18] and DreamBooth [15]. Concretely, Text Inversion [18] represents a new subject by learning an extra identifier word and adding this word to the dictionary of the text encoder. DreamBooth [15] binds rare new words with specific subjects through few-shot fine-tuning the whole Imagen [6] model. To compose multiple new concepts together, Custom [16] chooses to only optimize the parameters of the cross-attention in the StableDiffusion [5] model to represent new concepts and then joint trains for the combination of multiple concepts. In addition, Cones [17] associates customized subjects with activating a small cluster of neurons in the diffusion model. Although both Custom [16] and Cones [17] have explored combination multi-subject customization, they suffer from high training costs and low success rates along with the increased number of subjects. In this work, we study how to efficiently represent a particular subject as well as how to appropriately compose different subjects. Specifically, learning a residual on top of the base embedding can represent a new concept, and the introduction of layout into the attention map can help the model generate more accurate user-specified subjects. We find these design choices lead to better results in directly composing different subjects than joint training.
Spatial guidance in diffusion models. To further enhance the controllability of synthesizing images, some works [28-32] have tried to explore how to guide the generation process by more spatial information. Composer [28] directly adds spatial information as a condition input during the training phase. ControlNet [29] and T2I-Adapters [30] add spatial information to the pre-trained model by training a new adapter. Prompt-to-prompt [31] presents a training-free edit method by editing the cross-attention. In addition, a diffusion-based image translation [32] keeps the generated spatial structure by limiting the cross-attention map. Inspired by these works, we also adopt the layout as the spatial guidance for subject arrangement that can well appoint and separate the location of different subjects in the image, significantly alleviating the interference across them.
Given a set of subjects and their photos (3-5 for each) from different views, we aim to generate new images of any combination containing those subjects vividly and precisely. We accomplish this by combining subject-specific residual token embeddings with a pre-trained diffusion model and guiding the generation process with a layout. The overall framework is presented in Fig. 2. Specifically, we represent each subject as a residual token embedding shifted from its base category. Adding the residual token embedding to the base category embedding can yield the corresponding subject in the generated images. We present how to get this residual token embedding in Sec. 3.2. At inference time, subjects failing to show up and the subject attribute getting confused among subjects are two key problems in multi-subject customized generation. To address these issues, we present a method of composing subjects by leveraging layout guidance in Sec. 3.3.
3.1 Text-conditioned diffusion model
Diffusion models learn a data distribution by the gradual denoising of a variable sampled from a Gaussian distribution. This corresponds to learning the reverse process of a fixed-length Markov chain. In text-to-image tasks, the training objective of a conditional diffusion model  $\epsilon_{\theta}$  can be simplified as a reconstruction loss,
$$
L _ {\text {r e c}} = \mathbb {E} _ {\mathbf {x}, \mathbf {c}, \epsilon \sim \mathcal {N} (0, 1), t} [ \| \epsilon_ {\theta} (\mathbf {x} _ {t}, E (\mathbf {c}), t) - \epsilon \| _ {2} ^ {2} ], \tag {1}
$$
(b)
Figure 2: Illustration of the proposed approach. (a) We first learn a residual token embedding (e.g.,  $\Delta_{\text{dog}}^{\text{custom}}$ ) on top of the base embedding to register a user-specified subject, which allows composing various subjects arbitrarily without further model tuning. (b) Given a layout as the spatial guidance, we then arrange the subjects by rectifying the activations in cross-attention maps, which enables the control of the location of each subject and reduces the interference between them.
where  $t \sim \mathcal{U}([0,1])$  is the time variable,  $E$  is a pre-trained text encoder and  $\mathbf{x}_t = \alpha_t\mathbf{x} + \sigma_t\epsilon$  is a noised image from the ground-truth image  $\mathbf{x}$ . The parameters  $\alpha_{t}$  and  $\sigma_{t}$  are coefficients formulating the forward diffusion process. The model  $\epsilon_{\theta}$  is conditioned on the text embedding  $E(\mathbf{c})$  and  $t$ . The text embedding  $E(\mathbf{c})$  is injected into the model  $\epsilon_{\theta}$  through the cross-attention mechanism. At inference time, the network  $\epsilon_{\theta}$  is sampled by iteratively denoising  $\mathbf{x}_T \sim \mathcal{N}(0,\mathbf{I})$  using either deterministic samplers [33-35] or stochastic sampler [23].
3.2 Representing subjects with residual token embedding
Representing subjects with residual text embedding. Our goal is first to represent each subject with a residual text embedding among the output domain of the text encoder. An ideal residual text embedding  $\Delta^{\mathrm{custom}}$  that can robustly shift the raw category to a specific subject. For example, the model  $\epsilon_{\theta}$  with embedding input  $(E("a\text{photo of dog}) + \Delta_{\text{dog}}^{\mathrm{custom}})$  can truly generate a photo of specific "dog". One way to get this objective is to calculate an embedding direction vector [32] from a source (original) text encoder to the target (fine-tuned) text encoder  $E^{\mathrm{custom}}$ . The fine-tuned text encoder  $E^{\mathrm{custom}}$  needs to be able to customize subject  $s$  combined with the original diffusion model  $\epsilon_{\theta}$ . Similarly as DreamBooth [15],  $E^{\mathrm{custom}}$  can be trained with the subject-preservation loss, as
$$
L _ {\mathrm {s u b}} \left(E ^ {\text {c u s t o m}}\right) = \mathbb {E} _ {\left(\mathbf {x}, \mathbf {c}\right) \sim D _ {s}, \epsilon \sim \mathcal {N} (0, 1), t} \left[ \| \epsilon_ {\theta} \left(\mathbf {x} _ {t}, E ^ {\text {c u s t o m}} (\mathbf {c}), t\right) - \epsilon \| _ {2} ^ {2} \right], \tag {2}
$$
where  $D_{s} = \{(\mathbf{x}_{j}^{s},\text{"a photo of } s^{\prime \prime})|\mathbf{x}_{j}^{s}\in X^{s}\}$  is the reference few-shot data of subject  $s$
Regularization with a text-embedding-preservation loss. The residual text embedding obtained according to the previous section can only perform single-subject customized generation. Since those residual text embeddings are applied to the entire text, any two of them can admit significant conflicts so that they cannot be combined together directly while carrying out inference. Therefore, we propose a text-embedding-preservation loss to make the residual text embedding mainly act on the text embedding regarding the subject token. The core idea is to minimize the difference between  $E^{\mathrm{custom}}$  and  $E$  for tokens apart from the subject token  $s$ . Take the "dog" case above as an example, we sample 1,000 sentences  $C_{\mathrm{dog}} = \{\mathbf{c}^i\}_{i=1}^{1000}$  containing the word "dog" using ChatGPT [36], like "a dog on the beach", and then minimize the difference between  $E^{\mathrm{custom}}$  and  $E$  for all the token besides "dog". In detail, given any caption (e.g.  $\mathbf{c} =$  "a dog on the beach"), we split its text embedding into a sequence  $(E(\mathbf{c}) = (E(\mathbf{c})_{\mathrm{a}}, E(\mathbf{c})_{\mathrm{dog}}, \dots, E(\mathbf{c})_{\mathrm{beach}}))$ . Then we wish  $\| E(\mathbf{c})_p - E_p^{\mathrm{custom}(\mathbf{c})} \|_2^2 = 0$  for any  $p$  that is not equal to "dog". Namely, the text-embedding-preservation loss is a regularization, as
$$
L _ {\mathrm {r e g}} \left(E ^ {\text {c u s t o m}}\right) = \mathbb {E} _ {\mathbf {c} \sim C _ {\mathrm {d o g}}} \left[ \sum_ {p \in \mathbf {c}, p \neq s} \| E ^ {\text {c u s t o m}} (\mathbf {c}) _ {p} - E (\mathrm {c}) _ {p} \| _ {2} ^ {2} \right], \tag {3}
$$
Algorithm 1 N-Subject Customization with Layout Guidance
Require: Prompt c, customized set  $S = \{s_i\}_{i=1}^N \subset \mathbf{c}$ , pre-trained residual token embeddings  $\{\Delta_{s_i}^{\text{custom}}\}_{i=1}^N$ , guidance layout  $\mathbf{M} = \{\mathbf{M}_s : s \in S\}$ .
1: Edit the text embedding:  $E^{\mathrm{final}}(\mathbf{c}) = E(\mathbf{c}) \oplus \{\Delta_{s_i}^{\mathrm{custom}}\}_{i=1}^N$ ;
2: Define guidance layout for each  $s \in S$  from  $\mathbf{M}$ :
$$
\mathbf {M} _ {s} (i, j) = \left\{ \begin{array}{l l} \gamma^ {+} & (i, j) \in R _ {s} ^ {\text {s h o w}} \\ \gamma^ {-} & (i, j) \in R _ {s} ^ {\text {i r r e l e v a n t}} \\ 0 & \text {O t h e r w i s e} \end{array} \right.
$$
3: Sample  $\mathbf{x}_{\mathbf{T}}\sim \mathcal{N}(0,\mathbf{I})$
4: for  $t = T, T - 1, \ldots, 1$  do
5:  $\mathbf{CA} \gets \epsilon_{\theta}(\mathbf{x}_t, E^{\mathrm{final}}(\mathbf{c}), t)$ ;
6:  $\mathbf{CA}_{\mathrm{edited}} \gets \mathrm{EditedCA}(\mathbf{CA}, \mathbf{M}, \mathbf{c})$ ;
7:  $\mathbf{x}_{t - 1}\gets \epsilon_{\theta}(\mathbf{x}_t,E^{\mathrm{final}}(\mathbf{c}),t)\{\mathbf{CA}_{\mathrm{edited}}\} .$
8: end for
where  $p$  traverses all tokens inside sentence  $\mathbf{c}$  except the subject token  $s$ . Our complete training objective then comes as
$$
L = L _ {\text {s u b}} + \lambda L _ {\text {r e g}}, \tag {4}
$$
where  $\lambda$  controls for the relative weight of the text-embedding-preservation term. As shown in Fig. 2a, after the customized text encoder is obtained, we derive the residual token embedding of "dog" via computing the average shift of  $E^{\mathrm{custom}}(\mathbf{c})_{\mathrm{dog}}$  over these 1,000 sentences from  $E(\mathbf{c})_{\mathrm{dog}}$ , as
$$
\Delta_ {\mathrm {d o g}} ^ {\text {c u s t o m}} = \frac {1}{| C _ {\mathrm {d o g}} |} \cdot \sum_ {\mathbf {c} \in C _ {\mathrm {d o g}}} \left(E ^ {\text {c u s t o m}} (\mathbf {c}) _ {\mathrm {d o g}} - E (\mathbf {c}) _ {\mathrm {d o g}}\right). \tag {5}
$$
Inference with residual token embedding. The residual token embedding we get aforementioned can be used directly in any subject combinations involving them without further tuning. As shown in Fig. 2b, when we do customized generation with  $N$  specific subjects  $s_1, s_2, \dots, s_N$ , all we need is to fetch the pre-computed  $\Delta_{s_1}^{\text{custom}}$ ,  $\Delta_{s_2}^{\text{custom}}$ ,  $\dots$ ,  $\Delta_{s_N}^{\text{custom}}$  and add them to the token embedding, as
$$
E ^ {\text {f i n a l}} (\mathbf {c}) _ {s _ {i}} = E (\mathbf {c}) _ {s _ {i}} + \Delta_ {s _ {i}} ^ {\text {c u t o m}}, i = 1 \dots , N. \tag {6}
$$
In fact, the operation in Eq. (6) is all in the token dimension. This characteristic endows our method with significant convenience and high efficiency for large-scale applications. On the one hand, any pre-trained residual token embedding  $\Delta_{i}^{custom}$  can be used repeatedly and combined with another  $\Delta_{j}^{custom}$ . On the other hand, for each subject, we merely need to store a float32 vector, getting rid of storing large parameters as in previous methods [15-17].
3.3 Composing subjects with layout guidance
The text-to-image diffusion models [4-6] commonly inject the text embedding  $E(\mathbf{c})$  to its diffusion model  $\epsilon_{\theta}$  via the cross-attention mechanism. The attention map among a cross-attention layer is  $\mathbf{CA} = (\mathbf{W}_Q \cdot \varphi(\mathbf{x}_t)) \cdot (\mathbf{W}_K \cdot E(\mathbf{c}))$ , where  $\varphi(\mathbf{x}_t)$  denotes the transformed image feature and  $\mathbf{W}_Q, \mathbf{W}_K$  denotes the parameters for computing query and key. The cross-attention map directly affects the spatial layout of the final generation [31]. Below we will discuss how to improve the quality of customized generation by rectifying the activations in the cross-attention map.
Strengthening the signal of target subject. One issue in multi-subject customization is that some subjects may fail to show up. We argue that this is caused by insufficient activations in the cross-attention map of these subjects. To avoid this, we choose to strengthen the signal of the target subject in the region where we want it to show up.
Weakening the signal of irrelevant subject. Another issue in multi-subject customization is that the subject attribute gets confused among subjects, i.e. the subjects in generated images may contain characteristics from the other subjects. We argue that this is due to the overlapping activation regions of different subjects in the cross-attention map. To avoid this, we choose to weaken the signal of each subject appearing in the region of the other subjects.
Layout-guided iterative generation process. Combining the above two ideas, we present a method to guide the generation process according to a pre-defined layout  $\mathbf{M}$ . In practice, we define the layout  $\mathbf{M}$  as a set of subject bounding boxes and then get the guidance layout  $M_{s}$  for each subject  $s$ . In detail, as shown in Fig. 2b we divide  $M_{s}$  into different regions: we set the value of  $M_{s}$  to a positive value  $\gamma^{+} \in \mathbb{R}^{+}$  in the region where we want the subject  $s$  to show up (denote as  $R_{s}^{\mathrm{show}}$ ) and set the value of  $M_{s}$  to a negative value  $\gamma^{-} \in \mathbb{R}^{-}$  in the region that is irrelevant to the subject  $s$  (denote as  $R_{s}^{\mathrm{irrelevant}}$ ). At the inference time, we replace all the output of cross-attention with edited results at every generation step, as
$$
\operatorname {E d i t e d C A} (\mathbf {C A}, \mathbf {M}, \mathbf {c}) = \operatorname {S o f t m a x} (\mathbf {C A} \oplus \left\{\eta (t) \cdot \mathbf {M} _ {s _ {i}} | i = 1, \dots , N \right\}) \cdot \left(\mathbf {W} _ {V} \cdot E (\mathbf {c})\right), \tag {7}
$$
where  $\oplus$  denotes the operation that adds the corresponding dimension of CA and M, which is also visualized in Fig. 2b and  $\eta(t)$  is a concave function controlling the edit intensity at different time  $t$ . The implementation details refer to Algorithm 1.
4.1 Experimental setups
Datasets. For fair and unbiased evaluation, we select subjects from previous papers [15, 18, 16, 17] spanning various categories for a total of 15 customized subjects. It consists of two scenes, five pets and eight objects. We perform extensive experiments on various combinations of subjects, explaining the superiority of our approach.
Evaluation metrics. We evaluate our approach with two following metrics for customized generation proposed in Textual Inversion [18]. (1) Image similarity, which measures the visual similarity between the generated images and the target subjects. For multi-subject generation, we calculate the image similarity of the generated images and each target subject separately and finally calculate the mean value. (2) Textual similarity, which evaluates the average CLIP [24] similarity between all generated images and their textual prompts. To this end, we use a variety of prompts with different settings to generate images, including modifying the scenes, attributes, and relation between subjects.
Baselines. To evaluate our generation quality, we compare our approach with three state-of-art competitors, i.e., DreamBooth [15] that fine-tunes all parameters in diffusion model; Custom diffusion [16] that optimizes the newly added word embedding in text encoder and a few parameters in diffusion model, namely the key and value mapping from text to latent features in the cross-attention; and Cones [17] that finds a small cluster of neurons in diffusion model corresponding each customized subject. As Custom diffusion and Cones demonstrated, we omit Textual Inversion [18] as it performs much less competitively. And the implementation details of our approach and those of baselines are also reported in the Appendix A.