AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation
Authors: Lianyu Pang
Jian Yin
Baoquan Zhao
Feize Wu
Fu Lee Wang
Qing Li
Xudong Mao
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models
Personalized Text-to-Image Generation
Personalization
Primary area: diffusion_based_models

Abstract: Recent advances in text-to-image models have enabled high-quality personalized image synthesis based on user-provided concepts with flexible textual control. In this work, we analyze the limitations of two primary techniques in text-to-image personalization: Textual Inversion and DreamBooth. When integrating the learned concept into new prompts, Textual Inversion tends to overfit the concept, while DreamBooth often overlooks it. We attribute these issues to the incorrect learning of the embedding alignment for the concept. To address this, we introduce AttnDreamBooth, a novel approach that separately learns the embedding alignment, the attention map, and the subject identity across different training stages. We also introduce a cross-attention map regularization term to enhance the learning of the attention map. Our method demonstrates significant improvements in identity preservation and text alignment compared to the baseline methods.

AttnDreamBooth Towards Text-Aligned
Personalized Text-to-Image Generation
Lianyu Pang
Jian Yin
Baoquan Zhao'
Feize Wu'
Fu Lee Wang"
Qing Li
Xudong Mao
Sun Yat-sen University Hong Kong Metropolitan University
The Hong Kong Polytechnic University
http://atnddeambbootggggghhht
Abstract
Recent advances in text-to-image models have enabled high-quality personalized image synthesis based user-provided concepts with flexible textual control. In this work, we analyze the limitations of two primary techniques in text-to- image personalization: Textual Inversion and DreamBooth. When integrating the learned concept into new prompts, Textual Inversion tends to overfit the concept, while DreamBooth often overlooks it. We attribute these issues to the incorrect learning of the embedding alignment for the concept. To address this, we introduce AttnDreamBooth, a novel approach that separately learns the embedding alignment, the attention map, and the subject identity across different training stages. We also introduce cross-attentie the attention map. Our method demonstrates sigmificant improvements in identity preservation and text alignment compared to the baseline methods.
map regularization term to enhance the learning of
on astronout flocts weightlessly gravity environment. surrounded by celestial bodies and galaxies
Cyberpunk style of robot against the bockdrap of
cowboy drows its tiny nevolver in dusty town showdown, surrounded by cacti and a saloon
knight droped in armor, riding a horse and galloping through the fields
An oil pointing of dressed as musketeer French town
futuristic citysccpe night. by neon lights
Input Sample
assassin's walkinn street of long shadow in sunlit, Venice, surrourded by the of merchants and tourists
Jedi
Captain America stonding ruins of the city. surrounded smake
painting of boatman propping boat in the lake in the Monet
painting vintage steompunk out automaton, complete with gears and complex mechanical devices
empty desert
Figure 1: Our method enables text-aligned text-to-image personalization with complex prompts.
'Corresponding author (xudong.xdmaode gmaiiimcmmmm...mmmmmm
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Text-to-image personalization [24, 71, 50] is the task of customizing a pre-trained diffusion model to produce images of user-provided concepts in novel scenes styles. By providing several examples of a new concept, personalization techniques enable users to employ novel prompts to generate personalized images containing that concept. Current personalization techniques primarily fall into two categories: the first approach involves inverting the new concept into the textual embedding [24]; the second approach involves fine-tuning the diffusion model learn the new concept [71]. Personal- ization techniques aim to generate high-quality images of user-provided concepts, achieving high identity preservation and text alignment. However, despite the significant progress in personaliza- tion techniques, balancing the trade-off between identity preservation and text alignment remains challenge for current approaches.
Figure 2 shows the personalization results from Textual Inversion [24] and DreamBooth [71]. Textual Inversion tends to generate images that focus primarily the learned concept, often neglecting other elements of the prompt. In contrast, DreamBooth appears to overlook the learned concept, producing images that more influenced by other prompt tokens. These issues can be attributed to the incorrect learning of embedding alignment for the new concept, i.e., the embedding of the new concept is not functionally compatible with the embeddings of existing tokens.
Based on these observations, our approach aims to properly learn not only the subject identity but also the embedding alignment and the attention map for the concept. Our key insights as follows: 1) In the early stages of optimization, Textual Inversion effectively learns the embedding alignment but tends overfit after extensive optimization steps; 2) DreamBooth accurately captures the subject identity but struggles with learning the embedding alignment.
In this paper, we propose method named AttnDreamBooth, which separates the learning processes of the embedding alignment, the attention map, and the subject identity. Specifically, our approach consists of three main training stages, as illustrated in Figure 3. First, we optimize the textual embedding to learn the embedding alignment while preventing the risk of overfitting, which results in coarse attention map for the new concept. Next, fine-tune the poss-attention layers of the U-Net to refine the attention map. Lastly, we fine-tune the entire U-Net to capture the subject identity. Note that the text encoder remains fixed throughout all training stages to preserve its prior knowledge of contextual understanding.
Furthermore, we introduce cross-attention map regularization term to enhance the learning of the attention map. Throughout the three training stages, we use consistent training prompt, "a photo of [V] [super-category]" where [V] and [super-category] denote the tokens for the new concept and its super-category, respectively. Our attention map regularization term encourages similarity between the attention maps of the new concept and its super-category
Input
Textual Inversion
DrcamBooth
Ours
Output
"[V]"
"Drawing"
Output
"[VI" "Drawing"
Output
"[V]"
"Drawing"
"Manga drawing of [V] can"
Output
"[V]"
"Box"
Output
"[V]"
"Box"
Output
"[V]"
"Box"
"A [V] toy inside a box"
Figure 2: Analysis of two principal methods. We visualize the cross-attention maps corresponding the new concept and other tokens in the prompt. Textual Inversion [24] tends to overfit the textual embedding of the learned concept, resulting in incorrect attention map allocations to other tokens (e.g., "drawing" "box"). In contrast, DreamBooth [71] appears to overlook the learned concept, producing images primarily based on other tokens.

---

Text Encoder
+
M
Stage Trainable
Sef Attenttiiion
Croes Altention
Se Caf--Attentoon
Resheet
ReeNet
Stage Trainable
"doll"
frog
Stage Trainable
Diffusion U-Net
Figure 3: Overview of AttnDreamBooth Our method consists of three training stages. In Stage 1, we optimize the textual embedding of the new concept align its embedding with existing tokens. In Stage 2, we fine-tune the cross-attention layers to refine the attention map. In Stage 3, we fine- tune the entire U-net capture the subject identity. Moreover, we introduce a cross- cross-attention map regularization term to guide the learning of the attention map.
To demonstrate the effectiveness of AttnDreamBooth, compare it with four state-of-the-art baseline methods through both qualitative and quantitative evaluations. Our method achieves superior performance in terms of identity preservation and text alignment compared the baselines. More importantly, AttnDrcamBooth enables a variety of text-aligned personalized generations with complex prompts.
2 Related Work
Text-to-Image Generation. Generative models are designed to create new samples that resemble the patterns observed in their training data. There are various types of generative models, including VAEs [47, 76, 14J. GANs [27, 6, 44]. auto-regressive models [66, 91], flow-based models [21, 48], and diffusion models [75, 33, 59, 18]. These models can be enhanced by conditioning on text prompts, which are known as text-to-image models [68, 66, 60, 20, 23, 16, 5]. Recent advancements [73, 67, 70] text-to-image generation, powered by training on extremely large-scale datasets, have demonstrated an impressive ability to generate diverse and generalized outputs.
Texa-to-Image Personalization. Leveraging the impressive capabilities of diffusion models, text- to-image personalization involves adapting pre-trained diffusion models to capture new concepts depicted several given images. Pioncering works approach this by inverting the concept into the textual embedding [24], or by fine-tuning the diffusion model [71]. However, these methods often struggle balance the trade-off between identity preservation and text alignment, and typically require substantial time for optimization. To overcome these limitations, some studies focus on enhancing the identity preservation of the concept [81, 1, 94, 36, 31, 43, 40], while others aim to improve text alignment [78, 3, 4, 90, 37]. Additionally, there is growing trend of research attempting to accelerate the personalization process, either by reducing the number of tuning parameters [50, 29, 54, 34, 28, 57], or by pre-training on large datasets [85, 74, 39, 2, 25, 10, 72, 51, 87, 56, 11, 55]. Given the widespread interest in human synthesis, many studies also concentrate on the personalized synthesis of human faces [92, 62, 89, 53, 79, 83, 8, 49, 61, 45, 86, 17, 13, 82, 12].
Cross-Attention Control. The cross-attention layers [70] have been shown to play crucial role in diffusion models. The control of cross- tross-attention layers has proven effective in variety of tasks, including image editing [32], compositional synthesis [22, 52, 7, 26, 46], and layout- controlled synthesis [63, 9, 88]. In text-to-image personalization, several studies [50, 87, 4, 78, 30, 42, 58, 93] also have explored the control cross-attention layers. Custom Diffusion [50] illustrates how incorrect attention maps of the learned concepts can lead unsuccessful synthesis. FastComposer [87] and Break-A-Scent [4] propose using segmentation masks the target concepts to guide the learning of the attention maps, thereby enhancing text alignment, especially in scenarios involving multiple concepts. Perfusion [78] identifies the attention overfitting issue and addresses by fixing the cross-attention key matrices of the target concepts to their super-category tokens.
Multi-Stage Personalization. Several studies [24, 54, 4, 41, 38] have explored combining the strengths of different methods into more efficient models through multi-stage approach. Inspired by

---

PTI [69], Textual Inversion [24] investigated a tto-stage approach to enhance identity preservation by first optimizing the textual embedding and then fine-tuning the diffusion model to better capture the subject identity. Similarly, MagiCapture [38] first optimizes the textual embedding and then applies LoRA [34] in the U-Net. Break-A-Seen [4] proposes initially optimizing the textual embedding with high learning rate, followed by fine-tuning both the U-Net and the text encoder using significantly lower learning rate. Our method differs in several aspects. First, the motivation for our first stage (i.e., optimizing the textual embedding) differs from previous methods. Specifically, we focus on learning the embedding alignment while mitigating the risk of overfitting, thus significantly reducing the optimization steps and lowering the learning rate. Second, we decompose the learning process into three stages: learning the embedding alignment, refining the attention map, and capturing the subject identity. Third, we introduce cross-attention map regularization to guide the learning of the attention map in a self-supervised manner.
Preliminaries
Latent Diffusion Models. Our approach is based on the publicly available Stable Diffusion model, a type of Latent Diffusion Model (L.D)) [70] for text-to-image generation. In LDM, an autoencoder utilized to provide lower-dimensional representational space, where an encoder & transforms an image x into a latent representation = = (x)), and a decoder 2 reconstructs the image from this latent æ. Additionally, a Denoising Diffusion Probabilistic Model (DDPM) [33] is employed to produce latent codes within the latent space of the autoencoder. To generate images from text, the model leverages conditioning vector c(y), derived from given text prompt y. The training objective of LDM is given by:
code, i.e., (=E(x)
daifuuson E2=-(()) [|e Eo (2,, c1))
(1)
where the denoising network Eg is tasked with recovering the original latent code žy from the noised latent code 2,, given a specific timestep and the conditioning vector c(y).
Textual Inversion. Textual Inversion (TT) [24] personalizes a pre-trained diffusion model by encoding the target concept into the textual embedding. Given several images of a target concept, TI introduces a new token S, and its associated textual embedding vs to represent the concept. The learning process of TI involves initializing U, with a coarse descriptor and then optimizing it to minimize the diffusion objective (Eq. 1).
DreamBooth. DreamBooth (DB) [71] learns the target concept by fine-tuning the pre-trained diffusion model. Given several images of target concept, DB labels all the images with the prompt "a [V] [super-category]", where [V] is a rare token in the vocabulary. The learning process of DB involves fine-tuning the entire U-Net (and possibly the text encoder) using the diffusion objective (Eq. 1) combined with a prior preservation loss [71].
4 Method
In this section, we first analyze the problems associated with Textual Inversion and DreamBooth, as discussed in Section 4.1. To address these issues, we propose a novel method named BotnDreamBooth, as detailed Section 4.2. To further enhance text alignment, we introduce a cross-attention map regularization term in Section 4.3.
4.1 Analysis of Existing Methods
Problems and Analysis. As illustrated in Figure Textual Inversion and DreamBooth encounter distinct challenges when integrating the learned concept into novel prompts. For Textual Inversion, the generated images often excessively focus on the learned concept, overlooking other prompt tokens. To investigate this issue, we present the attention map visualization using DAAM [77] for different tokens Figure 2. This visualization reveals an embedding misalignment issue in novel compositions containing the concept, leading to incorrect attention map allocations for other tokens. A typical example is shown where the attention map corresponding to the "drawing" token focuses incorrect regions. This misalignment occurs because Textual Inversion tends to overfit

---

Input
Output
"[VT"
"Monet"
Before fine-tuning
After fine-tuning
(b)
Figure 4: Analysis of TI+DB. Column (a) demonstrates that TI+DB neglects the learned concept when integrating it into new prompt, "A painting a [V] toy in the style of Monet". Column (b) shows the generated images based single word prompt, "[VT", both before and after fine-tuning, using the diffusion model without fine-tuning, These images are notably similar to each other, which indicates that the learned textual embedding remains largely unchanged from its initial state.
the input embedding of the text encoder, responsible for managing the contextual understanding of the prompt. Conversely, images generated by DreamBooth sometimes focus solely other prompt tokens, neglecting the learned concept. This occurs because DreamBooth uses a rare token for the new concept while keeping its textual embedding fixed, thereby leading to insufficient learning of the embedding alignment for the new concept.
A Naive Solution. As analyzed previously, Textual Inversion and DreamBooth exhibit distinet issues related to the embedding alignment: Textual Inversion tends to overfit the embedding align- ment for the new concept, while DreamBooth demonstrates insufficient learning of the embedding alignment. straightforward solution to combine Textual Inversion with DreamBooth by jointly tuning the textual embedding and the U-Net, method we denote as TI+DB. We observe that TI+DB enhances performance over Textual Inversion or DreamBooth individually. However, it still tends to neglect the learned concept when integrating it into prompts. This issue arises from the slow update of the textual embedding relative to the U-Net. As illustrated in Figure 4, the learned textual embedding remains very close to its initial state. Furthermore, calculate the cosine similarity between the learned and initial embeddings, which averages about 0.9997, indicating that TI+DB still suffers from insufficient learning of the embedding alignment.
4.2 AttnDreamBooth
To address the issues described in Section 4.1, we propose a method named AttoDreamBooth, inspired by two key observations. First, while Textual Inversion often fails capture the subject identity and tends to overfit the embedding alignment for the new concept, can effectively learn the embedding alignment in the very early stages optimization. However, these early stages, the model only learns a coarse cross-attention map for the new concept. Second, although DreamBooth fails to learn the embedding alignment, it can accurately capture the subject identity. Based on these observations, we propose to decompose the personalization process into three training stages: 1) learning the embedding alignment; refining the attention map; and 3) acquiring the subject identity. An overview of our proposed ArtnDreamBoot is illustrated in Figure 3.
Learning the Embedding Alignment. As previously stated, learning the embedding alignment for the new concept is critical for properly allocating the cross-attention maps for novel prompts, which in turn influences the text alignment of the personalized generation results. To achieve this, we optimize the input textual embedding of the text encoder, since the text encoder manages the contextual understanding of the prompt. However, as analyzed in Section 4. this approach is prone to overfitting the embedding, leading to an embedding misalignment issue. Therefore, our objective this stage is learn the embedding alignment while minimizing the risk of overfitting. To this end, we adapt Textual Inversion [24] with three main modifications. First, we significantly reduce the number of optimization steps (to 60 steps in our experiments) and lower the learning rate (to 10 10-3) 3). Second, we introduce a cross- cross-attention map regularization (see Section 4.3) to guide the learning of the cross-attention map. Third, to facilitate the incorporation the cross-attention map regularization, we set the training prompt as "a photo of a [V] [super-category]" To prevent overfitting, we stop the optimization at very early stages, thereby resulting coarse cross-attention map for the new concept, as depicted in Figure 5. A full analysis of attention map allocations for each token is
5

---

Input
Prompt
Stage
Stage 2
Stage 3
[]] toy
inside a box
Figure 5: Results after each training stage. We present the generations along with the attention maps of "[V]" for each stage. In stage 1, the model properly aligns the embedding of [V] with other tokens, "inside box", but learns a very coarse attention map and subject identity. In stage 2, the model refines the attention map and subject identity. In stage 3, the model accurately captures the identity of the concept.
presented in Appendix E. The cross- aross-attention map, as well as the subject identity, are addressed in subsequent steps.
Refining the Cross-Attention Map. To mitigate the embedding misalignment issue, model initially learns a relatively coarse cross-attention map for the new concept. At this stage, we focus on refining the cross-attention map. Since these attention maps embedded within the cross-attention layers, inspired by Custom Diffusion [50], we fine-tune all the cross-attention layers in the U-Net. Additionally, we employ the proposed cross-attention map regularization (see Section 4.3) to aid in refining the attention map. Furthermore, we keep the textual embedding and the text encoder fixed to prevent further embedding misalignment.
Capturing the Subject Identity. As illustrated in Figure 5, the previous stage produces images that are similar to the target concept but still exhibit significant distortions. Therefore, in the third stage, following DreamBooth [71], we unfreeze all layers of the U-Net to more accurately capture the subject identity of the target concept. We choose not to adopt the prior preservation loss [71], as we empirically find that leads to poor identity preservation and requires significantly more training steps. A detail discussion of our models with or without the prior preservation loss could be found in Appendix H. Moreover, similar to the previous stage, we keep the textual embedding and the text encoder fixed to prevent embedding misalignment, and we continue to apply the cross- cross-attention map regularization to guide the learning of the attention map.
4.3 Cross-Attention Map Regularization
We set the training prompt as "a photo of a [V] (super-category]", where [V] and [super-category denote the tokens for the new concept and its super-category. respectively. To enhance the learning of the attention map, we introduce regularization term that encourages similarity between the attention maps of [V] and [super-category]. This regularization term serves two purposes. First, since the new concept and its super-category belong to the same object category, the attention map of the super-category token can serve reference for the new concept. Second, since [V] and [super-category] are used together to describe the new concept when integrating it into new prompts, the attention maps of [V] and [super-categor]] should refer the same region.
Formally, for the 16 attention maps {M1 My..., M16} from 16 different cross-attention layers, we minimize the squared differences in the mean and variance of the attention map values for [V] and (super-category] as follows:
Cagg Ay (M - He( (MM category + do [ - o OM
where ((M1:16) and (M1111)) denote the mean and variance of all the values across the 16 attention maps, respectively. This constraint helps ensure that the new concept exhibits similar level of concentration or dispersion in the attention map as the super-category token. Note that we avoid directly applying the constraint to the attention map values themselves because we empirically find that such constraint is too restrictive and difficult to optimize.

---

Input
TI+DB
NeTI
ViCo
OFT
Ours
An oil painting
a [V] sloth dressed as musketeer
in an old French town
[V] doll as a Jedi casting long shadow in sunlit, empty desert
A [V] fluffy walking in the rainy streets
Light seeps out from the inside
of a clear [V] teapot under the moonlight on
serene beach
A painting of [V] toy floating on lake under the full moon's glow in the style of Monet
Figure 6: Qualitative comparison. We present four images generated by our method and two images from each of the baseline methods, including TI+DB [24, 71], NeTI [1], ViCo [30], and OFT [64]. Our method demonstrates superior performance in text alignment and identity preservation compared these baselines.

---

A black [V] toy wearing
[V] toy wearing chef kitchen with meat vegetables the rable
A [V] wearing police cup a policce car
[V] toy priest in blue robes
App icon
Input Sample
sunglasses the heach
laughing [V] toy
purple [V] furby under the mystical auroru borealis
[V] farby against the back- drop of futuri- stic cityscape remote Arctic
A yellow [V] furby cobblestone street an old Faropean town, with historical architecture
A [V] furby amidst bustling street market surrounded by vibrant and textures
A black [V] Turby bathed the golden light sunset serene beach
Input Sample
night illurnintted lights
landscape
A [V] bear alop high cliff overlooking stormy seas
A [V] bear surrounded by Buttering butterflies in a meadow
A [V] beur in the reflection of cracked antique mürror
A [V] bear perehed on a city rooltop at sunset
A [V] bear floating in weightlessness of space
Input Sample
Figure 7: Examples of personalized generations obtained using AttDDremmoothh
5
Experiments
In this section, we first present the implementation details of our method. Subsequently, we evaluate its performance by conducting comparative analysis with four state-of-the-arr personalization methods. Lastly, we conduct ablation study to demonstrate the effectiveness of each sub-module.
5.1 Implementation and Evaluation Setup
Implementation Details. Our implementation is based on the publicly available Stable Diffusion V2.1 [70]. The textual embedding of the new concept is initialized using the embedding of the super-category token. We keep fixed batch size of 8 across all training stages but vary the learning rates and training steps. Specifically, we train with learning rate of 10-3 for 60 steps in stage followed by a learning rate of 2 x 10-5 for 100 steps in stage 2, and conclude with learning rate 0. and in stage 1, respectively, and
of 10-6 for 500 steps stage ' and A, are adjusted to and in subsequent stages. All experiments are conducted on a single Nvidia A100 GPU. The training process of our method takes about 20 minutes learn concept.
Evaluation Setup. We compare our method with four state-of-the-art personalization methods, including TI+DB [24, 71], NeTT [1], ViCo [30], and OFT [64]. The implementation details of the baseline methods are provided in Appendix A. We collect 22 concepts from TI [24] and DB [71]. For the quantitative evaluation, each method is evaluated using set of 24 text prompts, see Appendix B for complete list. These prompts cover background change, environment interaction, concept color change, and artistic style.
5.2 Results
Qualitative Evaluation. In Figure we present visual comparison of personalized generation for various concepts. We employ set of complex prompts for evaluation, where prompt simultanc-

---

Table Quantitative comparisons. "Iden- tity" denotes the identity preservation, and "Text" denotes the text alignment.
Table 2: User study. We asked the participants to select the image that better preserves the identity and matches the prompt.
Methods
Identityt
Text
Baselines
Prefer Baseline
Prefer Ours
TI+DB [24, 71]
0.7017
0.2578
TI+DB [24, 71]
32.0%
68.0%
NeTI [1]
0.6901
0.2522
NeTI [1]
20.6%
79.4% %
ViCo [30]
0.7507
0.2106
ViCo [30]
16.6%
83.4%
OFT [64]
0.7257
0.2445
OFT [64]
22.4%
72.6 %
furs-fast
0.7268
0.2536
Ours
0.7257
0.2532
ously incorporates several editing elements such as style change (e.g.. "oil painting"), scene change (e.g., "old French town"), and appearance change (e.g., "dressed as musketeer"). As observed, ViCo tends to overfit the new concept, failing to compose it in novel scenes or styles. Conversely, TI+DB sometimes overlooks the learned concept, producing images that solely reflect other prompt tokens. NeTI and OFT also struggle to achieve text-aligned generations, especially when the prompts are complex. Our method, AttnDreemBooth, is the only method that successfully generates identity- preserved and text-aligned personalized images for these complex prompts. Figures I and 7 show more personalized generations using complex prompts from our method. Additional qualitative results can be found in Appendix C.
Quantitative Evaluation. We conduct quantitative evaluation of each method in terms of identity preservation and text alignment. Identity preservation measured by the cosine similarity between the CLIP [65] embeddings of generated and real images, while text alignment is measured by the cosine similarity between the CLIP embeddings of generated images and their corresponding prompts. Each method is evaluated using 24 text prompts, generating 32 images per prompt. The results presented in Table 1. TI+DB excels in text alignment but performs poorly in identity preservation. This is consistent with the qualitative observation that TI+DB often neglects the learned concept and generates images based solely on other prompt tokens. In contrast, ViCo achieves the best identity preservation but ranks lowest in text alignment, indicative of its tendency to overfit the new concept. Besides these two extreme cases, our approach exhibits superior performance in both identity preservation and text alignment.
Fast Version of Our Method. The average training time using method is about 20 minutes. To reduce the training time, we developed fast version of our method by increasing the learning rate while simultaneously decreasing both the training steps and the batch size for the third training stage. Originally. this stage involves performing 500 steps with a learning rate of 2 x 10 6 and a batch size of 8. The fast version now completes training in just 200 steps with learning rate of 1 x 10 5 and a batch size of 4. These adjustments significantly reduce the training time from 20 minutes to an average of 6 minutes. Interestingly, this fast version maintains performance comparable to our original model, likely because the first two stages provide a convenient starting point, allowing for higher learning rate in the third stage. Notably, the fast version performs similarly the original model on short prompts, as shown in Table 1, but it shows slight degradation on complex prompts, as depicted in Figure 10.
User Study. We further evaluate our method by conducting a user study. Personalized images are generated using various prompts and concepts for each method. each question of the study, participants are presented with an input image and a text prompt, along with two generated images: one from our method and another from a baseline method. Participants are asked to select the image that better achieves identity preservation and text alignment. We collected total of 700 responses from 35 participants, as presented in Table The results demonstrate clear preference for our method.

---

Input
Prompt
w/o Stage
w/o Stage 2
w/ Stage 3
w/ Ree
Full
An oil painting of a [V] sloth dressed musketeer in an old French town
A painting of a [V] fluffy the style of Monet
Figure 8: Ablation study. We compare models trained without optimizing the textual embedding (w/o Stage 1), without fine-tuning the cross-attention layers (w/o Stage 2), without fine-tuning the U-Net (w/o Stage 3), and without the cross-altention map regularization (w/o Reg). As can be observed, all sub-modules are essential for achieving identity-preserved and text-aligned personalized generations.
5.3 Ablation Study
In this section, we evaluate the effectiveness of each sub-module within our framework. Specifically, we conduct ablation study by separately removing each training stage the attention map regularization term. Figure presents a visual comparison of personalized images generated by each variant. The results indicate that all sub-module are crucial for achieving ddentity-preservee and text- aligned personalized generations. Specifically, the model without optimizing the textual embedding (w/o Stage 1) tends to neglect the learned concept generate it with significant distortions due to insufficient learning of the embedding alignment. Models without fine-tuning the cross-attention layers (w/o Stage 2) without the regularization term (w/o Reg) suffer from degraded text alignment or identity preservation. The model without finettuning the U-Net (w/o Stage 3) leads to significant degradation in identity preservation. Additional ablation study results are provided in Appendix F. Similar behavior is observed in the guantitative ablation study, as detailed in Table
Conclusions and Limitations
In this paper, we identified and analyzed the embedding misalignment issue encountered by Textual Inversion and DreamBooth. Our proposed method, named AttnDreamBooth addresses this issue by decomposing the personalization process into three stages: learning the embedding alignment, refining the attention map, and acquiring the subject identity. AttnDreamBooth enables and text-aligned text-to-image personalization, even with complex prompts.
preseeredd
In our experiments, we used consistent training steps across different concepts: however, we observed that performance could be further improved by tuning the training steps for specific concepts. This limitation might be addressed by adopting adaptive training strategies, which we leave for future work. A second limitation that our three-stage training method requires approximately 20 minutes on average to learn a concept, it involves fine-tuning all parameters in the U-Net for 500 steps. To mitigate this, we introduced fast version that reduces the training time approximately minutes while still maintaining performance comparable to the original model.
Acknowledgments
This work is supported by National Natural Science Foundation of China (No. 62176223 and No. 62302535), Guangdong Basic and Applied Basic Research Foundation (No. 155111515112897 and No. 2023A151501 and Zhuhai Basic and Applied Basic Research Foundation (No. 2320004002745).
10

---

References
[1] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text-to-image personalization. arXiv preprint wrXiv:2305./5399 2023.
[2] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano. Domain-agnostic tuning- ennnder for fast personalization of text-to-image models, arXiv preprint wxXv:2307.069225 2023.
[3] Moab Arar, Andrey Voynov, Amir Hertz, Omri Avrahami, Shlomi Fruchter, Yael Pritch, Daniel Cohen-Or, and Ariel Shamir. Palp: Prompt aligned personalization of text-to-image models. arXiv preprint arXiv:2401.061005 2024.
[4] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a- scene: Extracting multiple concepts from single image. arXiv preprint 2rX5 2023.
[5] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming- Yu Liu. ediff-i: text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint vrXiv:2211.013224 2022.
[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprini urXiv::809) 1096, 2018
[7] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. In SIGGRAPHI, 2023,
[8] Li Chen, Mengyi Zhao, Yiheng Liu, Mingxu Ding, Yangyang Song, Shizun Wang, Xu Wang, Hao Yang. Jing Liu, Kang Du, and Min Zheng. Photoverse: Tuning-free image customization with text-to-image diffusion models. arXiv preprint arXiv:2309. 05793, 2023.
[9] Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In WACV, 2024.
[10] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming- Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. arXiv preprint arXiv:2304.000.80 2023.
[11] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: sero-shot object-level image customization. arXiv preprint arXiv:2307,0944 09,47,, 2023.
[12] Zhuowei Chen, Shancheng Fang, Wei Liu, Qian He, Mengqi Huang, Yongdong Zhang, and Zhendong Mao. Dreamidentity: Improved editability for efficient face-identity preserved image generation. arXiv preprint arX
2023.
[13] Jiaxiang Cheng, Pan Xie, Xin Xia, Jiashi Li, Jie Wu, Yuxi Ren, Huixia Li, Xuefeng Xiao, Min Zheng, and Lean Fu. Resadapter: Domain consistent resolution adapter for diffusion models. arXiv preprini arXiv:2403. 2404004400.00 2024.
[14] Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In ICLR, 2021.
[15] Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva. On the detection of synthetic images generated by diffusion models. In ICASSP, 2023.
[16] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. In ECCV, 2022.
[17] Siying Cui, Jiankang Deng, Jia Guo, Xiang An, Yongle Zhao, Xinyu Wei, and Ziyong Feng. Idadapter: Learning mixed features for tuning-free personalization of text-to-image models. arXiv preprint 2403 2024.
11

---

[18] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NeurIPS, 2021.
[19] Ganggui Ding, Canyu Zhao, Wen Wang, Zhen Yang, Zide Liu, Hao Chen, and Chunhua Shen. Freecustom: Tuning- foning-free customized image generation for multi-concept composition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9089 9899,9098,00 2024.
[20] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. In NeurlPS, 2021.
[21] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
[22] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In ICLR, 2023.
[23] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-e-scene: Scene-based text-to-image generation with human priors. ECCV, 2022.
[24] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar arXiv:2208.006
2022.
[25] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Encoder-based domain tuning for fast personalization of text-to-image models. TOG, 2023.
[26] songwei Ge, Taesung Park, Jun-Yan Zhu, and Jia-Bin Huang. Expressive text-to-image generation with rich text. In ICCV, 2023.
[27] Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[28] Yuchao Gu. Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, and Mike Zheng Shou. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models, arXiv preprint arXiv:2305. 18292, 2023.
[29] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning, arXiv preprint arXiv:2303. 13305, 2023.
[30] Shaozhe Hao, Kai Han, Shihao Zhao, and Yee K. Wong. Vico: Detail-preserving visual condition for personalized text-to-image generation. arXiv preprint arXiv:2306.0097) 2023.
[31] Xingzhe He, Zhiwen Cao, Nicholas Kolkin, Lantao Yu, Helge Rhodin, and Ratheesh Kalarot. A data perspective enhanced identity preservation for diffusion personalization. arXiv preprint vrXiv:2311.04311 2023.
[32] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- trompptt image editing with cross attention control. In ICLR, 2023.
[33] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[34] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[35] Hexiang Hu, Kelvin CK Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, et al. Instruct-imageet Image generation with multi-modal instruction. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 4754-4763. 2024.
12

---

[36] Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, and Qian He. Dreamtuner: Single image is enough for subject-driven generation. arXiv preprint arXiv:2312.1669,, 2023.
[37] Mengqi Huang, Zhendong Mao, Mingeong Liu, Qian He, and Yongdong Zhang, Realcustom: Narrowing real text word for real-time open-domain text-to-image customization, arXiv preprini arXiv:2403.00483 2024.
[38] Junha Hyung, Jaeyo Shin, and Jaegul Choo. Magicapture: High- nigh-resolution multi-concept portrait customization. arXiv preprint urXiv:2309.068955 2023.
[39] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu- Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304. 020::22240.0642, 2023.
[40] Jiaxiu Jiang, Yabo Zhang, Kailai Feng, Xiaohe Wu, and Wangmeng Zuo. Mc Multi-concept guidance for customized multi-concept generation. arXiv preprint
2024.
[41] Yuming Jiang, Tianxing Wu, Shuai Yang. Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy. and Ziwei Liu. Videobooth: Diffusion-basse based video generation with image prompts. In CVPR, 2024.
[42] Chen Jin, Ryutaro Tanno, Amrutha Saseendran, Tom Diethe, and Philip Teare. An image is worth multiple words: Learning object level concepts using multi-concept prompt learning. arXiv preprint arXiv:2310. 12274, 2023.
[43] Maxwell Jones, Sheng-Yo Yu Wang, Nupur Kumari, David Bau, and Yan Zhu. Customizing text-to-image models with a single image pair. arXiv preprint arXiv:2405.01536, 2024.
[44] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[45] Chanran Kim, Jeongin Lee, Shichang Joung, Bongmo Kim, and Yeul-Min Baek. Instantfamily: Masked attention for zero-shot multi-id image generation. arXiv preprint arXiv:2404.19427, 2024.
[46] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Woo Ha, and Jun- Yan Zhu. Dense rext-to-image generation with attention modulation. In ICCV, 2023.
[47] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[48] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NeurlPS, 2018
[49] Zhe Kong, Yong Zhang, Tianyu Yang, Tao Wang, Kaihao Zhang, Bizhu Wu, Guanying Chen, Wei Liu, and Wenhan Luo. Omg: Occlusion-friendly personalized multi-concept generation in diffusion models. arXiv preprint 2rXiv:2403.10993 2024.
[50] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In CVPR, 2023.
[51] Dongxu Li, Junnan Li, and Steven C. H. Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. arXiv preprint arXiv:2305, 0tt.:230511472, 2023.
[52] Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing. In BMYC, 2023.
[53] Zhen Li, Mingdeng Cao, Xintao Wang. Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked embedding. arXiv preprint arXiv:2312.04461, 2023.
[54] LoRA. Low-rank adaptation for fast text-to-image diffusion fine-tuning. http:///gitubbccomm/oooossm////p 2022.
13

---

[55] Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. oopject-iiffusiin.openn domain personal- ized text-to-image generation without test-time fine-tuning, arXiv preprint arXiv:2307.11410, 2023.
[56] Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv preprini arXiv:2303.09319, 2023.
[57] Shyam Marjit, Harshit Singh, Nityanand Mathur, Sayak Paul, Chia-Mu Yu, and Pin- Yu Chen. Diffusekrona: A parameter efficient fine-tuning method for personalized diffusion model. arXiv preprint arXiv:2402. 17412, 2024.
[58] Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, and Seunggyu Chang. Dreammatcher: Appearance matching self-attention for semantically-consiste text-to-image personalization. arXiv preprint arXiv:2402.09812, 2024.
[59] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021.
[60] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprini a 2021.
[61] Daniil Ostashev, Yuwei Fang, Sergey Tulyakov. Kfir Aberman, et al. Moa: Mixture-oo attention for subject-context disentanglement in personalized image generation. arXiv preprint MxXi:2244411165. 2024.
[62] Lianyu Pang. Jian Yin, Haoran Xie, Qiping Wang, Qing Li, and Xudong Mao. Cross initializa- tion for personalized text-to-image generation. arXiv preprint urXiv:23/2.15505 2023.
[63] Quynh Phung, songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. arXiv preprint arXiv:2306.05427 2023.
[64] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In NeurlPS, 2024.
[65] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[66] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zhro-shot text-to-image generation. In ICML, 2021.
[67] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
[68] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In ICML, 2016.
[69] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. TOG, 2022.
[70] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[71] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
[72] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wad- hwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models, arXiv preprint arXiv:2307.069453 2023.
14

---

[73] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes. Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[74] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03444 03411, 2023.
[75] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
[76] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In NeurlPS, 2016.
[77] Raphael Tang, Linging Liu, Akshat Pandey, Zhiying Jiang. Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the DAAM: Interpreting stable diffusion using cross attention. In ACL, 2023.
[78] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In SIGGRAPH, 2023.
[79] Dani Valevski, Danny Wasserman, Yossi Matias, and Yaniy Leviathan. FaceO: Instantaneously conditioning a text-to-image model on face. arXiv preprint arXiv:2306.0666 06638, 2023.
[80] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: Statc-of-the-art diffusion models. 2022.
[81] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.00 09522, 2023.
[82] Qinghe Wang, Xu Jia, Xiaomin Li, Taiqing Li, Liqian Ma, Yunzhi Zhuge, and Huchuan Lu. Stableidentity: Inserting anybody into anywhere at first sight. arXiv preprint 1xXi:2401..19755 2024.
[83] Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-preserving generation seconds. arXiv preprint arXiv:2401.07512 2024.
[84] Sheng-Yu Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. Cnn- generated images are surprisingly easy to spot... for now. In CVPR, 2020.
[85] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint 2xX.::3332.13444 2023.
[86] Yi Wu, Ziqiang Li, Heliang Zheng, Chaoyue Wang, and Bin Li. Infinite-id: preesttd personalization via id- semantinc decoupling paradigm. arXiv preprint arXiv:2403. 11781, 2024.
[87] Guangxuan Xiao, Tianwei Yin, William T. Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-fre multi-subject image generation with localized attention. arXiv preprint arXiv:2305. 10431, 2023.
[88] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffu- sion. In ICCV, 2023.
[89] Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, and Bin Fu. Facestudio: Put your face everywhere in seconds. arXiv preprint arXiv:2312.02666 2226266,,, 2023.
[90] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint wrxiv:2308.0672.1 2023.
[91] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022.
15

---

[92] Ge Yuan, Xiaodong Cun, Yong Zhang, Maomao Li, Chenyang Qi, Xintao Wang, Ying Shan, and Huicheng Zheng. Inserting anybody in diffusion models via celeb basis. In NewrIPS, 2023.
[93] Yanbing Zhang, Mengping Yang, Qin Zhou, and Zhe Wang. Attention calibration for disentan- gled text-to-image personalization. In CVPR, 2024.
[94] Yufan Zhou, Ruiyi Zhang, Tong Sun, and Jinhui Xu. Enhancing detail preservation for customized text-to-image generation: regularization-fue approach. arXiv preprint a335, 2023.
16

---

A Implementation Details of Baselines
We compare our method with four baseline methods, including TH+DB [24, 71J, NeTI [1], ViCo [30], and OFT [64]. For TI+DB, implement it based the diffusers library [80] without employing the prior preservation loss. We perform 660 training steps, which matches the total number of steps for our method, with learning rate of x 10-6 and batch size of 8. For the other baselines, use the official implementations and follow the hyper-parameetrss described in their papers.
In the Appendix, further compare method with four other bascline methods, including DreamMatcher [58], FreeCustom [19]. SuTI [10], and Instruct- Patruct-Imagen [35]. For DreamMatcher and their official implementations. Due to the unavailability of open-source models for SuTTI and Instruct-Imagen we rely on the examples provided in their papers for comparison.
FreeCustom,
B Text Prompts
In Table list all 24 text prompts used in the quantitative evaluation. These prompts range modifications, including background change, environment interaction, concept color change, and artistic style.
C Additional Qualitative Results
In Figures 9, 10, and 11, we provide additional qualitative comparisons to the baseline methods across wide range of prompts. Furthermore, Figure 12 presents a qualitative comparison of our method with Textual Inversion (TI), DreamBooth (DB), and two different configurations TI+DB: 1) first TI and then DB (TI-DB), and 2) first DB and then TI (DB-+TI). In Figure 13, we provide additional qualitative results generated by AttnDreamBooth.
D Single Image Personalization
In this section, compare AttnDreamBooth with the baseline methods when only single image used for training. In Figure 14, we present the generation results each method under this challenging setting. Our method demonstrates superior text alignment and identity preservation compared to the baselines.
E
Attention Maps for Each Token
In the main text, we provide the cross-attention maps of Textual Inversion and DreamBooth in Figure and the cross-attention map of "[V]" for each stage Figure 5. To vividly demonstrate the efficacy of our method in training the cross-attention layer, we present the generated images along with the cross-attention maps for each token in the prompt in Figure 15. As can be seen, our method accurately assigns the attention maps for each token, demonstrating correct embedding alignment for the new concept.
F Additional Ablation Study
In Figure 16, we provide additional ablation study results for each variant of our method. Table presents the quantitative results of our ablation study. Specifically, the model without Stage achieves better text alignment but significantly poorer identity preservation compared the full model. This is because, without sufficient training of the textual embedding, the model tends to overlook the learned concept generate it with significant distortions. Please note that the text alignment score calculated without considering the new concept: therefore, omitting the new concept can inadvertently boost this score. Similarly, models without Stage or Stage 3 also exhibit higher text alignment scores but lower identity preservation scores, due to insufficient learning of the attention maps and the subject identity, respectively. Additionally, the model without the regularization term shows degraded text alignment.
17

---

G Different Version of Stable Diffusion
visual comparison between our models with SD1.5 or SD2.1 is presented in Figure 17. As shown, the model with SD2.1 achieves superior performance in text alignment and identity preservation. Nevertheless, our method is also effective for SD1.5, and outperforms the baseline methods.
H Prior Preservation Loss
We present visual comparison between models with without the prior preservation loss in Figure 18. The results show that incorporating the prior preservation loss leads to degradation in identity preservation.
I User Study
As described in Section 5.2, we conducted a user study to evaluate our method against the baseline methods. Here, we present the details of this user study. Figure 19 shows example question from the user study. Given a concept image and text prompt, along with two generated images (one from our method and another from baseline method), participants were asked select the image that better preserves the identity of the concept image and aligns with the text prompt. The results are presented in Table 2.
Societal Impact
Similar to existing text-to-image personalization techniques, our approach provides broader users access to effectively fine-tuning large-scale pre-trained diffusion models. By enabling users to personalize these models with their own data, approach be used for numerous applications, including image editing, artistic creations, and industrial production. However, the use of generative techniques comes with risks, such the creation of misleading false information. To mitigate these concerns, it vital to develop effective methods for identifying fake generations [84, 15].
K Licenses for Pre-trained Models and Datasets
Our implementation is based the publicty available Stable Diffusion V2.1 [70], which is under the CreativeML Open RAIL++-M License. The datasets used for evaluation are from TI [24] and DB [71]. The data from DB is under the Unsplash license, while the license information for the data from TI is not available online.
18

---

Table 3: The prompts used in the quantitative evaluation.
photo a [V] [category]
photo of [V] [category] in Times Square
a photo of two [V] [category] on table
a [V] [category] in the jungle
stone wall in the countryside a [V] [category] on brick pathway
[V] [category]
garden
[V] [category] on a pile of fallen leaves in forest
[V] [category] at picnic spot with checkered blanket
a [V] [category] nestled among rocks
[V] [category] inside basket
[V] [category] inside metal cage
a [V] [category] drenched the rainy streets
[V] [category] in a grassy park with sunglasses
a [V] [category] floats on the water
a [V] [category] covered by snow
red [V] [category] wearing bowtie
a purple [V] [category]
black [V] [category]
[V] [category] latte art
pencil drawing a [V] [category]
manga drawing of [V] [category]
watercolor painting of [V] [category]
vector art of [V] [category]
a painting of a [V] [category] in the style of Monet
Table 4: Quantitative ablation study.
Methods
Identity Preservation
Text Alignment?
W/o Stage
0.7031
0,2595
W/o Stage
0.7145
0,2541
W/o Stage
0,6821
0.2650
W/o Reg
0.7269
0,2502
Full Model
0.7257
0.2532
19

---

Input
TI+DB
NeTI
ViCo
OFT
Ours
In a futuristic cityscape, a [V] can transformed robotic entity, buzzing with neon lights and wires
An ice sculpture a ['] bird on a beach at night illuminated by neon lights
[V] bear is building a sandcastle sunny beach while tiny crabs scuttle around and seagulls fly overhead
A plant
grows in broken [V] pot
A green [V] bowl full of
milk on cobblestone
street in an old European town
Figure 9: Additional qualitative comparison. We present four images generated by our method and two images from each of the baseline methods, including TI+DB [24, 71J. NeTT [1], ViCo [30], and OFT [64]. Our method demonstrates superior performance text alignment and identity preservation compared to these baselines.
20

---

Input
DM
FreeCustom
Ours-fast
Ours
An oil painting of a [V] sloth dressed as a musketeer in old French town
[] doll a Jedi casting long shadow in sunlit, empty desert
A [V] fluffy walking in the rainy streets'
Light seeps out from the inside
a clear [V] teapot under the moonlight on serene beach
painting of a [V] toy floating on the Jake under the full moon's glow in the style of Monet
Figure 10: Qualitative results of the fast version of our method, compared with DreamMatcher
(DM) [58] and FreeCustom [19].
21

---

Input
SuTI
Ours
Input
SuTI
Ours
"[VI under the stage lights"
"[V] cating iee-cream in bowl"
"[V] with silver-tipped loes kicking football"
back view of [V] watching
TV show about birds"
Input
Instruct-Img.
Ours
Input
Instruct-Img.
Ours
Sou Apply
"[V] reading book with a pink glasses on"
"[V snifting berry bowl"
Figure 11: Qualitative comparison to SuTT [10] and Dustruct-Imagen [35] using the examples from
their papers.
Input
TI
DB
TI-DB
DB -I
Ours
"An painting a [V] sloth dressed as musketeer in an old town'
"A V doll as Jedi casting long shadow in sunlit empty desert"
"Light seeps out from the inside of a clear V] under the moonlight on a serene beach"
"A [] fluffy walking the rainy streets"
Figure 12: Qualitative comparison of our method with Textual Inversion (TI), DreamBooth (DB), and different configurations of TI+DB: 1) first TI and then DB (TI->DB), and 2) first DB and then Tl (DB -T)..).
22

---

A [V] bear presenting poster at conference with people around
A [V] bear delivering his graduation speech at the podium
A purple [V] bear writing paper in the conference room
A red [V] bear holding his accepted paper the jungle
A black [V] bear sitting on his suitcase the airport
Input Sample
A [V] pot made out of pure gold with a metallic luster
A plant grows in broken [V] pot
Water pouring out a [[V] pot
A cube shaped [V] pot
A clear [V] pot full of milk
Input Sample
A [V] clock in whimsical, enchanted forest, surrounded by fairies and soft magical light, fantasy illustration
A [V] clock illuminated by the soft glow candle, nears window on rainy night
[V] clock floating on the water with cyberpunk cityscape in the background
[V] clock embedded the bark of an ancient oak tree with sunset in the background
A [V] clock in ancient library books scattered around
Input Sample
A [V] dog surfing wave, wearing a floral Ici
A [V] dog floating the water, wearing sunglasses
A [V] dog covered by snow New York city
A [V] dog burns in the fire a burning wood
A wet [V] dog drenched in the rainy streets
Input Sample
[V] dog lounging in hammock tropical beach, wearing sunglasses
A [V] dog caught in gentle snowfall a serene winter landscape
A [V] dog amidst the ruins of an ancient, forgotten civilization
A [V] dog wearing life jacket on a boat
A [V] dog nestled within vibrant coral underwater
Input Sample
Figure 13: Additional qualitative results by AtnDreamBoothh
23

---

Input
TI+DB
NeTI
ViCo
OFT
Ours
[ bird perched on the branch
aged oak tree, watching the sunrise
A [V] bear a hammock strung between two palm trees, swaying gently in the tropical breeze
A [V] wolf is reading book with cup of hot cocoa nearby
A [V] clock latte art
A [V] bird float water
Figure 14: Single image personalization results. We present four images generated by our method
and two images from each of the baseline methods, including TI+DB [24, 71]. NeTI [i], ViCo [30], and OFT [64]. Our method shows better text alignment and identity preservation than the baselines.
24

---

Input
Stage
Output
"a"
"[V]"
"toy" Stage 2
"inside"
"a"
"box"
A [V] toy inside box
Output
"a"
"[V]"
"toy" Stage 3
"inside"
"a" TM
"box"
Output
"a"
"[V]"
"toy"
"inside"
"a" tm
"box
Figure 15: The attention maps for each token after each training stage.
Input
w/o Stage
w/o Stage
w/o Stage
w/o Reg
Full Model
[V] toy as priest blue robes, the cathedral
[V] doll as Jedi casting a long shadow sunlit, empty desert
Figure 16: Additional ablation study results.
Input
NeTI
ViCo
OFT
Ours-SD1.5
Ours-SD2.
"An oil pamtiig
[V] sloth dressed a musketeer in an French town' "
"A painting of [V] toy floating the lake under the full moon's glow in the style of Monet" Figure 17: Results of our model using SD1.5.
25

---

Input
With prior preservation loss
Without prior preservation loss
"An oil painting of a [V] sloth dressed as
a musketeer in an old French town"
"A [V] doll as a Jedi casting long shadow in sunlit empty desert"
Figure 18: Results of models with without the prior preservation loss.
1. Concept image (denoted [V]]:
Appatal
Text prompt: [V] burs the fire".
Please select the image from the options below that better preserves the identity of the concept image shown above and aligns with the text prompt provided.
A
Figure 19: An example question of the user study. Given a concept image and prompt, along with two generated images, participants are asked to select the image that better preserves the identity of the concept image and aligns with the text prompt.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our method in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
There is no theory in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided the implementation details of our method in Section 5.1,
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification:
We have made the code publicly available with sufficient instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaeccccccccccccctlt / sssssssoiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We have provided the experimental details in Sections 5.1, A, and
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Error bars not applicable the experiments conducted in this paper. The qualitative comparison is the most important evaluation for this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the information on the computer resources in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///eurrppp.ep/ http:///ccrrp.pcpppppciiiiieeii htte//ublicceeeeee dt
csGuidel ines?
Answer: [Yes]
Justification: Our paper conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential impacts of our work in Section J.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or pre-trained models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have discussed the licenses of existing assets in Section K.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We have provided the full text of instructions given to participants and screen- shots in Section I.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: In our country, there no equivalent organization for research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "sRAiIMlIwo", "reviewer": "Reviewer_ZpWY", "summary": "The submission proposes AttnDreamBooth for text-to-image personalization. It addresses the limitations of existing methods, Textual Inversion and DreamBooth, by separating the learning process into three stages: embedding alignment, attention map refinement, and subject identity capture. The method aims to improve identity preservation and text alignment in generated images.\n\n# Key Contributions:\n- Introduces an approach for text-to-image personalization.\n- Addresses the limitations of existing methods by separating the learning process.\n- Demonstrates improved performance in terms of identity preservation and text alignment.\n- Provides a comprehensive analysis of the proposed method through qualitative and quantitative evaluations.", "questions": "- Comparison with multiple recent baselines is required. It would be great if the authors can provide this.", "limitations": "- While the proposed approach is novel, the core idea of decomposing the personalization process into multiple stages is not entirely groundbreaking. A more comprehensive exploration of the relationship between the proposed method and existing work would be beneficial.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-The paper identifies a key challenge in text-to-image personalization and provides a novel solution.\n- The proposed method, AttnDreamBooth, demonstrates superior performance in both identity preservation and text alignment compared to Textual Inversion and Dreambooth.\n- The work contributes to advancing the field of text-to-image personalization by offering a more effective approach to balancing the trade-off between identity and text alignment.", "weaknesses": "- My main concern is comparison with new existing work. There have been several recent works after Textual inversion and Dreambooth that have significantly advanced this area. Some of them like SuTI (NeurIPS 2023), Instruct Imagen (CVPR 2024), HyperDreambooth (CVPR 2024) are also zero-shot which do not require any fine-tuning during evaluation. \n\n- It is critical for this work to compare with the state-of-the-art papers in this area. I have listed a few above. However, I am sure that there could be more papers in the past two years in this area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive comments. We response the reviewer’s concerns as follows.\n\n>**W1: My main concern is comparison with new existing work. There have been several recent works after Textual inversion and Dreambooth. Some of them like SuTI (NeurIPS 2023), Instruct Imagen (CVPR 2024), HyperDreambooth (CVPR 2024).**\n\nWe would like to clarify some aspects regarding the baseline methods described in our paper. As indicated in Figure 6 of the main paper and Figure 8 of the appendix, our comparisons are not limited to Textual Inversion and DreamBooth. We also include evaluations against other recent methods such as OFT (NeurIPS 2023) [1] and NeTI (SIGGRAPH Asia 2023) [2]. We adopted these two methods as baselines because they are among the state-of-the-art and are open-source.\n\nThank you for the suggestion to compare our method with SuTI [3], Instruct-Imagen [4], and HyperDreamBooth [5]. In Figure R1 of the attached PDF, we present comparisons with SuTI and Instruct-Imagen. Due to the unavailability of open-source models for these two methods, we use the examples provided in their papers for comparison. As shown, our model achieves superior performance in text alignment compared to these methods. For instance, in the example of \"[V] fancy boot with silver-tipped toes kicking a football\", our model modifies only the toes to be silver, whereas SuTI modifies the entire boot. Regarding HyperDreamBooth, it focuses on personalizing human faces, whereas our approach targets general objects. Personalization of human faces is beyond the scope of our paper. We will include these comparisons in our revised paper.\n\n>**W2 and Q1: It is critical for this work to compare with the state-of-the-art papers in this area. I have listed a few above. However, I am sure that there could be more papers in the past two years in this area.**\n\nThanks for the suggestion. As illustrated in the response to point W1, our current comparison includes two recent state-of-the-art methods: OFT (NeurIPS 2023) [1] and NeTI (SIGGRAPH Asia 2023) [2]. In addition to the above comparison to SuTI and Instruct-Imagen, we further include comparisons with two other open-source models, DreamMatcher (CVPR 2024) [6] and FreeCustom (CVPR 2024) [7], in Figure R2 of the attached PDF. As can be observed, our method achieves superior performance in both identity preservation and text alignment compared to these methods. We will add these comparisons to the revised paper.\n\n>**Limitation: While the proposed approach is novel, the core idea of decomposing the personalization process into multiple stages is not entirely groundbreaking. A more comprehensive exploration of the relationship between the proposed method and existing work would be beneficial.**\n\nThe relationship between our method and existing multi-stage methods is discussed in the 'Multi-Stage Personalization' section of Related Work (i.e., Lines 95 - 108). Our method differs from existing methods in several aspects. Firstly, the motivation for our first stage (i.e., optimizing the textual embedding) differs from existing methods, where we focus on learning the embedding alignment while mitigating the risk of overfitting. Consequently, we significantly reduce the optimization steps to just 60 and lower the learning rate. Secondly, we decompose the learning process into three stages: learning the embedding alignment, refining the attention map, and capturing the subject identity. Thirdly, we utilize the super-category token to guide the attention map learning in a self-supervised manner throughout all training stages. We will include more multi-stage methods, such as MagiCapture [8], for comparison, making a more comprehensive discussion about multi-stage personalization methods.\n\n**References**\n\n[1] Zeju Qiu et. al. \"Controlling Text-to-Image Diffusion by Orthogonal Finetuning\". NeurIPS, 2023.\n\n[2] Yuval Alaluf et. al. \"A Neural Space-Time Representation for Text-to-Image Personalization\". SIGGRAPH Asia, 2023.\n\n[3] Wenhu Chen et. al. \"Subject-driven Text-to-Image Generation via Apprenticeship Learning\". NeurIPS, 2023.\n\n[4] Hexiang Hu et. al. \"Instruct-Imagen: Image Generation with Multi-modal Instruction\". CVPR, 2024.\n\n[5] Nataniel Ruiz et. al. \"HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Models\". CVPR, 2024.\n\n[6] Jisu Nam et. al. \"DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization\". CVPR, 2024.\n\n[7] Ganggui Ding et. al. \"FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition\". CVPR, 2024.\n\n[8] Junha Hyung et. al. \"Magicapture: High-resolution multi-concept portrait customization\". AAAI, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LbYU2gijXd", "reviewer": "Reviewer_hyCc", "summary": "The author proposed a method to generate high-quality personalized images. First, a textual embedding is learned, then the cross-attention layers is  finetund to refine attention map during learning the  textual embedding, finally, the entire U-Net is trained to capture the subject identity.", "questions": "1. Where does the super-category label come from? Is it a part of the annotation?\n2. I am curious how the  TI+DB perform if it is optimized step by step like AttnDreamBooth, e.g. first tuning the textual embedding and then the U-Net or first tuning U-Net then optimizing the embedding.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is welll-written and easy to understand.\n2. The proposed method achieve competitive results comparing with SOTAs.\n3. Using the super-category attention map is an interesting idea to calibrate the [V] attention map in a self-supervised manner.", "weaknesses": "1. The proposed method needs a costly test-time optimization to generate personalized images. The 3 stage finetuning requires expensive computation (20 minutes in A100)\n2. Even though the Attention Map Regularization idea is interesting, it seems to be the only critical contribution in this paper. The authors combine some techniques from off-the-shelf method including CostumDiffusion/DB/TI, and conducted them step by step with proposed -attention map regularization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive comments. We response the reviewer’s concerns as follows.\n\n>**W1: The proposed method needs a costly test-time optimization to generate personalized images. The 3 stage finetuning requires expensive computation (20 minutes in A100).**\n\nIndeed, computational overhead is a limitation of our method. To address this, we explored a simple yet effective strategy to reduce the training time. This involves increasing the learning rate while simultaneously decreasing both the training steps and the batch size for our third training stage, which is notably the most time-consuming phase. Specifically, the third stage of our original model performs 500 steps with a learning rate of 2e-6 and a batch size of 8. The fast version now completes training in just 200 steps with a learning rate of 1e-5 and a batch size of 4. This adjustment significantly reduces the training time from 20 minutes to 6 minutes on average. Interestingly, this fast version maintains performance comparable to our original model, likely because the first two stages provide a convenient starting point, allowing for a higher learning rate in the third stage. The qualitative evaluation is provided in Figure R4 of the attached PDF, and quantitative results are detailed in the table below. We observed that the fast version model performs very closely to the original model for short prompts (e.g., it even slightly outperforms the original model in the quantitative evaluation), but it slightly underperforms for complex prompts (e.g., the second and fourth examples in Figure R4).\n\n| Methods              | Identity Preservation | Text Alignment | Training Time |\n| :---------------- | :------: | :----: | :----: | \n| NeTI [1]  \t\t| 0.6901  | 0.2522 | 13 minutes |\n| Ours-fast       |  0.7268   | 0.2536 | 6 minutes |\n| Ours-original |  0.7257   | 0.2532 | 20 minutes |\n\nWe thank hyCc for this point and will definitely include these results of this fast version model in the revised paper!\n\n>**W2: Even though the Attention Map Regularization idea is interesting, it seems to be the only critical contribution in this paper. The authors combine some techniques from off-the-shelf method including CostumDiffusion/DB/TI, and conducted them step by step with proposed attention map regularization.**\n\nWe appreciate your recognition of our attention map regularization's contribution. In addition to this regularization, we would like to clarify other key contributions of our paper. First, we identify a crucial insight in text-to-image personalization: the textual embedding of the new concept should be aligned with and seamlessly integrated into existing tokens, unlike existing studies that mainly focus on learning the new concept itself. Second, based on this insight, we propose a three-stage personalization process that is more than just a combination of existing techniques. The first stage is designed to learn the embedding alignment while mitigating the risk of overfitting, thus reducing the optimization steps to just 60. This approach results in a very coarse attention map and identity for the new concept. Subsequently, we refine the attention map by fine-tuning the cross-attention layers in the second stage, followed by fine-tuning the U-Net to capture the concept's identity in the third stage.\n\n>**Q1: Where does the super-category label come from? Is it a part of the annotation?**\n\nYes, the dataset provides a coarse descriptor for each concept. In fact, many approaches, such as Textual Inversion, DreamBooth, and NeTI, also require a super-category label to initialize the textual embedding of the new concept or to provide prior knowledge. Regarding the use of the super-category label in our attention map regularization, this method does not necessitate a precise super-category label, as it does not enforce a strong constraint between the new concept and the super-category token. Instead, we impose a flexible constraint that enforces similarity in the mean and variance of the attention map values, as illustrated in Eq. (2) of the main paper. This strategy aims to encourage the new concept to exhibit a level of concentration or dispersion in the attention map similar to that of the super-category token.\n\n>**Q2: I am curious how the TI+DB perform if it is optimized step by step like AttnDreamBooth, e.g. first tuning the textual embedding and then the U-Net or first tuning U-Net then optimizing the embedding.**\n\nIn Figure R5 of the attached PDF, we present the results of the two suggested settings for TI+DB: 1) first tuning the textual embedding and then the U-Net (denoted as TI -> DB), and 2) first tuning the U-Net and then the textual embedding (denoted as DB -> TI). As shown, both models fail to generate text-aligned images. While the TI -> DB setting improves performance compared to using TI or DB individually, it still suffers from overfitting issues. The DB -> TI setting performs very closely to the DB model alone. In contrast, our method successfully generates images that preserve concept identity and align with the text.\n\n**References**\n\n[1] Yuval Alaluf et. al. \"A Neural Space-Time Representation for Text-to-Image Personalization\". SIGGRAPH Asia, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oYAhsS23rV", "reviewer": "Reviewer_U1BU", "summary": "This paper proposes a method to enhance the performance of personalizing text-to-image models by appropriately combining textual inversion approach, which learns new embeddings, and DreamBooth approach, which fine-tunes model weights. They demonstrate the effectiveness of their approach through qualitative evaluation and human evaluation.", "questions": "When conducting analysis in Figure 1, how many steps was DreamBooth trained for to obtain the results? It seems that with sufficient training steps, DreamBooth might exhibit different results.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Research on appropriately combining textual inversion and DreamBooth methods has been needed in this field, and they propose a reasonable and clear method for this.\n* The experimental results are good.", "weaknesses": "* (minor) The proposed method seems quite similar to the Magicapture [1] approach in that it separates embedding learning and weight fine-tuning and conducts regularization on attention. However, this paper does not cover a comparison with Magicapture. \n* (minor) The ablation study is not conducted quantitatively and relies on a single generation scenario.\n\n[1] Hyung, Junha, Jaeyo Shin, and Jaegul Choo. \"Magicapture: High-resolution multi-concept portrait customization.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive comments. We response the reviewer’s concerns as follows.\n\n>**W1: (minor) The proposed method seems quite similar to the Magicapture [1] approach in that it separates embedding learning and weight fine-tuning and conducts regularization on attention. However, this paper does not cover a comparison with Magicapture.**\n\nThanks for the suggestion to compare our method with MagiCapture [1]. Indeed, MagiCapture also employs a multi-stage learning strategy that first optimizes the textual embedding and then jointly finetunes the textual embedding and U-Net. However, our method differs in several key aspects. First, the motivation for our first stage (i.e., optimizing the textual embedding) differs from MagiCapture. We focus on learning the embedding alignment while mitigating the risk of overfitting, thus significantly reducing the optimization steps to just 60. In contrast, MagiCapture optimizes the textual embedding for 1200 steps in the first stage. Second, the attention regularization in MagiCapture is applied with the help of a user-provided mask, while we utilize the super-category token to guide the attention map learning in a self-supervised manner. Third, our learning process is divided into three stages: learning the embedding alignment, refining the attention map, and capturing the subject identity. Additionally, as MagiCapture is designed for integrating subject and style concepts, and our method focuses solely on personalizing subject concepts, a direct comparison of performance between these two methods is not feasible. We will include this comparison with MagiCapture in the revised paper.\n\n\n>**W2: (minor) The ablation study is not conducted quantitatively and relies on a single generation scenario.**\n\nThank you for pointing this out. The table below presents the quantitative results of our ablation study. Specifically, the model without Stage 1 achieves better text alignment but significantly poorer identity preservation compared to the full model. This is because, without sufficient training of the textual embedding, the model tends to overlook the learned concept or generate it with significant distortions. Please note that the text alignment score is calculated without considering the new concept; therefore, omitting the new concept can inadvertently boost this score. Similarly, models without Stage 2 or Stage 3 also exhibit higher text alignment scores but lower identity preservation scores, due to insufficient learning of the attention maps and the subject identity, respectively. Additionally, the model without the regularization term shows degraded text alignment. Regarding qualitative evaluation, more generated images are provided in Figure 13 of the Appendix. We will include this quantitative ablation study and more generated images in the revised paper.\n\n| Methods              | Identity Preservation | Text Alignment |\n| :---------------- | :------: | ----: |\n| w/o Stage 1    |   0.7031   | 0.2595 |\n| w/o Stage 2    |   0.7145   | 0.2541 |\n| w/o Stage 3    |  0.6821   | 0.2650 |\n| w/o Reg          |  0.7269   | 0.2502 |\n| Full Model |  0.7257   | 0.2532 |\n\n>**Q1: When conducting analysis in Figure 1, how many steps was DreamBooth trained for to obtain the results? It seems that with sufficient training steps, DreamBooth might exhibit different results.**\n\nIn Figure 1, we follow NeTI [2] to perform 500 steps with a batch size of 4 for training DreamBooth. The results of training DreamBooth for more steps are provided in Figure R3 of the attached PDF. Indeed, for certain examples (e.g., \"Manga drawing of a [V] can\"), DreamBooth can generate text-aligned images after 1,000 training steps. However, for many other examples (as shown in Figure R3), DreamBooth still tends to overlook the new concept even after 5,000 training steps. In contrast, our method successfully generates text-aligned images for these prompts. We appreciate this point and will include this discussion in the revised paper.\n\n**References**\n\n[1] Junha Hyung et. al. \"Magicapture: High-resolution multi-concept portrait customization\". AAAI, 2024.\n\n[2] Yuval Alaluf et. al. \"A Neural Space-Time Representation for Text-to-Image Personalization\". SIGGRAPH Asia, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JaTRgCjwl1", "reviewer": "Reviewer_FYJx", "summary": "This paper proposes a new method for personalized image generation, decompose the personalization process into three training stages and introducing a cross-attention map regularization term.", "questions": "1.\tDo the constraints imposed between [V] and [super-category] across all layers in Eq. 2 potentially restrict the diversity? Considering that [super-category] represents the original identity information and [V] introduces diversity.\n\n2.\tWhy was SD2.1 chosen, and how do other models perform?\n\n3.\tIn line 197, it is mentioned that retaining the prior preservation loss leads to poor results. Can you provide experimental results to support this?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript is well-written.\n\nThe authors propose to address the intrinsic issues of two classical and up-to-date methods, Textual Inversion and DreamBooth, by aligning the new concept from the prompt while preserving the original subject identity. This approach tackles a significant problem.", "weaknesses": "1.\tSome parts of the manuscript are slightly verbose. For instance, the introduction section already introduces the existing problems of Textual Inversion and DreamBooth, along with their corresponding analysis and naïve solution. However, these points are reiterated in Section 4.1 and the first paragraph of Section 4.2 without adding any new information.\n\n2.\tWhat does the term 'this approach' refer to in Line 174?\n\n3.\tIn general, the proposed method tends to be a bag of tricks with some customized hype-parameters.\n\n4.\tImportant comparisons are lacking in the manuscript, specifically the comparison with Textual Inversion (TI) and DreamBooth (DB) individually, in addition to the comparison with the combined approach of TI+DB.\n\n5.\tThis method uses cross-attention map for regularization, which results in a high time cost; on a NVIDIA A100, training requires 660 steps, taking 20 minutes. In contrast, DreamBooth requires only 5 minutes to train for 1000 steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive comments. We response the reviewer’s concerns as follows.\n\n>**W1: The problems and analysis of Textual Inversion and DreamBooth introduced in the Introduction section are reiterated in Section 4.1**\n\nThanks for this point. The current version of Section 4.1 provides a more detailed problem analysis of existing methods than the Introduction section. We agree that minimizing repetition can enhance the clarity and conciseness of the paper. To improve conciseness, we will revise Sections 1 and 4.1 as follows: 1) remove the description of the naive solution from Section 1; 2) condense the analysis of existing methods in Section 1; 3) streamline the problem description in Section 4.1; and 4) enhance Section 4.1 with a more detailed analysis and deeper insights.\n\n>**W2: What does the term 'this approach' refer to in Line 174?**\n\nIn Line 174, \"this approach\" refers to Textual Inversion, which involves optimizing the input textual embedding of the text encoder to learn the new concept. Thank you for this point, and we will revise Lines 173-174 as: \"To achieve this, we choose to optimize the input textual embedding of the text encoder, as Textual Inversion does, given that the text encoder manages the contextual understanding of the prompt. However, as analyzed in Section 4.1, this approach is prone to overfitting the textual embedding, resulting in an embedding misalignment issue.\"\n\n>**W3: The proposed method tends to be a combination of existing methods.**\n\nWe would like to clarify the key contributions of the paper. First, we identify a crucial insight in text-to-image personalization: the textual embedding of the new concept should be aligned with and seamlessly integrated into existing tokens, unlike existing studies that mainly focus on learning the new concept itself. Second, based on this insight, we propose a three-stage personalization process that is more than just a combination of existing techniques. The first stage is designed to learn the embedding alignment while mitigating the risk of overfitting, thus we significantly reduce the optimization steps to just 60. This results in a very coarse attention map and identity for the new concept. Subsequently, we refine the attention map by fine-tuning the cross-attention layers in the second stage, followed by fine-tuning the U-Net to capture the concept's identity in the third stage. Third, we propose an attention map regularization that utilizes the super-category token to guide the attention map learning in a self-supervised manner. Thanks for pointing this out, and we will improve the statements about the contributions in the revised paper.\n\n>**W4: Comparisons to Textual Inversion (TI) and DreamBooth (DB) individually.**\n\nThanks for the suggestion. Comparisons to TI and DB are provided in Figure R5 of the attached PDF. As shown, our model achieves superior performance in both identity preservation and text alignment. We will add this comparison to the revised paper.\n\n>**W5: The proposed attention map regularization results in a high time cost. The method takes 20 minutes for 660 steps, while DreamBooth takes only 5 minutes for 1000 steps.**\n\nIndeed, computational overhead is a limitation of our method. We would like to clarify that the attention map regularization is not the most time-comsuming aspect, adding an average of 67 seconds. Instead, the third training stage consumes the most time. To mitigate this, we developed an effective strategy to reduce the training time. Due to the limited text length, please refer to our global response for the details. Our fast version model significantly reduces the training time from 20 minutes to 6 minutes, while maintaining performance comparable to the original model. We thank FYJx for this point, and will definitely include the results of this fast version model in the revised paper!\n\n>**Q1: Do the constraints imposed between [V] and [super-category] in Eq. 2 potentially restrict the diversity? Considering that [super-category] represents the original identity information and [V] introduces diversity.**\n\nWe assume that the \"diversity\" refers to the distinctive characteristics of the new concept [V], but please correct us if we are wrong. Our proposed constraint does not restrict the diversity of [V], as it does not enforce a strong constraint between [V] and [super-category]. Instead, we apply a flexible constraint that enforces similarity in the mean and variance of the attention map values. This strategy aims to encourage [V] to exhibit a level of concentration or dispersion in the attention map similar to that of [super-category].\n\n>**Q2: Why was SD2.1 chosen, and how do other models perform?**\n\nWe empirically find that SD2.1 achieves better performance than SD1.5 for our method. SD2.1 is widely used in text-to-image personalization models, such as AnyDoor [1], ADI [2], and IDAdapter [3]. A visual comparison between our models with SD1.5 or SD2.1 is presented in Figure R6 of the attached PDF. As shown, the model with SD2.1 achieves superior performance in text alignment and identity preservation. Nevertheless, our method is also effective for SD1.5, and it outperforms the baseline methods. We will add this discussion to the revised paper.\n\n>**Q3: It is mentioned that retaining the prior preservation loss leads to poor results. Can you provide experimental results to support this?**\n\nThanks for pointing this out. We present a visual comparison between models with or without the prior preservation loss in Figure R7 of the attached PDF. The results show that incorporating the prior preservation loss leads to degradation in identity preservation. We will add this comparison to the revised paper.\n\n[1] AnyDoor: Zero-shot Object-level Image Customization, 2023\n\n[2] Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation, 2023\n\n[3] IDAdapter: Learning Mixed Features for Tuning-Free Personalization of T2I Models, 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their constructive and thoughtful feedback. We are encouraged that the reviewers find our idea novel (ZpWY) and interesting (hyCc), and our method reasonable and clear (U1BU). We are pleased that they consider our results to be good (U1BU) and competitive compared to state-of-the-art approaches (hyCc). We appreciate their recognition of our paper as well-written and easy to understand (hyCc, FYJx). Moreover, ZpWY, U1BU, and FYJx recognize that our approach addresses a key problem in text-to-image personalization, while ZpWY confirms that our approach advances the field.

**[Response to a Common Concern]**

Below, we address a common concern regarding the training time of our model. Point-to-point responses are included as a reply to each reviewer.

**1. Concern regarding the training time of our model**

Indeed, computational overhead is a limitation of our method. To address this, we explored a simple yet effective strategy to reduce the training time. This involves increasing the learning rate while simultaneously decreasing both the training steps and the batch size for our third training stage, which is notably the most time-consuming phase. Specifically, the third stage of our original model performs 500 steps with a learning rate of 2e-6 and a batch size of 8. The fast version now completes training in just 200 steps with a learning rate of 1e-5 and a batch size of 4. This adjustment significantly reduces the training time from 20 minutes to 6 minutes on average. Interestingly, this fast version maintains performance comparable to our original model, likely because the first two stages provide a convenient starting point, allowing for a higher learning rate in the third stage. The qualitative evaluation is provided in Figure R4 of the attached PDF, and quantitative results are detailed in the table below. We observed that the fast version model performs very closely to the original model for short prompts (e.g., it even slightly outperforms the original model in the quantitative evaluation), but it slightly underperforms for complex prompts (e.g., the second and fourth examples in Figure R4).

| Methods              | Identity Preservation | Text Alignment | Training Time |
| :---------------- | :------: | :----: | :----: | 
| NeTI [1]  	    | 0.6901  | 0.2522 |  13 minutes |
| Ours-fast       |  0.7268   | 0.2536 | 6 minutes |
| Ours-original |  0.7257   | 0.2532 | 20 minutes |

We thank the reviewers for this point, and will definitely include these results of this fast version model in the revised paper!

**[Additional Experimental Results]**

We summarize our additional experimental results below. Please refer to the attached PDF file for the figures. In the following, please note that Figure R* denotes the figure in the attached PDF.
1. **Comparison to SuTI [1] and Instruct-Imagen [2].** In Figure R1, we present a visual comparison with SuTI and Instruct-Imagen. Due to the unavailability of open-source models for these two methods, we compare with the examples provided in their papers.
2. **Comparison to DreamMatcher [3] and FreeCustom [4].** In Figure R2, we present a visual comparison with DreamMatcher and FreeCustom.
3. **Fast version of our model.** In Figure R4, we provide the results of our fast version model, which significantly reduces the training time from 20 minutes to 6 minutes. 
4. **Quantitative ablation study.** The results of the quantitative ablation study are detailed in the response to reviewer U1BU's point W2.
5. **DreamBooth with more training steps.** In Figure R3, we provide the results of training DreamBooth with more training steps.
6. **Different settings of TI+DB.** In Figure R5, we present a visual comparison with two different settings for TI+DB: 1) first tuning the textual embedding and then the U-Net, and 2) first tuning the U-Net and then the textual embedding.
7. **Comparison to using TI or DB individually.** In Figure R5, we present a visual comparison to using TI or DB individually.
8. **Our model using SD1.5.** In Figure R6, we present the results of our model using SD1.5.
9. **Our model with the prior preservation loss.** In Figure R7, we present a visual comparison of our models with or without the prior preservation loss.

All the above additional experimental results will be added to the main text or appendix of the revised paper.

**References**

[1] Wenhu Chen et. al. "Subject-driven Text-to-Image Generation via Apprenticeship Learning". NeurIPS, 2023.

[2] Hexiang Hu et. al. "Instruct-Imagen: Image Generation with Multi-modal Instruction". CVPR, 2024.

[3] Jisu Nam et. al. "DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization". CVPR, 2024.

[4] Ganggui Ding et. al. "FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition". CVPR, 2024.

Meta-review: This paper proposes a new method for personalized high-quality image generation, called AttnDreamBooth.

Intially, reviewers pointed out some concerns such as less originality, ambigous description, lack of comparisons while respecting the importance of the tasks addressed and its method efficacy, with three positive scores and one rejection score. 

In particular, AC carefully read R-ZpWY's comments considering his/her rejection score.
Then, AC found that most methods that the reviewer suggested to use for comparisons are publihsed after NeurIPS deadline or are not available. 

Considering the situation and the contributions of this paper, AC thinks this paper is enough to be presented in NeurIPS, so recommends accepting this paper. 

AC asks the authors reflect all important comments of reviewers in the cam-ready version.
Decision: Accept (poster)