AttnDreamBooth: Towards Text-Aligned Personalized Text-to-Image Generation

Authors: lianyu  pang
jian  yin
baoquan  zhao
feize  wu
fu lee wang
qing  li
xudong  mao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:54:46

Abstract: Recent advances in text-to-image models have enabled high-quality personalized image synthesis based on user-provided concepts with flexible textual control. In this work, we analyze the limitations of two primary techniques in text-to-image personalization: Textual Inversion and DreamBooth. When integrating the learned concept into new prompts, Textual Inversion tends to overfit the concept, while DreamBooth often overlooks it. We attribute these issues to the incorrect learning of the embedding alignment for the concept. To address this, we introduce AttnDreamBooth, a novel approach that separately learns the embedding alignment, the attention map, and the subject identity across different training stages. We also introduce a cross-attention map regularization term to enhance the learning of the attention map. Our method demonstrates significant improvements in identity preservation and text alignment compared to the baseline methods.

Main paper: 
 introduction: 
 Text-to-image personalization [24, 71, 50] is the task of customizing a pre-trained diffusion model to produce images of user-provided concepts in novel scenes or styles. By providing several examples of a new concept, personalization techniques enable users to employ novel prompts to generate personalized images containing that concept. Current personalization techniques primarily fall into two categories: the first approach involves inverting the new concept into the textual embedding [24] ; the second approach involves fine-tuning the diffusion model to learn the new concept [71] . Personalization techniques aim to generate high-quality images of user-provided concepts, achieving high identity preservation and text alignment. However, despite the significant progress in personalization techniques, balancing the trade-off between identity preservation and text alignment remains a challenge for current approaches.Figure 2 shows the personalization results from Textual Inversion [24] and DreamBooth [71] . Textual Inversion tends to generate images that focus primarily on the learned concept, often neglecting other elements of the prompt. In contrast, DreamBooth appears to overlook the learned concept, producing images that are more influenced by other prompt tokens. These issues can be attributed to the incorrect learning of embedding alignment for the new concept, i.e., the embedding of the new concept is not functionally compatible with the embeddings of existing tokens.Based on these observations, our approach aims to properly learn not only the subject identity but also the embedding alignment and the attention map for the new concept. Our key insights are as follows: 1) In the early stages of optimization, Textual Inversion effectively learns the embedding alignment but tends to overfit after extensive optimization steps; 2) DreamBooth accurately captures the subject identity but struggles with learning the embedding alignment.In this paper, we propose a method named AttnDreamBooth, which separates the learning processes of the embedding alignment, the attention map, and the subject identity. Specifically, our approach consists of three main training stages, as illustrated in Figure 3 . First, we optimize the textual embedding to learn the embedding alignment while preventing the risk of overfitting, which results in a coarse attention map for the new concept. Next, we fine-tune the cross-attention layers of the U-Net to refine the attention map. Lastly, we fine-tune the entire U-Net to capture the subject identity. Note that the text encoder remains fixed throughout all training stages to preserve its prior knowledge of contextual understanding.Furthermore, we introduce a cross-attention map regularization term to enhance the learning of the attention map. Throughout the three training stages, we use a consistent training prompt, "a photo of a [V] [super-category]", where [V] and [super-category] denote the tokens for the new concept and its super-category, respectively. Our attention map regularization term encourages similarity between the attention maps of the new concept and its super-category.Output "[V]" "Drawing" Output "[V]" "Drawing" Output "[V]" "Drawing" "Manga drawing of a [V] can" Output "[V]" "Box" Output "[V]" "Box" Output "[V]" "Box" "A [V] toy inside a box" Figure 2 : Analysis of two principal methods. We visualize the cross-attention maps corresponding to the new concept and other tokens in the prompt. Textual Inversion [24] tends to overfit the textual embedding of the learned concept, resulting in incorrect attention map allocations to other tokens (e.g., "drawing" or "box"). In contrast, DreamBooth [71] appears to overlook the learned concept, producing images primarily based on other tokens. To demonstrate the effectiveness of AttnDreamBooth, we compare it with four state-of-the-art baseline methods through both qualitative and quantitative evaluations. Our method achieves superior performance in terms of identity preservation and text alignment compared to the baselines. More importantly, AttnDreamBooth enables a variety of text-aligned personalized generations with complex prompts. 
related work: 
 Text-to-Image Generation. Generative models are designed to create new samples that resemble the patterns observed in their training data. There are various types of generative models, including VAEs [47, 76, 14] , GANs [27, 6, 44] , auto-regressive models [66, 91] , flow-based models [21, 48] , and diffusion models [75, 33, 59, 18] . These models can be enhanced by conditioning on text prompts, which are known as text-to-image models [68, 66, 60, 20, 23, 16, 5] . Recent advancements [73, 67, 70] in text-to-image generation, powered by training on extremely large-scale datasets, have demonstrated an impressive ability to generate diverse and generalized outputs.Text-to-Image Personalization. Leveraging the impressive capabilities of diffusion models, textto-image personalization involves adapting pre-trained diffusion models to capture new concepts depicted in several given images. Pioneering works approach this by inverting the concept into the textual embedding [24] , or by fine-tuning the diffusion model [71] . However, these methods often struggle to balance the trade-off between identity preservation and text alignment, and typically require substantial time for optimization. To overcome these limitations, some studies focus on enhancing the identity preservation of the concept [81, 1, 94, 36, 31, 43, 40] , while others aim to improve text alignment [78, 3, 4, 90, 37] . Additionally, there is a growing trend of research attempting to accelerate the personalization process, either by reducing the number of tuning parameters [50, 29, 54, 34, 28, 57] , or by pre-training on large datasets [85, 74, 39, 2, 25, 10, 72, 51, 87, 56, 11, 55] .Given the widespread interest in human synthesis, many studies also concentrate on the personalized synthesis of human faces [92, 62, 89, 53, 79, 83, 8, 49, 61, 45, 86, 17, 13, 82, 12] .Cross-Attention Control. The cross-attention layers [70] have been shown to play a crucial role in diffusion models. The control of cross-attention layers has proven effective in a variety of tasks, including image editing [32] , compositional synthesis [22, 52, 7, 26, 46] , and layoutcontrolled synthesis [63, 9, 88] . In text-to-image personalization, several studies [50, 87, 4, 78, 30, 42, 58, 93] also have explored the control of cross-attention layers. Custom Diffusion [50] illustrates how incorrect attention maps of the learned concepts can lead to unsuccessful synthesis. FastComposer [87] and Break-A-Scene [4] propose using segmentation masks of the target concepts to guide the learning of the attention maps, thereby enhancing text alignment, especially in scenarios involving multiple concepts. Perfusion [78] identifies the attention overfitting issue and addresses it by fixing the cross-attention key matrices of the target concepts to their super-category tokens.Multi-Stage Personalization. Several studies [24, 54, 4, 41, 38] have explored combining the strengths of different methods into more efficient models through a multi-stage approach. Inspired by PTI [69] , Textual Inversion [24] investigated a two-stage approach to enhance identity preservation by first optimizing the textual embedding and then fine-tuning the diffusion model to better capture the subject identity. Similarly, MagiCapture [38] first optimizes the textual embedding and then applies LoRA [34] in the U-Net. Break-A-Scene [4] proposes initially optimizing the textual embedding with a high learning rate, followed by fine-tuning both the U-Net and the text encoder using a significantly lower learning rate. Our method differs in several aspects. First, the motivation for our first stage (i.e., optimizing the textual embedding) differs from previous methods. Specifically, we focus on learning the embedding alignment while mitigating the risk of overfitting, thus significantly reducing the optimization steps and lowering the learning rate. Second, we decompose the learning process into three stages: learning the embedding alignment, refining the attention map, and capturing the subject identity. Third, we introduce a cross-attention map regularization to guide the learning of the attention map in a self-supervised manner. 
preliminaries: 
 Latent Diffusion Models. Our approach is based on the publicly available Stable Diffusion model, a type of Latent Diffusion Model (LDM) [70] for text-to-image generation. In LDM, an autoencoder is utilized to provide a lower-dimensional representational space, where an encoder E transforms an image x into a latent representation z = E(x), and a decoder D reconstructs the image from this latent code, i.e., D(E(x)) ≈ x. Additionally, a Denoising Diffusion Probabilistic Model (DDPM) [33] is employed to produce latent codes within the latent space of the autoencoder. To generate images from text, the model leverages a conditioning vector c(y), derived from a given text prompt y. The training objective of LDM is given by:EQUATIONwhere the denoising network ε θ is tasked with recovering the original latent code z 0 from the noised latent code z t , given a specific timestep t and the conditioning vector c(y).Textual Inversion. Textual Inversion (TI) [24] personalizes a pre-trained diffusion model by encoding the target concept into the textual embedding. Given several images of a target concept, TI introduces a new token S * and its associated textual embedding v * to represent the concept. The learning process of TI involves initializing v * with a coarse descriptor and then optimizing it to minimize the diffusion objective (Eq. 1).DreamBooth. DreamBooth (DB) [71] learns the target concept by fine-tuning the pre-trained diffusion model. Given several images of a target concept, DB labels all the images with the prompt "a [V] [super-category]", where [V] is a rare token in the vocabulary. The learning process of DB involves fine-tuning the entire U-Net (and possibly the text encoder) using the diffusion objective (Eq. 1) combined with a prior preservation loss [71] . 
method: 
 In this section, we first analyze the problems associated with Textual Inversion and DreamBooth, as discussed in Section 4.1. To address these issues, we propose a novel method named AttnDreamBooth, as detailed in Section 4.2. To further enhance text alignment, we introduce a cross-attention map regularization term in Section 4.3.Problems and Analysis. As illustrated in Figure 2 , Textual Inversion and DreamBooth encounter distinct challenges when integrating the learned concept into novel prompts. For Textual Inversion, the generated images often excessively focus on the learned concept, overlooking other prompt tokens. To investigate this issue, we present the attention map visualization using DAAM [77] for different tokens in Figure 2 . This visualization reveals an embedding misalignment issue in novel compositions containing the concept, leading to incorrect attention map allocations for other tokens. A typical example is shown where the attention map corresponding to the "drawing" token focuses on incorrect regions. This misalignment occurs because Textual Inversion tends to overfit shows the generated images based on a single word prompt, "[V]", both before and after fine-tuning, using the diffusion model without fine-tuning. These images are notably similar to each other, which indicates that the learned textual embedding remains largely unchanged from its initial state.the input embedding of the text encoder, responsible for managing the contextual understanding of the prompt. Conversely, images generated by DreamBooth sometimes focus solely on other prompt tokens, neglecting the learned concept. This occurs because DreamBooth uses a rare token for the new concept while keeping its textual embedding fixed, thereby leading to insufficient learning of the embedding alignment for the new concept.A Naive Solution. As analyzed previously, Textual Inversion and DreamBooth exhibit distinct issues related to the embedding alignment: Textual Inversion tends to overfit the embedding alignment for the new concept, while DreamBooth demonstrates insufficient learning of the embedding alignment. A straightforward solution is to combine Textual Inversion with DreamBooth by jointly tuning the textual embedding and the U-Net, a method we denote as TI+DB. We observe that TI+DB enhances performance over Textual Inversion or DreamBooth individually. However, it still tends to neglect the learned concept when integrating it into new prompts. This issue arises from the slow update of the textual embedding relative to the U-Net. As illustrated in Figure 4 , the learned textual embedding remains very close to its initial state. Furthermore, we calculate the cosine similarity between the learned and initial embeddings, which averages about 0.9997, indicating that TI+DB still suffers from insufficient learning of the embedding alignment.To address the issues described in Section 4.1, we propose a method named AttnDreamBooth, inspired by two key observations. First, while Textual Inversion often fails to capture the subject identity and tends to overfit the embedding alignment for the new concept, it can effectively learn the embedding alignment in the very early stages of optimization. However, at these early stages, the model only learns a coarse cross-attention map for the new concept. Second, although DreamBooth fails to learn the embedding alignment, it can accurately capture the subject identity. Based on these observations, we propose to decompose the personalization process into three training stages: 1) learning the embedding alignment; 2) refining the attention map; and 3) acquiring the subject identity. An overview of our proposed AttnDreamBooth is illustrated in Figure 3 .Learning the Embedding Alignment. As previously stated, learning the embedding alignment for the new concept is critical for properly allocating the cross-attention maps for novel prompts, which in turn influences the text alignment of the personalized generation results. To achieve this, we optimize the input textual embedding of the text encoder, since the text encoder manages the contextual understanding of the prompt. However, as analyzed in Section 4.1, this approach is prone to overfitting the embedding, leading to an embedding misalignment issue. Therefore, our objective at this stage is to learn the embedding alignment while minimizing the risk of overfitting. To this end, we adapt Textual Inversion [24] with three main modifications. First, we significantly reduce the number of optimization steps (to 60 steps in our experiments) and lower the learning rate (to 10 -3 ). Second, we introduce a cross-attention map regularization (see Section 4.3) to guide the learning of the cross-attention map. Third, to facilitate the incorporation of the cross-attention map regularization, we set the training prompt as "a photo of a [V] [super-category]". To prevent overfitting, we stop the optimization at very early stages, thereby resulting in a coarse cross-attention map for the new concept, as depicted in Figure 5 . A full analysis of attention map allocations for each token is presented in Appendix E. The cross-attention map, as well as the subject identity, are addressed in subsequent steps.Refining the Cross-Attention Map. To mitigate the embedding misalignment issue, our model initially learns a relatively coarse cross-attention map for the new concept. At this stage, we focus on refining the cross-attention map. Since these attention maps are embedded within the cross-attention layers, inspired by Custom Diffusion [50] , we fine-tune all the cross-attention layers in the U-Net. Additionally, we employ the proposed cross-attention map regularization (see Section 4.3) to aid in refining the attention map. Furthermore, we keep the textual embedding and the text encoder fixed to prevent further embedding misalignment.Capturing the Subject Identity. As illustrated in Figure 5 , the previous stage produces images that are similar to the target concept but still exhibit significant distortions. Therefore, in the third stage, following DreamBooth [71] , we unfreeze all layers of the U-Net to more accurately capture the subject identity of the target concept. We choose not to adopt the prior preservation loss [71] , as we empirically find that it leads to poor identity preservation and requires significantly more training steps. A detail discussion of our models with or without the prior preservation loss could be found in Appendix H. Moreover, similar to the previous stage, we keep the textual embedding and the text encoder fixed to prevent embedding misalignment, and we continue to apply the cross-attention map regularization to guide the learning of the attention map.We set the training prompt as "a photo of a [V] [super-category]", where [V] and [super-category] denote the tokens for the new concept and its super-category, respectively. To enhance the learning of the attention map, we introduce a regularization term that encourages similarity between the attention maps of [V] and [super-category]. This regularization term serves two purposes. First, since the new concept and its super-category belong to the same object category, the attention map of the super-category token can serve as a reference for the new concept. Second, since [V] and [super-category] are used together to describe the new concept when integrating it into new prompts, the attention maps of [V] and [super-category] should refer to the same region.Formally, for the 16 attention maps {M 1 , M 2 ..., M 16 } from 16 different cross-attention layers, we minimize the squared differences in the mean and variance of the attention map values for [V] and [super-category] as follows:EQUATIONwhere µ(M 1:16 ) and σ 2 (M 1:16 ) denote the mean and variance of all the values across the 16 attention maps, respectively. This constraint helps ensure that the new concept exhibits a similar level of concentration or dispersion in the attention map as the super-category token. Note that we avoid directly applying the constraint to the attention map values themselves because we empirically find that such a constraint is too restrictive and difficult to optimize. We present four images generated by our method and two images from each of the baseline methods, including TI+DB [24, 71] , NeTI [1] , ViCo [30] , and OFT [64] . Our method demonstrates superior performance in text alignment and identity preservation compared to these baselines.  
experiments: 
 In this section, we first present the implementation details of our method. Subsequently, we evaluate its performance by conducting a comparative analysis with four state-of-the-art personalization methods. Lastly, we conduct an ablation study to demonstrate the effectiveness of each sub-module.Implementation Details. Our implementation is based on the publicly available Stable Diffusion V2.1 [70] . The textual embedding of the new concept is initialized using the embedding of the super-category token. We keep a fixed batch size of 8 across all training stages but vary the learning rates and training steps. Specifically, we train with a learning rate of 10 -3 for 60 steps in stage 1, followed by a learning rate of 2 × 10 -5 for 100 steps in stage 2, and conclude with a learning rate of 2 × 10 -6 for 500 steps in stage 3. λ µ and λ σ are set to 0.1 and 0 in stage 1, respectively, and are adjusted to 2 and 5 in subsequent stages. All experiments are conducted on a single Nvidia A100 GPU. The training process of our method takes about 20 minutes to learn a concept.Evaluation Setup. We compare our method with four state-of-the-art personalization methods, including TI+DB [24, 71] , NeTI [1] , ViCo [30] , and OFT [64] . The implementation details of the baseline methods are provided in Appendix A. We collect 22 concepts from TI [24] and DB [71] . For the quantitative evaluation, each method is evaluated using a set of 24 text prompts, see Appendix B for a complete list. These prompts cover background change, environment interaction, concept color change, and artistic style.Qualitative Evaluation. In Figure 6 , we present a visual comparison of personalized generation for various concepts. We employ a set of complex prompts for evaluation, where one prompt simultane- 20.6% 79.4% ViCo [30] 16.6% 83.4% OFT [64] 22.4% 72.6%ously incorporates several editing elements such as style change (e.g., "oil painting"), scene change (e.g., "old French town"), and appearance change (e.g., "dressed as a musketeer"). As observed, ViCo tends to overfit the new concept, failing to compose it in novel scenes or styles. Conversely, TI+DB sometimes overlooks the learned concept, producing images that solely reflect other prompt tokens. NeTI and OFT also struggle to achieve text-aligned generations, especially when the prompts are complex. Our method, AttnDreamBooth, is the only method that successfully generates identitypreserved and text-aligned personalized images for these complex prompts. Figures 1 and 7 show more personalized generations using complex prompts from our method. Additional qualitative results can be found in Appendix C.Quantitative Evaluation. We conduct a quantitative evaluation of each method in terms of identity preservation and text alignment. Identity preservation is measured by the cosine similarity between the CLIP [65] embeddings of generated and real images, while text alignment is measured by the cosine similarity between the CLIP embeddings of generated images and their corresponding prompts. Each method is evaluated using 24 text prompts, generating 32 images per prompt. The results are presented in Table 1 . TI+DB excels in text alignment but performs poorly in identity preservation. This is consistent with the qualitative observation that TI+DB often neglects the learned concept and generates images based solely on other prompt tokens. In contrast, ViCo achieves the best identity preservation but ranks lowest in text alignment, indicative of its tendency to overfit the new concept. Besides these two extreme cases, our approach exhibits superior performance in both identity preservation and text alignment.Fast Version of Our Method. The average training time using our method is about 20 minutes. To reduce the training time, we developed a fast version of our method by increasing the learning rate while simultaneously decreasing both the training steps and the batch size for the third training stage. Originally, this stage involves performing 500 steps with a learning rate of 2 × 10 -6 and a batch size of 8. The fast version now completes training in just 200 steps with a learning rate of 1 × 10 -5 and a batch size of 4. These adjustments significantly reduce the training time from 20 minutes to an average of 6 minutes. Interestingly, this fast version maintains performance comparable to our original model, likely because the first two stages provide a convenient starting point, allowing for a higher learning rate in the third stage. Notably, the fast version performs similarly to the original model on short prompts, as shown in Table 1 , but it shows slight degradation on complex prompts, as depicted in Figure 10 .User Study. We further evaluate our method by conducting a user study. Personalized images are generated using various prompts and concepts for each method. In each question of the study, participants are presented with an input image and a text prompt, along with two generated images: one from our method and another from a baseline method. Participants are asked to select the image that better achieves identity preservation and text alignment. We collected a total of 700 responses from 35 participants, as presented in Table 2 . The results demonstrate a clear preference for our method. In this section, we evaluate the effectiveness of each sub-module within our framework. Specifically, we conduct an ablation study by separately removing each training stage or the attention map regularization term. Figure 8 presents a visual comparison of personalized images generated by each variant. The results indicate that all sub-modules are crucial for achieving identity-preserved and textaligned personalized generations. Specifically, the model without optimizing the textual embedding (w/o Stage 1) tends to neglect the learned concept or generate it with significant distortions due to insufficient learning of the embedding alignment. Models without fine-tuning the cross-attention layers (w/o Stage 2) or without the regularization term (w/o Reg) suffer from degraded text alignment or identity preservation. The model without fine-tuning the U-Net (w/o Stage 3) leads to significant degradation in identity preservation. Additional ablation study results are provided in Appendix F. Similar behavior is observed in the quantitative ablation study, as detailed in Table 4 . 
conclusions and limitations: 
 In this paper, we identified and analyzed the embedding misalignment issue encountered by Textual Inversion and DreamBooth. Our proposed method, named AttnDreamBooth, addresses this issue by decomposing the personalization process into three stages: learning the embedding alignment, refining the attention map, and acquiring the subject identity. AttnDreamBooth enables identity-preserved and text-aligned text-to-image personalization, even with complex prompts.In our experiments, we used consistent training steps across different concepts; however, we observed that performance could be further improved by tuning the training steps for specific concepts. This limitation might be addressed by adopting adaptive training strategies, which we leave for future work. A second limitation is that our three-stage training method requires approximately 20 minutes on average to learn a concept, as it involves fine-tuning all parameters in the U-Net for 500 steps. To mitigate this, we introduced a fast version that reduces the training time to approximately 6 minutes while still maintaining performance comparable to the original model. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have provided the information on the computer resources in Section 5.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the potential impacts of our work in Section J.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not release data or pre-trained models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have discussed the licenses of existing assets in Section K.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We compare our method with four baseline methods, including TI+DB [24, 71] , NeTI [1] , ViCo [30] , and OFT [64] . For TI+DB, we implement it based on the diffusers library [80] without employing the prior preservation loss. We perform 660 training steps, which matches the total number of steps for our method, with a learning rate of 2 × 10 -6 and a batch size of 8. For the other baselines, we use the official implementations and follow the hyper-parameters described in their papers.In the Appendix, we further compare our method with four other baseline methods, including DreamMatcher [58] , FreeCustom [19] , SuTI [10] , and Instruct-Imagen [35] . For DreamMatcher and FreeCustom, we use their official implementations. Due to the unavailability of open-source models for SuTI and Instruct-Imagen, we rely on the examples provided in their papers for comparison.In Table 3 , we list all 24 text prompts used in the quantitative evaluation. These prompts cover a range of modifications, including background change, environment interaction, concept color change, and artistic style.In Figures 9, 10 , and 11, we provide additional qualitative comparisons to the baseline methods across a wide range of prompts. Furthermore, Figure 12 presents a qualitative comparison of our method with Textual Inversion (TI), DreamBooth (DB), and two different configurations of TI+DB: 1) first TI and then DB (TI→DB), and 2) first DB and then TI (DB→TI). In Figure 13 , we provide additional qualitative results generated by AttnDreamBooth.In this section, we compare AttnDreamBooth with the baseline methods when only a single image is used for training. In Figure 14 , we present the generation results of each method under this challenging setting. Our method demonstrates superior text alignment and identity preservation compared to the baselines.In the main text, we provide the cross-attention maps of Textual Inversion and DreamBooth in Figure 2 and the cross-attention map of "[V]" for each stage in Figure 5 . To vividly demonstrate the efficacy of our method in training the cross-attention layer, we present the generated images along with the cross-attention maps for each token in the prompt in Figure 15 . As can be seen, our method accurately assigns the attention maps for each token, demonstrating correct embedding alignment for the new concept.In Figure 16 , we provide additional ablation study results for each variant of our method. Table 4 presents the quantitative results of our ablation study. Specifically, the model without Stage 1 achieves better text alignment but significantly poorer identity preservation compared to the full model. This is because, without sufficient training of the textual embedding, the model tends to overlook the learned concept or generate it with significant distortions. Please note that the text alignment score is calculated without considering the new concept; therefore, omitting the new concept can inadvertently boost this score. Similarly, models without Stage 2 or Stage 3 also exhibit higher text alignment scores but lower identity preservation scores, due to insufficient learning of the attention maps and the subject identity, respectively. Additionally, the model without the regularization term shows degraded text alignment.A visual comparison between our models with SD1.5 or SD2.1 is presented in Figure 17 . As shown, the model with SD2.1 achieves superior performance in text alignment and identity preservation. Nevertheless, our method is also effective for SD1.5, and it outperforms the baseline methods.We present a visual comparison between models with or without the prior preservation loss in Figure 18 . The results show that incorporating the prior preservation loss leads to degradation in identity preservation.As described in Section 5.2, we conducted a user study to evaluate our method against the baseline methods. Here, we present the details of this user study. Figure 19 shows an example question from the user study. Given a concept image and a text prompt, along with two generated images (one from our method and another from a baseline method), participants were asked to select the image that better preserves the identity of the concept image and aligns with the text prompt. The results are presented in Table 2 .Similar to existing text-to-image personalization techniques, our approach provides broader users access to effectively fine-tuning large-scale pre-trained diffusion models. By enabling users to personalize these models with their own data, our approach can be used for numerous applications, including image editing, artistic creations, and industrial production. However, the use of generative techniques comes with risks, such as the creation of misleading or false information. To mitigate these concerns, it is vital to develop effective methods for identifying fake generations [84, 15] .Our implementation is based on the publicly available Stable Diffusion V2.1 [70] , which is under the CreativeML Open RAIL++-M License. The datasets used for evaluation are from TI [24] and DB [71] . The data from DB is under the Unsplash license, while the license information for the data from TI is not available online. Figure 9 : Additional qualitative comparison. We present four images generated by our method and two images from each of the baseline methods, including TI+DB [24, 71] , NeTI [1] , ViCo [30] , and OFT [64] . Our method demonstrates superior performance in text alignment and identity preservation compared to these baselines. We present four images generated by our method and two images from each of the baseline methods, including TI+DB [24, 71] , NeTI [1] , ViCo [30] , and OFT [64] . Our method shows better text alignment and identity preservation than the baselines.Input Stage 1Output "a" "[V]" "toy" "inside" "a"" "box" Stage 2 A [V] toy inside a box Output "a" "[V]" "toy" "inside" "a"" "box" Stage 3Output "a" "[V]" "toy" "inside" "a"" "box" • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: Our paper conforms to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: This paper proposes a new method for personalized high-quality image generation, called AttnDreamBooth.

Intially, reviewers pointed out some concerns such as less originality, ambigous description, lack of comparisons while respecting the importance of the tasks addressed and its method efficacy, with three positive scores and one rejection score. 

In particular, AC carefully read R-ZpWY's comments considering his/her rejection score.
Then, AC found that most methods that the reviewer suggested to use for comparisons are publihsed after NeurIPS deadline or are not available. 

Considering the situation and the contributions of this paper, AC thinks this paper is enough to be presented in NeurIPS, so recommends accepting this paper. 

AC asks the authors reflect all important comments of reviewers in the cam-ready version.

Reviews: review: paper_summary: paper_summary: The submission proposes AttnDreamBooth for text-to-image personalization. It addresses the limitations of existing methods, Textual Inversion and DreamBooth, by separating the learning process into three stages: embedding alignment, attention map refinement, and subject identity capture. The method aims to improve identity preservation and text alignment in generated images.

# Key Contributions:
- Introduces an approach for text-to-image personalization.
- Addresses the limitations of existing methods by separating the learning process.
- Demonstrates improved performance in terms of identity preservation and text alignment.
- Provides a comprehensive analysis of the proposed method through qualitative and quantitative evaluations.
strength_weakness: strength_weakness: - My main concern is comparison with new existing work. There have been several recent works after Textual inversion and Dreambooth that have significantly advanced this area. Some of them like SuTI (NeurIPS 2023), Instruct Imagen (CVPR 2024), HyperDreambooth (CVPR 2024) are also zero-shot which do not require any fine-tuning during evaluation. 

- It is critical for this work to compare with the state-of-the-art papers in this area. I have listed a few above. However, I am sure that there could be more papers in the past two years in this area.
questions: questions: - Comparison with multiple recent baselines is required. It would be great if the authors can provide this.
limitations: limitations: - While the proposed approach is novel, the core idea of decomposing the personalization process into multiple stages is not entirely groundbreaking. A more comprehensive exploration of the relationship between the proposed method and existing work would be beneficial.
paper_summary: paper_summary: The author proposed a method to generate high-quality personalized images. First, a textual embedding is learned, then the cross-attention layers is  finetund to refine attention map during learning the  textual embedding, finally, the entire U-Net is trained to capture the subject identity.
strength_weakness: strength_weakness: 1. The proposed method needs a costly test-time optimization to generate personalized images. The 3 stage finetuning requires expensive computation (20 minutes in A100)
2. Even though the Attention Map Regularization idea is interesting, it seems to be the only critical contribution in this paper. The authors combine some techniques from off-the-shelf method including CostumDiffusion/DB/TI, and conducted them step by step with proposed -attention map regularization.
questions: questions: 1. Where does the super-category label come from? Is it a part of the annotation?
2. I am curious how the  TI+DB perform if it is optimized step by step like AttnDreamBooth, e.g. first tuning the textual embedding and then the U-Net or first tuning U-Net then optimizing the embedding.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a method to enhance the performance of personalizing text-to-image models by appropriately combining textual inversion approach, which learns new embeddings, and DreamBooth approach, which fine-tunes model weights. They demonstrate the effectiveness of their approach through qualitative evaluation and human evaluation.
strength_weakness: strength_weakness: * (minor) The proposed method seems quite similar to the Magicapture [1] approach in that it separates embedding learning and weight fine-tuning and conducts regularization on attention. However, this paper does not cover a comparison with Magicapture. 
* (minor) The ablation study is not conducted quantitatively and relies on a single generation scenario.

[1] Hyung, Junha, Jaeyo Shin, and Jaegul Choo. "Magicapture: High-resolution multi-concept portrait customization." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.
questions: questions: When conducting analysis in Figure 1, how many steps was DreamBooth trained for to obtain the results? It seems that with sufficient training steps, DreamBooth might exhibit different results.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a new method for personalized image generation, decompose the personalization process into three training stages and introducing a cross-attention map regularization term.
strength_weakness: strength_weakness: 1.	Some parts of the manuscript are slightly verbose. For instance, the introduction section already introduces the existing problems of Textual Inversion and DreamBooth, along with their corresponding analysis and naïve solution. However, these points are reiterated in Section 4.1 and the first paragraph of Section 4.2 without adding any new information.

2.	What does the term 'this approach' refer to in Line 174?

3.	In general, the proposed method tends to be a bag of tricks with some customized hype-parameters.

4.	Important comparisons are lacking in the manuscript, specifically the comparison with Textual Inversion (TI) and DreamBooth (DB) individually, in addition to the comparison with the combined approach of TI+DB.

5.	This method uses cross-attention map for regularization, which results in a high time cost; on a NVIDIA A100, training requires 660 steps, taking 20 minutes. In contrast, DreamBooth requires only 5 minutes to train for 1000 steps.
questions: questions: 1.	Do the constraints imposed between [V] and [super-category] across all layers in Eq. 2 potentially restrict the diversity? Considering that [super-category] represents the original identity information and [V] introduces diversity.

2.	Why was SD2.1 chosen, and how do other models perform?

3.	In line 197, it is mentioned that retaining the prior preservation loss leads to poor results. Can you provide experimental results to support this?
limitations: limitations: N/A