Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting

Authors: weili  zeng
qi  zhu
pengzhi  chu
weiming  zhao
xiaokang  yang
yichao  yan
zhuo  chen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-22 08:27:27

Abstract: Text-to-image (T2I) customization aims to create images that embody specific visual concepts delineated in textual descriptions. 
However, existing works still face a main challenge, \textbf{concept overfitting}.
To tackle this challenge, we first analyze overfitting, categorizing it into concept-agnostic overfitting, which undermines non-customized concept knowledge, and concept-specific overfitting, which is confined to customize on limited modalities, \ie, backgrounds, layouts, styles. To evaluate the overfitting degree, we further introduce two metrics, \ie, Latent Fisher divergence and Wasserstein metric to measure the distribution changes of non-customized and customized concept respectively.
Drawing from the analysis, we propose Infusion, a T2I customization method that enables the learning of target concepts to avoid being constrained by limited training modalities, while preserving non-customized knowledge. Remarkably, Infusion achieves this feat with remarkable efficiency, requiring a mere \textbf{11KB} of trained parameters. 
Extensive experiments also demonstrate that our approach outperforms state-of-the-art methods in both single and multi-concept customized generation.

Main paper: 
 introduction: 
 In the era of multi-modality, text-to-image (T2I) generation [29, 31, 33, 35] has experienced rapid growth, showcasing highly imaginative works. Meanwhile, T2I customization rises as a potential need in image generation, aiming to create images that embody specific visual concepts aligned with textual descriptions. Users only need to provide a set of similar images whose pertinent visual concepts, e.g, pets, toys, or styles, are subsequently extracted by a T2I model. These identified concepts are then seamlessly interwoven into the users' textual descriptions for customized generation.Typically, T2I customization methods can be classified into two categories: inversion-based and fine-tuning. Specifically, the inversionbased methods [11, 47] represent concepts by learning additional conceptual words. While this approach offers flexibility, it introduces a significant overfitting challenge, as the injected concepts permeate every part of the cross-attention module with textual information. Another approach, fine-tuning method [34] , keeps the original architecture and optimizes model parameters with a set of customized image data. However, a large parameter space that is fitted by a small dataset leads to severe overfitting. Besides, the extensive size of these models often necessitates substantial storage and computation resources. Although Parameter-efficient tuning (PET) approaches [21, 24] shrink the fine-tuning into only a small subset of parameters, they still fail to eliminate the overfitting issue.Comprehensively considering the previous methods, current T2I customization still faces three challenges: 1) Concept overfitting takes away general generation capabilities of foundational T2I models from customized models. 2) Lack of appropriate metrics to quantify the impact of overfitting on customized models. 3) Cumbersome deployment prevents users from seamlessly switching between customized mode and regular mode as desired. The first column on the left is the target concept, and the right is the non-customized results. The generated "cat" consistently exhibits black spotted stripes, while the "teddy" consistently presents a doll-like form, both sharing similar backgrounds.Figure 3 : Example of concept-specific overfitting. Customization with the prompt "a photo of a ⟨cat⟩", we reveal that all prior methods generate cat images with a similar size, pose, or background to the training data.We deeply explore the overfitting problem and categorize it into concept-agnostic overfitting and concept-specific overfitting. Concept-agnostic overfitting is depicted in Figure 2 . When learning the concepts of ⟨cat⟩ and ⟨teddy⟩, customization undermines original generative ability for non-customized concept. Concept-specific overfitting is depicted in Figure 3 . During the customization process of T2I model, it confuses target concept with limited training diversities, resulting in a diminution of generative diversity and consequently impacts the textual controllability.Based on the analysis of the above two categories of overfitting, we introduce "Latent Fisher divergence" and "Wasserstein metric" to measure the distribution changes of non-customized and customized concept respectively. The latent Fisher divergence quantifies the score difference of the non-customized concept between the original model and the customized model. The Wasserstein metric quantifies the deviation of the customized distribution from its original generative distribution under a super-class concept, serving as a measure of diversity.To overcome the aforementioned challenges of overfitting, we propose the Infusion method, a T2I customization method that enables the learning of target concepts to avoid being constrained by limited training diversities. Infusion can also flexibly inject customized concepts into original models while preserving their original knowledge. Our novel insight lies in the decoupling of the attention map and value feature in each cross-attention module. As depicted in Figure 4 , we implemented a dual-stream structure that shares pipeline parameters. The foundational T2I pipeline performs regular T2I generation. To harness its generative capabilities, we imitate the operations in P2P [17] , and use the attention maps in the foundational T2I pipeline to replace the attention maps in the customized T2I pipeline. We introduce target concepts by learning a residual value embedding in the customized pipeline. All the learnable embeddings for a concept are highly lightweight, only 11KB, enabling seamless integration into the pipeline in a flexible and plug-and-play manner.In summary, our contributions can be delineated as follows:• We analyze concept-agnostic and concept-specific overfitting, and propose employing "Latent Fisher divergence" and "Wasserstein metric" to quantify their impact. • We propose Infusion, a method that decouples attention maps and value features in each cross-attention module, fully utilizing the capability of the original T2I model. • Experiments show that Infusion enables plug-and-play singleconcept and multi-concept generation and achieves superior results compared to state-of-the-art methods. 
related work 2.1 text-to-image generation: 
 Early Text-to-Image (T2I) models primarily relied on Generative Adversarial Networks [6, 14, 22, 41, 51, 56] , yet often exhibited limitations in terms of diversity. Recently, diffusion models [19, 38-40, 53, 54] , coupled with classifier-free guidance [20] , have emerged as leaders in this domain. Their remarkable generative performance is particularly noteworthy when applied to large-scale internet datasets [27, [31] [32] [33] . This fact drives us to fully exploit the prior knowledge of these foundational models for T2I customization.The advent of contrastive multimodal models [30, 50] , exemplified by CLIP [30] , has introduced a transformative paradigm shift in the realm of image editing [3-5, 13, 28] . Significantly, these models empower global or localized image manipulation through the exclusive utilization of textual prompts. This paradigm has engendered a series of text-based image editing methods [7, 17, 23, 26, 52] , among which the most relevant to our work is the prompt-to-prompt (P2P) [17] . P2P achieves image editing by replacing or modifying the attention map of the cross-attention module. Similarly, our approach employs attention map replacement for image customization. The difference is that P2P operates on virtual generated concepts, while we can operate on real concepts.Text-to-Image (T2I) customization aims to generate customized concepts aligned with text description. Textual Inversion [11] is the pioneering method in this realm, which introduces an innovative approach to learning new word embeddings for representing specific concepts without tuning model parameters. Despite its plug-and-play adaptability, its efficacy remains constrained. Dream-Booth [34] employs class nouns along with unique identifiers to represent target concepts but requires the comprehensive training of model parameters. While achieving high conceptual fidelity, this approach is susceptible to overfitting. Subsequent methodologies, incorporating low-rank updates or just tuning a few parameters, aim to address this limitation [15, 21, 24] . Notably, Custom Diffusion [24] focuses on updating the weights of cross-attention and introduces regularization sets to mitigate overfitting. Perfusion [42] introduces a decoupling of cross-attention into "what" and "where" pathways. It leverages a Rank-one Model editing method [25] to adjust the weights of key and value projection matrices, thereby alleviating concerns associated with overfitting. Another research trajectory involves using extensive customized data for pre-training encoders, enabling training-free inference [2, 10, 12, 48, 49, 55] . 
analysis of concept overfitting: 
 In this section, based on T2I diffusion model, we analyze two types of overfitting in the customized diffusion model: concept-agnostic overfitting and concept-specific overfitting. We subsequently define two metrics corresponding to each type of overfitting.We applied our method to the Stable Diffusion (SD) model [1] .During the training phase, the encoder E maps the input image 𝑥 ∈ X to latent code 𝑧 = E (𝑥). Subsequently, the decoder D, is tasked with reconstructing the input image, aiming for D (E (𝑥)) ≈ 𝑥.Diffusion models are then introduced to the latent space. The forward process perturbs the input to a noisy variant 𝑧 𝑡 , at each time step 𝑡. Given a conditional prompt 𝑦, the latent diffusion model 𝜀 𝜃 is to minimize the loss function:EQUATIONwhere 𝜀 𝜃 is a time-conditional UNet [36] comprising self-attention layers and cross-attention layers. 𝜏 (𝑦) is randomly replaced by an empty input ∅. This training objective can be construed as a form of denoising score matching [45] , with 𝜀 𝜃 (𝑧 𝑡 , 𝑡) serving as an approximation to the score ∇ 𝑧 log 𝑝 (𝑧 𝑡 ) of the latent distribution.Stable Diffusion employs a cross-attention mechanism to incorporate textual modal inputs. As previously outlined, the operation 𝜏 𝜃 projects the text condition 𝑦 into an intermediate representation denoted as 𝜏 𝜃 (𝑦) ∈ R 𝐿×𝑑 𝜏 . Subsequently, the cross-attention operation of the 𝑖-th layer network is expressed as:EQUATIONwhich is utilized to integrate textual information into the UNet network. The constituents are defined as:EQUATIONwhere Concept-agnostic overfitting often stems from alterations in the foundational model parameters aimed at customization, thereby compromising the model's performance on non-customized concepts. As shown in Figure 5 , our toy foundational model is a fourpeak hybrid Gaussian model, where each peak represents a different concept. The process of customized tuning can be viewed as the model learning a specialized distribution under a particular concept. Here, we simplify it as learning a linear distribution under the Gaussian peak in the top-left corner. It can be observed that when employing a tuning method similar to Dreambooth [34] , the generated distributions for non-customized concepts become compact. The shrinking of these distributions implies a reduction in the diversity of generated samples under non-customized concepts.𝑄 𝑖 ∈ R 𝑁 2 𝑖 ×𝑑 , 𝐾 𝑖 ∈ R 𝐿×𝑑 ,Therefore, for T2I models, we infer that concept-agnostic overfitting is primarily related to the offset of the non-customized score. Specifically, let 𝑐 represent the customized concept and 𝑐 represent non-customized concepts. 𝜃 , 𝜃 ′ are the parameters of the original model and customized model. As depicted in Figure 7 , the gradient field described by ∇ 𝑧 log 𝑝 𝜃 (𝑧|𝑐), gradually contracting during training, leading to a more convergent field ∇ 𝑧 log 𝑝 𝜃 ′ (𝑧|𝑐). To quantify this effect, we define the latent Fisher divergence as definition 3.1, which measures the distance between two models' score under the non-customized concepts.Definition 3.1 (Latent Fisher divergence). Based on the fact that 𝑐 is only a small part of a vast image concept, and the training principles of denoise score matching [45] , the Fisher divergence between 𝑝 𝜃 (𝑧|𝑐) and 𝑝 𝜃 ′ (𝑧|𝑐) can be approximated by:EQUATION)In essence, this metric assesses the degree to which the model's inherent generative capacity is compromised throughout the customization process.Concept-specific overfitting primarily stems from the confusion learning between customized concepts and limited training diversities. As illustrated in Figure 6 , we also employ a toy Gaussian model for clarification, where each peak region represents different diversities within the same concept. The original foundational model is capable of generating 25 different diversities within a similar concept, each representing variations such as diverse backgrounds, layouts, and styles. The customized target is also to learning a linear distribution. Typically, customized training datasets only contain a subset of these diversities. We represent this with the utilization of only five Gaussian distributions. It is evident from Figure 6 that, during the training process, the model gradually reduces the number of diversities and ultimately converges to only include those present in the training dataset. However, as shown Figure 6 (e), our ultimate goal is to represent linear concepts as comprehensively as possible across all diversities within the model.Therefore, during the learning process of T2I model, we infer that the model confuses the target concept with other diversities such as background, layout, style, etc., leading to concept-specific overfitting. Based on the manifold hypothesis [16] , each concept's generative distribution can be regarded as a probability manifold embedded within the entire generative space, with the distributions under different diversities of each concept considered as distinct sub-manifolds embedded within this manifold. To quantify this overfitting, it becomes necessary to measure the distance between these sub-manifolds before and after customization training. To achieve this, we introduce the Wasserstein metric, as defined in definition 3.2. Definition 3.2 (2-Wasserstein metric [44] ). Let (X, 𝑑) be a Polish metric space. For any two probability measures 𝜇, 𝜈 on X, 𝜇 and 𝜈 are respective marginal distributions over X, law (𝑋 ) = 𝜇, law (𝑌 ) = 𝜈. 𝑑 (x, y) = ∥x -y∥ 2 is Euclidean distance. 𝜋 (𝑥, 𝑦) is the set of all probabilistic couplings in (X × X) with marginals 𝜇 and 𝜈. The 2-Wasserstein distance between 𝜇 and 𝜈 is defined by the formula:EQUATIONHowever, directly compute Wasserstein metric in high-dimensional space is difficult. So in this work, inspired by Fréchet Inception Distance [18] , we fits a Gaussian distribution to the latent of SD model for each distribution and then computes the 2-Wasserstein distance, between those Gaussians. 
method: 
 To address the aforementioned two types of overfitting, we aim to retain the generative capacity of the foundational T2I model and exploit its generative diversity for customization. For instance, to generate a customized image with the prompt "a ⟨cat⟩ is reading a book". An intuitive strategy is that we generate an initial image by foundational T2I model first. During this process, we can apply a transformation to replace the depicted "cat" with the desired customized "⟨cat⟩". Note that the use of "⟨⟩" is merely for written distinction, and we do not undertake additional learning of unique identifiers. As shown in Figure 4 , the "cat" was selectively replaced with a customized concept, while preserving the constancy of other elements. In this way, we retain the T2I model's generative capacity, avoid concept-agnostic overfitting, and simultaneously harness its diversity to mitigate the risk of concept-specific overfitting.Following [42] , we can partition the cross-attention module into two pathways: the "where" pathway for the position and geometric attributes of objects in the final image, and the "what" pathway dictating the features incorporated into each spatial region, thus controls the content appearing in the final image. Specifically, expanding the expression denoted by Equation 2, we derive the following:EQUATIONHere, 𝑚 𝑖 𝑘 ∈ R 𝑁 2 𝑖 ×1 denotes the attention map corresponding to the 𝑘-th token of the input prompt and 𝑣 𝑖 𝑘 ∈ R 1×𝑑 represents the value feature associated with the same token. As shown in Figure 4 , there is usually minimal overlap between attention maps of the foundational T2I model. Various methods [9, 43] exist can be directly combined with Infusion to further prevent their overlap, here we just consider a simplified scenario. Thus, in conjunction with Equation 6, we observe that independent attention maps determine the positional presentation of their corresponding tokens in the image. The content they present is determined by the respective values.Under the condition where the conceptualized generation content remains fixed, the augmentation of spatial layout and posture in the generated entities is imperative to amplify the diversity of customized generation. Furthermore, we posit that the foundational T2I model, untainted by customized training, inherently encapsulates the spatial diversity. To fully harness the diversity, our insight is that its attention maps should be decoupled for customized generation.It is somewhat akin to the mechanisms employed in P2P, we extract attention maps from the foundational T2I model's generation under identical prompts, utilizing them to replace the attention maps in the customized generation process. However, P2P is constrained to the substitution of concepts exclusively within the generative scope of the model.The essence of Infusion lies in preserving the cross-attention maps of the foundational T2I model and subsequently learning and infusing concepts during each time step 𝑡 along the "what" pathway.For example, consider the training image 𝐼 and its corresponding text description 𝑦 ("a photo of a cat"). We feed them into both the foundational T2I pipeline (F-pipeline) and the customized T2I pipeline (C-pipeline). These two pipelines share fixed foundational model parameters. Differently, to facilitate the learning of customized concepts, a trainable embedding is introduced in the "what" pathway of the C-pipeline. Specifically, a residual embedding Δ cat 𝑖 is incorporated into the value feature of the "cat" token. This embedding represents the customized concept and fulfills the transformation from the original category to a specific object. Consequently, the attention module within the C-pipeline can be structured as follows:EQUATIONThe subscript 𝑜𝑏 𝑗 denotes the index of the concept token in the prompt. Vectors with "-" denote those from the C-pipeline, while vectors without "-" originate from the F-pipeline. It is noteworthy that 𝑣 𝑖 𝑘 = 𝑣 𝑖 𝑘 when 𝑘 ≠ obj, or in other words, 𝑉 differs from 𝑉 only in the value feature corresponding to the token ⟨obj⟩.Finally, the trainable parameters Δ 𝑖 ⟨obj⟩ are optimized with the customization loss without additional regularization:EQUATIONwhere 𝑥 is the reference few-shot data of ⟨obj⟩, 𝜃 ′ is the parameters of the customized model.Similarly, during the inference stage, attention maps from the Fpipeline are employed to replace the attention maps within each cross-attention module in the C-pipeline. The previously obtained residual value embedding can be effortlessly applied to value features of the corresponding concept token. For a prompt involving𝑆 specific concepts ⟨obj 1 ⟩, ⟨obj 2 ⟩, • • • , ⟨obj 𝑆 ⟩, it is sufficient to re- trieve the pre-computed Δ 𝑖 ⟨obj 1 ⟩ , Δ 𝑖 ⟨obj 2 ⟩ , • • • , Δ 𝑖 ⟨obj 𝑆 ⟩and integrate them into the value embedding of the 𝑖-th layer, as follows:EQUATIONCombining Equation 7with Equation 9, we can finally generate single-concept or multi-concept customized images. sectionExperimentsExperiment Setup. We utilize the pre-trained StableDiffusion (SD)-v1.5 model [1] for ablation studies and comparisons. In each layer of the Unet's cross-attention, we train the residual concept Δ ⟨obj⟩ ∈ R 1×𝑑 for the corresponding value feature, where 𝑑 = 768. The optimization of these embeddings is carried out with a learning rate of 0.01 and a batch size of 4. During the inference process, we employ a DDIM sampler with a sampling step size of 𝑇 = 50, and leverage classifier-free guidance with a guiding scale of 𝑠 = 8. We utilize datasets from prior works [24, 34] , encompassing diverse subjects such as toys, animals, buildings, and personal items.Baseline. We benchmark our method against state-of-the-art competitors, including Textual Inversion, DreamBooth, DreamBoothlora, Custom Diffusion, and Perfusion. For Textual Inversion and DreamBooth(lora), we utilize their Diffusers versions [46] , and for Custom Diffusion and Perfusion, we employ their official implementations with experimental parameters configured following official recommendations.Overfitting Evaluation. Referring to Section 3, we employ the Latent Fisher divergence and Wasserstein metric to assess the concept-agnostic overfitting and concept-specific overfitting on the customized model. Latent Fisher divergence is computed based on a random sample of 50000 examples from LAION-400M [37] . We sample time steps uniformly and measure the distance between the outputs of 𝜖 𝜃 and 𝜖 𝜃 ′ . To evaluate concept-specific overfitting, we generate 1000 samples under 8 concepts based on 20 prompts and calculate the 2-Wasserstein distance between 𝑝 𝜃 (𝑧|𝑥, 𝑐) and 𝑝 𝜃 ′ (𝑧|𝑥, 𝑐).Customization Evaluation. Following previous works [34, 48] , we employ CLIP-I, CLIP-T, and DINO-I to evaluate the model ability for customized image generation. CLIP-T is to assess the alignment between generated images and prompts. CLIP-I evaluates the similarity between generated samples and training data in the CLIP feature space. DINO-I computes the cosine similarity between the ViTS/16 DINO [8] embeddings of training images and generated images.To ensure fair comparison of the anti-overfitting capabilities among different customized models, we adjusted the learning rate and batch size to achieve near-optimal performance for each method at the same number of training steps. Additionally, we continued training each model after convergence to assess the extent of overfitting. As depicted in Figures 10 and Figures 11, Infusion demonstrates robust resistance to concept overfitting compared to other methods. As illustrated in Figure 12 , we present the generated results of Infusion at training steps 100, 200, 400, 1000 and 2000. It is evident that even at higher training steps, Infusion keeps producing content highly aligned with the given text. This sets it apart from other methods that require precise control over training steps to prevent overfitting. In contrast, DB(Dreambooth)-lora, CD(Customized Diffusion), and TI(Textual Inversion) exhibit increasing susceptibility to concept-specific overfitting with continued training, becoming increasingly confined to the training diversities, and losing control over textual consistency. Perfusion performs better in this regard, but its consistency with customized concepts is notably inferior to Infusion. Moreover, since Infusion can directly generate noncustomized concepts through the F-pipeline, it remains unaffected by concept-agnostic overfitting.Figure 8 illustrates a series of generated images produced by our approach and its competitors, including Stable Diffusion (SD), for single-concept customization. To rigorously assess the models' generative capabilities, we deliberately selected a set of imaginative descriptive prompts. Notably, DreamBooth and Textual Inversion 9 : Multi-concept generation. We show the results of multi-concept. Infusion demonstrates superior performance in generating works that exhibit higher levels of imagination and fidelity to customized concepts compared to Custom Diffusion exhibit pronounced susceptibility to overfitting, emphasizing conceptual consistency limits their imagination. Perfusion, on the other hand, demonstrates commendable imaginative capacities, yet tends to overlook the generation of customized concepts. Custom Diffusion, while achieving performance comparable to our method, still lacks some diversity.In contrast, Infusion adeptly balances textual expression and concept fidelity. Furthermore, when compared with the generated results of SD, it becomes evident how our method leverages the "where" pathway and "what" pathway to infuse concepts. Our approach effectively preserves the SD's generated background and layout while injecting customized concepts onto the corresponding objects. This strategy allows us to harness the imaginative capacity of SD for customization and avoid overfitting.Figure 9 showcases the performance of our method in multiconcept generation. In this context, we only present the results of Infusion and Custom Diffusion, as the performance of other competitors in this task significantly lags behind these two. We observe that Custom Diffusion tends to overlook some concepts during multi-concept customization. In comparison, our method excels in crafting works that are not only more imaginative but also more faithful to the customized concepts.Table 1 : Quantitative Results. We compared different baseline methods across various metrics, and Infusion achieved the highest text alignment score.CLIP-T ↑ CLIP-I ↑ DINO ↑ Storage DreamBooth [34] 0.255 0.838 0.721 3.3 GB DreamBooth-lora [46] 0.264 0.791 0.615 3.29 MB Textual Inversion [11] 0.240 0.780 0.584 3KB Custom Diffusion [24] 0.256 0.800 0.643 57.1MB Perfusion [42] 0.242 0.729 0.505 100KB Infusion (Ours) 0.267 0.816 0.697 11KBOur quantitative analysis primarily focuses on general objects customization. Utilizing 20 concept dataset from [24, 34] and the same 25 prompts as in [34] , we generated 5 images for each prompt. Table 1 demonstrates our method's superiority in text alignment. While it may not surpass Dreambooth and Custom Diffusion in image alignment and DINO score, we attribute this discrepancy to inherent limitations in the metrics. These metrics consider background information similarity during computation, which may not align with the goals of our customized generation approach. The simplicity of testing prompts also limits a comprehensive assessment, future work will include more intricate test texts to evaluate the model's generative capabilities thoroughly. User Study. We evaluate the average preferences of 21 participants for Infusion and other baseline methods. Each participant responds to 32 questions about quality comparison. Table 2 indicates users' general preference for Infusion in terms of text alignment and conceptual fidelity. 
conclusion and limitations: 
 In this work, we analyze two types of overfitting and introduce "Latent Fisher divergence" and "Wasserstein metric" to quantify them. Our proposed Infusion preserves the inherent generative capacity of the original T2I model while offering flexible, plugand-play usage for customization. Infusion demonstrates excellent performance in overall learning of target concepts. However, for tasks that require high-fidelity preservation of detailed textures, Infusion exhibits certain limitations. In such cases, there is a tradeoff between diversity and fidelity, necessitating the sacrifice of some diversity in exchange for higher fidelity, such as training more residual embeddings. 
 
 Appendix:  


Decision: This paper ultimately received three "borderline accept" scores. All reviewers were satisfied with the "novelty and positive results" of the paper. After the rebuttal, reviewers QQKJ and 4wNo felt that their questions were well addressed. Reviewer Wh2t thinks that "the description and the analysis" still need further improvement. Based on the above, I recommend accepting this paper, and strongly suggest that the authors further improve the paper according to Reviewer Wh2t's feedback.

Reviews: review: paper_summary: paper_summary: This paper introduces a method called Infusion aimed at mitigating overfitting in text-to- image (T2I) diffusion models. The authors identify two primary types of overfitting: concept-agnostic overfitting, which affects non-customized concepts, and concept- specific overfitting, which limits the diversity of generated images. To address these issues, Infusion employs a dual-stream structure that decouples the attention maps and value features in each cross-attention module. This approach leverages the foundational model's generative capacity while introducing customized concepts through a lightweight embedding process. The method uses metrics like Latent Fisher divergence and the Wasserstein metric to evaluate overfitting and demonstrates superior performance compared to state-of-the-art methods through extensive experiments.
strength_weakness: strength_weakness: 1. Adequate analysis: The paper conducts adequate analysis on two types of overfitting in the customized diffusion model: concept-agnostic overfitting and concept-specific overfitting. And it define two metrics corresponding to each type of overfitting.
2. Novelty: Infusion decouples the attention mapping and value features, thereby leveraging the diverse generation modalities of the original T2I model to mitigate concept-specific overfitting
3. Adequate Evaluation:The paper provides comprehensive evaluations using both quantitative metrics and qualitative assessments. Extensive comparisons with state-of-the-art methods and user studies bolster the credibility of the results. The use of metrics tailored to measure overfitting is particularly noteworthy.
limitations: limitations: 1. Application: The paper somehow solve the overfitting problem, but will it have bad influence on the image quality? As shown in Fig 8, the visualization of Infusion is not as good as dreambooth in the third row in the upper right: the duck toy losses its face. The author is suggested to show the quantitive comparison of the fid and clip score compared with existing methods to evaluate the t2i quality.
2. Multi-concept results: all the images in the figure9 in third cow has some unpleasant airtifacts, (such as the face of the cat in the first row, the light in a strange cat shape in the air in the second line and the lamp head with the teapot base), I wonder it doesn't really work in practical applications.
3. Poor Writing: The paper seems to written in a rush, there are obvious typos and organization errors. Such as:
a. line 93-97 "Generate the Correct Terms for Your Paper." b. Line 202 :"As c in the Figure"
c. the line 397-400 shouldn't be included in section 3.2 cause it also introduce the Concept-specific Overfitting.
paper_summary: paper_summary: This paper analyzes the overfitting issue in text-to-image customization tasks, classifying it into two categories based on whether the overfitting phenomenon is related to the specific concept needing customization. It introduces two metrics to measure these overfitting categories during text-to-image customization. Furthermore, the paper proposes a model named Infusion to address these issues.
strength_weakness: strength_weakness: 1. Compared to existing state-of-the-art methods, the proposed method demonstrates performance improvement according to the presented generated images and it is simple but effective.

2. The paper provides an in-depth exploration of the overfitting issue in text-to-image customization and visualizes it through two simple experiments, presenting the issue clearly.
limitations: limitations: 1. The terminology used in the paper is confusing, particularly the term "training modalities," which appears frequently. It is inappropriate to use this phrase to describe the diversity of training data since modalities typically refer to data types such as video, image, text, depth, etc.

2. Several symbols in the paper lack proper explanations, making the equations difficult to understand. For example, the symbols in equation (5) in L486-L490 are not adequately explained.

3. There are some unclear descriptions in the paper. For instance, in L272-L274, it mentions “the difference is that P2P ...”, but the concept of the "virtual generated concept" in this sentence is confusing and lacks further explanation. Additionally, P2P can also be applied to edit real images through an existing inversion process.

4. There are issues with the proposed two metrics and the analysis of overfitting in S3:

(1) The paper uses two simplified experiments to imitate text-to-image customization with diffusion models. The scientific basis for this simplification is unclear.

(2) The metrics for concept-agnostic and concept-specific overfitting are both based on calculating the difference between two distributions. It is unclear why they are not derived from the same metric for distribution distance measurement.

(3) In L487-L479, it mentions, “To quantify ...”, but the method for obtaining the distribution of the target concept is unclear. The target concept cannot be sampled from the foundational distribution of the pre-trained diffusion model, and it is impossible to present all images with the target concept. In this paper, this distribution is estimated by sampling data with generalized text related to the target concept in the foundational distribution, which is confusing.

5. There are issues with the experiments in S5.2. The analysis of Figures 10 and 11 is not appropriate. For example, the phrases "excellent fidelity to textual consistency" in L670-L672 and “losing control ..." in L676 are inappropriate. Figures 10 and 11 only reflect the overfitting condition of the model and do not demonstrate the consistency between the text prompts and generated images.
paper_summary: paper_summary: This paper conducts a detailed analysis of the overfitting problem that occurs in customized finetuning. The authors interpret the problem theoretically and claim that the overfitting problem mainly consists of concept-agnostic overfitting and concept-specific overfitting. This paper proposes “Latent Fisher divergence” and “Wasserstein metric” to quantify them, and the experimental results show the visually superior performance.
strength_weakness: strength_weakness: 1. Experimental results show the better performance than other costomized finetuning methods, e.g., dreambooth, textual inversion, custom diffusion, etc.
2. The proposed method achieves this feat with remarkable efficiency, requiring a mere 11KB of trained parameters.
3. The paper is well organized and analyzes the problem both theoretically and experimentally.
limitations: limitations: After reading the paper, I have some doubts as follows:
1. In fig.2 and fig.3, why does the proposed method (Infusion) have such differentiated output under the same prompt (a photo of cat)? In fig.2, "a photo of cat" corresponds to diverse cat images, while in fig.3, "a photo of cat" corresponds to output images of the same cat category.
2. In fig.8, in the images generated by Infusion, why is the overall brightness of the object much brighter compared to the concept images, especially in the example of teapot.
3. This paper introduces “Latent Fisher divergence” and “Wasserstein metric”, but lacks an ablation study on these two. I wonder which core technology is responsible for the performance improvement of this method.
4. Generally speaking, how many concept images does this method require to achieve the effect of generating realistic images? What's the relationship between the performance and the number of the concept images?