ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance
Authors: Jiannan Huang
Jun Hao Liew
Hanshu Yan
Yuyang Yin
Yao Zhao
Humphrey Shi
Yunchao Wei
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion
Subject-Driven Personalization
Primary area: generative models

Abstract: Recent text-to-image customization works have proven successful in generating images of given concepts by fine-tuning diffusion models on a few examples. However, tuning-based methods inherently tend to overfit the concepts, resulting in failure to create the concept under multiple conditions (*e.g.*, headphone is missing when generating "a <sks>`dog wearing a headphone"). Interestingly, we notice that the base model before fine-tuning exhibits the capability to compose the base concept with other elements (*e.g.*, "a dog wearing a headphone"), implying that the compositional ability only disappears after personalization tuning. We observe a semantic shift in the customized concept after fine-tuning, indicating that the personalized concept is not aligned with the original concept, and further show through theoretical analyses that this semantic shift leads to increased difficulty in sampling the joint conditional probability distribution, resulting in the loss of the compositional ability. Inspired by this finding, we present **ClassDiffusion**, a technique that leverages a **semantic preservation loss** to explicitly regulate the concept space when learning a new concept. Although simple, this approach effectively prevents semantic drift during the fine-tuning process of the target concepts. Extensive qualitative and quantitative experiments demonstrate that the use of semantic preservation loss effectively improves the compositional abilities of fine-tuning models. Lastly, we also extend our ClassDiffusion to personalized video generation, demonstrating its flexibility.

Published as a conference paper at ICLR 2025
CLASSDIFFUSION: MORE ALIGNED PERSONALIZATION TUNING WITH EXPLICIT CLASS GUIDANCE
Jiannan Huang!.2. Jun Hao Liewa Hanshu Yan
Yuyang Yin 1.2
Yao Zhao 1.2 Humphrey Shi Yunchao Wei
Institute of Information Science, Beijing Jiaotong University
2 Visual Intelligence X International Joint Laboratory of the Ministry of Education
8 ByteDance Inc. SHI Labs@ Georgia Tech
ABSTRACT
Recent text-to text-to-image customization works have proven successful in generating images of given concepts by fine-tuning the diffusion models on a few examples. However, tunning-basse
methods inherently tend to overfit the concepts, resulting in failure to create the concept under multiple conditions (e.g.. headphone is missing when generating "a <sks> dog wearing headphone"). Interestingly, we notice that the base model before fine-tuning exhibits the capability to compose the base concept with other elements (e.g.. "a dog wearing headphone"), implying that the compositional ability only disappears after personalization tuning. We observe semantic shift in the customized concept after fine-tuning, indicating that the personalized concept is not aligned with the original concept, and further show through theoretical analyses that this semantic shift leads increased difficulty in sampling the joint conditional probability distribution, resulting in the loss of the compositiona ability. Inspired by this finding, we present ClassDiffusion, a technique that leverages semantic preservation loss to explicitly regulate the concept space when learning the new concept. Although simple, this approach effectively prevents semantic drift during the fine-tuning process on the target concepts. Extensive qualitative and quantitative experiments demonstrate that the use of semantic preservation loss effectively improves the compositional abilities fine-tuning models. Lastly, we also extend our ClassDiffusion to personalized video generation, demonstrating its flexibility.
INTRODUCTION
Thanks to the rapid progress in the diffusion model [31, 48, 55, 59, 63, 65, 67, 68, 72, 92, 95], the field of text-to-image generation has achieved significant progress in recent years. The leading text-to-image models [1. 20, 34, 38, 40, 66, 80] have been successful in generating high-fidelity images that align well with textual inputs. Recently, significant part of the research [2, 4, 5, 8, 11, 28, 36, 60, 79, 88, 93, 94] has changed their focus from creating high-quality images to improving control over the generated images. Among these works, important and widely explored research domain is subject-driven personalized generation, which aims to generate new images for a specific concept given
reference images of that concept.
Existing personalization methods [1, 9, 20, 34, 38, 40, 51, 66, 78, 80, 81, 85] can generate images that closely resemble the concept by fine-tuning the base text-to-image model in a specific image set. However, all tuning-based models will inherently suffer from the over-fitting introduced by this process, which leads to weakening in the compositional ability of the model. For example, when generating "a <sks> dog wearing headphone", though the given dog is well reconstructed, the headphone is always missing (Fig. 1). This feature affects the diversity of the generated output in practical use. A commonly accepted explanation within the community [20, 29, 66, 75] attributes this phenomenon to overfitting given a limited number of images. However, the fundamental cause of this overfit remains unexplored. In this work, our aim is to investigate the underlying causes behind the overfitting.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Stable Diffusion
CustomDiffusion
Ours
Fine Tune
(V*) dog wearing a headphone
Figure 1: The base Stable Diffusion (SD) possesses the capbility to compose the concept of a dog and headphone, generating dog wearing headphone. However, we notice that this compositional generation capability is lost during personalization tuning. For example, when using Custom Diffusion (CD) [38], the headphone missing despite the target corgi generated successfully. On the other hand, our method can successfully compose the target corgi with the headphone.
Upon initial examination, it appears that the model diminishes some of its original capabilities after personalization tuning. Taking Stable Diffusion (SD) [74] as an example, from Fig. I, we observe that the base SD model indeed has the ability to combine the concepts of dog and headphone. However, after fine-tuning, the model struggles to achieve compositional generation; for instance, while the target concept <sks> (dog) can be generated successfully, the headphone is missing. We hypothesized that the decline in this compositional ability stems from the semantic drift of the target concept away from its superclass target during fine-tuning. To better understand this, we conduct some empirical analysis by visualizing the CLIP text-space and cross-attention map activation area in Fig. 3a, 3b. In addition, also perform theoretical analysis and find that the root cause lies in the semantic bias that reduces the entropy of the probability of the composed conditions, which significantly increases the difficulty to simultaneously sample the target concept combined with other elements.
Based on our experimental findings and theoretical analysis, we introduce ClassDiffusion to address the issue of weakening compositional capacity after fine-tuning. Fig. 2 shows the performance of our method. Our method uses semantic preservation loss to explicitly guide the model to restore the semantic imbalance that arises during the fine-tuning stage. In particular, it narrows the gap between the text embeddings of the target concept and respective superclass in the textual space. Despite its simplicity, the proposed loss can successfully recover the compositional ability shown in Fig. 1. Therefore, distinct from prevalent loss design in the community which seek to migrate the overfitting in tuning- based models, while maintaining the concept of customized subject. Extensive experiments have demonstrated the effectiveness of our method in restoring the compositional generation capability of the hase model. Furthermore, we explore the potential of our approach in personalized video synthesis, showcasing its ability recovering the semantical space of the generative model. In addition, we found that the CLIP-T metric can hardly reflect the actual performance of personalized generation. Therefore, we introduce the BLIP2-T metric, a more equitable and effective evaluation metric for this particular domain. To summarize, the contributions of our work are:
method enhances the model's capacity of following the text prompt
We offer a thorough examination to understand why existing mmiing-based subject-driven personalized methods inherently suffer from the loss of compositional ability. This is elucidated through both experimental observations and theoretical analysis.
We propose ClassDiffusion, a simple technique to recover the compositional capabilities lost during personalized tuning.
Extensive experiments demonstrate that the proposed technique achieves improved person- alization ability in image and video generation tasks.
2
RELATED WORK
Text-to-Imagg Generation and Its Control Text-to-image generation is designed to generate high- quality, high-fidelity images that are aligned with textual prompts. This field has been under research
2

---

Published as a conference paper at ICLR 2025
a dog
a
useS computer To write book
holds published book
wins in Litenature
dog wearing sunglasses
a sunglasses
being sold on counter
a shopping basket
wearing by woman
being thrown onto bed
A qualitative result of two small stories produced by our model. The above showcases bear's literary journey: from reading a book ultimately earning a Nobel Literature Prize. The below shows the fate of a sunglasses. Finally, the bear gets the sunglasses. It shows a potential application due to our model's high performance.
Figure 2:
for an extended period. Recently, the field of Text-to-image generation has made significant progress with extensive research Generative Adversarial Network (GAN) [23, 24, 53, 98], Variational Autoencoder (VAE) [6, 13, 37, 76], and Diffusion models [31, 48, 55, 59, 63, 65, 67, 68. 72, 95]. The diffusion models achieve a new state-of-the-ar (SOTA) unconditional image generation. Numerous works [12, 33, 43, 46, 55, 61-63, 67, 91, 96, 99] have been done to make the image generated by diffusion models more aligned with the textual prompts. Among them, Stable Diffusion [65] is widely recognized model in the field, utilizes cross-attention mechanism to integrate textual conditions into the image generation process and employs the Latent Diffusion Model, which maps the image to latent space [65]. Our research is based the Stable Diffusion framework due to its adaptability and wide use in the community. Furthermore, different methods exist for controlling generative models. The primary categories for controlling generative models typically [1, 20, 34, 38, 40, 66, 80), Additional Sparse conditions [2, 4, 5, 8, 11, 28, 36, 49, 60, 79, 88, 93, 94], Brain-guided [3, 10, 19, 47, 54, 57, 74], Hound-Guided [60, 89], and some universe control [42, 49, 90]. Text-guided control utilizes textual descriptions directly influence the outcome, guiding the model based on specific verbal instructions. Our method focused on the text-guided controllable generative model.
include Text-guided [7, 18, 22, 60, 64], Image-guides
Subject-Driven Personalized Generation Subject-driven personalized generation is focused on creating images based on reference images. Recent works [1, 9, 20, 21, 34, 38, 40, 51, 58, 66, 73, 78, 80, 81, 85, 87] have explored techniques for producing striking resemblance images in multiple ways. One of the primary ways is to fine-tune the base text-to -mage models. Furthermore, there has been significant effort in research [14, 29, 32, 35, 38, 39, 71, 80, 84] aimed at integrating various concepts in personalization. While striking resemblance images are produced, fine-tuning the base model on small set of images leads to overfitting, resulting in unexpected issues. One prevalent issue discussed in prior research is the decrease in diversity. Recent studies have proposed various methods to address these issues. For instance, DreamBooth [66] introduced the -pecsSSpccific Priori Loss, which effectively addresses diversity reduction by recovering the class's prior knowledge. However, does not effectively maintain the ability of the model follow the text prompt. Another common issue is the inability to generate images under multiple conditions. Some Recent Research [29, 30, 40, 50, 75] proposed some methods to migrate this appearance. However, the underlying reasons for this phenomenon have not been thoroughly investigated. Our research endeavors to explore these reasons and develop solutions to overcome this challenge.
3 METHOD
3.1 PRELIMINARY
Text-to-Image Diffusion Model Stable Diffusion [65] is widely used in image generation task. For any input image, Stable Diffusion first transforms it into latent representation r using the encoder of variant auto-encoder [37]. For any input image, Stable Diffusion first transforms it into
3

---

Published as a conference paper at ICLR 2025
1e.5
Stable Dilfusion
fire: tuming
Custom Diffusion
Steps
dag
Essgee-ciios
wittt
dAS degg
Bon-tuiin
600
huppy
1200
(a) Comparison of distances in CLIP text space. (b) Visualization of the cross-attention map activation area.
Figure 3: (a) Each dot represents the position of a phrase combining an adjective and "dog" in the CLIP text-space. After tneettnnngg customized concepts move further away from the the distribution of super-class. (b) Visualization results of cross-attention map activation maps corresponding to the dog token. The bar chart on the right shows the average activation level in the dog area. Experiments show that the activation strengths of the corresponding classes decrease with the increase of the learning rate and the total number of training steps. These demonstrate that the customized concepts likely no longer belong to the super-class, resulting in a loss of super-class semantic information, such as wearing headphone.
latent representation æ using the encoder E of a variant auto-encoder [37]. The diffusion process then operates on by incrementally introducing noise, resulting in fixed-length Markov chain represented as æ T22 ..., 2r, where the chain's length. Stable Diffusion uses UNet architecture to learn the reverse of this diffusion process, predicting a denoised version of the latent input Ia at each timestep é from I to T. In the context of text-to- toi-tiggoeee generation, the text prompts' conditioning information y encoded into intermediate representation (y c, where Ta is pre-trained CLIP [61] text encoder. The primary objective in training this text-to-image diffusion model involves optimizing this transformation and prediction process, and it can be expressed
Crecon = Ez,g,.,t [|ee c0( t, To (1))
where c and c9 represent the noise samples from the standard Gaussian noise c NO 1) and predicted noise residual, respectively.
Subject-driven Diffusion Model Although text-to-image models have achieved remarkable per- formance, their controllability is limited. To personalize the generated outputs, DreamBooth[66) fine-tunes the diffusion U-Net to several target concept images. Custom Diffusion[38] introduces new modifier token V* in front of the category name and optimizes only the key and value matrices in the cross-attention layers, thereby improving efficiency.
3.2 EXPERIMENTAL ANALYSIS
We begin by observing simple experimental test cases to realize that the loss of compositional ability after personalization tuning is a common phenomenon. We then analyze the underlying logic through visualizations of the CLIP text-space and cross-attention strength map. Finally, we conduct a theoretical analysis to support our hypothesis.
Simple experimental test cases. As shown in Fig. 1, we observe loss of compositional ability after fine-tuning. We then conduct additional test cases and find that the headphone concept is not the only one affected; other concepts also experience a similar loss. Furthermore, this situation occurs in both the dog case and other classes.
Semantic drift in CLIP text-space. To elucidate the reasons, we project text into the CLIP text space and use two dimensions for simplified visualization in Fig. 3a. Each dot represents phrases composing the super-class ("dog" in this case) with different adjectives (e.g., "a cute dog", "a white dog" etc.). Before fine-tunin,, the customized concepts (<sks> dog) have special meaning and at a similar distance from the super-class as other words. After fine-tuning, we observe significant increase in the distance of the target concept from its corresponding super-class. This indicates that
4

---

Published as a conference paper at ICLR 2025
the semanties of the personalized concepts change during fine-tuning, In short, the model increasingly fails to recognize that personalized concepts belong to the dog category. This shift may lead to an inability access the knowledge associated with the super-class(lke wearing a headphone). More details can be found in Appendix D, E.
Reduction of cross-attention activation strength. We further investigate the model by visualizing the cross-attention layers in Fig. 3b. The attention maps indicate the activation area of super-class words in cross-attention layers. It shows that while the "dog" token activates the relevant region in the image, its activation level is notably lower than that of the pre-trained model. Furthermore, its activation level decreases with the increase epochs and the learning rate. These findings align with our observations in the CLIP text space and provide support for hypothesis that the customized concepts are increasingly not recognized as part of the super-class during fine-tuning.
Next, we theoretically analyze why the semantic drift of personalized phrases results in the weakening of the compositional ability from the respective reduction in the entropy composable conditional probability.
3.3 THEORETICAL ANALYSIS
Drawing on the insights of [45], a trained diffusion model can be seen as implicitly defining an Energy-Based Model (EBM) [17]. This perspective allows us to build on prior research in composing EBMs and adapting them for use in diffusion models. Building on the work of [16] in the context of generating images with multiple attributes and Bayes' theorem, the conditional probability can be decomposed as:
p(z//etasss C1, ... (i) p(æ, Celass: C1, 02. ..., 03) pC ( (x)(())
II p(es(z)
GET
p(cc)p(alcs)
= II GET p(x)
where T is all the set of conditions in prompts except for the class, p(c; represents the probability of occurrence of condition 6; in the training dataset and can be regarded as a constant for large-scale pre-training models. p(q/(a) represents implicit classifier, denoting the probability of categorizing a concept as Celass. Specifically, p(Cotaes/z) (z) represents specific implicit classifier for the super- category. Thus, we have:
p(c)p(x))) RE p(x)
p(r|Cetass, C1, .G. x p (CCataso=))p(2 II
Denoted p(x:) I p(ccp(zle)) p(z) as d(xr,, p p(Cetnsslz)) as q(x), and p(xlc,, .2....,c.) ..) as a(x). The
entropy of is calculated as:
H(a) = Σ c(())(<)) + log(d(x))]
(5)
After fine-tuning, the components of d(x)) change only slightly and can be treated as unchanged, and the implicit classifier po (Cetass)a)) (x) changes Py (Celass (r), Thus the difference in entropy before and after can be expressed as:
AH = Σ (o(r))(x) [log(ι( + log(d(x))
(6)
Σ ao (x)d(x)) log(ggr (z)) + log(d(x))))
d(x) Σ (1) log log go ( (x) ) log 9 (z)
+ log d(xx) (2) 9o (x))
5

---

Published as a conference paper at ICLR 2025
Headphane
Headphove
Headphone
Headphane
Dog Headphone
bog
Headppong
Headppone
V" Deg
Dog
Dog
Figure 4: The orange and green point sets represent the distributions of dogs and headphones, respectively, and their overlapping regions represent their joint probability distributions. During the tuning process, the conditional distribution of dogs and headphones shrinks, which gradually increases the difficulty of sampling. Unlike the Prior Preservation Loss (PPL. in DreamBooth [66], which aims maintain class diversity, proposed Semantic Preservation Loss (SPL.) focuses recovering the semantic space of the customized concept. This approach enables our method to synthesize images that are more consistent with the text prompt.
Based observations in Fig. 3a, 3b show that (2) 48' (z), combining the properties of probability theory and the monotonically decreasing nature of x log x at (0,1), we have:
qe (z) log qo (x) go (x) log qu (x) < O; log d(x) <
(8)
Thus, we have:
AH(a) 0
As result, it is more difficult sample from our demanded conditional distributions under Celass; C1. .t...., Cj conditions than before the fine-tuning, leading to the phenomenon that the com- bining ability is weakened after the fine-tuning. We will discuss the theoretical reasons here in more detail in Appendix B. Fig. 4 illustrates the changes in the distribution during this process that lead to weakening of the compositional generation capability.
The diminished combinatorial capacity is due to the increased difficulty in sampling from joint conditional probabilities caused by shifts the semantics customisation concepts. Therefore, in order to reduce the difficulty of sampling in joint conditional probability distributions, we need to recover the original semantic space of the text, i.e. to recover the semantic distance between custom concepts and superclasses in the semantic space that has been distanced by the fine-tuning process. So, the next section, we present our proposed semantic preservation loss to mitigate the semantic drift that occurred during fine-tuning.
3.4 SEMANTIC PRESERVATION LOSS
As analyzed above, the key challenge lies in preserving semantic information during fine-tuning to reduce the difficulty of sampling from the joint conditional distribution. To address this, we propose novel loss function aimed at constraining semantic variation throughout the fine-tuning process.
Specifically, considering that there are N special tokens, each representing customized concept. During the process, the embeddings associated with these tokens are fine-tuned to align with the target concepts. Our loss function designed to minimize the semantic distance between the phrase containing the special token (c.g.. photo of a V dog) and the phrase containing only the class word (e.g., photo
dog). By labeling the training prompt as Ptp (e.g., a photo of a V* dog), and the class prompt as Pop (e.g., a photo of dog), Egp in Stable Diffusion's 's semantic space. The semantic preservation loss (SPL) is calculated by the sum of the cosine distance of their text embeddings. Formally speaking, our proposed SPL can be
text encoder to get their embeddings Esp and

---

Published as a conference paper at ICLR 2025
photo of
cls
photo of
y dog
Trainable
Frozen
Text transformer
Text transformer
Semantic Preservation Loss
BOS
EOS
ReaNet
Denoising U-Net
Reconstruction Loss
Figure 5: The framework of ClassDiffusion. The personalization fine-tuning strategy is based on Custom Diffusion [39], which primarily fine-tunes the K and parameters the transformer block. Our semantic preservation loss (SPL) is calculated by measuring the cosine distance between text features extracted from the same text transformer (using EOS tokens as text features following CLIP) for phrases with personalized tokens and phrases with only super-class.
expressed by the following equation:
N L Lup ΣΣΣ De (Fisc Fe))
(10)
where B represents the batch size, L denotes the hidden dimensions of the text encoder, and D. implies the cosine distance. We can represent the final training objective as:
L
Crecon XC
(11)
The overview of our proposed model is shown in Fig. 5
EXPERIMENTS
4.1 EXPERIMENT DETAILS
Implementation details Our method is built Stable Diffusion V1.5, with learning rate 10 10-6, and batch size 2 for fine-tuning. We used 500 optimization steps for a single concept and 800 for multiple concepts, respectively. During inference, the guidance scale set to 6.0 and the inference steps are set to 100. The semantical preservation loss weight is set to 1.0 during all experiments. All experiments are conducted on 2 RTX4090 GPUs. Our method uses min for the generation of single concepts and ~ 1l min for the generation of multiple concepts.
To better preserve the semantic space, we compute SPL between text embeddings embedded in the semantic space of the Stable Diffusion model. Therefore, we utilize the CLIP [61] text encoder from Stable Diffusion v1.5 [63], specifically lar--vittlagepatteeee of phrases. Following common practice, we use the End of Sequence (EOS) token to represent the semantics of embeddings.
[47], to extract the text embeddings
Baselines We compare our method with state-of-the-arr (SOTA) competitors, including Dream- Booth [66], Textual Inversion [20], Custom Diffusion [38], NeTI [1], SVDiff [29]. For DreamBooth, CustomDiffusion, and Textual Inversion, we used the diffusers [77] version of the implementation. For NeTI, we use its official implementation. Given that SVDiff does not have an official open-source repository. For SVDiff, recommendations of the official paper. To ensure faimess of comparison, all these baselines are built on Stable Diffusion V1.5.
the implementation of [69]. All training parameters follow the
7

---

Published as a conference paper at ICLR 2025
Textual Inversion
Custem Diffusion
Source
Curs
DreamBooth
NeTT
SVDif
of cat wearing headphone
A photo of a dog wearing chef"s hat photo flower
teapor the of moutain
A duck_toy wearing firefighter's outfit
Figure 6: Qualitative comparison between our method and baselines with single given concept. Our method generates images that align with the prompts, surpassing all baselines.
Datasets Following previous work [29, 66, 75], we conduct quantitative experiments on DreamBooth Dataset [66]. It contains 30 objects including both live objects and non-live objects. In addition, used images from the Textual Inversion Dataset [20] and CustomConcept101 [38] in qualitative experiments.
Evaluation metrics We assess our approach using three metries: CLIP-I, CLIP-T, and DINO-I. CLIP-I calculates the visual similarity between the produced images and the target concept images by utilizing CLIP [61] visual features. CLIP-T evaluates the similarity between text prompts and images. one baseline contains the special token S*, it will be replaced with a prior class word. In the case of DINO-I, evaluate the cosine similarity between the ViT-S/16 DINO [56] embeddings of the generated images and the concept images. Further, we note the impact of CLIP's outdated performance the fairness of the evaluation. Therefore, we introduce the BLIP2-T Score, which calculates the similarity between text features extracted from BLIP2's Q-former and image features extracted from Vision Encoder as score. This metric is designed by calculating the similarity between image and text embeddings extracted by the BLIP2 model. Our approach involved utilizing Transformer [82] implementation and the fine-tuned weights of BLIP2-IMT on CoCo [44], with VITAI. [15]. This new metric aims to offer more equitable and efficient evaluation measure for future studies in this field. Empirical findings from various studies [25, 40, 41, 70, 83, 86] indicate that BLIP2 outperforms CLIP significantly in the assessment of text-image alignment.
4.2 QUALITATIVE & QUANTITATIVE EXPERIMENTS
Qualitative Experiments We compare our method with DreamBooth [66], Textual Inversion [20], NeTI [1], SVDiff [29], and Custom Diffusion [38] on challenging prompts. The results depicted in Fig. 6 demonstrate the outcomes obtained from these prompts. Fig. shows a story of given dog and sunglasses. The experimental findings indicate substantial superiority of approach over other techniques regarding aligmment with text prompts, without any decline similarity to the specified concept. More qualitative results are shown in Appendix. Also, conduct
8

---

Published as a conference paper at ICLR 2025
Source Image
Ours
photo bear sitting front of bazn
dog wearing backpack
wearing classes
CD
Figure 7: Qualitative comparison between method and Custom Diffusion(CD) in multiple concepts. Our method has better text alignment than custom diffusion.
qualitative experiments with multiple concepts the combinations <cat>, <sunglasses> <bear>, <barn>; <dog>, <backpack>; Fig. 7 shows the results of the experiments. The experiments show that method can be aligned with prompts better than custom diffusion in multi-concept generation.
Quantitative Experiments Following the previous work, used 20 concepts for quantitative experiments. For Single Concept text similarity metrics (CLIP-T, BLIP2-T), we followed the 25 prompts used in [66]. sampling 20 images per prompt. The results of the experiment are shown in Tab. 1. Experimental results show that our method obtains new SOTA on each text similarity metric, indicating that we have good compositional generation capability.
Table 1: Quantitative Results on all Metrics and Results of the User Study. The last two columns display the win rates of our method compared other approaches in the user study, evaluated in terms of text similarity and image similarity. A win rate exceeding 50% (highlight by v) indicates that our method outperforms the compared methods on the corresponding metric, as judged from human perspective.
User-T Win Rate
User-I Win Rate
Method
CLIP-TT
CLIP-If
DINO-I1
BLIP2-T
TIFA
DreamBooth [66]
0.249
0.855
0.700
0.295
0.559
95.4%
42.1% 1%
Textual Inversion [20]
0.242
0.825
0.631
0.308
0.505
95.1% /
75.0% /
Single Concept
Custom Diffusion [38]
0.286
0.837
0.693
0.416
0.746
79.1% /
40.0%
NeTT [1]
0.290
0.838
0.648
0.329
0.607
78.8%
70.0% /
SVDiff
0.293
0.834
0.606
0.418
0.835
56.6%
95.8% /
Our
0.300
0.828
0.673
0.460
0.843
Multiple Concepts
Custom Diffusion [38]
0.282
0.813
0.636
0.380
Our
0.320
0.821
0.604
0.477
4.3
ADDITIONAL EXPERIMENTS
User Study We also performed user study to validate the effectiveness of our method. We used the same set images generated the Section 4.2 for user study, details of which are available in Appendix. The results of the user study located in Tab. 1. The numbers in the table indicate what percentage our method is considered by humans to be superior to the compared methods. The result of the user study shows that our method outperforms all methods in text similarity (> 50%).

---

Published as a conference paper at ICLR 2025
Weiaht
Sip
900
750
1500
Figure 8: Generation results for the prompt "a photo of dog wearing a headphone" with different step counts and SPI. weights. All results are generated using the same random seed.
dog ruming on street
fiotting
cot chesing ball
Figure 9: Result of generated videos, showing good textual alignment and similarity of given concepts.
Although our method does not outperform all methods in image similarity, given the high CLIP-I scores, our method still produces images that are highly consistent with the given concept.
Personalized Video Generation We investigate the implementation of our method in personalized video generation. We utilized AnimateDiff V2 [26, 27] for video generation, configuring parameters to a resolution of 51125112 guidance scale of 7.5, and 25 inference steps. The outcomes of the video technique produces videos
generation process illustrated in Fig. 9. Utilizing AnimateDDfff, that exhibit strong textual and conceptual coherence without the need for additional training. This demonstrates that approach, which aligns personalized phrases with superclass-centric semantics, can generate engaging videos with dynamic generation capabilities stemming from pre-training, along with the ability to transition across corresponding domains.
A Abalation Experiments We studied the influence of different weights of semantic preservation loss (SPL). The results show that higher SPL weights preserve combining ability better. At s SPL weight of 100, all steps successfully depict "wearing a headphone." However, at lower weights of 0. and 0.01, the headphone details diminish by 750 steps and disappear by 1500 steps. On the other hand, lower SPL weights restore specific concept features more effectively.
CONCLUSIONS
In this work, highlight the problem of weakened compositional ability due individualized fine-tuning and provide an analysis of the causes of this problem from experimental observations and information information-theoretico theoretic perspectives. We discovered that this weakening effect primarily attributed to the semantic shift of the customized concepts throughout the fine-tuning process. As the model undergoes fine-tuning, the representations of these concepts gradually drift away from their original meanings, leading to a misalignment with the intended semantics. This semantic drift complicates the model's ability to accurately sample from the joint conditional distribution, ultimately hindering its performance in generating concepts. We then introduce a new approach, termed ClassDiffusion, which mitigates the weakening of compositional ability by restoring the original semantic space. Finally, present comprehensive experimental results showcasing the efficacy of ClassDiffusion and the fresh perspectives offers on interconnected fields.
understanding the intended outcomes based on the fine-tuned
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This work was supported in part by the National Key R&D Program of China under Grant No. 2022YFC3310200, the National Natural Science Foundation of China NN4.9247000, U23A20314), Beijing Natural Science Foundation (No. L242022), ), and the Fundamental Research Funds for the Central Universities (No. 2024XKRC082)
REFERENCES
[1] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text-to-image personalization. ACM Transactions on Graphics (TOG), 42(6):1-10, 2023.
[2] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18370-18380, 2023.
[3] Yunpeng Bai, Xintao Wang, Yan-pei Cao, Yixiao Ge, Chun Yuan, and Ying Shan. Dreamdiffu- sion: Generating high-quality images from brain eeg signals. arXiv preprint arXiv:2306. 2023.
[4] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, et al. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint 1eXiv:2211.013244 2022.
[5] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/VV Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.
[6] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang. Ming- Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint 2rXiv:2301.00704 2023.
[7] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4))1-10, 2023.
[8] Kai Chen, Enze Xie, Zhe Chen, Lanqing Hong, Zhenguo Li, and Dit-Yan Yeung. Integrating geometric control into text-to-image diffusion models for high-quality detection data generation via text prompt. arXiv preprint arXiv:2.366.0400 2023.
[9] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval- augmented text-to-image generator. arXiv preprint arXiv:2209. 22t.22999,444,, 2022.
[10] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 22710-22720, 2023.
[11] Jiaxin Cheng, Xiao Liang, Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdif- fuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv:2302.088 08908, 2023.
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[13] Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 14:119822-19835 2021.
[14] Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text- to-image generation via positive-negative prompt-tuning. arXiv preprint arX1v:2211.11337, 2022.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.

---

Published as a conference paper at ICLR 2025
An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint Stt::201001192299 2020.
[16] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-6647, 2020.
[17] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
[18] Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training- structured diffusion guidance for compositional -kt-to-immaee synthesis. arXiv preprint arXiv:2212.088 05.0032, 2022.
[19] Honghao Fu, Zhiqi Shen, Jing Jih Chin, and Hao Wang. Brainvis: Exploring the bridge between brain and visual signals via image reconstruction. arXiv preprint arXiv:23 14871, 2023.
[20] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
[21] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Designing an encoder for fast personalization of text-to-image models. arXiv e-prints, pages arXiv-2302, 2023.
[22] Songwei Ge, Taesung Park, Jun- Zhu, and Jia-Bin Huang. Expressive text-to-image generation with rich text. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7545-7556, 2023.
[23] Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3224-3234, 2019.
[24] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[25] Paul Grimal, Hervé Le Borgne, Olivier Ferret, and Julien Tourille. Tiam a metric for evaluating alignment in text-to-image generation, 2024.
[26] Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311.16993, 2023.
[27] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. International Conference on Learning Representations, 2024.
[28] Cusuh Ham, James Hays, Jingwan Lu, Krishna Kumar Singh, Zhifei Zhang, and Tobias Hinz. Modulating pretrained diffusion models for multimodal image synthesis. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[29] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
[30] Yutong He, Alexander Robey, Naoki Murata, Yiding Jiang, Joshua Williams, George Pappas, Hamed Hassani, Yuki Mitsufuji. Ruslan Salakhutdinov, and Zico Kolter. Automated box prompt engineering for personalized text-to- eext-to-image generation. arXiv preprint vrXiv:2403.19100 2024.
[31] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in nearal information processing systems, 33:6840-685 2020.
[32] Hexiang Hu, Kelvin C. K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, Ming- Wei Chang, and Xuhui Jia. Instruct- imagen: Image generation with
instruction, 2024.
[33] Yihan Hu, Yiheng Lin, Wei Wang, Yao Zhao, Yunchao Wei, and Humphrey Shi. Diffusion for natural image matting. In European Conference on Computer Vision, pages 181-199. Springer, 2025.
12

---

Published as a conference paper at ICLR 2025
[34] Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, and Qian He. Dreamtuner: Single image is enough for subject-driven generation. arXiv preprint arXiv:2312.13691, 2023.
[35] Sangwon Jang, Jaehyeong Jo, Kimin Lee, and Sung Ju Hwang. Identity decoupling for multi- subject personalization of text-to-image -oagge models, 2024.
[36] Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd:
A native pegletoon-uuides diffusion model for human image generation. In Proceedings of the
IEEE/CVF International Conference on Computer Vision, pages 15988-15998, 2023.
[37] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114 14, 2013.
[38] Nupur Kumari. Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.
[39] Gihyun Kwon, Simon Jenni, Dingzeyu Li, Joon- Joon-Young Lee, Jong Chul Ye, and Fabian Caba Heilbron. Concept weaver: Enabling multi-concept fusion in text-to-image models, 2024.
[40] Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-ro-image generation and editing. Advances Neural Information Processing Systems, 36, 2024.
[41] Jumnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-iman pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742, PMLR, 2023.
[42] Wei Li, Xue Xu, Jiachen I Liu, and Xinyan Xiao. Unimo-g: Unified image generation through multimodal conditional diffusion, 2024.
[43] Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Plataniotis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models. arXiv preprint vrXiv:2405.166455 2024.
[44] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Ziticck. Microsoftt coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 6-12, 2014, Proceedings, Part 13, pages 740-755. Springer, 2014.
[45] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439, Springer, 2022.
[46] Xihui Liu, Dong Huk Park, Samanch Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 289-299, 2023.
[47] Yizhuo Lu, Changde Du, Qiongyi Zhou, Dianpeng Wang, and Huiguang He. Minddiffuser: Con- trolled image reconstruction from human brain activity with semantic and structural diffusion. In Proceedings of the s ACM International Conference on Multimedia, pages 5899-5908, 2023.
[48] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378. 2023.
[49] Yuanhuiyi Lyu, Xu Zheng, and Lin Wang. Image anything: Towards reasoning- coherent and training-free multi-modal image generation, 2024.
[50] Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion Open domain personal- ized text-to-image generation without test-time fine-tuning. arXiv preprint arXiv:2307. 11410, 2023.
[51] Yiyang Ma, Huan Yang, Wenjing Wang, Jianlong Fu, and Jiaying Liu. Unified latent diffusion for joint subject and text conditional image generation. arXiv preprint 2303::200000313 2023.
[52] David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
13

---

Published as a conference paper at ICLR 2025
[53] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint ar1iv:1411. 1784, 2014.
[54] Pengyu Ni and Yifeng Zhang. Natural image reconstruction from fmri based on self-supervised representation learning and latent diffusion model. In Proceedings of the 15th International Conference on Digital Image Processing, pages 1-9,20 2023.
[55] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprini arXiv: 10741, 2021.
[56]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprini arXiv:2304.077 19719,3,00 2023.
[57] Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using generative latent diffusion. Scientific Reports, 13(1):15666, 2023.
[58] Jae Wan Park, Sang Hyun Park, Jun Young Koh, Junha Lee, and Min Song. Cat: Contrastive adapter training for personalized image generation, 2024.
[59] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.00 00952. 2023.
[60] Can Qin, Ning Yu, Chen Xing. Shu Zhang, Zeyuan Chen, Stefano Ermon, Yun Fu, Caiming Xiong, and Ran Xu. Gluegen: Plug and play multi-modal encoders for x-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23085- 23096, 2023.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[62] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(10))1-67, 2020.
[63] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2XX::22044006125, 1(2):3, 2022.
[64] Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems, 36, 2024.
[65] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10699 June 2022.
[66] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-drivens generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[67] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphacl Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 1136449-36499 2022.
[68] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation, 2023.
[69] Mingkai Shing. Svdiff: Stochastic video diffusion for conditional video generation. https:
/ //github. com mkkshing/ svd wwwifff-yyy ff- 2023.
[70] Jaskirat Singh and Liang Zheng. Divide, evaluate, and refine: Evaluating and improving text-to-image alignment with iterative vqa feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
14

---

Published as a conference paper at ICLR 2025
[71] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li. et al. Styledrop: Text-to-image generation in any style. arXiv preprint arXiv:2306. 00986, 2023.
[72]
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500. 2020.
[73] Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, and Xiao Yang. Moma: Multimodal Ilm adapter for fast personalized image generation, 2024.
[74] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453-14463, 2023.
[75] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings. pages 1-11, 2023.
[76] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive mod- cling: Scalable image generation via next-scale prediction. arXiv preprint WvXi:2244.029005 2024.
[77] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-ar diffusion models. https:/ /tapp:://giie /tttttbttb.u//// com/ hugging face / di diffusers, 2022.
[78] Anton Voronov, Mikhail Khoroshikh, Artem Babenko, and Max Ryabinin. Is this loss informa- tive? faster text-to-image customization by tracking objective dynamics. Advances in Neural Information Processing Systems, 36, 2024.
[79] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or.
Sketch-guided text-to-image diffusion models. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1-11, 2023.
[80] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.09522 2023.
[81] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943-
15953, 2023.
[82] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: ofate-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics.
[83] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-iman image synthesis. arXiv preprint vrXiv:2306.09344 2023.
[84] You Wu, Kean Liu, Xiaoyue Mi, Fan Tang, Juan Cao, and Jintao Li. U-vap: User-specified visual appearance personalization via decoupled self augmentation. arXiv preprint 2xXiv22403.202111 2024.
[85] Chendong Xiang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. A closer look at parameter- efficient tuning in diffusion models. arXiv preprint arXiv:2303. 230.:3888.18 2023.
[86] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
[87] Zhongcong Xu, Jianfeng Zhang. Jun Hao Liew, Hanshu Yan, Jia- Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint 1.. 2023.
15

---

Published as a conference paper at ICLR 2025
[88] Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14256-14266, 2023.
[89] Yue Yang, Kaipeng Zhang, Yuying Ge, Wenqi Shao, Zeyue Xue, Yu Qiao, and Ping Luo. Align, adapt and inject: Sound-guided unified image generation. arXiv preprint arXiv:2306. 1504, 2023.
[90] Yuyang Yin, Dejia Xu, Chuangchuang Tan, Ping Liu, Yao Zhao, and Yunchao Wei. Cle diffusion: Controllable light enhancement diffusion model. Proceedings of the 31st ACM International 2023.
Conference Multimedia, pages
[91] Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225, 2023.
[92] Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency, 2024.
[93] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13174-23184 2023.
[94] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-imaae diffusion models, 2023.
[95] Shiwei Zhang. Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjic Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint 1..xiv:2311.04444 2023.
[96] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and
Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint 23X1v:2305.13077, 2023.
[97] Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Tong- Yee Lee, Oliver Deussen, and Changsheng Xu. Prospect: Prompt spectrum for sarribuee-waan personalization of diffusion models. ACM Transactions on Graphics (TOG), 42(6):244:1- 244:14, 2023.
[98] Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based based generative adversarial network. arXiv preprint 1rXivv1609.03122 2016
[99] Yiming Zhong, Xiaolin Zhang, Yao Zhao, and Yunchao Wei. Dreamlem: Towards high quality text-to-3d generation via latent consistency model. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1731-1740, 2024.
16

---

Published as a conference paper at ICLR 2025
A VISUALIZATION AFTER FINE TUNNING
In section 3, we present visualization results for the text space and the cross-attention layer of other methods, highlighting the semantic bias that emerges the text space, leading to decline in compositional ability. To further affirm the effectiveness of our method and our hypothesis, we also visualize the textual feature space and the cross-attention layer with our method this section. The visualization results opposed to 26 out of 71 before fine-tume tine-tuning and 67 out of 71 for other methods. comparison with the visualizations in Fig. 3b and Fig. 3a reveals that model effectively addresses the semantic drift in the text space.
depicted in Fig. 10. The ranking of the distance is 36 out of 71, as
Aury
soury dOg
CANA BON
NAY 4-
Evendl
Figure 10: Visualization results after fine-tuning of our approach.
B ENTROPY REDUCTION DURING THE FINE-TUNING
In section 3.3, we provide a solid theoretical analysis for the weakening of compositional ability due to the semantic drift from the perspective of information theory and probability distributions. In this section, we will discuss it in a more detailed way.
In the field of subject-driven personalization generation, manifest phenomena caused by overfitting: weakening of diversity in classes of given concepts and weakening of compositional ability. In addition the calculations mentioned in the main body of the text, the entropy of combined conditional probability can also be calculated as conditional entropy:
H(Xler.c2....
Make the given concept into series of specific conditions: Csl...., Cass each condition describes one of the features the given concept. The entropy after fine-tuning will be:
H(Xlc1, ......, Cn, C.l...,, c.t) = H(X ...,))......))
c.)) I(X ... Guj Csl, ..s)
(2)
Where represent the mutual information, According to [52], have:
[[X]....,, Gni Cal: Col....... Cos) 2
(3)
H(X H(X|a,,2 02... Cn. Col. ...., Coi) H(X Qq..,,..
(4)
17

---

Published as a conference paper at ICLR 2025
Course & Poor Performance
Source
bear plushic
plushic
toy
animal
Visualization of generating "a <center word sitting on the chair.. This <center word> is used in both prompt and class token. Experiments show that fine-grained center word will benefit our proposed method.
Figure 11:
However, the entropy reduction here different from the entropy reduction in the main text. The entropy reduction here leads to reduction in the diversity of the generated images. Specifically, when the cue word is "a photo of a dog", the image generated is closer to the given concept than to the diversity of dogs.
C FINE-GRAINED EXPERIMENTS
At the of approach, want the semantics of personalized phrases to be closer to the category-centered words. In this section, we explore the effect of different category cetegory-centered words on the results for the same given concept. Fig. 11 shows the training results using different category- centered words. The results show that the use of different center words leads to significant differences in performance, and fine-grained center word benefits method. Also, it indicates the importance of recovering the semantical space of the customized concept.
D PROMPT USED IN THE VISUALIZATION OF CLIP TEXT SAMPLE SPACE
In this section, we provide a realistic visualization of the schematic in Fig. 3a, and discuss the prompt generate the 70 phrases that include adjectives and super-categories "dog", and the whole 71 adjectives.
The realistic visualization of the schematic is:
The prompt we use is:
Please help me generate some adjectives that can describe an attribute of dog in a photo.
The adjectives we use are:
happy dark fast hard weak good
beautiful bright old soft
sad big
tall
short
small warm
young
slow heavy light strong
cold
bad
rich poor
thick thin
expensive
cheap quiet
loud clean boring healthy right wrong
dirty
smart stupid
interesting
new
old-fashioned
safe
dangerous
sick
easy
difficult
high
low
near
far
early busy free
late
wet
dry
careful
careless friendly
unfriendly
important energetic calm
unimportant
famous
unknown
excited
serious funny fired
18

---

Published as a conference paper at ICLR 2025
Trut Sample Cetributiss using Pretrainee CUP butt vooder
Teat Sumpple Ditribation uring fine-ure CUP But Frooder
the
dlg
SPOriS
wod
AN died High
dig
ScPoggarie
buty dage
Atp
Rank:26/71
Rank:67/71
Figure 12: Visualization of the CLIP sample space. Using ChatGPT, created 70 phrases containing adjectives related to the superclass of dogs. Subsequently, text features derived from these phrases were processed through the CLIP text encoder, downscaled, and their distance from the central point of the superclass (representing dog image) was calculated. The comparison between the pre- pre-trained model (illustrated in the left figure) and the fine-tuned model (depicted in the right figure) indicates that in the pre-trained model, phrases with special tokens ranked 26 out of 71, while in the fine-tuned model, they ranked 67 out of 71. Moreover, it is evident that phrases containing special tokens are situated further away from the central point of the superclass.
E 2D TEXT SPACE DISTANCE CALCULATION
In this section, we discuss how we visualized CLIP's text sample space in Fig. 3a. First, we collect 70 phrases that combine adjectives with words representing the class of given concepts (e.g.. "a happy dog" or "a cool dog"). Using the CLIP text encoder, we extract text embeddings for these phrases. To visualize the semantic space and intuitively track modifications within it, these high-dimensional embedding vectors to 2D. An overview of this result are shown in Fig.3(a), meanwhile we show the real experimental results in Fig. 12. Formally, we the adjectives generated which are described in Section. D as the initialize set S, and use the following pseudocode to get a 2D point T:
t-SNE reduce
Algorithm 1 Algorithm to Convert Character Set to 2D Point Set
1: Input: Initial character set S
2: Output: 2D point set T, Distance set Dis
3: E <- CLIP text encoder encoding(S)
Encode the character set to an encoding set
4: T <- TSNE(E)
Dimensionality reduction of the encoding set to a point set i= 1,2,..., ... [T]} > Calculate the 2D distance to Telase for each point
5: Dis < [ Tetaeel
in T'
6: return T, Dis
F MULTI-CONCEPTS EXPERIMENTS
In this section, we demonstrate the ability of ClassDiffusion to generate multiple concepts, specifically 3 concepts in model. Fig 13 shows the result of this experiment. The experiments demonstrate that ClassDiffusion generates high-quality results when combining multiple concepts, validating the effectiveness of proposed methods.
19

---

Published as a conference paper at ICLR 2025
Source Images
ClassDiffusion( Our)
Figure 13: Qualitative result of generating three concepts.
G MORE BASELINE COMPARISON
To further evaluate the performance of our proposed method, we further introduce a new baseline Prospect [97] which aims at solving similar problems we observe. The result of the experiment is shown in Tab. 2. The quantitative results below show that our model achieves superior performance.
Models
CLIP-TY
CLIP-It
DINO-11
Prospect 0.294
0.815
0.588
ClassDiffusion((rrs)
0.300
0.828
0.673
Table 2: Quantitative results comparing with Prospect
H QUANTITATIVE ABLATION OF SPL WEIGHT
In this section, we conduct quantitative ablation experiment the choose of SPL weight. Tab. shows the result of the experiment result. This result shows that CLIP-T becomes higher (increase in the ability follow prompts) and DINO-I decreases(decrease in the ability to customize concepts) with increasing SPI. weight, which is consistent with our expectations. Meanwhile, we find that SPL is loss function that is insensitive weight. Combined with the qualitative observation in Fig. we prefer to choose as the SPL weight.
I DETAILS OF USER STUDY
We offer users comprehensive user study guide that includes user selection criteria which is shown in Fig. 14. Additionally, to maintain faimess, we positioned our method alongside the baseline method randomly to prevent users from showing bias towards either method. Our percentages in Tab. are obtained by calculating the number that chose our model better as a percentage of the overall number that made preference.
20

---

Published as a conference paper at ICLR 2025
User Study
Study xXxx-xxx.com
After entering the correctokker, you will see interaction similar to the following:
bowl in the jungle
Ranh Appias
Right
Equally good
The images shown the should make decision
1. one of the image aligned with the prompt and another isn't, choose the align one. prampt,
images aligned with the promot.
3. you can't make a choice (l.o. you think both diagrams are equally good/bad for the prompt) Equally good.
For the example image, should choose "left better". This because although both the left and right images have bowla, the left image has jungle and the right doea not.
Or will see an interaction similar the following:
Reference Image
Image
butter
Right
Equally good
image Equally good
not make decision.
Figure 14: User Study Guide, which describes the user selection criteria and provides an example for reference.
21

---

Published as a conference paper at ICLR 2025
SPL weight
CLIP-T1
DINO-I
0.01
0.299
0.677
0.1
0,300
0.674
0,300
0.673
0.300
0.665
100
0.301
0.661
Table Performance metrics for different SPL. weights.
MORE QUALITATIVE RESULT
In Fig. 6, one generated image is provided for each prompt. Fig. 15 presents more images generated from the prompts, thereby reinforcing the efficacy of approach.
Textual Inversion
Custom
Source
Ours
DreamBooth
NeTI
SVDiff
Diffusson
A photo of a cat wearing a headphone
photo of dog wearing chef's hat
A photo of flower in vose
teapot on the mountain
A duck_toy wearing firefighter's outfit
Figure 15: Qualitative result of same prompts the main text.
K LIMITATION
In this section, discuss the limitations of our work. Our work able generate images that aligned with the given prompt while keeping the features of the given concept. However, there are two major limitations in work:
22

---

Published as a conference paper at ICLR 2025
Considering that reconstruction of the human face is fine-grained, and the phrase "a photo of a human" or "a photo of a human face" can not include extensive information about humans. Whether our work transfer to human-driven personalized generation remains explored.
For objects that have combination of categories, choosing an appropriate center word requires some experimentation.
L
SOCIAL IMPACT
The advancements in text-to-image customization through fine-tuning diffusion models, evidenced by work ClassDiffusion, have significant social implications. By enhancing the compositional capabilities of these models, our approach can contribute to a variety of fields, including digital content creation, and education. In the realm of digital content creation, ClassDiffusion enables artists, designers, and marketers to generate more precise and complex images based on textual descriptions. This improvement reduces the time and effort required produce customized visual content, fostering creativity and innovation. It also allows for the seamless incorporation of personalized elements into digital artworks, advertising materials, and user-generated content, thereby enhancing engagement and satisfaction. Class can also be powerful tool in educational settings. Educators can use this technology to create illustrative materials that are tailored specific learning objectives. For instance, teachers could generate images that accurately depict historical events, scientific concepts, literary scenes, making learning more interactive and engaging for students. Furthermore, this technology can aid in the development of educational content for diverse learning needs, including materials for students with disabilities.
While the advancements in text-to-image generation hold promise, it is essential to address the ethical considerations associated with their use. Ensuring that these models are free from biases and do not perpetuate harmful stereotypes crucial. Our work ClassDiffusion includes measures to mitigate semantic drift, which helps maintain the integrity and accuracy of generated content. Continuous evaluation and updates are necessary uphold these standards and ensure the technology benefits society as a whole.
23


Reviews:
[{"review_id": "9BTNiBsFqT", "reviewer": "Reviewer_pV4W", "summary": "This work delves into the captivating realm of text-to-image customization. However, current tuning-based methods often struggle with a common pitfall: they tend to overfit to the limited concepts presented in their training datasets. This overfitting can significantly diminish the model's compositional abilities. The authors highlight that this decline is mainly due to the semantic shifts that occur with customized concepts during the fine-tuning process.  To address this issue, the authors introduce an approach called ClassDiffusion. This technique revitalizes the model's compositional strength by restoring its connection to the original semantic space. Through a series of thorough experiments, the authors demonstrate the effectiveness of ClassDiffusion and the fresh insights it brings to interconnected fields. The results not only showcase its prowess but also pave the way for exciting advancements in the domain.", "questions": "see weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors unveil the intriguing underlying cause behind the overfitting of SD. Rather than a typical case of overfitting, they suggest it’s more about semantic drift—a perspective I find quite compelling, though I still believe overfitting truly occurs.\n\n2. This paper is not only well-written but also flows smoothly, making it a pleasure to read.\n\n3. The inclusion of additional relevant baselines demonstrates the authors' thorough research and deep engagement with the topic.\n\n4. The authors provide a wealth of comprehensive experimental results, reinforcing the strength of their proposed ideas.", "weaknesses": "1. I agree that semantic drift occurs during fine-tuning. This drift arises because the model is an intricate network filled with numerous parameters. When we fine-tune with only few samples, the adjustments to these parameters are minimal, yet the model still retains its generative capabilities. In my view, semantic loss may struggle to restore the original semantic space, as it doesn’t preserve the nuances of the drift encountered. Theoretically, utilizing the entire training text data could mitigate this issue.\n\n2. While I think that the proposed method is quite straightforward and effective,  it should be explained further.\n\n3. The figures in this paper could use some enhancement—except for Figure 4, which stands out nicely. The other figures are somewhat simplistic, and certain visuals, like Figure 2, lack clarity when paired with their captions.\n\n4. Although the writing is commendable, both the tables and figures require refinement. For instance, in Table 1, every similarity is bolded, which could lead to confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lAiubSxdUy", "reviewer": "Reviewer_Jqzc", "summary": "The manuscript introduces \"ClassDiffusion,\" a methodology for personalizing text-to-image generation. Leveraging a semantic preservation loss, this method facilitates rapid and efficient concept customization, and prevent the semantic drift during the fine-tuning process. The results demonstrate that the proposed work is competitive with leading frameworks in various image generation tasks.", "questions": "Given the concerns mentioned, particularly around the method's scalability to more complex multi-subject personalizations and the rationale behind the loss choices, I recommend a \"marginally below the acceptance threshold\" for this paper. Enhancements in demonstrating multi-subject capabilities, clarity in embedding visualization, and justification for the choice of technology could potentially elevate the manuscript to meet publication standards.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed work observes the reason why the existing methods suffer from the issue of compositional ability with experiments and theoretical analysis.\n\n2. The authors have developed a framework for generating personalized images that effectively ensure the compositional capabilities for concept personalization tuning.\n\n2. The paper provides experimental results, including both quantitative and qualitative assessments, showcasing the superior performance of the framework. The results clearly highlight the effectiveness of the proposed method in facilitating personalized image generation.", "weaknesses": "1. In Figure 5, the method utilizes the semantic preservation loss. However, it is unclear how it would perform with fine-grained subjects (two dogs or two cats with different breed). Clarification on whether the method can effectively manage such fine distinctions and a visual representation of joint embeddings for both similar and diverse subjects would be beneficial.\n\n2. Recent methodologies [1, 2] have demonstrated the capability to learn multi-concept personalization, it remains uncertain if the proposed work can handle multiple personalized instances, particularly for contexts involving up to five subjects. Although quantitative results for two subjects are provided in the paper, the absence of qualitative results for three or more subjects in both the main text and appendix is a notable omission. Including these results would substantiate the method's capability in more complex scenarios.\n\n[1] Liu, Zhiheng, et al. \"Cones 2: Customizable image synthesis with multiple subjects.\" arXiv preprint arXiv:2305.19327 (2023).\n\n[2] Yeh, Chun-Hsiao, et al. \"Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition.\" arXiv preprint arXiv:2402.15504 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JyhFjjnKFp", "reviewer": "Reviewer_FpQ7", "summary": "This paper addresses a critical issue in text-to-image personalization models: the loss of compositional ability after fine-tuning. The authors identify that after personalization tuning, models struggle to generate images that combine the personalized concept with other elements (e.g., a personalized dog wearing headphones). Through experimental and theoretical analysis, they attribute this to semantic drift of the personalized concept from its superclass. They propose ClassDiffusion, which introduces a semantic preservation loss to maintain alignment between personalized concepts and their superclass during fine-tuning.", "questions": "1 Include more detailed ablation studies to validate the design choices in the semantic preservation loss.\n\n2 Expand the evaluation section with more quantitative metrics specifically designed to measure compositional ability.\n\n3 Provide more extensive comparisons with other state-of-the-art methods that attempt to address similar issues. Include more diverse case studies beyond the dog-headphone example to demonstrate the generality of the approach.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The paper provides a thorough investigation of a significant problem in personalization models, supported by both empirical observations and theoretical analysis.\n\n2 The proposed solution is elegantly simple yet effective, making it practical for implementation.\n\n3 The experimental results demonstrate clear improvements over existing methods.", "weaknesses": "1 The paper lacks comprehensive ablation studies to demonstrate the impact of different components of the semantic preservation loss.\n\n2 While the authors mention extending their method to video generation, this aspect feels somewhat underdeveloped and could benefit from more detailed exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "88B6oizepq", "reviewer": "Reviewer_BPUN", "summary": "For text-to-image customization task, the authors suggest that all tuning-based models will inherently suffer from the over-fitting introduced by this process, which leads to weakening in the compositional ability of the model. They hypothesize that the decline in this compositional ability stems from the semantic drift of the target concept away from superclass target during fine-tuning. They analyze the experimental and theoretical aspects in turn, and introduce ClassDiffusion to address the issue of weakening compositional capacity after fine-tuning.", "questions": "1. The paper mentions that \"CLIP-T metric can hardly reflect the actual performance of personalized generation\" but CLIP is still used to calculate Loss (Equation 10). The rationale for using CLIP for loss calculation, despite its limitations for evaluation, should be explained.\n\n2. The paper mentions \"higher SPL weights preserve combining ability better\" and \"lower SPL weights restore specific concept features more effectively\", why is the weight of SPL set to 1 in dataset A. Is this the optimal value?  An ablation study or justification should be provided for the choice of 1 as the SPL weight, and how they balanced the trade-off between preserving combining ability and restoring specific concept features should be discussed.\n\n3. The writing is somewhat redundant. For example, it is not clear how significant Figure 2 is. The Figure 1 is already shows the performance of proposed method.\n\n4. The paper mentions that \"we found that the CLIP-T metric can hardly reflect the actual performance of personalized generation\", but does not clearly explain how this is found. Whether it is based on references to other works is very confusing. Evidence or references should be provided to support their claim about the limitations of the CLIP-T metric for evaluating personalized generation. This would strengthen the justification for introducing a new metric.\n\n5. The paper mentions \"we introduce the BLIP2-T metric\", is this metric proposed by the authors? It should be clarified whether this is a novel metric proposed by the authors or whether it's based on existing work. If it's novel, more details on its development and validation are requested.\n\n6. Is Figure 3a an experimental result or a schematic diagram? If it is a schematic diagram rigorous notation is recommended.  More detailed captions or annotations should be provided to improve clarity.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors explain what they work do with a vivid example, which is easy for readers to understand.\n2. The authors theoretically analyze the sampling difficulty based on the joint conditional probability distribution.\n3. The paper is written logically and clearly, from experimental phenomenon, theoretical proof, and solution method respectively.\n4. The authors extend the proposed method to the video field.", "weaknesses": "1. The paper mentions that “CLIP-T metric can hardly reflect the actual performance of personalized generation” but CLIP is still used to calculate Loss (Equation 10).\n2. The paper mentions “higher SPL weights preserve combining ability better” and “lower SPL weights restore specific concept features more effectively”, why is the weight of SPL set to 1 in dataset A. Is this the optimal value? It needs to be further discussed.\n3. The writing is somewhat redundant. For example, it is not clear how significant Figure 2 is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)