ClassDiffusion: More Aligned Personalization Tuning with Explicit Class Guidance

## Abstract

Abstract Recent text-to-image customization works have been proven successful in generating images of given concepts by fine-tuning the diffusion models on a few examples.
However, these methods tend to overfit the concepts, resulting in failure to create the concept under multiple conditions ( e.g. , headphone is missing when generating “a <sks> dog wearing a headphone”).
Interestingly, we notice that the base model before fine-tuning exhibits the capability to compose the base concept with other elements ( e.g. , “a dog wearing a headphone”), implying that the compositional ability only disappears after personalization tuning.
Inspired by this observation, we present ClassDiffusion,
a simple technique that leverages a semantic preservation loss to explicitly regulate the concept space when learning the new concept. Despite its simplicity, this helps avoid semantic drift when fine-tuning on the target concepts. Extensive qualitative and quantitative experiments demonstrate that the use of semantic preservation loss effectively improves the compositional abilities of the fine-tune models. In response to the ineffective evaluation of CLIP-T metrics, we introduce BLIP2-T metric, a more equitable and effective evaluation metric for this particular domain. We also provide in-depth empirical study and theoretical analysis to better understand the role of the proposed loss. Lastly, we also extend our ClassDiffusion to personalized video generation, demonstrating its flexibility. Our project page are https://classdiffusion.github.io/ .

## 1 Introduction

Thanks to the rapid progress in the diffusion model [, , , , , , , , , ], the field of text-to-image generation has achieved significant progress in recent years. Leading text-to-image models [, , , , , , ] have been successful in generating high-fidelity images that align well with textual inputs. Recently, a significant portion of research  [, , , , , , , , , , , ] has transitioned their focus from creating high-quality images to enhancing control over the generated images. Among these works, an important and widely explored research domain is subject-driven personalized generation, which aims to generate new images for specific concept given some reference images of that concept.

Existing personalization methods  [, , , , , , , , , , , , ] can generate images that closely resemble the concept by fine-tuning the base text-to-image model in a specific image set. However, this process leads to a weakening in the model’s compositional ability. For example, when generating “a <sks> dog wearing a headphone”, though the given dog is well reconstructed, the headphone is always missing (Fig. 1). A commonly accepted explanation within the community [, , , ] attributes this phenomenon to overfitting given limited number of images. Nonetheless, the fundamental cause of this overfitting remains unexplored. In this work, we aim to investigate the underlying causes behind the overfitting.

Figure: Figure 1: The base Stable Diffusion (SD) possesses the capbility to compose the concept of a dog and headphone, generating a dog wearing a headphone. However, we notice that this compositional generation capability is lost during personalization tuning. For example, when using CustomDiffusion (CD) [], the headphone is missing despite the target corgi is generated successfully. On the other hand, our method can successfully compose the target corgi with headphone.
Refer to caption: x1.png

Upon initial examination, it appears that the model diminishes some of its original capabilities after personalization tuning.
Taking Stable Diffusion (SD) [] as an example, from Fig. 1, we notice that the base SD model actually has acquired the skill to compose the concept of dog and a headphone.
Nevertheless, after fine-tuning, it struggles to achieve similar levels of compositional generation (e.g., headphone is missing although the target concept <sks> dog can be generated successfully).
We hypothesized that the decline in this compositional ability stems from the semantic drift of the target concept away from its superclass during fine-tuning.
To better understand this, we conduct some empirical analysis by visualizing the text space and cross-attention layers in Fig. 2, 2.
In addition, we also perform theoretical analysis and find that the root cause lies in the semantic bias that reduces the entropy of the probability of the composed conditions, which significantly increases the difficulty to simultaneously sample the target concept combined with other elements.

Based on our experimental findings and theoretical analysis, we introduce ClassDiffusion to tackle the issue of weakening compositional capacity post-fine-tuning. This method uses semantic preservation loss to explicitly guide the model to restore the semantic imbalance that arises during the fine-tuning stage. In particular, it narrows the gap between the text embeddings of the target concept and its respective superclass in the textual space. Despite its simplicity, the proposed loss can successfully recover the compositional ability as shown in Fig. 1.
Extensive experiments have demonstrated the effectiveness of our method to restore the base model’s compositional generation capability. Furthermore, we explore the potential of our approach in personalized video synthesis, showcasing its flexibility. In addition, we found that CLIP-T metric can hardly reflect the actual performance of personalized generation. We therefore introduce BLIP2-T metric, a more equitable and effective evaluation metric for this particular domain. To summarize, the contributions of our work are:

- •
We offer a thorough examination to understand why existing tuning-based subject-driven personalization methods suffer from the loss of compositional ability. This is elucidated through both experimental observations and theoretical analysis.
- •
We propose ClassDiffusion, a simple technique to recover the compositional capabilities lost during personalization tuning by adding a simple semantic preservation loss.
- •
We proposed BLIP2-T Score, a fairer, better-performing evaluation metric for evaluating image-text alignment.

## 2 Related Work

#### Text-to-Image Generation and Its Control

Text-to-image generation is designed to generate high-quality, high-fidelity images that are aligned with textual prompts. This field has been under research for an extended period. Recently, the field of Text-to-image generation has made significant progress with extensive research in Generative Adversarial Network (GAN)  [, , , ], Variational Autoencoder (VAE)  [, , , ], and Diffusion models  [, , , , , , , , , ]. The diffusion models achieve a new state-of-the-art (SOTA) in unconditional image generation. Numerous works [, , , , , , ] have been done to make the image generated by diffusion models more aligned with the textual prompts.
Among them, Stable Diffusion  [] is a widely recognized model in the field, utilizes a cross-attention mechanism to integrate textual conditions into the image generation process and employs the Latent Diffusion Model, which maps the image to latent space []. Our research is based on the Stable Diffusion framework due to its adaptability and wide use in the community. Furthermore, different methods exist for controlling generative models. The primary categories for controlling generative models typically include Text-guided  [, , , , ], Image-guided  [, , , , , , ], Additional Sparse conditions [, , , , , , , , , , , , ], Brain-guided  [, , , , , , ], Sound-Guided [, ], and some universe control [, , ].
Text-guided control utilizes textual descriptions to directly influence the outcome, guiding the model based on specific verbal instructions.
Our method focused on the text-guided controllable generative model.

#### Subject-Driven Personalized Generation

Subject-driven personalized generation is focused on creating images based on reference images. Recent works [, , , , , , , , , , , , , , , ] have explored techniques for producing striking resemblance images in multiple ways. One of the primary ways is to fine-tune the base text-to-image models.
Furthermore, there has been a significant effort in research  [, , , , , , , , ] aimed at integrating various concepts in personalization. While producing striking resemblance images, fine-tuning the base model on a small set of images leads to overfitting, resulting in unexpected issues. One prevalent issue discussed in prior research is the decrease in diversity. Recent studies have proposed various methods to address these issues. For instance, DreamBooth  [] introduced the Class-Specific Priori Loss, which effectively addresses diversity reduction by recovering the class’s prior knowledge.
Another common issue is the inability to generate images under multiple conditions. Some Recent Research  [, , , , ] proposed some methods to migrate this appearance.
However, the underlying reasons for this phenomenon have not been thoroughly investigated. Our research endeavors to explore these reasons and develop solutions to overcome this challenge.

Figure: (a)
Refer to caption: x2.png

Figure: Figure 3: During the personalization tuning process, as the distribution of dogs shrinks, the conditional distribution of dogs and headphones also shrinks. This gradually increases the difficulty to sample in this distribution, leading to the weakening of the compositional generation capability. Our ClassDiffusion mitigates this by incorporating semantic preservation loss (SPL) to minimize the semantic drift of the personalized concept from its superclass.
Refer to caption: x4.png

## 3 Method

### 3.1 Preliminary

#### Text-to-Image Diffusion Model

Stable Diffusion is used by almost all of the methods in the
area of subject-driven personalization generation, which is developed from the Latent Diffusion Model (LDM)  []. For any input image, Stable Diffusion first transforms it into a latent representation $x$ using the encoder $\varepsilon$ of a Variational Autoencoder  []. The diffusion process then operates on $x$ by incrementally introducing noise, resulting in a fixed-length Markov Chain represented as $x_{1},x_{2},...,x_{T}$, where $T$ is the chain’s length. Stable Diffusion utilizes a UNet architecture to learn the reverse of this diffusion process, predicting a denoised version of the latent input $x_{t}$ at each timestep $t$ from 1 to $T$. In the context of text-to-image generation, the text prompts’ conditioning information $y$ is encoded into an intermediate representation $\tau_{\theta}(y)=c$, where $\tau_{\theta}$ is a pre-trained CLIP  [] text encoder. The primary objective in training this text-to-image diffusion model involves optimizing this transformation and prediction process, and it can be expressed as:

$$ $\mathcal{L}_{recon}=\mathbb{E}_{x,y,\mathbf{\epsilon},t}\left[\left\|\epsilon- \epsilon_{\theta}(x_{t},t,\tau_{\theta}(y))\right\|^{2}_{2}\right]$ (1) $$

where $\epsilon$ and $\epsilon_{\theta}$ represent the noise samples from the standard Gaussian noise $\epsilon\sim\mathcal{N}(0,1)$ and predicted noise residual, respectively.

### 3.2 Experimental Analysis

We begin by conducting some empirical analysis to investigate the reason behind the lost of compositional ability after personalization tuning.

Semantic drift in CLIP text-space.
We first examine the CLIP text sample space before and after fine-tuning. We visualize the CLIP text space in Fig. 2 and notice that customized concepts are more distant from the class center. Specifically, among the 71 phrases composing the super-category (“dog” in this case) with different adjectives (e.g., "a cute dog", "a tall dog" etc.),
we observe a significant increase in distance of the target concept from its corresponding superclass.
This indicates that the semantics of the personalized concepts have changed during fine-tuning, which potentially leads to an inability to access the knowledge associated with the super-categories learned during pre-training.
More details can be found in Appendix D,  E.

Reduction of cross-attention strength.
We further investigate the model by visualizing the cross-attention layers in Fig. 2.
It shows that, while the "dog" token activates the relevant region in the image, its activation level is notably lower than that of the pre-trained model. These findings align with our observations in the CLIP text space and provide support for our hypothesis regarding this discrepancy.
Next, we theoretically analyze why the semantic drift of personalized phrases results in the weakening of the compositional ability from the respective of reduction in the entropy of composable conditional probability.

### 3.3 Theoretical Analysis

Drawing on the insights of  [], a trained diffusion model can be seen as implicitly defining an Energy-Based Model (EBM)  []. This perspective allows us to build on prior research in composing EBMs and adapting them for use in diffusion models. Building on the work of  [] in the context of generating images with multiple attributes and Bayes’ theorem, the conditional probability can be decomposed as:

$$ $\displaystyle p(x|c_{class},c_{1},c_{2},\dots,c_{i})\propto p(x,c_{class},c_{1 },c_{2},...,c_{3})$ $\displaystyle=p(c_{class}|x)p(x)\prod_{i\in T}p(c_{i}|x)$ (2) $\displaystyle=p(c_{class}|x)p(x)\prod_{i\in T}\frac{p(c_{i})p(x|c_{i})}{p(x)}$ (3) $$

Figure: Figure 4: Overview of our proposed ClassDiffusion. Our semantic preservation loss (SPL) is calculated by measuring the cosine distance between text features extracted from the same text transformer (using EOS tokens as text features following CLIP) for phrases with personalized tokens and phrases with only superclasses.
Refer to caption: x5.png

where $T$ is all the set of conditions in prompts except for the class, $p(c_{i})$ represents the probability of occurrence of condition $c_{i}$ in the training dataset and can be regarded as a constant for large-scale pre-training models. $p(c_{i}|x)$ represents an implicit classifier, denoting the probability of categorizing a concept as $c_{class}$. Specifically, $p(c_{class}|x)$ represents a specific implicit classifier for the super-category. Thus, we have:

$$ $\displaystyle p(x|c_{1},c_{2},\dots,c_{i})\propto p(c_{class}|x)p(x)\prod_{i \in T}\frac{p(c_{i})p(x|c_{i})}{p(x)}$ (4) $$

Denoted $p(x)\prod_{i\in T}\frac{p(c_{i})p(x|c_{i})}{p(x)}$ as $d(x)$, $p(c_{class}|x)$ as $q(x)$, and $p(x|c_{1},c_{2},\dots,c_{i})$ as $a(x)$. The entropy of $a$ is calculated as:

$$ $\mathrm{H}(a)=-\sum_{x}q(x)d(x)[\log(q(x))+\log(d(x))]$ (5) $$

After fine-tuning, The components of $d(x)$ change only slightly and can be treated as unchanged, and the implicit classifier $p_{\theta}(c_{class}|x)$ changes to $p_{\theta^{{}^{\prime}}}(c_{class}|x)$, Thus the difference in entropy before and after can be expressed as:

$$ $\displaystyle\bigtriangleup\mathrm{H}$ $\displaystyle=\sum_{x}q_{\theta}(x)d(x)\left[\log(q_{\theta}(x))+\log(d(x))\right]$ (6) $\displaystyle\quad-\sum_{x}q_{\theta^{{}^{\prime}}}(x)d(x)\left[\log(q_{\theta ^{{}^{\prime}}}(x))+\log(d(x))\right]$ $\displaystyle=d(x)\sum_{x}\{\left[q_{\theta}(x)\log q_{\theta}(x)-q_{\theta^{{ }^{\prime}}}(x)\log q_{\theta^{{}^{\prime}}}(x)\right]$ (7) $\displaystyle\quad+\log d(x)\left[q_{\theta}(x)-q_{\theta^{{}^{\prime}}}(x) \right]\}$ $$

Based on our observations in Fig. 2, 2 we can show that $q_{\theta}(x)>q_{\theta^{{}^{\prime}}}(x)$, combining the properties of probability theory and the monotonically decreasing nature of $x\log x$ at (0,1), we have:

$$ $\displaystyle q_{\theta}(x)\log q_{\theta}(x)-q_{\theta^{{}^{\prime}}}(x)\log q _{\theta^{{}^{\prime}}}(x)<0;\log d(x)<0$ (8) $$

Thus, we have:

$$ $\displaystyle\bigtriangleup\mathrm{H}(a)<0$ (9) $$

As a result, it is more difficult to sample from our demanded conditional distributions under $c_{\text{class}},c_{1},...,c_{i}$ conditions than before the fine-tuning, leading to the phenomenon that the combining ability is weakened after the fine-tuning. We will discuss the theoretical reasons here in more detail in Appendix C.
Fig.  3 illustrates the changes in the distribution during this process that lead to a weakening of the compositional generation capability.

Inspired by the empirical study and theoretical analysis above, in the next section, we present our proposed semantic preservation loss for mitigating the semantic drift occurred during fine-tuning.

### 3.4 Semantic Preservation Loss

Figure: Figure 5: Qualitative comparison between our method and baselines with single given concept. Our method generates images that align with the prompts, surpassing all baselines.
Refer to caption: extracted/5624185/singleconcept.png

Recent personalization techniques are based on tuning and creating a link between reference images and a unique token denoted as <new1>. During the tuning process, the embeddings associated with this token are fine-tuned to align with the specific concept being targeted. When dealing with multiple concepts during training, a total of $N$ special tokens are utilized, with each one designated to represent a distinct concept.
Our method is designed to reduce the semantic gap between the customized concepts and he center of the super-categories distribution in text space. Fig.  4 shows an overall pipeline of our method.
By labeling the phrase containing a special token as $P_{SC}$ and the phrase containing only a class token as $P_{C}$, using $\mathbf{E}_{SC}$ and $\mathbf{E}_{C}$ to represent the embeddings outputs by the CLIP encoder of the personalized phrases and the class phrases respectively, the semantic preservation loss can be expressed by the following equation,

$$ $\mathcal{L}_{cls}=\mathsf{\sum^{\mathit{N}}\sum^{\mathit{b}}\sum^{\mathit{L}}} D_{c}(\mathbf{E}_{SC},\mathbf{E}_{C})$ (10) $$

where $b$ represents for the batch size, $L$ denotes the length of embeddings for CLIP. $D_{c}$ implies the cosine distance. Therefore, the final optimization objective is:

$$ $\mathcal{L}=\mathcal{L}_{recon}+\lambda\mathcal{L}_{cls}$ (11) $$

where $\lambda$ controls the weight of semantic preservation loss.

## 4 Experiments

Figure: Figure 6: Qualitative comparison between our method and Custom Diffusion(CD) in multiple concepts. Our method has better text similarity than custom diffusion.
Refer to caption: extracted/5624185/multiple.png

### 4.1 Experiment Details

Implementation details Our method is built on Stable Diffusion V1.5, with a learning rate $10^{-6}$, and batch size 2 for fine-tuning. We use 500 optimization steps for single concept and 800 for multiple concepts, respectively. During inference, the guidance scale is set to 6.0, and inference steps are set to 100.
All experiments are conducted on 2$\times$RTX4090 GPUs. Our method uses $\sim$ 6 min for single concept generation and $\sim$ 11 min for multiple concept generation.

Baselines
We compare our method with state-of-the-art (SOTA) competitors including DreamBooth [], Textual Inversion [], Custom Diffusion [], NeTI [], SVDiff [].
For DreamBooth, CustomDiffusion, and Textual Inversion, we used the diffusers [] version of the implementation. For NeTI, we use its official implementation. Given SVDiff doesn’t have an official open-source repository. For SVDiff, we use the implementation of  []. All training parameters follow the recommendations of the official paper.
To ensure the fairness of comparison, all of these baselines are built on Stable Diffusion V1.5.

Datasets
Following previous work [, , ], we conduct quantitative experiments on DreamBooth Dataset. It contains 30 objects including both live objects and non-live objects.
Also, we use images from Textual Inversion Dataset and CustomConcept101 in qualitative experiments.

Figure: Figure 7: Qualitative result of a small story with given concepts.
Refer to caption: x6.png

Evaluation metrics
We assess our approach using three metrics: CLIP-I, CLIP-T, and DINO-I. CLIP-I calculates the visual similarity between the produced images and the target concept images by utilizing CLIP visual features. CLIP-T evaluates the similarity between text prompts and images. If one baseline contains the special token S*, it will be replaced with a prior class word. In the case of DINO-I, we evaluate the cosine similarity between the ViT-S/16 DINO [] embeddings of the generated images and the concept images.
Further, we note the impact of CLIP’s outdated performance on the fairness of the evaluation. Therefore we introduce BLIP2-T Score, which calculates the similarity between text features extracted from BLIP2’s Q-former and image features extracted from Vision Encoder as a score.
Our approach involved utilizing the Transformer [] implementation and the fine-tuned weights of BLIP2-IMT on CoCo [], with ViT/L []. This new metric aims to offer a more equitable and efficient evaluation measure for future studies in this field. Empirical findings from various studies [, , , , , ] indicate that BLIP2 outperforms CLIP significantly in the assessment of text-image alignment.

### 4.2 Qualitative & Quantitative Experiments

**Table 1: Quantitative Results on all metrics and Results of the User Study**
|  | Method | CLIP-T$\uparrow$ | CLIP-I$\uparrow$ | DINO-I$\uparrow$ | BLIP2-T $\uparrow$ | Text<br>Similarity<br>$\uparrow$ | Image<br>Similarity<br>$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Single<br>Concept | DreamBooth [] | 0.249 | 0.855 | 0.700 | 0.295 | 95.4% | 42.1% |
| Textual Inversion [] | 0.242 | 0.825 | 0.631 | 0.308 | 95.1% | 75.0% |  |
| Custom Diffusion [] | 0.286 | 0.837 | 0.693 | 0.416 | 79.1% | 40.0% |  |
| NeTI [] | 0.290 | 0.838 | 0.648 | 0.329 | 78.8% | 70.0% |  |
| SVDiff [] | 0.293 | 0.834 | 0.606 | 0.418 | 56.6% | 95.8% |  |
| \cdashline2-8 | Our | 0.300 | 0.828 | 0.673 | 0.460 | - | - |
| Multiple<br>Concept | Custom Diffusion [] | 0.282 | 0.813 | 0.636 | 0.380 | - | - |
| \cdashline2-8 | Our | 0.320 | 0.821 | 0.604 | 0.477 | - | - |

Qualitative Experiments We compare our method with DreamBooth, Textual Inversion, NeTI, SVDiff, and Custom Diffusion on challenging prompts. The results depicted in Fig. 5 demonstrate the outcomes obtained from these prompts. Fig. 7 shows a story of a given dog and sunglasses. The experimental findings indicate a substantial superiority of our approach over other techniques regarding alignment with text prompts, without any decline in similarity to the specified concept.
More qualitative results are shown in Appendix. G. Also, we conduct qualitative experiments with multiple concepts on the combinations <cat>, <sunglasses>; <bear>,<barn>; <dog>,<backpack>; Fig. 6 shows the results of the experiments. The experiments show that our method can be aligned with prompts better than custom diffusion in multi-concept generation.

Quantitative Experiments Following the previous work, we used 20 concepts for quantitative experiments. For Single Concept text similarity metrics (CLIP-T, BLIP2-T), we followed the 25 prompts used in  [], sampling 20 images per prompt. The results of the experiment are shown in Tab. 1. Experimental results show that our method obtains new SOTA on each text similarity metric, indicating that we have good compositional generation capability.

Figure: Figure 8: Result of generated videos, showing good textual alignment and similarity of given concepts.
Refer to caption: x7.png

Figure: Figure 9: Generation results for the prompt “a photo of a dog wearing a headphone” with different step counts and SPL weights. All results are generated using the same random seed.
Refer to caption: x8.png

### 4.3 Additional Experiments

User Study We also performed user study to validate the effectiveness of our method. We used the same set of images generated in the Section 1 for user study, details of which are available in Appendix. F. The results of the user study are located in Tab. 1. The result of the user study shows that our method outperforms all methods in text similarity. Although our method does not outperform all methods in image similarity, given the high CLIP-I scores, our method still produces images that are highly consistent with the given concept.

Personalized Video Generation We investigate the implementation of our method in personalized video generation. We utilized AnimateDiff V2 [, ] for video generation, configuring parameters to a resolution of 512$\times$512, a guidance scale of 7.5, and 25 inference steps. The outcomes of the video generation process are illustrated in Fig. 8. Utilizing AnimateDiff, our technique produces videos that exhibit strong textual and conceptual coherence without the need for additional training. This demonstrates that our approach, which aligns personalized phrases with superclass-centric semantics, can generate engaging videos with dynamic generation capabilities stemming from pre-training, along with the ability to transition across corresponding domains.

Abalation Experiments We studied the influence of different weights of semantic preservation loss (SPL). The results show that higher SPL weights preserve combining ability better. At s SPL weight of 100, all steps successfully depict "wearing a headphone." However, at lower weights of 0.1 and 0.01, the headphone details diminish by 750 steps and disappear by 1500 steps. On the other hand, lower SPL weights restore specific concept features more effectively.

## 5 Conclusions

In this work, we highlight the problem of weakened compositional ability due to individualized fine-tuning and provide an analysis of the causes of this problem from experimental observations and information-theoretic perspectives. We then introduce a new approach, termed ClassDiffusion, which mitigates the weakening of compositional ability by restoring the original semantic space. Finally, we present comprehensive experimental results showcasing the efficacy of ClassDiffusion and the fresh perspectives it offers on interconnected fields.

## Appendix A Visualization after Fine Tunning

In section 3, we present visualization results for the text space and the cross-attention layer of other methods, highlighting the semantic bias that emerges in the text space, leading to a decline in compositional ability. To further affirm the effectiveness of our method and our hypothesis, we also visualize the textual feature space and the cross-attention layer with our method in this section. The visualization results are depicted in Fig. 10. The ranking of the distance is 36 out of 71, as opposed to 26 out of 71 before fine-tuning and 67 out of 71 for other methods. A comparison with the visualizations in Fig. 2 and Fig. 2 reveals that our model effectively addresses the semantic drift in the text space.

Figure: Figure 10: Visualization results after fine-tuning of our approach.
Refer to caption: x9.png

## Appendix B Fine-grained experiments

At the core of our approach, we want the semantics of personalized phrases to be closer to the category-centered words. In this section, we explore the effect of different category-centered words on the results for the same given concept. Fig. 11 shows the training results using different category-centered words. The results show that the use of different center words leads to significant differences in performance.

Figure: Figure 11: Visualization of generating “a <center word> sitting on the chair.”. This <center word> is used in both prompt and class token.
Refer to caption: x10.png

## Appendix C Entropy reduction during the fine-tuning

In section 3.3, we provide a solid theoretical analysis for the weakening of compositional ability due to the semantic drift from the perspective of information theory and probability distributions. In this section, we will discuss it in a more detailed way.

In the field of subject-driven personalization generation, two manifest phenomena are caused by overfitting: weakening of diversity in classes of given concepts and weakening of compositional ability. In addition to the calculations mentioned in the main body of the text, the entropy of combined conditional probability can also be calculated as conditional entropy:

$$ $\mathrm{H}(X|c_{1},c_{2},\dots,c_{i})$ (1) $$

Make the given concept into a series of specific conditions: $c_{s1},\dots,c_{si}$, each condition describes one of the features of the given concept. The entropy after fine-tuning will be:

$$ $\mathrm{H}(X|c_{1},c_{2},\dots,c_{n},c_{s1},\dots,c_{si})=\mathrm{H}(X|c_{1} \dots,c_{n})-I(X|c_{1}\dots,c_{n};c_{s1},\dots,c_{si})$ (2) $$

Where $I$ represent the mutual information, According to  [], we have:

$$ $\displaystyle I(X|c_{1}\dots,c_{n};c_{s1},\dots,c_{si})$ $\displaystyle\geq 0$ (3) $\displaystyle\mathrm{H}(X|c_{1},c_{2},\dots,c_{n},c_{s1},\dots,c_{si})$ $\displaystyle<\mathrm{H}(X|c_{1},c_{2},\dots,c_{i})$ (4) $$

However, the entropy reduction here is different from the entropy reduction in the main text. The entropy reduction here leads to a reduction in the diversity of the generated images. Specifically, when the cue word is “a photo of a dog”, the image generated is closer to the given concept than to the diversity of dogs.

## Appendix D Prompt used in the visualization of CLIP text sample space

In this section, we provide a realistic visualization of the schematic in Fig. 2, and discuss the prompt to generate the 70 phrases that include adjectives and super-categories “dog”, and the whole 71 adjectives.

The realistic visualization of the schematic is:

Figure: Figure 12: Visualization of the CLIP sample space. Using ChatGPT, we created 70 phrases containing adjectives related to the superclass of dogs. Subsequently, text features derived from these phrases were processed through the CLIP text encoder, downscaled, and their distance from the central point of the superclass (representing a dog image) was calculated. The comparison between the pre-trained model (illustrated in the left figure) and the fine-tuned model (depicted in the right figure) indicates that in the pre-trained model, phrases with special tokens ranked 26 out of 71, while in the fine-tuned model, they ranked 67 out of 71. Moreover, it is evident that phrases containing special tokens are situated further away from the central point of the superclass.
Refer to caption: x11.png

The prompt we use is:

*Please help me generate some adjectives that can describe an attribute of a dog in a photo.*

The adjectives we use are:

| beautiful | happy | sad | tall | short |
| --- | --- | --- | --- | --- |
| bright | dark | big | small | young |
| old | fast | slow | warm | cold |
| soft | hard | heavy | light | strong |
| weak | good | bad | rich | poor |
| thick | thin | expensive | cheap | quiet |
| loud | clean | dirty | smart | stupid |
| interesting | boring | new | old-fashioned | safe |
| dangerous | healthy | sick | easy | difficult |
| right | wrong | high | low | near |
| far | early | late | wet | dry |
| busy | free | careful | careless | friendly |
| unfriendly | important | unimportant | famous | unknown |
| excited | calm | serious | funny | tired |
| energetic |  |  |  |  |

## Appendix E 2D Text Space Distance Calculation

In this section, we discuss how we visualized CLIP’s text sample space in Fig. 2. First, we use the adjectives generated which are described in Section. D as the initialize set $S$, and use the following pseudocode to get a 2D point set $T$:

Figure: Algorithm 1 Algorithm to Convert Character Set to 2D Point Set

## Appendix F Details of User Study

We offer users a comprehensive user study guide that includes user selection criteria which is shown in Fig. 13. Additionally, to maintain fairness, we positioned our method alongside the baseline method randomly to prevent users from showing bias towards either method. Our percentages in Tab. 1 are obtained by calculating the number that chose our model better as a percentage of the overall number that made a preference.

Figure: Figure 13: User Study Guide, which describes the user selection criteria and provides an example for reference.
Refer to caption: extracted/5624185/user_study.png

## Appendix G More Qualitative Result

In Fig. 5, one generated image is provided for each prompt. Fig. 14 presents more images generated from the same prompts, thereby reinforcing the efficacy of our approach.

Figure: Figure 14: Qualitative result of same prompts the main text.
Refer to caption: extracted/5624185/singleconcept_appen.png

## Appendix H Limitation

In this section, we discuss the limitations of our work. Our work are able to generate images that are aligned with the given prompt while keeping the features of the given concept. However, there are two major limitations in our work:

- •
Considering that reconstruction of the human face is fine-grained, and the phrase “a photo of a human” or “a photo of a human face” can not include extensive information about humans. Whether our work can transfer to human-driven personalized generation remains explored.
- •
For objects that have a combination of categories, choosing an appropriate center word requires some experimentation.

## Appendix I Social Impact

The advancements in text-to-image customization through fine-tuning diffusion models, as evidenced by our work on ClassDiffusion, have significant social implications. By enhancing the compositional capabilities of these models, our approach can contribute to a variety of fields, including digital content creation, and education. In the realm of digital content creation, ClassDiffusion enables artists, designers, and marketers to generate more precise and complex images based on textual descriptions. This improvement reduces the time and effort required to produce customized visual content, fostering creativity and innovation. It also allows for the seamless incorporation of personalized elements into digital artworks, advertising materials, and user-generated content, thereby enhancing user engagement and satisfaction. ClassDiffusion can also be a powerful tool in educational settings. Educators can use this technology to create illustrative materials that are tailored to specific learning objectives. For instance, teachers could generate images that accurately depict historical events, scientific concepts, or literary scenes, making learning more interactive and engaging for students. Furthermore, this technology can aid in the development of educational content for diverse learning needs, including materials for students with disabilities.

While the advancements in text-to-image generation hold promise, it is essential to address the ethical considerations associated with their use. Ensuring that these models are free from biases and do not perpetuate harmful stereotypes is crucial. Our work on ClassDiffusion includes measures to mitigate semantic drift, which helps maintain the integrity and accuracy of generated content. Continuous evaluation and updates are necessary to uphold these standards and ensure the technology benefits society as a whole.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 8
presentation: 3
questions: see weaknesses
soundness: 3
strengths: 1. The authors unveil the intriguing underlying cause behind the overfitting of SD. Rather than a typical case of overfitting, they suggest it’s more about semantic drift—a perspective I find quite compelling, though I still believe overfitting truly occurs.

2. This paper is not only well-written but also flows smoothly, making it a pleasure to read.

3. The inclusion of additional relevant baselines demonstrates the authors' thorough research and deep engagement with the topic.

4. The authors provide a wealth of comprehensive experimental results, reinforcing the strength of their proposed ideas.
summary: This work delves into the captivating realm of text-to-image customization. However, current tuning-based methods often struggle with a common pitfall: they tend to overfit to the limited concepts presented in their training datasets. This overfitting can significantly diminish the model's compositional abilities. The authors highlight that this decline is mainly due to the semantic shifts that occur with customized concepts during the fine-tuning process.  To address this issue, the authors introduce an approach called ClassDiffusion. This technique revitalizes the model's compositional strength by restoring its connection to the original semantic space. Through a series of thorough experiments, the authors demonstrate the effectiveness of ClassDiffusion and the fresh insights it brings to interconnected fields. The results not only showcase its prowess but also pave the way for exciting advancements in the domain.
weaknesses: 1. I agree that semantic drift occurs during fine-tuning. This drift arises because the model is an intricate network filled with numerous parameters. When we fine-tune with only few samples, the adjustments to these parameters are minimal, yet the model still retains its generative capabilities. In my view, semantic loss may struggle to restore the original semantic space, as it doesn’t preserve the nuances of the drift encountered. Theoretically, utilizing the entire training text data could mitigate this issue.

2. While I think that the proposed method is quite straightforward and effective,  it should be explained further.

3. The figures in this paper could use some enhancement—except for Figure 4, which stands out nicely. The other figures are somewhat simplistic, and certain visuals, like Figure 2, lack clarity when paired with their captions.

4. Although the writing is commendable, both the tables and figures require refinement. For instance, in Table 1, every similarity is bolded, which could lead to confusion.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Given the concerns mentioned, particularly around the method's scalability to more complex multi-subject personalizations and the rationale behind the loss choices, I recommend a "marginally below the acceptance threshold" for this paper. Enhancements in demonstrating multi-subject capabilities, clarity in embedding visualization, and justification for the choice of technology could potentially elevate the manuscript to meet publication standards.
soundness: 3
strengths: 1. The proposed work observes the reason why the existing methods suffer from the issue of compositional ability with experiments and theoretical analysis.

2. The authors have developed a framework for generating personalized images that effectively ensure the compositional capabilities for concept personalization tuning.

2. The paper provides experimental results, including both quantitative and qualitative assessments, showcasing the superior performance of the framework. The results clearly highlight the effectiveness of the proposed method in facilitating personalized image generation.
summary: The manuscript introduces "ClassDiffusion," a methodology for personalizing text-to-image generation. Leveraging a semantic preservation loss, this method facilitates rapid and efficient concept customization, and prevent the semantic drift during the fine-tuning process. The results demonstrate that the proposed work is competitive with leading frameworks in various image generation tasks.
weaknesses: 1. In Figure 5, the method utilizes the semantic preservation loss. However, it is unclear how it would perform with fine-grained subjects (two dogs or two cats with different breed). Clarification on whether the method can effectively manage such fine distinctions and a visual representation of joint embeddings for both similar and diverse subjects would be beneficial.

2. Recent methodologies [1, 2] have demonstrated the capability to learn multi-concept personalization, it remains uncertain if the proposed work can handle multiple personalized instances, particularly for contexts involving up to five subjects. Although quantitative results for two subjects are provided in the paper, the absence of qualitative results for three or more subjects in both the main text and appendix is a notable omission. Including these results would substantiate the method's capability in more complex scenarios.

[1] Liu, Zhiheng, et al. "Cones 2: Customizable image synthesis with multiple subjects." arXiv preprint arXiv:2305.19327 (2023).

[2] Yeh, Chun-Hsiao, et al. "Gen4Gen: Generative Data Pipeline for Generative Multi-Concept Composition." arXiv preprint arXiv:2402.15504 (2024).

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1 Include more detailed ablation studies to validate the design choices in the semantic preservation loss.

2 Expand the evaluation section with more quantitative metrics specifically designed to measure compositional ability.

3 Provide more extensive comparisons with other state-of-the-art methods that attempt to address similar issues. Include more diverse case studies beyond the dog-headphone example to demonstrate the generality of the approach.
soundness: 3
strengths: 1 The paper provides a thorough investigation of a significant problem in personalization models, supported by both empirical observations and theoretical analysis.

2 The proposed solution is elegantly simple yet effective, making it practical for implementation.

3 The experimental results demonstrate clear improvements over existing methods.
summary: This paper addresses a critical issue in text-to-image personalization models: the loss of compositional ability after fine-tuning. The authors identify that after personalization tuning, models struggle to generate images that combine the personalized concept with other elements (e.g., a personalized dog wearing headphones). Through experimental and theoretical analysis, they attribute this to semantic drift of the personalized concept from its superclass. They propose ClassDiffusion, which introduces a semantic preservation loss to maintain alignment between personalized concepts and their superclass during fine-tuning.
weaknesses: 1 The paper lacks comprehensive ablation studies to demonstrate the impact of different components of the semantic preservation loss.

2 While the authors mention extending their method to video generation, this aspect feels somewhat underdeveloped and could benefit from more detailed exploration.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The paper mentions that "CLIP-T metric can hardly reflect the actual performance of personalized generation" but CLIP is still used to calculate Loss (Equation 10). The rationale for using CLIP for loss calculation, despite its limitations for evaluation, should be explained.

2. The paper mentions "higher SPL weights preserve combining ability better" and "lower SPL weights restore specific concept features more effectively", why is the weight of SPL set to 1 in dataset A. Is this the optimal value?  An ablation study or justification should be provided for the choice of 1 as the SPL weight, and how they balanced the trade-off between preserving combining ability and restoring specific concept features should be discussed.

3. The writing is somewhat redundant. For example, it is not clear how significant Figure 2 is. The Figure 1 is already shows the performance of proposed method.

4. The paper mentions that "we found that the CLIP-T metric can hardly reflect the actual performance of personalized generation", but does not clearly explain how this is found. Whether it is based on references to other works is very confusing. Evidence or references should be provided to support their claim about the limitations of the CLIP-T metric for evaluating personalized generation. This would strengthen the justification for introducing a new metric.

5. The paper mentions "we introduce the BLIP2-T metric", is this metric proposed by the authors? It should be clarified whether this is a novel metric proposed by the authors or whether it's based on existing work. If it's novel, more details on its development and validation are requested.

6. Is Figure 3a an experimental result or a schematic diagram? If it is a schematic diagram rigorous notation is recommended.  More detailed captions or annotations should be provided to improve clarity.
soundness: 3
strengths: 1. The authors explain what they work do with a vivid example, which is easy for readers to understand.
2. The authors theoretically analyze the sampling difficulty based on the joint conditional probability distribution.
3. The paper is written logically and clearly, from experimental phenomenon, theoretical proof, and solution method respectively.
4. The authors extend the proposed method to the video field.
summary: For text-to-image customization task, the authors suggest that all tuning-based models will inherently suffer from the over-fitting introduced by this process, which leads to weakening in the compositional ability of the model. They hypothesize that the decline in this compositional ability stems from the semantic drift of the target concept away from superclass target during fine-tuning. They analyze the experimental and theoretical aspects in turn, and introduce ClassDiffusion to address the issue of weakening compositional capacity after fine-tuning.
weaknesses: 1. The paper mentions that “CLIP-T metric can hardly reflect the actual performance of personalized generation” but CLIP is still used to calculate Loss (Equation 10).
2. The paper mentions “higher SPL weights preserve combining ability better” and “lower SPL weights restore specific concept features more effectively”, why is the weight of SPL set to 1 in dataset A. Is this the optimal value? It needs to be further discussed.
3. The writing is somewhat redundant. For example, it is not clear how significant Figure 2 is.

Meta-review: The manuscript received ratings of 6, 6, 8, and 5. Reviewers appreciated that the manuscript studies the underlying cause behing overfitting of Stable Diffusion and the reasons behing the struggle of existing works with respect to the issue of compositional ability. Reviewers also rasied some concerns in the initial review, including additional details about the utilization of the semantic preservation loss, effectiveness of the proposed method to handle multiple personalized instances, additional comparisons using the quantitative metrics specifically designed to measure compositional ability, and more clarity with respect to figure 3a. Authors provided a rebuttal to address the concerns of the reviewers, including further explanation of the proposed method (e.g., rationale behind the loss choice), improving caption & clarification about the figure, more details about embedding visualization, and quantitative experiments on customizing multiple concepts. Post-rebuttal, three reviewers remained positive mentioning that most of their concerns are addressed in the rebuttal.  Given the reviewers comments, rebuttal and discussions, the recommendation is accept. Authors are strongly encouraged to take into consideration reviewers feedback when preparing the revised manuscript.

Decision: Accept (Poster)