Assessing Open-world Forgetting in Generative Image Model Customization

## Abstract

Abstract Recent advances in diffusion models have significantly enhanced image generation capabilities. However, customizing these models with new classes often leads to unintended consequences that compromise their reliability. We introduce the concept of open-world forgetting to emphasize the vast scope of these unintended alterations, contrasting it with the well-studied closed-world forgetting , which is measurable by evaluating performance on a limited set of classes or skills.
Our research presents the first comprehensive investigation into open-world forgetting in diffusion models, focusing on semantic and appearance drift of representations. We utilize zero-shot classification to analyze semantic drift, revealing that even minor model adaptations lead to unpredictable shifts affecting areas far beyond newly introduced concepts, with dramatic drops in zero-shot classification of up to 60%. Additionally, we observe significant changes in texture and color of generated content when analyzing appearance drift.
To address these issues, we propose a mitigation strategy based on functional regularization, designed to preserve original capabilities while accommodating new concepts. Our study aims to raise awareness of unintended changes due to model customization and advocates for the analysis of open-world forgetting in future research on model customization and finetuning methods. Furthermore, we provide insights for developing more robust adaptation methodologies.

## 1 Introduction

Recent advancements in image generation have led to the development of remarkably powerful foundational models capable of synthesizing highly realistic and diverse visual content. Techniques such as Generative Adversarial Networks (GANs) (Goodfellow et al., ), and more recently autoregressive models (Yu et al., ), Rectified Flows (Liu et al., ), and Denoising Diffusion Probabilistic Models (DDPMs) (Ho et al., ), have each contributed to significant progress in the field. These methods offer unique strengths in sample quality, diversity, and controllability. Among them, diffusion models have gained particular prominence due to their recent successes and growing influence, especially in enabling text-based image generation (Shonenkov et al., ; Ramesh et al., ) and complementary multimodal conditioning (Zhang & Agrawala, ; Mou et al., ), making them a key focus in current research and applications.

Given the high-quality image generation capabilities of these models, a major focus of research has been on how to efficiently incorporate new content and adapt them to new tasks and domains. To address this challenge, various state-of-the-art transfer learning methods have been introduced. These include finetuning approaches such as DreamBooth (Ruiz et al., ) and CustomDiffusion (Kumari et al., ), which allow models to learn new concepts effectively. Conditioning-based methods like ControlNet (Zhang & Agrawala, ) and IP-Adapters (Ye et al., ) enable precise control over generated images by incorporating additional guidance signals.
Prompt methods like Prompt-to-Prompt (Hertz et al., ) and Textual Inversion (Gal et al., ) enable semantic image editing and learning new concepts without modifying the base model.
Parameter-efficient techniques such as Low-Rank Adaptations (LoRA) (Hu et al., ) have shown great promise, allowing for rapid adaptation with minimal computational overhead (Blattmann et al., ; Shi et al., ).
These techniques enable models to learn new concepts effectively, even with few examples.

These methods for adapting diffusion models (Ruiz et al., ; Kumari et al., ) mainly rely on transfer learning and primarily focus on finetuning model weights to accommodate newly introduced data. However, they lead to unforeseen changes in the model’s behavior, which can have significant implications, altering the model’s existing knowledge, and skills, or the alignment between language and visual content within the network. The field of continual learning has long studied the issue of *catastrophic forgetting* in neural networks when these aim to adapt to new data (often referred to as new tasks) (Kirkpatrick et al., ; De Lange et al., ). Traditionally, it has focused on what we term *closed-world forgetting*, where evaluation is limited to a fixed set of classes encountered in previously learned tasks or skills (similar to concept editing). In contrast, modern foundation models introduce a new challenge: measuring *open-world forgetting*. Here, the model’s prior knowledge is vast, making it difficult to assess what has been forgotten or altered during the adaptation process.

In this paper, we focus on two popular personalization methods, namely Dreambooth (Ruiz et al., ) and CustomDiffusion (Kumari et al., ), for a case study of open-world forgetting. These techniques are especially relevant, as they only add very little new knowledge to the network: a single new concept represented by a small set of typically 3-5 images. Although one might expect that finetuning the model with such limited data would have minimal impact on the vast knowledge of the foundation model (e.g., Stable Diffusion), our analysis reveals that even these small updates can lead to highly detrimental consequences. As Figure 1 illustrates, finetuning can drastically alter the image representation of concepts seemingly unrelated to the training images. The complexity of the forgetting underscores the need for a better understanding of how and where it occurs. Without this understanding, finetuned models risk becoming less reliable, less robust, and ultimately less trustworthy, particularly in safety-critical applications where precision and predictability are paramount.

We propose to analyze open-world forgetting from several perspectives. First, we examine *semantic drift* using the recent observation that diffusion models can function as zero-shot classifiers; we propose to compare zero-shot capacity of models before and after adaptation on a set of image classification data sets. Second, we analyze *appearance drift* by evaluating changes in color and perceptual measurements before and after adaptation. Lastly, we assess the extent of forgetting in closely related concepts (*local drift*) versus unrelated concepts. To address these three aspects of drift, we explore a straightforward, yet effective mitigation strategy by introducing a regularization technique during the training of new concepts. In conclusion, the main contributions of this work are:

- •
We are the first to systematically analyze *open-world forgetting* in diffusion models due to model adaptation. Results show that even when adapting to very small domains, the consequences can be highly detrimental.
- •
We propose two approaches to analyze open-world forgetting, which are designed to assess *semantic* and *appearance drift* caused by the adaptation. We leverage the zero-shot classification capabilities of diffusion models to measure the semantic drift, and observe drastic performance drops (of over 60% for some classes). Appearance drift analysis confirms that customization leads to considerable changes in intra-class representation, color, and texture.
- •
We introduce a method to mitigate open-world forgetting, addressing the challenges of observed drift in text-to-image (T2I) models.
This method aims to preserve the original model’s capabilities while allowing for effective customization. Experiments confirm that it greatly reduces both the semantic and appearance drift caused by open-world forgetting.

## 2 Related work

#### Text-to-image diffusion model adaptation.

Text-to-image (T2I) diffusion model adaptation is also referred to T2I personalization or subject-driven image generation.
This aims to adapt a given model to a new concept by providing a few images and binding the new concept to a unique token.
As a result, the adaptation model can generate various renditions of the new concept guided by text prompts.
Depending on whether the adaptation method is finetuning the T2I model, they are categorized into two main streams.
One of the most representative methods focuses on learning new concept tokens while freezing the T2I generative backbones. Textual Inversion (TI) (Gal et al., ) is a pioneering work focusing on finding new pseudo-words by performing personalization in the text embedding space.
The following works (Dong et al., ; Daras & Dimakis, ; Voynov et al., ; Han et al., ) continue to improve this technique stream.
Another stream is finetuning the T2I generative models while updating the modifier tokens. One of the most representative methods is DreamBooth (Ruiz et al., ), where the pre-trained T2I model learns to bind a modified unique identifier to a specific subject given 3$\sim$5 images, while it also updates the T2I model parameters. Custom Diffusion (Kumari et al., ) and other approaches (Han et al., ; Chen et al., ; Shi et al., ) follow this pipeline and further improve the quality of the generation.

Finetuning methods often achieve state-of-the-art performance but introduce forgetting in large T2I models. While research focuses on improving new concept generation, it overlooks continuous model updating and forgetting mitigation. Recent works (Sun et al., ; Smith et al., ) address token forgetting but neglect other impacts of finetuning, such as semantic drifting in color, appearance, and visual recognition, which this paper explores.

#### Assessing forgetting.

The main challenge of continual learning is to learn incrementally and accumulate knowledge of new data while preventing *catastrophic forgetting*, which is defined as a sudden drop in performance on previously acquired knowledge (McCloskey & Cohen, ; McClelland et al., ). The vast majority of studies on continual learning focus on, what we here call, closed-world forgetting, where the knowledge of the network can be represented by its performance on a limited set of classes (Lopez-Paz & Ranzato, ; De Lange et al., ; Masana et al., ). However, as argued in the introduction, the growing importance of starting from large pretrained models (also known as foundation models), which have a vast prior knowledge, requires new techniques to assess forgetting. The forgetting of large language models (LLMs) during continual finetuning has received some attention in recent years, showing the importance of pretraining to mitigate forgetting (Cossu et al., ), however, they mainly evaluate on down-stream-task performance Scialom et al. ().
To the best of our knowledge, open-world forgetting has not yet been systematically analyzed for text-based image generation models which multi-modal nature can further worsen the impact of forgetting due to misalignment of the modalities.

## 3 Customization of Diffusion Models

In this section, we briefly introduce text-to-image (T2I) models and the two main customization methods we will evaluate during our analysis. In addition, we introduce an alternative regularization method to further mitigate forgetting.

### 3.1 Diffusion Models

Diffusion models are a class of generative models that generate data by gradually denoising a sample from a pure noise distribution. The process is modeled in two stages: a forward process and a reverse process. In the forward process, Gaussian noise is iteratively added to data samples, typically over $T$ steps, forming a Markov chain. At each step, the transition is defined as:

$$ $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t }}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}),$ (1) $$

where $\beta_{t}$ controls the noise schedule.

The reverse process denoises the data by learning the conditional probability $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$, typically parameterized by a neural network $\epsilon_{\theta}(\mathbf{x}_{t},t)$ that predicts the added noise at each step. The model is trained by minimizing a simple mean-squared error between the true and predicted noise:

$$ $L(\theta)=\mathbb{E}_{t,\mathbf{x}_{0},\epsilon}\left[\left\|\epsilon-\epsilon _{\theta}(\mathbf{x}_{t},t)\right\|^{2}\right].$ (2) $$

Text-to-image diffusion models employ an additional conditioning vector $\mathbf{c}=\mathcal{E}(P)$ generated using a text encoder $\mathcal{E}$ and a text prompt $P$.
These models have gained prominence for their ability to generate high-quality, diverse samples, often outperforming other generative models like GANs and VAEs in terms of mode coverage and sample quality (Ho et al., ; Song et al., ).

### 3.2 Customization approaches

Diffusion models often require finetuning for specific domains or user needs. This involves introducing new conditioning mechanisms or retraining on specialized datasets. This paper applies two adaptation methods to evaluate finetuning’s impact on image generation models.

Dreambooth (Ruiz et al., ) enables personalization of diffusion models by finetuning them with a small set of images. It reuses an infrequent token of the vocabulary to represent a unique subject, allowing the model to generate images of the subject in varied contexts or styles. This approach induces language drift and reduced output diversity in the model, which is mitigated by replaying class-specific instances alongside the subject training, called prior preservation loss. The final training objective reads

$$ $\mathbb{E}_{\mathbf{\epsilon},\mathbf{x},\mathbf{c},t}[w_{t}\lVert\mathbf{ \epsilon}-\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},\mathbf{c},t)\rVert+ \lambda w_{t^{\prime}}\lVert\mathbf{\epsilon}^{\prime}-\mathbf{\epsilon}_{ \theta}(\mathbf{x}_{t^{\prime}}^{\text{pr}},\mathbf{c}^{\text{pr}},{t^{\prime} })\rVert],$ (3) $$

where $\lambda$ is a weighting parameter, and $\mathbf{x}_{t}^{\text{pr}}$ and $\mathbf{c}^{\text{pr}}$ come from the prior dataset. DreamBooth is especially useful for personalized content generation where subject fidelity is critical.

Custom diffusion (Kumari et al., ) is another approach aimed at efficiently finetuning diffusion models with minimal data and compute.
This method observes that the cross-attention layer parameters undergo the most change during personalization, so they propose to only update the key and value projections in these layers.
It introduces a token into the text-encoder representing a unique subject, rather than reusing an old one.
By freezing the majority of the model’s parameters and focusing updates on a few key layers, Custom Diffusion facilitates rapid customization with less degradation in image quality.
Prior preservation loss is maintained, since language drift is still experienced otherwise.

Customized Model Set In our experiments, we will evaluate Dreambooth and Custom Diffusion. We adapt both these models to ten different concepts based on 5 images per concept. The concepts are ‘lamp’, ‘vase’, ‘person2’,‘person3’,‘cat’,‘dog’,‘lighthouse’,‘waterfall’,‘bike’ and ‘car’ taken from CustomConcept101 (Kumari et al., ). We will refer to these ten models for both DreamBooth and Custom Diffusion as the *Customized Model Set*.

### 3.3 Drift Correction

The two studied approaches, Dreambooth (Ruiz et al., ) and Custom Diffusion (Kumari et al., ), apply finetuning to adapt to the new data: they mainly focus on how good the learned model is on the target data, and do not study the possible detrimental effects for other classes. The Dreambooth method includes a method called prior regularization, which by replaying general instances of the concept being learned (see Eq. 3), helps to prevent the model from overfitting to the new data and ensures that the representation of the superclass remains stable. This same mitigation strategy is also applied in custom diffusion (Kumari et al., ).

In this paper, we propose another regularization technique that can be applied during new concept learning. The method is remarkably simple and is motivated from continual learning literature. This field has proposed a variety of methods to counter forgetting during the learning of new concepts (De Lange et al., ).
Regularization methods aim to regularize the learning of new concepts in such a way that it does not change weights which were found relevant for previous tasks. The field differentiates between parameter regularization
methods, like EWC (Kirkpatrick et al., ) which directly learn an importance weight for all the network parameters, or functional (or data) regularization, like Learning-without-Forgetting (Li & Hoiem, ; Pan et al., ) which regularizes the weights indirectly by imposing a penalty on changes between the (intermediate) outputs of a previous and current model.

We propose to apply a functional regularization loss to the network during the training of new concepts. Our loss, called *drift correction loss*, constrains the difference between the outputs of the pre-trained and fine-tuned models when the new concept is not present in the prompt. It has the following form:

$$ $\mathbb{E}_{\mathbf{\epsilon},\mathbf{x},\mathbf{c},t}[w_{t}\lVert\mathbf{ \epsilon}-\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},\mathbf{c},t)\rVert+ \lambda w_{t^{\prime}}\lVert\mathbf{\epsilon}_{\theta^{*}}(\mathbf{x}_{t^{ \prime}}^{\text{pr}},\mathbf{c}^{\text{pr}},{t^{\prime}})-\mathbf{\epsilon}_{ \theta}(\mathbf{x}_{t^{\prime}}^{\text{pr}},\mathbf{c}^{\text{pr}},{t^{\prime} })\rVert],$ (4) $$

where the second term is the distillation loss, $\lambda$ is a relative weighting parameter and $\mathbf{\epsilon}_{\theta^{*}}$ is the base model.
This loss helps to maintain consistency in the model’s internal representations while allowing it to learn new information effectively. For the training process, we choose instances from the same class as the concept being learned, similar to those used by prior regularization. The change between our proposed drift correction method (Eq. 4) and the existing prior regularization (Eq. 3) is that we do not require the finetuned network to estimate the true forward noise, but instead we want it to estimate the same noise as the original starting network. We will see that this small change significantly improves stability and mitigates forgetting.

In our evaluation, we provide results for *DreamBooth (DB)* which includes the prior regularization, for *Dreambooth with Drift Correction (DB-DC)* which also includes the prior regularization and for *DreamBooth with Drift Correction without the prior regularization (DB-DC$\backslash$pr)*. Similarly, we show results for the various variants of Custom Diffusion (*CD, CD-DC*, and *CD-DC$\backslash$pr*).

## 4 Open-World Forgetting in Generative Model Adaptation

In this section, we explore the effects of finetuning on foundational image generation models, particularly how even slight modifications can significantly impair the model’s ability to retain previously acquired knowledge. We hypothesize that this degradation affects not only the model’s performance on newly introduced tasks, but also its capacity to accurately reproduce or classify previously learned concepts. Given the broad scope of knowledge encompassed by the pretrained model, we refer to this phenomenon as *open-world forgetting*.

As an initial experiment, to assess open-world forgetting, we evaluate both the original unaltered model (called *base model* from now on) and the *Customized Model Set* on 10,000 user prompts from DiffusionDB (Wang et al., ) dataset (prompt examples are provided in Appendix B.1).
Specifically, we measure the change of the resulting images using the cosine distance between CLIP-I encodings (Radford et al., ) when generating images with the same prompt and seed. Distances in the CLIP-I embedding are related to semantic similarity between images, with smaller distances indicating more similar visual content and larger distances suggesting more significant differences in the generated images. The distribution is plotted in Figure 2. For a detailed description of our experimental setup, please refer to Appendix B.1. It is important to note that a personalization method that does not alter the model would yield identical image outputs, resulting in a plot density concentrated at $1$.

When considering Figure 2a, we observe that even though most of the prompts from DiffusionDB are not related with the selected trained concepts, there is a significant part of the distribution that is shifted to the left. This shows indeed that the representations of the original model have changed. Furthermore, further analysis shows that open-world forgetting significantly alters the output in different ways, as illustrated by the samples in Figure 2a. For instance, a sampled pair from the most dissimilar outputs (purple triangle) shows a complete change in content, colors, and scene composition that no longer matches the prompt. In contrast, a very similar pair (yellow star) closely adheres to the original model’s output, with only changes in color or details. Interestingly, when looking at Figure 2b where we apply the proposed Drift Correction to DreamBooth, the distribution shifts to the right, showing that the drift has been reduced considerably.

Figure: Figure 2: Similarity (measured as cosine distance in CLIP-I embedding space) between models before and after adaptation. Each curve represents one of the 10 models from the Customized Model Set. a) Results with DreamBooth adaptation (includes prior regularization). b) Results with DreamBooth with Drift Correction. For more results see Appendix A.
Refer to caption: x2.png

To better assess the impact of open-world forgetting, we propose to categorize the effects into two distinct types: semantic and appearance drift.
Semantic drift implies a change at the class or object level, where one concept is effectively misencoded as another. Appearance drift, on the other hand, refers to shifts in the appearance of a concept that do not necessarily imply a change in recognition (e.g., alterations in color, texture, or scene composition). It is important to note that these two categories are highly correlated, and changes in either of them impact the other.

### 4.1 Semantic Drift

Semantic drift refers to alterations in a model’s representation that cause the generation of semantically divergent content following customization. In the experiment depicted in Figure 2a, almost all prompts exhibit some level of drift, with a notable long tail of highly dissimilar generations. Many of these pronounced deviations have resulted in the generation of content that semantically no longer align with the input prompt.

To evaluate how semantic drift affects generative models, we use a straightforward approach: we utilize the model’s internal representations on different classification tasks (Mittal et al., ; Tang et al., ). It is based on a recent insight that showed that diffusion models can be directly applied for zero-class classification, by leveraging the conditional likelihood estimation of the model. Concretely, we use Diffusion Classifier (Li et al., ), where a posterior distribution over classes $\{\mathbf{c}_{i}\}_{i=1}^{N}$ is calculated as:

$$ $p_{\theta}(\mathbf{c}_{i}\mid\mathbf{x})=\frac{\exp\{-\mathbb{E}_{t,\epsilon}[ \|\epsilon-\epsilon_{\theta}(\mathbf{x}_{t},\mathbf{c}_{i})\|^{2}]\}}{\sum_{j= 1}^{N}\exp\{-\mathbb{E}_{t,\epsilon}[\|\epsilon-\epsilon_{\theta}(\mathbf{x}_{ t},\mathbf{c}_{j})\|^{2}]\}}.$ (5) $$

Monte Carlo sampling is performed over $t_{i}$ and $\epsilon$ to obtain a classifier from a text-to-image model $\epsilon_{\theta}$.

While this method offers a simple, parameter-free approach to evaluating semantic drift, it is worth noting that alternative techniques have been proposed to assess the representation space of diffusion models. These include linear probing on activations (Xiang et al., ), analysis of hierarchical features (Mukhopadhyay et al., ), and methods requiring a preliminary likelihood maximization stage (Chen et al., ). However, these alternatives often involve additional computational steps or are subject to specific settings, potentially limiting their applicability or introducing complexity to the evaluation process.

We conducted zero-shot classification experiments across multiple datasets spanning diverse domains to quantify the semantic drift of the models. We perform two measurements. First, we measure the *average zero-shot classification score* for the various models (the results are averaged over the 10 models of the Customized Model Set). Second, we establish the performance of the original pretrained model as the baseline, and measure the presence of semantic drift by calculating the drop in accuracy from the baseline. We also report the *worst class drop* which is the drop in accuracy of the class that has suffered the largest deterioration due to the adaptation.
For further details on the classification method, please refer to Appendix B.2.

**Table 1: Average zero-shot classification using the T2I models of the Customized Model Set for several image classification datasets. Worst class drop between parenthesis.**
|  | CIFAR10 | STL10 | Flowers | Pets | ObjectNet | Food | Aircraft |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Base Model | 81.60 | 93.00 | 50.00 | 86.87 | 28.50 | 71.09 | 23.40 |
| \hdashlineDB | 75.92 (32.40) | 91.30 (18.60) | 46.61 (64.00) | 82.61 (36.43) | 25.26 (56.00) | 65.48 (56.00) | 19.36 (58.00) |
| DB-DC | 80.98 (14.00) | 93.36 (4.40) | 49.29 (42.00) | 86.64 (17.14) | 27.72 (42.00) | 69.07 (44.00) | 21.42 (48.00) |
| DB-DC$\backslash$pr | 80.60 (14.00) | 92.94 (5.20) | 49.06 (40.00) | 86.37 (16.43) | 27.45 (46.00) | 68.79 (44.00) | 21.54 (44.00) |
| \hdashlineCD | 79.98 (17.00) | 91.40 (12.20) | 47.65 (66.00) | 83.46 (33.57) | 25.75 (58.00) | 65.25 (56.00) | 19.44 (58.00) |
| CD-DC | 82.36 (9.00) | 93.02 (5.00) | 49.33 (42.00) | 86.37 (16.43) | 27.91 (42.00) | 69.19 (44.00) | 21.94 (46.00) |
| CD-DC$\backslash$pr | 82.04 (10.80) | 92.76 (6.00) | 49.16 (44.00) | 86.70 (20.00) | 27.77 (42.00) | 68.99 (44.00) | 21.56 (48.00) |

**Table 2: Concept fidelity (DINO, CLIP-I) and prompt fidelity (CLIP-T). Drift Correction maintains fidelity across metrics.**
|  | DINO | CLIP-I | CLIP-T |
| --- | --- | --- | --- |
| DB | 0.42 | 0.68 | 0.79 |
| DB-DC | 0.43 | 0.68 | 0.78 |
| DB-DC$\backslash$pr | 0.43 | 0.68 | 0.78 |
| \hdashlineCD | 0.44 | 0.69 | 0.79 |
| CD-DC | 0.44 | 0.69 | 0.79 |
| CD-DC$\backslash$pr | 0.44 | 0.69 | 0.79 |

The results in Table 1 are surprising, average zero-shot classification accuracy drop significantly on all the datasets: adapting a huge generative image foundation model to just five images of a new concept has a vast impact throughout the latent space of the diffusion models. When applying DreamBooth, average zero-shot performance drops by over 4% on CIFAR10, Pets, Food and Aircraft. If we look at individual classes, the impact can be much larger. As indicated by the *worst class drop*, for some classes, zero-shot performance drops by over 60% (e.g. ‘vacuum cleaner’ gets recognized as ‘microwave’, ‘drill’ or ‘laptop’). We show that these drops in performance are mitigated to a large extent by our alternative Drift Correction results (see DB-DC and CD-DC results) and their average zero-shot classification scores are in general within 1% of the base model. Removing the prior regularization from our method (see DB-DC$\backslash$pr and CD-DC$\backslash$pr) leads to only slightly lower results, showing the impact of our proposed regularization method. Also, worst class drop significantly reduces when applying DC, but for some datasets remains still high.

Furthermore, to measure generation quality, diversity, and alignment between image and text of the learned concepts, we employ metrics adapted from personalization methods: DINO, CLIP-I, and CLIP-T. In Table 2 we can see that the proposed regularization method DC does not negatively impact the image generation quality of the learned concepts.(^1^11The results with standard deviations for Table 1 and 2 are provided in Appendix B.)

### 4.2 Appearance Drift

While open-world forgetting does not always result in significant changes to the core content of the image, as shown in Figure 2a, it notably affects intra-class variation, color distribution, and texture characteristics. We define these collective changes as appearance drift, a phenomenon that alters the model’s representation space in subtle yet impactful ways.
Figure 3 demonstrates two key aspects of appearance drift; intra-class and contextual variation (first row), where different customizations of the base model lead to changes in car brand and background, while maintaining the overall concept of ‘car’. Color shift (second row), where the color palette of the generated images changes significantly, even when the intra-class characteristics and background remain relatively constant.

Appearance drift manifests through alterations in visual attributes at varying degrees of intensity. Finetuning can cause the model to reinterpret these visual features, leading to inconsistencies between original and newly generated outputs. Although initially subtle, appearance drift can substantially impact customized models. For example, when attempting to learn and generate a set of new concepts within the same context (e.g., for synthetic dataset creation or advertising purposes), each customization of the base model may result in color and content changes. This variability makes it challenging to achieve consistent results across multiple iterations.
Moreover, as the customization process alters the model’s manifold, the resulting model becomes less reliable in domains outside the scope of the customization training images. This limitation highlights the importance of understanding and mitigating appearance drift in applications of fine-tuned text-to-image models.

Figure: Figure 3: Appearance drift as consequence of customization. a) chromaticity plot of pixels of three realization of the prompts (’photo of a car/cow’) and the same seed with different models, namely b) the base model, c) model adapted to lighthouse and d) model adapted to bike.
Refer to caption: x3.png

#### How to measure appearance drift?

Quantifying appearance drift presents unique challenges due to the inherent variability in text-to-image model outputs. Traditional metrics like LPIPS (Zhang et al., ) and DIST (Ding et al., ) are designed for image pair comparisons. However, the inherent variability in T2I model outputs means that images generated from the same prompt can vary significantly due to changes in seed, model weights, and prompt interpretation. Comparing just two images fails to capture the full range of possible outputs and does not adequately represent the model’s capabilities or biases. Consequently, conclusions drawn from such limited comparisons may lack statistical significance.

To address this variability, the research community has employed metrics that measure distances between probability distributions (e.g., FID (Heusel et al., ), KID (Bińkowski et al., )) of real-world observations and generated data(^2^22In general FID and KID require a set of real images for comparison. In our study, we consider the images generated by the original model as the “real” set, as we are measuring the shift from this initial distribution.). In addition to these metrics, we also propose a new metric that directly measures the color drift between image sets.

#### Color Drift Index.

With the aim to introduce a clearly interpretable difference measure for the color content of the generated images, we propose a novel approach that measures specific properties in pixel space. Our method focuses on color assessment, as traditional image generation metrics tend to be more sensitive to intra-class and texture variations. Inspired by natural image statistics, we introduce the Color Drift Index (CDI), which estimates the color distribution associated with a specific concept by analyzing a large number of generated images.

We utilize the CIE chromaticity diagram(^3^33Our approach offers the added benefit of being applicable across multiple color spaces.), where each pixel color is projected onto a lobe-shaped space representing all visible colors.
Given a set of images $I=\{\mathbf{x}_{i}\}$ and their density distribution in the CIE chromaticity diagram $p^{\text{CIE}}(I)$, we calculate the CDI as the Wasserstein distance (Panaretos & Zemel, ) between the color distributions of two sets of images:

$$ $\text{CDI}(I_{a},I_{b})=W_{p}(p^{\text{CIE}}(I_{a}),p^{\text{CIE}}(I_{b})).$ (6) $$

Figure: Figure 4: Appearance drift as consequence of customization measured with (left) Color Drift Index (CDI) and (right) Kernel Inception Distance (KID). The orange and green line represent the distance between the base model and the customized model. The blue line is a control line, representing the distance between two sets of images generated from different seeds both with the base model. Lines close to the origin are better.
Refer to caption: x4.png

We evaluate the appearance drift using the CDI together with KID (FID results are presented in Appendix B.3).
We conducted a comprehensive experiment adopting the carefully curated selection of common concepts from the dataset of Torralba & Oliva ().
For each prompt, we generated 1,000 images using both original and the ten models from the Customized Model Set.
Figure 4 presents the mean values of the metrics across several adaptations, providing a visual representation of the differences captured by each measure. For a detailed overview of the results, including individual model performances, refer to Table 8 in the appendix.
To help interpret the KID and CDI scores, we provide a control setting (blue line). In this configuration, we measure CDI and KID between images generated with the base model but using different seeds (functioning as a lower bound). If we are sampling from the same distribution, the base model should yield lower distances (approaching zero as the number of samples grows) than the customized models (DB and CD).

The results in Figure 4 reveal two key insights.
First, the base model consistently produces smaller distance values compared to DB, confirming that the distribution of each concept is indeed changing due to appearance drift. Second, each concept is affected differently by the drift, attributable to the fact that each concept relates to different parts of the model’s manifold. Furthermore, as demonstrated in Appendix C, the magnitude and nature of the drift vary as a function of the content in the replay buffer and training images. Also, importantly, Figure 4 shows that our proposed method (DC) significantly reduces the impact of the appearance drift introduced by customization methods. Especially, the drift measure in KID is considerably reduced.

These findings underscore the complexity and pervasiveness of appearance drift in fine-tuned text-to-image models. They highlight the need for robust mitigation strategies and careful consideration when deploying customized models in real-world applications, emphasizing the importance of ongoing research in this area to ensure the reliability and consistency of generated outputs.

### 4.3 Local Drift

In this paper, we have focused on drift throughout the whole diffusion model manifold. Previous works, especially those in the machine unlearning community (Gandikota et al., ), have concentrated on *local drift*. When removing a concept from a model, it is believed to mainly impact the representation of closely related concepts (hence the name local drift).
Our findings suggest that the effects of finetuning are more pervasive than previously thought, potentially influencing the model’s understanding and representation of far-away categories as well as close-by (local) categories.

Figure: Figure 5: Similarity (measured as cosine distance in CLIP-I embedding space) and perceptual metrics between models before and after adaptation. For each concept trained, we evaluate closely related concepts to measure the local drift. a) Results with DreamBooth adaptation (includes prior regularization). b) Results with DreamBooth with Drift Correction. c) Color Drift Index (CDI) and Kernel Inception Distance (KID). For more results see Appendix A.
Refer to caption: x5.png

Here, we repeated our experiments from Section 4.1 and 4.2 to measure the local semantic and local appearance drift. For this setup, we generated 1,000 samples of the closest concepts (superclasses) to each trained model (see Appendix B.1 for the details) and evaluated the CLIP-I, CDI, FID, and KID metrics. As Figure 5a shows, the semantic drift is showing a significant shift towards the left, indicating that local drift is more pronounced. For appearance drift, Figure 5c depicts a more uniform color and KID shift over all the models; this shows that related concepts are affected with a similar magnitude by appearance drift. Again the application of our proposed Drift Correction method greatly reduces both the local semantic drift (as measured in Figure 5b and it almost removes the local appearance drift as measured by KID, even though some color drift remains (see Figure 5c).

## 5 Discussion and Conclusion

Our investigation into unintended consequences of generative model adaptation reveals several key contributions: 1) We observe that finetuning foundational generative models leads to substantial *open-world forgetting*, manifesting as both semantic and appearance drift. 2) We propose to measure semantic drift by considering zero-shot classification on a wide range of image classification tasks, and we propose a novel metric to measure color drift that combined with KID measures appearance drift. 3) We propose a technique to mitigate open-world forgetting using functional regularization. This method shows promise in preserving foundational knowledge while allowing effective customization. These findings provide a guidance and initial step to tackle open-world forgetting and developing more robust and reliable customization techniques.

The last couple of years has seen a growing number of foundation models being released. These models have been enthusiastically adapted by the community, and are often the basis for further finetuning operations to new domains, tasks, etc. The main purpose of our paper is to contribute to the awareness of the unintended changes in the original foundation model knowledge that can occur when adapting these. This phenomenon, coined open-world forgetting, is complicated to assess, and we have proposed several ways to assess it for customized generative models. We hope that our results motivate more research in this direction. Open-world forgetting resembles the finding of a needle in a haystack due to the vast amount of prior knowledge of the foundation models, and we think that other research directions which actively optimize to find the most affected areas might be required. Finally, we consider extending our methodology to other forms of model adaptation, such as unlearning techniques as exiting future work.

Limitations. While our study provides valuable insights into open-world forgetting in generative model adaptation, there are several limitations to consider. First, inherently to the nature of open-world, we cannot exhaustively evaluate the entire conceptual space of the model. Our analysis is limited to a subset of concepts and may not capture all potential instances of forgetting. Second, our study has mainly focused on evaluating the impact of diffusion model customization. Even though we represent a mitigation technique for open-world forgetting (Drift Correction) we think that there are many more ways to address this problem (especially considering the vast continual learning literature) which have not considered in our study.

## Ethical Statement

We acknowledge the potential ethical implications of deploying generative models, including issues related to privacy, data misuse, and the propagation of biases. All models used in this paper are publicly available, as well as the base training scripts. We will release the modified codes to reproduce the results of this paper. We also want to point out the potential role of customization approaches in the generation of fake news, and we encourage and support responsible usage of these models. Finally, we think that awareness of open-world forgetting can contribute to safer models in the future, since it encourages a more thorough investigation into the unpredictable changes occurring when adapting models to new data.

## Reproducibility Statement

To facilitate reproducibility, we will make the entire source code and scripts needed to replicate all results presented in this paper available after the peer review period. We will release the code for the novel color metric we have introduced. We conducted all experiments using publicly accessible datasets. Elaborate details of all experiments have been provided in the Appendices.

## Appendix A More examples

### A.1 Open-world forgetting

In Figure 6, we present more samples of generated images with appearance drift.

Figure: Figure 6: Several examples of appearance drift with DreamBooth. Images generated from the same initial seed.
Refer to caption: x6.png

### A.2 Local drift

In Figure 7 we present examples of local drift for the concept ”dog”.

Figure: Figure 7: Several examples of local drift with DreamBooth. Images generated from the same initial seed. Note that the variety in viewpoint and breeds reduces significantly.
Refer to caption: x7.png

## Appendix B Experiment details

This section outlines our experimental setup, including datasets, metrics, and training configurations.

### B.1 Semantic Drift Evaluation

#### Datasets.

To evaluate open-world forgetting, we select a random subset of 10,000 user prompts from DiffusionDB 2M (Wang et al., ) (Table 3).
For adaptation training and evaluation, we choose a subset of 10 concepts from CustomConcept101 (Kumari et al., ), namely decoritems_vase2, decoritems_lamp1, person_2, person_3, pet_cat5, pet_dog4, transport_bike, transport_car2, scene_lighthouse, scene_waterfall. Each concept contains approximately 3-5 images.
For superclass evaluation, we create a dataset of 10 synonyms with respect to each concept, which can be found in Table 4.

**Table 3: DiffusionDB subset sample prompts. Shorter prompts selected for visualization purposes.**
| DiffusionDB prompts |
| --- |
| “dafne keen, mad max, cinematic shot, 8k resolution” |
| “creepy horror movie characters, fog, rain, volumetric lighting, beautiful, golden hour, sharp focus, highly detailed, cgsociety” |
| “the railroad is a place of death. it’s where the forgotten and the damned go to die. it’s a place of dark secrets and hidden terror. photorealistic ” |
| “samurai jack johnny bravo by salvador dali ” |
| “Film still of Emma Watson as Princess Leia in Star Wars (1977)” |
| “a detailed figure of indigo montoya, first 4 figures, detailed product photo ” |
| “a hyper scary pokemon, horror, creepy, big budget horror movie, by zdzisław beksinski, by dorian cleavenger ” |
| “the war between worlds extremely detailed claymation art, dark, moody, foggy ” |
| “a painting of Hatsune Miku by H. R. Giger, highly detailed, 4k digital art” |
| “a redneck with wings and horns wearing sunglasses and snake skin smoking a blunt, detailed, 4 k, realistic, picture ” |
| “fantasy art 4 k ultra detailed photo caricature walter matthau as an fighter pilot ” |
| “ CG Homer Simpson as Thanos, cinematic, 4K” |
| “Full body portrait of Raven from Teen Titans (2003), digital art by Sakimichan, trending on ArtStation” |
| “bigfoot walking down the street in downtown Bremerton Washington” |
| “garden layout rendering with flowers and plants native to ottawa canada ” |
| “a beautiful planet of guangzhou travel place of interest, chill time. good view, exciting honor. by david inshaw ” |
| “an oil painting of Dwayne Johnson instead of Mona Lisa in the famous painting The Joconde painted by Leonardo Da Vinci” |
| “film still of danny devito as mario in live action super mario bros movie, 4 k ” |
| “a beautiful artist’s rendition of what the stable diffusion algorithm dreams about ” |

**Table 4: Concept synonyms.**
| Concept | Synonyms |
| --- | --- |
| bike | pedal cycle, velociped, roadster, bicycle, push bike, pushbike, cycle,wheels, two-wheeler, pedal bike |
| car | jalopy, ride, auto, vehicle, coupe, wheels, automobile, sedan, hatchback, motocar |
| cat | feline, grimalkin, mouser, moggy, tabby, puss, kitty, kitten, pussycat, tomcat |
| dog | canine, hound, pup, pooch, fido, puppy, mutt, man’s best friend, doggy, cur |
| lamp | fixture, chandelier, light, illuminator, lantern, luminaire, glow, torch, sconce, beacon |
| lighthouse | light, coastal beacon, navigation light, pharos, seamark, watchover, beacon, guide light, light station, signal tower |
| person | gent, bloke, chap, gentleman, lad, guy, male, bro, fellow, dude |
| vase | urn, amphora, container, pitcher, carafe, receptacle, jar, vessel, pot, jug |
| waterfall | rapids, torrent, flume, cascade, spillway, cataract, plunge, chute, falls, deluge |

#### Metrics.

We employ three primary metrics to assess image generation quality. CLIP-I is calculated as the average pairwise cosine similarity between CLIP (Radford et al., ) embeddings of real and generated images. DINO uses the same pairwise cosine similarity method but with DINO (Caron et al., ) ViT-S16 embeddings. This metric is preferred over CLIP-I as it does not ignore differences between subjects of the same class. CLIP-T measures the CLIP embedding cosine similarity between the prompt and the generated image, and is used to evaluate prompt fidelity.

#### Training configuration.

We adapt models using publicly available scripts from Diffusers (von Platen et al., ) for Dreambooth(^4^44[https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora.py](https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora.py)) and Custom Diffusion(^5^55[https://github.com/huggingface/diffusers/blob/main/examples/custom_diffusion/train_custom_diffusion.py](https://github.com/huggingface/diffusers/blob/main/examples/custom_diffusion/train_custom_diffusion.py)) applied to Stable Diffusion v1.5 (Rombach et al., ). Both methods use prior regularization unless otherwise stated, which is designed to prevent drifting towards the training concept. The set of images for prior regularization is generated from the base model before training starts.
We use the LoRA script versions and refer to the resulting models as DB for DreamBooth and CD for Custom Diffusion. Full finetuning models exhibit the same or worse shortcomings as the LoRA models analyzed and are termed FT for finetuning, such as DB FT in Table 6.
Since DB and CD are similar and to ensure a fair comparison, both methods use similar training settings: a learning rate of 1e-4, batch size of 1, 500 training steps, and no augmentations. The prior regularization uses a weighting of 1 and comprises 200 samples of generated images with the prompt “{concept}” for each concept, using default generation settings. For the drift correction method described in Section 3.3, all settings remain the same, and the weighting parameter is set to $\lambda=10$.

### B.2 Diffusion Classifier

We employ the official released code of the Diffusion Classifier method(^6^66[https://github.com/diffusion-classifier/diffusion-classifier](https://github.com/diffusion-classifier/diffusion-classifier)). However, due to computational constraints, we modify some parameters for our explorations. We reduce the keep list to $(10,100)$ across all datasets while maintaining the trial list at $(5,1)$. This significantly reduces computational time while resulting in minimal percentual score uncertainty.
Additionally, datasets with many classes or samples were reduced to have a total number of samples of roughly 500 by random selection of the samples of each class. The datasets configuration can be seen in Table 5. It is worth noting that the original ObjectNet has 313 classes, but Diffusion Classifier only uses 113 for testing. We also use a fixed noise for consistent evaluations.

**Table 5: Dataset configurations to evaluate Diffusion Classifier.**
| Dataset | Food | CIFAR10 | Aircraft | Pets | Flowers | STL10 | ObjectNet |
| --- | --- | --- | --- | --- | --- | --- | --- |
| # classes | 101 | 10 | 100 | 37 | 102 | 10 | 113 |
| Samples / class | 5 | 50 | 5 | 14 | 5 | 50 | 5 |
| Total samples | 505 | 500 | 500 | 518 | 510 | 500 | 565 |

The standard deviations from Table 1 and 2 are included in Table 6 and 7, respectively.

**Table 6: Zero-shot classification using the T2I model. Personalized models suffer from degraded representations. Worst class drop between parenthesis. Scores with standard deviation across models.**
|  | Food | CIFAR10 | Aircraft | Pets | Flowers | STL10 | ObjectNet |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Base Model | 71.09 | 81.60 | 23.40 | 86.87 | 50.00 | 93.00 | 28.50 |
| DB FT | 61.50$\pm$5.95 | 69.86$\pm$6.79 | 16.04$\pm$3.81 | 79.11$\pm$5.29 | 43.18$\pm$6.58 | 87.04$\pm$4.63 | 21.52$\pm$3.41 |
| DB | 65.48$\pm$2.59 | 75.92$\pm$5.81 | 19.36$\pm$2.69 | 82.61$\pm$3.33 | 46.61$\pm$2.49 | 91.30$\pm$2.05 | 25.26$\pm$1.85 |
| DB-DC | 69.07$\pm$1.58 | 80.98$\pm$2.57 | 21.42$\pm$0.45 | 86.64$\pm$0.92 | 49.29$\pm$1.33 | 93.36$\pm$0.70 | 27.72$\pm$1.56 |
| DB-DC$\backslash$pr | 68.79$\pm$1.78 | 80.60$\pm$2.46 | 21.54$\pm$0.77 | 86.37$\pm$0.78 | 49.06$\pm$1.12 | 92.94$\pm$0.95 | 27.45$\pm$1.15 |
| CD | 65.25$\pm$2.76 | 79.98$\pm$4.21 | 19.44$\pm$1.97 | 83.46$\pm$2.97 | 47.65$\pm$2.34 | 91.40$\pm$1.60 | 25.75$\pm$2.16 |
| CD-DC | 69.19$\pm$1.73 | 82.36$\pm$1.91 | 21.94$\pm$1.51 | 86.37$\pm$1.28 | 49.33$\pm$1.16 | 93.02$\pm$0.97 | 27.91$\pm$1.30 |
| CD-DC$\backslash$pr | 68.99$\pm$1.73 | 82.04$\pm$2.23 | 21.56$\pm$1.94 | 86.70$\pm$1.22 | 49.16$\pm$1.31 | 92.76$\pm$0.85 | 27.77$\pm$1.58 |

As further illustration in Figure 8 of the zero-shot classification accuracy, we provide the results for one of the models, namely “decoritems_lamp1” for all the datasets for DreamBooth. We can observe that the adaptation leads to drops on most classes (identified in red) but can also occasionally result in a performance increase (in green).

Figure: Figure 8: Class-wise accuracy of base model and DreamBooth adaptation to the “decoritems_lamp1” concept for several data sets.
Refer to caption: x8.png

**Table 7: Concept fidelity (DINO, CLIP-I) and prompt fidelity (CLIP-T). Drift Correction maintains fidelity across metrics.concept evaluation**
|  | DINO | CLIP-I | CLIP-T |
| --- | --- | --- | --- |
| DB | 0.4241$\pm$0.1503 | 0.6764$\pm$0.1046 | 0.7896$\pm$0.0296 |
| DB-DC | 0.4283$\pm$0.1584 | 0.6817$\pm$0.1097 | 0.7799$\pm$0.0324 |
| DB-DC$\backslash$pr | 0.4315$\pm$0.1585 | 0.6841$\pm$0.1086 | 0.7776$\pm$0.0322 |
| CD | 0.4422$\pm$0.1378 | 0.6934$\pm$0.0902 | 0.7916$\pm$0.0266 |
| CD-DC | 0.4381$\pm$0.4381 | 0.6925$\pm$0.0888 | 0.7899$\pm$0.0280 |
| CD-DC$\backslash$pr | 0.4382$\pm$0.1351 | 0.6935$\pm$0.0872 | 0.7872$\pm$0.0264 |

### B.3 Appearance drift full tables

See Table 8 for the full results which have been used for the generation of Figure 4. The prompt are ‘0:Face’,‘1:Pedestrian’,‘2:Car’,‘3:Cow’,‘4:Hand’,‘5:Chair’,‘6:Mountain’, ‘7:Beach’,‘8:Forest’,‘9:Highway’,‘10:Street’,‘11:Indoor’,
‘12:Animal in natural scene’, ‘13:Tree in urban scene’, ‘14:Close-up person in urban scene’,‘15:Far pedestrian in urban scene’,‘16:Car in urban scene’,‘17:Lamp in indoor scene’, ‘18:empty prompt’.

**Table 8: Comparison of CDI, FID, and KID Values for custom vs custom-regularized methods for the prompts (similar to those in Figure 4).**
| Prompt | Vanilla | DB | DB-DC | CD | CD-DC |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | CDI | FID | KID | CDI | FID | KID | CDI | FID | KID | CDI | FID | KID | CDI | FID | KID |
| 00 | 0.10 | 24.75 | 0.00 | 0.65 | 43.74 | 2.00 | 0.23 | 21.70 | 0.16 | 0.48 | 37.01 | 1.22 | 0.24 | 24.90 | 0.29 |
| 01 | 0.14 | 44.55 | 0.00 | 1.01 | 79.78 | 3.22 | 0.43 | 46.16 | 0.50 | 0.82 | 72.44 | 2.63 | 0.33 | 51.68 | 0.79 |
| 02 | 0.12 | 19.70 | 0.01 | 0.35 | 28.31 | 0.90 | 0.18 | 18.37 | 0.12 | 0.31 | 27.14 | 0.58 | 0.19 | 20.11 | 0.15 |
| 03 | 0.09 | 29.17 | 0.01 | 0.36 | 41.81 | 1.22 | 0.14 | 27.40 | 0.13 | 0.40 | 42.25 | 1.13 | 0.23 | 31.62 | 0.37 |
| 04 | 0.17 | 35.93 | 0.01 | 0.59 | 40.77 | 0.59 | 0.23 | 30.53 | 0.07 | 0.57 | 42.65 | 0.70 | 0.32 | 33.76 | 0.22 |
| 05 | 0.14 | 19.65 | 0.01 | 0.76 | 28.15 | 0.61 | 0.27 | 17.93 | 0.08 | 0.58 | 26.45 | 0.57 | 0.34 | 20.09 | 0.17 |
| 06 | 0.09 | 34.22 | 0.00 | 0.48 | 52.80 | 1.46 | 0.26 | 34.55 | 0.32 | 0.57 | 60.90 | 2.16 | 0.34 | 41.52 | 0.72 |
| 07 | 0.14 | 34.89 | 0.05 | 0.36 | 42.58 | 0.64 | 0.25 | 29.45 | 0.14 | 0.41 | 48.98 | 1.19 | 0.22 | 33.66 | 0.26 |
| 08 | 0.10 | 25.50 | 0.00 | 0.48 | 52.42 | 2.79 | 0.25 | 26.78 | 0.57 | 0.42 | 55.33 | 2.95 | 0.29 | 32.76 | 1.01 |
| 09 | 0.12 | 27.51 | 0.01 | 0.51 | 41.82 | 1.73 | 0.30 | 28.22 | 0.30 | 0.43 | 41.37 | 1.40 | 0.33 | 27.80 | 0.52 |
| 10 | 0.10 | 24.34 | 0.00 | 0.62 | 47.28 | 2.78 | 0.34 | 26.27 | 0.68 | 0.69 | 44.67 | 2.09 | 0.42 | 28.39 | 0.72 |
| 11 | 0.06 | 33.56 | 0.02 | 0.38 | 46.44 | 1.39 | 0.19 | 30.86 | 0.27 | 0.53 | 52.92 | 1.86 | 0.26 | 35.79 | 0.53 |
| 12 | 0.11 | 29.75 | 0.01 | 0.43 | 43.92 | 1.20 | 0.24 | 28.67 | 0.21 | 0.41 | 49.51 | 1.69 | 0.27 | 36.02 | 0.68 |
| 13 | 0.13 | 20.26 | 0.00 | 0.66 | 44.59 | 1.50 | 0.64 | 20.36 | 0.47 | 0.62 | 43.14 | 1.40 | 0.42 | 30.33 | 0.37 |
| 14 | 0.12 | 40.60 | 0.01 | 0.53 | 47.32 | 0.58 | 0.33 | 36.76 | 0.16 | 0.60 | 48.05 | 0.66 | 0.33 | 38.68 | 0.18 |
| 15 | 0.07 | 36.90 | 0.01 | 0.96 | 60.22 | 1.96 | 0.48 | 38.81 | 0.51 | 0.75 | 59.16 | 1.92 | 0.46 | 40.97 | 0.51 |
| 16 | 0.09 | 26.65 | 0.00 | 0.68 | 37.07 | 1.02 | 0.41 | 26.12 | 0.26 | 0.66 | 36.38 | 0.90 | 0.41 | 28.21 | 0.29 |
| 17 | 0.13 | 27.73 | 0.01 | 0.59 | 32.44 | 0.67 | 0.30 | 25.44 | 0.23 | 0.52 | 33.93 | 0.74 | 0.27 | 25.77 | 0.14 |
| 18 | 0.18 | 59.52 | 0.02 | 0.29 | 62.02 | 0.59 | 0.23 | 37.99 | 0.00 | 0.31 | 68.52 | 0.93 | 0.21 | 48.00 | 0.13 |

## Appendix C Ablations

### C.1 Does finetuning lead to loss of diversity?

Finetuning large foundational models on a limited set of images (typically around 5) of a specific subject can lead to overfitting, a phenomenon observed in previous studies such as DreamBooth. This overfitting often results in a loss of diversity in generated images and a noticeable shift towards the characteristics of the training subject. While prior regularization techniques have been employed to mitigate this shift, they have not fully resolved the issue, as our analysis demonstrates.

To assess the impact of finetuning on diversity, we adapt the metric introduced in the DreamBooth study. This metric quantifies diversity by calculating the average Learned Perceptual Image Patch Similarity (LPIPS) between generated images of the same subject using identical prompts. A higher LPIPS score indicates greater diversity among the generated images.

Our proposed method not only improves the mitigation of subject shifting, as evidenced in Figures 4 and 5, but also maintains the diversity of the original model. To validate this, we conducted an extensive evaluation using 100 different prompts, each generating 100 images. These prompts were sourced from the DiffusionDB subset, as detailed in Appendix B.1.

Figure 9 presents the results of our diversity analysis. The data demonstrates that our method preserves diversity at a level comparable to, or even exceeding, previous approaches. This finding is particularly significant as it indicates that our technique not only addresses the shifting problem more effectively but does so without compromising the model’s ability to generate diverse outputs.

The preservation of diversity while improving subject fidelity represents a crucial advancement in finetuning methodologies for large generative models. It ensures that the fine-tuned model retains its creative capacity and versatility across a wide range of prompts and subjects, even as it gains enhanced capabilities in representing specific training subjects. This balance between specificity and diversity is essential for the practical application of fine-tuned models in various creative and technical domains.

Figure: Figure 9: Diversity evaluation using LPIPS. Drift Correction maintains the original diversity after adaptation.
Refer to caption: x9.png

### C.2 Increasing the buffer size reduces drift

To investigate the impact of buffer size on mitigating open-world forgetting, we conducted experiments varying the number of images in the replay buffer during model adaptation. Table 9 presents the results of this analysis, showing the effect of buffer size on both semantic drift (measured by zero-shot CIFAR10 classification accuracy) and appearance drift (measured by Color Drift Index, CDI).

**Table 9: Effect of the number of images in the buffer.**
| Metric | 0 | 50 | 100 | 200 | 500 | 1000 | 2000 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Acc CIFAR10 | 63.24$\pm$14.61 | 76.10$\pm$6.66 | 75.62$\pm$6.48 | 75.92$\pm$5.81 | 76.36$\pm$5.28 | 76.92$\pm$5.87 | 77.12$\pm$6.20 |
| CDI | 0.87 | 0.64 | 0.58 | 0.56 | 0.70 | 0.60 | 0.51 |

The experiment suggests that incorporating a replay buffer, even of modest size, is beneficial for mitigating open-world forgetting, particularly in terms of semantic drift. However, the benefits of increasing buffer size show diminishing returns, especially for semantic preservation. For appearance drift, while there is a general trend towards improvement with larger buffers, significant drift persists regardless of buffer size.

### C.3 Influence of the training images

Our experiments reveal that the characteristics of the training images used during model adaptation can significantly impact the nature and extent of appearance drift. To illustrate this effect, we conducted an experiment in Figure 10 focusing on how the background color in training samples influences the color distribution of generated images. The results show that the background color of the training images has a noticeable impact on the color distribution of the generated images, even when generating images of unrelated concepts.

Figure: Figure 10: Appearance drift variation as a function of background color in training samples
Refer to caption: x10.png

These findings highlight the importance of carefully considering the visual characteristics of training images when adapting generative models. The background, lighting, and overall composition of training samples can have far-reaching effects on the model’s output distribution, extending beyond the specific concept being learned.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: 1. Were alternative metrics for measuring semantic drift considered? For instance, could user evaluations on the semantic consistency of generated images be feasible?

2. Has the drift correction method been tested on more complex or diverse concepts beyond the simple categories presented? For example, in Figure 2a, “a sampled pair from the most dissimilar outputs (purple triangle) shows a complete change in content, colors, and scene composition that no longer matches the prompt.”, What happens if drift correction is applied in this case, do the colors and scene composition match the prompts better?
soundness: 2
strengths: The introduction of open-world forgetting is a novel perspective in the domain of diffusion model customization. While previous research often focuses on task-specific performance, this paper broadens the scope of forgetfulness evaluation, bringing strong innovation.
summary: This paper investigates the issue of “open-world forgetting” in generative image model customization, highlighting how fine-tuning models for new classes can inadvertently degrade performance on previously learned classes. The authors systematically analyze open-world forgetting through the introduction of semantic and appearance drift concepts, using zero-shot classification to measure semantic drift. To mitigate this issue, they propose a “drift correction” method, designed to reduce the model's tendency to forget prior knowledge when learning new concepts. Experiments demonstrate that drift correction significantly reduces semantic and appearance drift, enhancing the stability and reliability of customized models.
weaknesses: 1. The paper’s structure is confusing. For instance, lines 513-532 discuss contributions to open-world forgetting before defining it clearly, and these lines largely repeat lines 108-119, adding unnecessary redundancy.

2. Key metrics lack basic explanation in the main text, with details only in supplementary materials. This makes it hard for readers to grasp the evaluation approach without prior knowledge.

3. The paper focuses on how fine-tuning degrades original model capabilities but doesn’t assess how the proposed method impacts performance on the fine-tuning tasks.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is the customization method used in Figure 3 DB? Please clarify it in the revised version,
2. I recommend the author to increase the number of concepts in the evaluation. 10 concepts might be a "cherry pick".
3. Add more images generated by the model before and after applying your methods which would be intuitive and easy to understand.

Overall, I believe this paper is valuable to the community and I would be happy to raise the score if my concern can be well addressed.
soundness: 3
strengths: 1. The concept of open-world forgetting represents a novel extension to the field of model adaptation and customization, particularly in generative models. While catastrophic forgetting has been explored in closed-world contexts, applying it to open-world scenarios in text-to-image models is original and fills an important gap in understanding how fine-tuning affects broad model knowledge.
2. The proposed Color Drift Index (CDI) for assessing appearance drift is a new and creative metric tailored to the nuances of generative models. This provides a fresh approach to evaluating visual consistency, which goes beyond typical performance metrics in generative model research.
3. By providing quantitative results with the CDI and zero-shot classification performance across multiple models and concepts, the authors present strong empirical evidence for their findings, which enhances the reliability of their conclusions.
4. The use of functional regularization as a mitigation strategy is well-executed and shows practical effectiveness, backed by solid data on how it mitigates both semantic and appearance drift.
5. By proposing methods to measure and mitigate unintended model alterations, the work establishes a foundation for future research on safeguarding foundational knowledge in generative models, which will inspire other researchers to work in this direction in the future.
6. The proposed method is simple yet effective and the writing of the paper is clear and easy to follow.
summary: This paper introduces the concept of "open-world forgetting" to emphasize the vast scope of these unintended alterations during the model customization of foundation generative models (i.e., Stable Diffusion). The authors conduct a comprehensive investigation of the semantic and appearance drift of representations. To address these issues, this paper proposes a mitigation strategy based on functional regularization which preserves original capabilities while accommodating new concepts. The contribution of the paper can be summarized as follows:

1. $\textbf{Introduction of Open-World Forgetting Concept:}$ The paper defines open-world forgetting in pre-trained foundation generative image models and systematically assesses its impact, focusing on how models lose previously learned concepts after adaptation.

2. $\textbf{Semantic and Appearance Drift Analysis:}$ It introduces two evaluation approaches:
Semantic Drift: Uses zero-shot classification to measure changes in a model's semantic understanding.
Appearance Drift: Examines shifts in visual attributes like color and texture after customization, introducing the "Color Drift Index" (CDI) as a metric for quantifying these changes.

3. $\textbf{Mitigation Strategy:}$ The paper proposes a functional regularization technique aimed at reducing open-world forgetting, enabling models to integrate new concepts while retaining prior knowledge. This method proved effective in maintaining both semantic and appearance integrity in fine-tuned models.
weaknesses: The authors have mentioned some of the limitations in the paper:
1. The analysis is limited to a small subset of concepts and may not capture all potential instances of forgetting. 10 concepts are apparently not enough to reflect the effectiveness of the proposed method comprehensively. The scale is recommended to be scaled up to at least a hundred to thousand level.
2.  This study has mainly focused on evaluating the impact of diffusion model customization and only focuses on evaluating two very representative but not new (DB was proposed in 2022) works. The results would be more convincing if the authors evaluated their methods on more SOTA methods. For instance, as the author mentioned in the related works:

[1] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff:
Compact parameter space for diffusion fine-tuning. ICCV, 2023b.

[2] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W
Cohen. Subject-driven text-to-image generation via apprenticeship learning. arXiv preprint
arXiv:2304.00186, 2023b.

[3] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image
generation without test-time finetuning. arXiv preprint arXiv:2304.03411, 2023a.
3. The results in Table 1 and Table 2 are not significant enough.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. The paper successfully highlights the issue of open-world forgetting, which is less studied compared to closed-world scenarios, providing a valuable framework for further research in model customization.
2. The use of zero-shot classification to quantify semantic drift and the introduction of novel metrics for appearance drift offer robust tools for understanding model behavior post-customization.
summary: This paper presents a study on open-world forgetting during the customization of generative image models, specifically focusing on diffusion models like Stable Diffusion. The authors define open-world forgetting as the unintended changes in model behavior when adapted to new classes or content, distinguishing it from the more limited scope of closed-world forgetting. They explore how fine-tuning models with even small sets of data  can lead to significant drifts in both the semantic understanding and appearance of images generated by the model. Key methodologies discussed include using zero-shot classification to measure semantic drift and assessing changes in appearance through color and texture analysis. The paper also introduces a mitigation strategy through functional regularization to minimize these drifts while incorporating new concepts.
weaknesses: 1. While the study provides detailed insights into open-world forgetting, the generalizability of the findings across different types of generative models or broader sets of data remains unclear.
2. The mitigation strategy's effectiveness is potentially limited by the quality of the data used for fine-tuning and the subjective nature of assessing image quality and drift.
3. How does the model perform when fine-tuned with highly diverse or noisy datasets? Does the regularization technique maintain its effectiveness across such variations?
4. What is the long-term stability of models customized using these techniques? Do they continue to maintain reduced levels of forgetting with continued use or additional rounds of customization?
5. How do quantitative assessments of drift align with qualitative evaluations from human observers? Is there a significant disparity, and how can it be addressed?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the above Weaknesses.
soundness: 3
strengths: 1.The paper is well-written and easily comprehensible.

2.The proposed method is simple and straightforward.

3.The motivation for addressing both Appearance Drift and Semantic Drift is commendable.
summary: This paper delves into open-world forgetting in diffusion models,. Unveiling semantic and appearance drift, this paper proposes functional regularization to maintain original capabilities. Besides, this paper supports studying open-world forgetting and offers insights for stronger adaptation methods.
weaknesses: 1. This paper solely analyzes Appearance Drift and Semantic Drift in the context of OPEN-WORLD FORGETTING, leaving uncertainties about their occurrence in closed-world scenarios. 

2. The analysis in this paper focuses on two methods, Dreambooth and CustomDiffusion, overlooking recent approaches like BLIP-Diffusion [1] and HyperDreamBooth [2]. Do Appearance Drift and Semantic Drift commonly appear in these diffusion-based methods?

3. While this paper claims to mitigate Appearance Drift and Semantic Drift using a mitigation strategy, the relevant qualitative results are missing, and the effectiveness needs to be intuitively demonstrated.

4. The related work section requires updating. For instance, [2] and [3] in relation to this paper should be discussed and analyzed.

[1] BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing. In Neurips2023 (pp.30146-30166).

[2] Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. InCVPR2024 (pp. 6527-6536. 2024).

[3] Customization assistant for text-to-image generation. InCVPR2024 (pp. 9182-9191).

Meta-review: Summary: This paper investigates "open-world forgetting," a phenomenon observed during the customization of pre-trained generative image models, particularly diffusion models like Stable Diffusion. Open-world forgetting refers to unintended degradations in a model's performance on previously learned concepts when fine-tuned for new classes, contrasting with the narrower scope of closed-world forgetting. The study introduces two key evaluation metrics: zero-shot classification for measuring semantic drift and the "Color Drift Index" (CDI) for quantifying appearance drift through changes in visual attributes such as color and texture. To mitigate these challenges, the paper proposes a functional regularization technique that enables models to incorporate new concepts while preserving prior knowledge. Experimental results validate the effectiveness of this method in reducing semantic and appearance drift, enhancing the stability and reliability of customized models.

Strengths and Weaknesses: 

The reviewers generally acknowledge the significance of introducing open-world forgetting in diffusion model customization, along with establishing evaluation metrics and proposing a simple yet effective mitigation strategy, highlighting its contribution to addressing a critical gap in understanding the broader impacts of fine-tuning on model knowledge. However, they express concerns regarding the study's limited contributions and the lack of a convincing evaluation.

In particular, concerns are raised about the study’s applicability to broader datasets, different types of generative models, and more diverse or noisy datasets. The analysis is limited to a small subset of 10 concepts, raising questions about the comprehensiveness of the findings. The evaluation does not incorporate more recent approaches, leaving uncertainties about the generalizability of the method across diverse concepts and models. Additionally, while the paper focuses on how fine-tuning degrades original model capabilities, it does not adequately evaluate the proposed mitigation strategy's impact on fine-tuning task performance. Furthermore, the alignment between quantitative drift assessments and qualitative human evaluations also remain unclear. The paper does not explore whether the identified drifts are exclusive to open-world forgetting or if they also occur in closed-world scenarios.

During the discussion phase, the authors presented a scaled evaluation with a larger set of concepts, demonstrated the applicability of the proposed method to newer approaches with similar improvements, and supplemented their findings with a user study and additional visualizations. However, due to time constraints, the authors were unable to provide complete evaluation results across all datasets. The reviewers remained unconvinced, citing the perceived limited contributions of this work (Reviewer 698f). Additionally, Reviewer 5xU2 raised concerns about the reproducibility of the study, noting discrepancies in DreamBooth's results compared to typical behavior and inferior visualization results in the revised version (Figures 8 and 9) compared to the original submission.

The reviewers and ACs have discussed the paper. Three out of four reviewers suggested boardline acceptance, albeit with some remaining concerns. On the contrary, Reviewer 5xU2 expressed strong reservations. ACs, considering the compelling arguments from Reviewer Yt1v and remaining concerns from other reviewers (Reviewer 698f), lean towards the view that the authors need to conduct more comprehensive evaluations. It is recommended that the authors perform thorough and large-scale evaluations, especially using advanced diffusion models. Additionally, providing a clearer analysis of the results and enhancing missing details would strengthen the paper. Therefore, the paper is not ready for this ICLR. ACs encourage the authors to continue this line of work for future submission.

Decision: Reject