Assessing Open-world Forgetting in Generative Image Model Customization

Authors: Héctor Laria, Alex Gomez-Villa, Imad Eddine MAROUF, Kai Wang, Bogdan Raducanu, Joost van de Weijer
Primary area: generative models
Keywords: generative models, diffusion, open-world forgetting, semantic drift, appearance drift

Abstract:
Recent advances in diffusion models have significantly enhanced image generation capabilities. However, customizing these models with new classes often leads to unintended consequences that compromise their reliability. We introduce the concept of *open-world forgetting* to emphasize the vast scope of these unintended alterations, contrasting it with the well-studied *closed-world forgetting*, which is measurable by evaluating performance on a limited set of classes or skills.
Our research presents the first comprehensive investigation into open-world forgetting in diffusion models, focusing on semantic and appearance drift of representations. We utilize zero-shot classification to analyze semantic drift, revealing that even minor model adaptations lead to unpredictable shifts affecting areas far beyond newly introduced concepts, with dramatic drops in zero-shot classification of up to 60\%. Additionally, we observe significant changes in texture and color of generated content when analyzing appearance drift.
To address these issues, we propose a mitigation strategy based on functional regularization, designed to preserve original capabilities while accommodating new concepts. Our study aims to raise awareness of unintended changes due to model customization and advocates for the analysis of open-world forgetting in future research on model customization and finetuning methods. Furthermore, we provide insights for developing more robust adaptation methodologies.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730268999600

thread_type:
official_review

<thread_object>
creator: reviewer_698f

<content>
summary:
This paper delves into open-world forgetting in diffusion models,. Unveiling semantic and appearance drift, this paper proposes functional regularization to maintain original capabilities. Besides, this paper supports studying open-world forgetting and offers insights for stronger adaptation methods.

strengths:
1.The paper is well-written and easily comprehensible.

2.The proposed method is simple and straightforward.

3.The motivation for addressing both Appearance Drift and Semantic Drift is commendable.

weaknesses:
1. This paper solely analyzes Appearance Drift and Semantic Drift in the context of OPEN-WORLD FORGETTING, leaving uncertainties about their occurrence in closed-world scenarios. 

2. The analysis in this paper focuses on two methods, Dreambooth and CustomDiffusion, overlooking recent approaches like BLIP-Diffusion [1] and HyperDreamBooth [2]. Do Appearance Drift and Semantic Drift commonly appear in these diffusion-based methods?

3. While this paper claims to mitigate Appearance Drift and Semantic Drift using a mitigation strategy, the relevant qualitative results are missing, and the effectiveness needs to be intuitively demonstrated.

4. The related work section requires updating. For instance, [2] and [3] in relation to this paper should be discussed and analyzed.

[1] BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing. In Neurips2023 (pp.30146-30166).

[2] Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. InCVPR2024 (pp. 6527-6536. 2024).

[3] Customization assistant for text-to-image generation. InCVPR2024 (pp. 9182-9191).

questions:
Please see the above Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730482856938

thread_type:
official_review

<thread_object>
creator: reviewer_ab8e

<content>
summary:
This paper presents a study on open-world forgetting during the customization of generative image models, specifically focusing on diffusion models like Stable Diffusion. The authors define open-world forgetting as the unintended changes in model behavior when adapted to new classes or content, distinguishing it from the more limited scope of closed-world forgetting. They explore how fine-tuning models with even small sets of data  can lead to significant drifts in both the semantic understanding and appearance of images generated by the model. Key methodologies discussed include using zero-shot classification to measure semantic drift and assessing changes in appearance through color and texture analysis. The paper also introduces a mitigation strategy through functional regularization to minimize these drifts while incorporating new concepts.

strengths:
1. The paper successfully highlights the issue of open-world forgetting, which is less studied compared to closed-world scenarios, providing a valuable framework for further research in model customization.
2. The use of zero-shot classification to quantify semantic drift and the introduction of novel metrics for appearance drift offer robust tools for understanding model behavior post-customization.

weaknesses:
1. While the study provides detailed insights into open-world forgetting, the generalizability of the findings across different types of generative models or broader sets of data remains unclear.
2. The mitigation strategy's effectiveness is potentially limited by the quality of the data used for fine-tuning and the subjective nature of assessing image quality and drift.
3. How does the model perform when fine-tuned with highly diverse or noisy datasets? Does the regularization technique maintain its effectiveness across such variations?
4. What is the long-term stability of models customized using these techniques? Do they continue to maintain reduced levels of forgetting with continued use or additional rounds of customization?
5. How do quantitative assessments of drift align with qualitative evaluations from human observers? Is there a significant disparity, and how can it be addressed?

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527854264

thread_type:
official_review

<thread_object>
creator: reviewer_khxh

<content>
summary:
This paper introduces the concept of "open-world forgetting" to emphasize the vast scope of these unintended alterations during the model customization of foundation generative models (i.e., Stable Diffusion). The authors conduct a comprehensive investigation of the semantic and appearance drift of representations. To address these issues, this paper proposes a mitigation strategy based on functional regularization which preserves original capabilities while accommodating new concepts. The contribution of the paper can be summarized as follows:

1. $\textbf{Introduction of Open-World Forgetting Concept:}$ The paper defines open-world forgetting in pre-trained foundation generative image models and systematically assesses its impact, focusing on how models lose previously learned concepts after adaptation.

2. $\textbf{Semantic and Appearance Drift Analysis:}$ It introduces two evaluation approaches:
Semantic Drift: Uses zero-shot classification to measure changes in a model's semantic understanding.
Appearance Drift: Examines shifts in visual attributes like color and texture after customization, introducing the "Color Drift Index" (CDI) as a metric for quantifying these changes.

3. $\textbf{Mitigation Strategy:}$ The paper proposes a functional regularization technique aimed at reducing open-world forgetting, enabling models to integrate new concepts while retaining prior knowledge. This method proved effective in maintaining both semantic and appearance integrity in fine-tuned models.

strengths:
1. The concept of open-world forgetting represents a novel extension to the field of model adaptation and customization, particularly in generative models. While catastrophic forgetting has been explored in closed-world contexts, applying it to open-world scenarios in text-to-image models is original and fills an important gap in understanding how fine-tuning affects broad model knowledge.
2. The proposed Color Drift Index (CDI) for assessing appearance drift is a new and creative metric tailored to the nuances of generative models. This provides a fresh approach to evaluating visual consistency, which goes beyond typical performance metrics in generative model research.
3. By providing quantitative results with the CDI and zero-shot classification performance across multiple models and concepts, the authors present strong empirical evidence for their findings, which enhances the reliability of their conclusions.
4. The use of functional regularization as a mitigation strategy is well-executed and shows practical effectiveness, backed by solid data on how it mitigates both semantic and appearance drift.
5. By proposing methods to measure and mitigate unintended model alterations, the work establishes a foundation for future research on safeguarding foundational knowledge in generative models, which will inspire other researchers to work in this direction in the future.
6. The proposed method is simple yet effective and the writing of the paper is clear and easy to follow.

weaknesses:
The authors have mentioned some of the limitations in the paper:
1. The analysis is limited to a small subset of concepts and may not capture all potential instances of forgetting. 10 concepts are apparently not enough to reflect the effectiveness of the proposed method comprehensively. The scale is recommended to be scaled up to at least a hundred to thousand level.
2.  This study has mainly focused on evaluating the impact of diffusion model customization and only focuses on evaluating two very representative but not new (DB was proposed in 2022) works. The results would be more convincing if the authors evaluated their methods on more SOTA methods. For instance, as the author mentioned in the related works:

[1] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff:
Compact parameter space for diffusion fine-tuning. ICCV, 2023b.

[2] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W
Cohen. Subject-driven text-to-image generation via apprenticeship learning. arXiv preprint
arXiv:2304.00186, 2023b.

[3] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image
generation without test-time finetuning. arXiv preprint arXiv:2304.03411, 2023a.
3. The results in Table 1 and Table 2 are not significant enough.

questions:
1. Is the customization method used in Figure 3 DB? Please clarify it in the revised version,
2. I recommend the author to increase the number of concepts in the evaluation. 10 concepts might be a "cherry pick".
3. Add more images generated by the model before and after applying your methods which would be intuitive and easy to understand.

Overall, I believe this paper is valuable to the community and I would be happy to raise the score if my concern can be well addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730583188838

thread_type:
official_review

<thread_object>
creator: reviewer_5xu2

<content>
summary:
This paper investigates the issue of “open-world forgetting” in generative image model customization, highlighting how fine-tuning models for new classes can inadvertently degrade performance on previously learned classes. The authors systematically analyze open-world forgetting through the introduction of semantic and appearance drift concepts, using zero-shot classification to measure semantic drift. To mitigate this issue, they propose a “drift correction” method, designed to reduce the model's tendency to forget prior knowledge when learning new concepts. Experiments demonstrate that drift correction significantly reduces semantic and appearance drift, enhancing the stability and reliability of customized models.

strengths:
The introduction of open-world forgetting is a novel perspective in the domain of diffusion model customization. While previous research often focuses on task-specific performance, this paper broadens the scope of forgetfulness evaluation, bringing strong innovation.

weaknesses:
1. The paper’s structure is confusing. For instance, lines 513-532 discuss contributions to open-world forgetting before defining it clearly, and these lines largely repeat lines 108-119, adding unnecessary redundancy.

2. Key metrics lack basic explanation in the main text, with details only in supplementary materials. This makes it hard for readers to grasp the evaluation approach without prior knowledge.

3. The paper focuses on how fine-tuning degrades original model capabilities but doesn’t assess how the proposed method impacts performance on the fine-tuning tasks.

questions:
1. Were alternative metrics for measuring semantic drift considered? For instance, could user evaluations on the semantic consistency of generated images be feasible?

2. Has the drift correction method been tested on more complex or diverse concepts beyond the simple categories presented? For example, in Figure 2a, “a sampled pair from the most dissimilar outputs (purple triangle) shows a complete change in content, colors, and scene composition that no longer matches the prompt.”, What happens if drift correction is applied in this case, do the colors and scene composition match the prompts better?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732536729923

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General answers to reviewers 1/3

comment:
# General answer
We thank the reviewers for their positive and thoughtful feedback. The reviewers highlight our main contribution of introducing open-world forgetting as a novel extension to generative model adaptation (KHxH, AB8E, 5xU2), establishing a foundation for future research on safeguarding foundational knowledge (KHxH). They commend our methodology, particularly the Color Drift Index and zero-shot classification as robust tools for evaluating model behavior (KHxH, AB8E). The proposed functional regularization technique is praised for being simple yet effective in mitigating both semantic and appearance drift (KHxH, 698f), and the paper's clear presentation is appreciated (KHxH, 698f).

During the rebuttal period, we have substantially expanded our evaluation to address the main reviewer concerns:

1. **Scale of evaluation (5xU2, KHxH, AB8E)**: We want to emphasize the computational requirements for the computation of Table 1 are significant (e.g., zero-shot evaluation on 100 concepts for a single model and single (average-size) dataset takes around 2.5 days on 8x Nvidia 6000 Ada GPUs), therefore we could only include results on Pets dataset as of today. Similarly, to generate the 10s of thousands of images required to compute FID, KID, CDI of Table 8 and Figures 4, 5c, would take nearly two weeks on the same hardware.  
We have now extended the evaluation to all 101 concepts of the CustomConcept101 dataset [1]. These include concepts gathered from the internet across a variety of categories, like *toys, plushies, wearables, scenes, transport vehicles, furniture, home decor items, luggage, human faces, musical instruments, rare flowers, food items* or *pet animals*. 

| Method     | Pets $\uparrow$ (10 concepts)                      | Pets $\uparrow$ (101 concepts)                     |
|------------|-----------------------------------------|-----------------------------------------|
| Base Model | 86.87                                   |  86.87                                 |
| DB         | 82.61${\scriptsize \pm 3.33 \ (36.43)}$ | 82.87${\scriptsize \pm 2.80 \ (34.44)}$ |
| DB-DC      | 86.64${\scriptsize \pm 0.92 \ (17.14)}$ | 86.58${\scriptsize \pm 1.00 \ (16.55)}$ |
| CD         | 83.46${\scriptsize \pm 2.97 \ (33.57)}$ | 83.00${\scriptsize \pm 2.92 \ (35.55)}$ |
| CD-DC      | 86.37${\scriptsize \pm 1.28 \ (16.43)}$ | 86.18${\scriptsize \pm 1.20 \ (19.83)}$ |

The results from this expanded evaluation strongly validate our initial findings. The performance metrics remain remarkably consistent with our original Table 1 based on 10 concepts, showing a maximum change in performance of only 0.5%. This consistency demonstrates that representation drift occurs systematically, regardless of the concept type, and that our proposed drift compensation method effectively mitigates this effect across all categories. **We will include complete evaluation results across all datasets in the final version of the paper**.

|       | DINO $\uparrow$ (10)           | CLIP-I $\uparrow$ (10)         | CLIP-T $\uparrow$ (10)         | DINO $\uparrow$ (101)          | CLIP-I $\uparrow$ (101)        | CLIP-T $\uparrow$ (101)        |
|-------|--------------------------------|--------------------------------|--------------------------------|--------------------------------|--------------------------------|--------------------------------|
| DB    | 0.4241${\scriptsize \pm 0.1503}$ | 0.6764${\scriptsize \pm 0.1046}$ | 0.7896${\scriptsize \pm 0.0296}$ | 0.3926${\scriptsize \pm 0.1147}$ | 0.6746${\scriptsize \pm 0.0672}$ | 0.7641${\scriptsize \pm 0.0405}$ |
| DB-DC | 0.4283${\scriptsize \pm 0.1584}$ | 0.6817${\scriptsize \pm 0.1097}$ | 0.7799${\scriptsize \pm 0.0324}$ | 0.3913${\scriptsize \pm 0.1153}$ | 0.6739${\scriptsize \pm 0.0664}$ | 0.7570${\scriptsize \pm 0.0422}$ |
| CD    | 0.4422${\scriptsize \pm 0.1378}$ | 0.6934${\scriptsize \pm 0.0902}$ | 0.7916${\scriptsize \pm 0.0266}$ | 0.3899${\scriptsize \pm 0.1132}$ | 0.6763${\scriptsize \pm 0.0661}$ | 0.7759${\scriptsize \pm 0.0431}$ |
| CD-DC | 0.4381${\scriptsize \pm 0.1380}$ | 0.6925${\scriptsize \pm 0.0888}$ | 0.7899${\scriptsize \pm 0.0280}$ | 0.3753${\scriptsize \pm 0.1164}$ | 0.6675${\scriptsize \pm 0.0665}$ | 0.7760${\scriptsize \pm 0.0457}$ |
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: This paper investigates "open-world forgetting," a phenomenon observed during the customization of pre-trained generative image models, particularly diffusion models like Stable Diffusion. Open-world forgetting refers to unintended degradations in a model's performance on previously learned concepts when fine-tuned for new classes, contrasting with the narrower scope of closed-world forgetting. The study introduces two key evaluation metrics: zero-shot classification for measuring semantic drift and the "Color Drift Index" (CDI) for quantifying appearance drift through changes in visual attributes such as color and texture. To mitigate these challenges, the paper proposes a functional regularization technique that enables models to incorporate new concepts while preserving prior knowledge. Experimental results validate the effectiveness of this method in reducing semantic and appearance drift, enhancing the stability and reliability of customized models.

Strengths and Weaknesses: 

The reviewers generally acknowledge the significance of introducing open-world forgetting in diffusion model customization, along with establishing evaluation metrics and proposing a simple yet effective mitigation strategy, highlighting its contribution to addressing a critical gap in understanding the broader impacts of fine-tuning on model knowledge. However, they express concerns regarding the study's limited contributions and the lack of a convincing evaluation.

In particular, concerns are raised about the study’s applicability to broader datasets, different types of generative models, and more diverse or noisy datasets. The analysis is limited to a small subset of 10 concepts, raising questions about the comprehensiveness of the findings. The evaluation does not incorporate more recent approaches, leaving uncertainties about the generalizability of the method across diverse concepts and models. Additionally, while the paper focuses on how fine-tuning degrades original model capabilities, it does not adequately evaluate the proposed mitigation strategy's impact on fine-tuning task performance. Furthermore, the alignment between quantitative drift assessments and qualitative human evaluations also remain unclear. The paper does not explore whether the identified drifts are exclusive to open-world forgetting or if they also occur in closed-world scenarios.

During the discussion phase, the authors presented a scaled evaluation with a larger set of concepts, demonstrated the applicability of the proposed method to newer approaches with similar improvements, and supplemented their findings with a user study and additional visualizations. However, due to time constraints, the authors were unable to provide complete evaluation results across all datasets. The reviewers remained unconvinced, citing the perceived limited contributions of this work (Reviewer 698f). Additionally, Reviewer 5xU2 raised concerns about the reproducibility of the study, noting discrepancies in DreamBooth's results compared to typical behavior and inferior visualization results in the revised version (Figures 8 and 9) compared to the original submission.

The reviewers and ACs have discussed the paper. Three out of four reviewers suggested boardline acceptance, albeit with some remaining concerns. On the contrary, Reviewer 5xU2 expressed strong reservations. ACs, considering the compelling arguments from Reviewer Yt1v and remaining concerns from other reviewers (Reviewer 698f), lean towards the view that the authors need to conduct more comprehensive evaluations. It is recommended that the authors perform thorough and large-scale evaluations, especially using advanced diffusion models. Additionally, providing a clearer analysis of the results and enhancing missing details would strengthen the paper. Therefore, the paper is not ready for this ICLR. ACs encourage the authors to continue this line of work for future submission.