Title: Leveraging Catastrophic Forgetting to Develop Safe Diffusion Models against Malicious Finetuning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Model

AI Safety
Abstract: Diffusion models (DMs) have demonstrated remarkable proficiency in producing images based on textual prompts. Numerous methods have been proposed to ensure these models generate safe images. Early methods attempt to incorporate safety filters into models to mitigate the risk of generating harmful images but such external filters do not inherently detoxify the model and can be easily bypassed. Hence, model unlearning and data cleaning are the most essential methods for maintaining the safety of models, given their impact on model parameters.
However, malicious fine-tuning can still make models prone to generating harmful or undesirable images even with these methods.
Inspired by the phenomenon of catastrophic forgetting, we propose a training policy using contrastive learning to increase the latent space distance between clean and harmful data distribution, thereby protecting models from being fine-tuned to generate harmful images due to forgetting.
The experimental results demonstrate that our methods not only maintain clean image generation capabilities before malicious fine-tuning but also effectively prevent DMs from producing harmful images after malicious fine-tuning. Our method can also be combined with other safety methods to maintain their safety against malicious fine-tuning further.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the possibility of preventing the T2I models from malicious fine-tuning attacks. The authors draw their inspiration from contrastive learning and propose two ways of separating the safe distribution from the harmful distribution in the latent space of the T2I models (LT, NG). The authors have provided quantitative and qualitative results to verify the effectiveness of the algorithm.\n\nSTRENGTHS:\n(1) The proposed algorithm is well-motivated and clearly illustrated. I find the manuscript easy to follow.\n\n(2) Based on the visual results (Figure 1 and Figure 4), the proposed algorithm seems to be very effective.\n\n(3) The authors conduct experiments on two models, several datasets, and various scenarios verifying the effectiveness of the algorithm.\n\nWEAKNESSES:\nMajor:\n(1) My main concern about the proposed algorithm is whether a universally safe model can be achieved, i.e., a diffusion model that is unable to generate any images related to nudity, violence, politics, discrimination... Currently, the authors have only experimented with eliminating a single concept (sexual, violence). I am doubtful that the problem can be solved by simply gathering all the unsafe images into the harmful set and conducting the same training pipeline.\n\n(2) The numerical results fail to reveal a significant improvement, especially when it comes to the Aesthetic Score and the CLIP Score. Besides, the algorithm performs notably worse when removing the violence concept. I am also a little bit skeptical about the three human judges. Three judges is simply not enough for me for human evaluation and are the three people authors of this paper?\n\n(3) Appendix A is not clearly organized and no discussion is made. I cannot draw much information from the two tables listed in Appendix A.\n\nMinor:\n\n(1) Figure 3 is of low quality. Are you using the JPEG image directly?\n\n(2) I encourage the authors to display visual results covering more scenarios. For example, I would like to see whether the model modified by LT or NG can generate safe human images normally (like a portrait of a lady with cloth).\n\nI would like to adjust my score if the authors can clarify my concerns.\n\nQUESTIONS:\n(1) I would like to see whether the model modified by LT or NG can generate safe human images normally (like a portrait of a lady with cloth).\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of ensuring the safety of generative models (here text-to-image diffusion models) against malicious fine-tuning as well as the erasure of undesired concepts and capabilities. To this end, the proposed approach leverages catastrophic forgetting through contrastive learning The authors demonstrate the effectiveness of their method via experiments on erasing potential harmful capabilities (generating images displaying nudity and violence) and securing the model from malicious fine-tuning. While providing evidence based on text-to-image models, the paper also highlights the universality of the method.\n\nSTRENGTHS:\n- The paper tackles a significant issue in the field of generative models, extending beyond simple concept erasure to securing models from being maliciously fine-tuned. This is crucial for preventing the misuse of powerful open-source generative models. \n- The integration of contrastive learning with diffusion models is innovative and well-motivated.\n\nWEAKNESSES:\n- The soundness of the paper is undermined by several issues in the mathematical formulation. Specifically, Equations 4 and 5 contain undefined terms such as R, b, and alpha, making it difficult to fully understand and verify the proposed method. Further, the writing in many paragraphs is unclear, leading to difficulties in comprehending the methodology and results (e.g. see lines 246, 191). In the tables, the usage of bold values is inconsistent (sometimes missing, sometimes wrong (e.g. Table 2, the value 0.4421 should be bold instead of 0.4441)) \n- The use of the NSFW score as a metric is problematic, as it is not well-suited for evaluating the presents of violence, and there are more reliable alternatives, such as the Q16 classifier [Schramowski et al.]. Further, more reliable alternatives to classifying the display of nudity in images exist (Nudenet by [Praneet]). \n- Furthermore, the safety improvements shown in the experiments are only marginal, and there is a lack of comparison with other methods. \n\n\n\n[Schramowski et al.] Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? In FAccT, 2022\n\n[Praneet] Nudenet: Neural nets for nudity classification, detection and selective censorin, 2019\n\nQUESTIONS:\n- The term “unlearning model” is mentioned at line 168. Could you clarify what an unlearning model is and how their method applies it as a base model?\n- More details about the participants of the study for the Human Eval metric are needed. Could you provide this information to better understand the context and validity of these evaluations?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper, inspired by the phenomenon of catastrophic forgetting, proposes a training policy using contrastive learning to increase the latent space distance between clean and harmful data distribution, thereby protecting models from being fine-tuned to generate harmful images due to forgetting.\nTwo main steps for the method: 1)  transforming the latent variable distribution of images, 2) adding different noises to clean and harmful images to induce different changes in the distribution of images\nExperiments demonstrate that using the proposed method to fine-tune the SD model significantly improves its safety and prevents it from being maliciously fine-tuned.\n\nSTRENGTHS:\n- I like the idea of latent space manipulation for distancing harmful and clean image space, which make sense for better safety both for malicious prompt, safety detection, and malicious fine-tuning, since maximizing this distribution distance leads to catastrophic forgetting when the model is fine-tuned on harmful data.\n- The paper is well-written and easy to follow. The explanations are clear, and the methodology is presented in a step-by-step manner, making the complex concepts accessible to a broad audience.\n- The experiments showed that the proposed methods significantly improve the safety of diffusion models. The models trained with these methods exhibited resistance to generating harmful images even after malicious fine-tuning. Additional experiments demonstrated the robustness and universality of the proposed methods across different datasets and types of images.\n\nWEAKNESSES:\nI do not see major flaws in the paper.\nI would like to suggest the authors to discuss more on the performance trade-off. I understand that the authors focus on the safety part of the model, but I'm also curious beyond the CLIP score, what is the influence of your method on the quality of normal images? \nBeyond the CLIP score, it would be beneficial to include additional metrics or evaluations that assess the quality of normal images generated by the models. \nAnd would it also be possible for Hum. Eval for the quality difference? That would be quite convincing on the influence on the quality.\n\nQUESTIONS:\nSee weakenss.\n\nLIMITATIONS:\nThe authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

comments: SUMMARY:
This paper considers a scenario where malicious entities want to train a diffusion model for harmful content generation. To prevent the model from being finetuned to generate harmful content, this paper proposes to leverage the catastrophic forgetting mechanism to counteract the harmful finetuning. To trigger catastrophic forgetting, the authors proposed increasing the distance between the distributions of clean and harmful data using two methods: latent transformation and noise guidance.

STRENGTHS:
The attack scenario defined in this work is very relevant to real threats, as many diffusion models are fine-tuned to produce harmful content. A way to prevent diffusion models from malicious finetuning can potentially have a huge impact. 

The method proposed by the author that leverages catastrophic forgetting to produce a positive outcome is novel.

Evaluation results show the promise of this method.

WEAKNESSES:
The work does not show the performance degradation of clean images after harmful finetuning. For instance, the FID score can be included.
Results can be enhanced if the work includes more recent models, such as SD-XL or DiT.

The limitation section is not included in this paper.

minor: typo in L52 (constructive learning).

QUESTIONS:
I have one potential question related to the limitation of this method. What if the malicious entity also applies a different finetuning that tries to bring the latent space between the clean and harmful data closer, then performs standard finetuning?

LIMITATIONS:
Not included in this work. Suggestion in Questions section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-10