Direct Unlearning Optimization for Robust and Safe Text-to-Image Models

Authors: yong-hyun  park
sangdoo  yun
jin-hwa  kim
junho  kim
geonhui  jang
yonghyun  jeong
junghyo  jo
gayoung  lee
spm  uce esd
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 14:47:09

Abstract: Recent advancements in text-to-image (T2I) models have greatly benefited from large-scale datasets, but they also pose significant risks due to the potential generation of unsafe content. To mitigate this issue, researchers proposed unlearning techniques that attempt to induce the model to unlearn potentially harmful prompts. However, these methods are easily bypassed by adversarial attacks, making them unreliable for ensuring the safety of generated images. In this paper, we propose Direct Unlearning Optimization (DUO), a novel framework for removing NSFW content from T2I models while preserving their performance on unrelated topics. DUO employs a preference optimization approach using curated paired image data, ensuring that the model learns to remove unsafe visual concepts while retain unrelated features. Furthermore, we introduce an output-preserving regularization term to maintain the model's generative capabilities on safe content. Extensive experiments demonstrate that DUO can robustly defend against various state-of-the-art red teaming methods without significant performance degradation on unrelated topics, as measured by FID and CLIP scores. Our work contributes to the development of safer and more reliable T2I models, paving the way for their responsible deployment in both closed-source and open-source scenarios.

Main paper: 
 introduction: 
 In recent years, text-to-image (T2I) models [22, 47, 45, 40, 8, 7] have experienced significant advancements thanks to large-scale data and diffusion models. However, the large-scale web-crawled data, e.g., LAION [51] often include a significant amount of inappropriate and objectionable material, so T2I models trained on such data may pose a potential risk of generating harmful content such as Not Safe For Work (NSFW) content, copyright infringement, and a violation of privacy [46, 4] . Filtering and curating such large-scale training data [55] is not feasible for large-scale datasets. At the production level, service providers usually block inappropriately generated images with ad-hoc classifier [55] , but attackers can easily bypass the classifier by accessing the model directly.Natural unsafe prompts (e.g., "nudity", "naked") Unsafe Safe Image Space Unsafe Safe Safe prompts (e.g., "dressed", "clothed") Adversarial prompts (e.g., "zkj<(#*nud@>", soft embs)Figure 1 : Image-based unlearning. Prompt-based unlearning can be easily circumvented with adversarial prompt attack. On the other hand, image-based unlearning robustly produces safe images regardless of the given prompt. We use for publication purposes.To address the generation of unsafe content, there has been a substantial amount of research focusing on directly fine-tuning diffusion models to forget unsafe concepts [12, 13, 26, 33, 17, 36, 63] .Recent work in the field has primarily focused on prompt-based approaches to unlearning, which aim to induce models to behave as if they had been provided with safe prompts when an unsafe prompt is given [12, 26, 33, 63, 36, 13] . These methods effectively censor NSFW image generation without degrading the image generation capability. However, recent studies [42, 56, 60] warn that by introducing slight perturbations to the unsafe prompts, it is possible to circumvent these censoring mechanisms and generate undesirable images (Figure 1 ).Current unlearning methods are vulnerable to adversarial attacks because it is difficult to cover all NSFW concepts using prompts alone. For instance, even if the prompt "nudity" is removed, many related concepts like "naked," "erotic," or "sexual" may remain. Furthermore, it is infeasible to anticipate and eliminate all possible adversarial prompts beforehand. Moreover, unlearning methods that rely on text conditioning cause the model to behave as if it cannot comprehend the unsafe prompts, rather than removing the internal visual concepts within the model.To fundamentally address limitations, it is necessary to guide the model to remove unsafe visual features regardless of the prompt. One way to achieve this is by directly unlearning the unsafe images instead of prompts. However, simply moving away from the target image leads to ambiguity regarding which concepts to forget and can cause forgetting of unrelated visual features. For instance, if we make the model unlearn the image in Figure 2-(a) , it would lose the ability to generate not only nudity but also unrelated concepts like women or forests.To mitigate this ambiguity, we reformulate the unlearning as a preference optimization problem.Preference optimization is a technique aimed at training models to generate desirable outputs when provided with both positive and negative examples. Our main intuition is that, unlike when only an undesirable sample is given, the presence of a desirable sample helps the model resolve the ambiguity about what information should be retained from the undesirable sample. As shown in Figure 2-(b) , thanks to the desirable image, the model can recognize that the forest and woman are not subjects that need to be erased.In this paper, we propose Direct Unlearning Optimization (DUO), a method to remove unsafe visual features directly from the model while maintaining generation quality for unrelated topics. Specifically, we created paired data by using unsafe images and their counterparts, from which unsafe concepts were removed using the SDEdit [34] . Then, we employ Direct Preference Optimization (DPO) [44] to make the model favor the latter group while encouraging it to move away from the former group. As shown in Figure 2 -(b), preference optimization through a paired dataset explicitly specifies which features of a given image should be forgotten. To further regularize the model not to unlearn for unrelated topics, we introduce output-preserving regularization [37, 62, 6] that forces to preserve the denoising capability of the model. We call our framework as Direct Unlearning Optimization (DUO). We demonstrate that our method can robustly defend against various state-ofthe-art red teaming methods [56, 42] without significant performance degradation in unrelated topics, as measured by LPIPS [65] , FID [19] and CLIP scores [43] . Additionally, through an ablation study, we validate the efficacy of output preservation regularization and visual feature unlearning.  
related work: 
 Text-to-Image (T2I) models with Safety Mechanisms. The safety mechanisms designed to prevent undesirable behavior in T2I models can be classified into four categories: dataset filtering [55, 15] , concept unlearning [12, 13, 26, 33, 17, 36, 63] , in-generation guidance [50] , and post-generation content screening [1, 55, 3] . These methods are orthogonal to each other, and using them together can make the T2I generation system safer.Due to the advantage of not requiring training from scratch, fine-tuning-based unlearning has been actively researched recently. Pioneering works like ESD [12] and CA [26] train the model to generate the same noise regardless of whether an unsafe prompt is present or not. SPM [33] trains a onedimensional regulator instead of full model training. TIME [36] , UCE [13] and MACE [31] update prompt-dependent cross-attention model weights with closed-form equation. Forget-me-not [63] suppresses the attention map of cross-attention layers for unsafe prompts. SA [17] proposed an unlearning method based on continual learning, utilizing Elastic Weight Consolidation (EWC). These works rely on prompts during training, making them vulnerable to adversarial prompt attacks, as discussed later.Meanwhile, SafeGen [29] , as a concurrent work with ours, addresses similar concerns regarding adversarial prompts and utilizing only images for training. They use blurred images and guide a model using supervised learning. In contrast to their work, we utilize paired data generated with SDEdit [34] to enable the model to erase only unsafe concepts and employ preference optimization. Therefore, we have the advantage of not requiring separate benign data and producing erased results more closely resemble real images.Red-Teaming for T2I models. Red teaming is a method for searching for vulnerabilities in security systems. Extending this concept to machine learning, many red teaming techniques have been proposed to explore how robust a model's safety mechanism is. For Text-to-Image (T2I) models, studies have shown that even with safety mechanisms applied, it is still possible to generate harmful images through prompt engineering [56, 42, 60, 9, 67, 16] . Ring-A-bell [56] and SneakyPrompt [60] generate adversarial prompts in a black box scenario without access to the denoising model. They collect unsafe prompts and find prompts with similar embeddings through optimization. On the other hand, Concept Inversion [42] is a method that can be used in a white box scenario where access to the gradient of the denoising model is available. This method learns a special token ⟨c⟩ through textual inversion that contains information about unsafe images, and then uses it to generate unsafe images. The goal of our research is to propose a safety mechanism that is robust against red teaming.Preference optimization in T2I models. Direct Preference Optimization (DPO) [44] has been frequently used in language models as it enables preference-based model tuning without the need for reward models. DiffusionDPO [57] and D3PO [59] extend this approach to diffusion models, ensuring that generated images better reflect user preferences. Diffusion-KTO [28] replaces DPO with Kahneman-Tversky Optimization (KTO) [11] , eliminating the need for paired data. DCO [27] applies preference optimization to the personalization domain, ensuring that models preserve priors while generating personal images. Our work is the first to apply preference optimization to the unlearning problem and proposes task-specific data and regularization methods to fit unlearning task.  
method: 
 In this section, we introduce our framework named Direct Unlearning Optimization (DUO) consisting of Direct Preference Optimization (DPO) with synthetic paired data and output-preserving regularization. After providing a preliminary explanation of diffusion models, we detail the process of generating paired images, which are necessary for successful preference optimization for concept unlearning. Next, we provide a detailed explanation of applying direct preference optimization to the unlearning task using the generated paired images. Finally, we introduce output-preserving regularization loss, a novel strategy to maintain the prior distribution of the model.Diffusion models [52, 21, 53, 54, 32] are generative models that learn to generate images through an iterative denoising process. For an image x 0 , the diffusion process samples a noisy image x t from q(x t |x 0 ) = N ( √ α t x 0 , √ 1 -α t I). The magnitude of the added noise is determined by the noise schedule α t , and it increases as t becomes larger. We train the model ϵ θ (•) to denoise the noisy image x t = √ α t x 0 + √ 1 -α t ϵ by predicting the added noise ϵ.EQUATIONThis is called the denoising score matching (DSM) loss. It encourages the model to estimate the gradient of the log-probability of the noisy data, i.e., ϵ θ (x t ) ∝ ∇ log p(x t | c).Diffusion models can be conditioned on other modalities, such as text, to generate images that align with the given context [20, 35, 64] . In this case, the model learns to estimate ϵ(x t , c) ∝ ∇ log p(x t ), where c represents the conditioning information. This allows the model to generate images that are not only realistic but also semantically consistent with the provided text or other modalities.As we mentioned in Section 1, even images with unsafe concepts contain numerous unrelated visual features that need to be preserved (e.g., photo-realism, background, and humans). Therefore, a method is needed to selectively remove only the visual features associated with the targeted unsafe concept.Intuitively, if we provide the model with images that have identical unrelated visual features but differ in the targeted unsafe concept, the model can be guided to unlearn only the features related to the unsafe concept. Unfortunately, as shown in Figure 3 (b), naively generating paired images with different prompts often resulted in changes to the unrelated attributes of the images.We solve this by creating image pairs using SDEdit [34] , a method for image translation. The process of creating image pairs with SDEdit is as follows. Given an undesirable concept c -, e.g., naked, we first generate an unsafe image x - 0 using c -. Then, we add a certain amount of Gaussian noise to send it to x - t . Lastly, denoise x - t to create paired safe image x + 0 with negative guidance for c -. If possible, we also use positive guidance for c + , e.g., dressed, during the denoising process. Thanks to the characteristics of diffusion models, SDEdit allows us to generate images that have similar coarse features but either include or exclude the unsafe concept, ensuring that the concept unlearning process is focused on the specific elements we want to remove while preserving the rest of the image as shown in Figure 3 (c) . Now, we want to make model not to generate visual features in unsafe image x - 0 while still can generate safe image x + 0 . We solve this as a preference optimization problem. In the next subsection, we will discuss how we consider our generated paired dataset as preference data and apply Direct Preference Optimization (DPO) to our unlearning problem.In subsection 3.2, we ensure that the paired images only differed in the presence or absence of unsafe concepts while keeping other attributes the same. Using them, we need to guide the model to remove only the visual features associated with unsafe concepts. We employed a preference optimization method where the model is trained to move away from unsafe image x - 0 and towards their corresponding safe images x + 0 . Since the only difference between our dataset pairs x - 0 and x + 0 is the information related to the unsafe concept, the unrelated concepts remain unaffected.Suppose we are given a paired dataset {x + 0 , x - 0 }, where x + 0 and x - 0 denote the retain and forget sample, respectively. The probability of preferring x + 0 over x - 0 can be modeled by the reward function r(x):EQUATION)where σ is the sigmoid function. This model is known as the Bradley-Terry (BT) model [5] . The reward function can be learned using a binary cross-entropy loss to maximize likelihood:EQUATIONGiven this reward function, preference optimization aims to fine-tune the model to maximize the reward. To prevent the model p θ (x) from excessively forgetting its existing capabilities, we use KL-constrained reward optimization as our objective:EQUATIONwhere p ϕ denotes the pretrained model distribution, and the hyperparameter β controls the extent to which the model diverges from the prior distribution. The above equation has a unique closed-form solution p * θ [14, 25, 39, 41] :EQUATIONwhere Z = x0 p ϕ (x 0 ) exp(r(x 0 )/β) is the partition function. By manipulating this equation, we can express the reward function in terms of the fine-tuned and pretrained model distributions:EQUATIONPlugging this into the BT loss equation, we obtain the DPO loss [44] :EQUATIONDiffusion-DPO. There is a challenge in directly applying the DPO loss function (7) to diffusion models. Specifically, p θ (x 0 ) is not tractable as it requires marginalizing out all possible diffusion paths (x 1 , • • • , x T ). Diffusion-DPO [57] elegantly addresses this issue using the evidence lower bound (ELBO). First, we define a reward function R(c, x 0:T ) that measures the reward over the entire diffusion trajectory, which yields r(x 0 ) when marginalized across the trajectory:EQUATIONUsing R(x 0:T ), we can reformulate (4) for the diffusion paths x 0:T :EQUATIONFollowing a similar process as with the DPO objective, we can derive Diffusion-DPO objective to optimize p θ (x 0:T ):EQUATIONBy leveraging Jensen's inequality and the ELBO, this loss can be expressed as a combination of fully tractable denoising score matching losses (see the Appendix D for detailed derivation):EQUATIONIn Eq. ( 17), ∥ϵ -ϵ θ (x + t , t)∥ 2 2 -∥ϵ -ϵ ϕ (x + t , t)∥ 2 2operates as gradient descent for the preferred sample, while ∥ϵ -ϵ θ (x - t , t)∥ 2 2 -∥ϵ -ϵ ϕ (x - t , t)∥ 2 2operates as gradient ascent for the dispreferred sample.Unfortunately, in preliminary experiments, we discovered that DPO's KL divergence regularization alone is insufficient for prior preservation. The main reason was that the gradient ascent term in Eq. ( 17) does not simply unlearn x + 0 , but also decreases the model's denoising ability. To mitigate this issue, we introduced the following novel regularization term to maintain the diffusion model's denoising capability:EQUATIONThe above regularization maintains the output even when unlearning is performed and the image is completely noise, i.e., at x T . The reason for limiting it to t = T is that when image information remains, output preservation can interfere with unlearning. On the other hand, when t = T , only complete noise information remains in the image, so we expected that enforcing output preservation would not affect unlearning.Our final loss function, which incorporates the output preservation regularization, is as follows:L DUO ≜ -E x + t ∼q(x + t |x + 0 ),x - 0 ∼q(x - t |x - 0 ),x T ∼N (0,I) log σ -β ∥ϵ -ϵ θ (x + t , t)∥ 2 2 -∥ϵ -ϵ ϕ (x + t , t)∥ 2 2 -∥ϵ -ϵ θ (x - t , t)∥ 2 2 + ∥ϵ -ϵ ϕ (x - t , t)∥ 2 2+ λL prior (13) 4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments setup: 
 Unlearning setup. We use Stable Diffuson 1.4v (SD1.4v) with a LoRA [23, 48] rank of 32 with the Adam optimizer for fine-tuning. We observe that it is difficult to unlearn violence with a single LoRA, because it has various subcategories. To address this, we decompose it into four distinctive concepts: blood, suffering, gun, and horror. We apply DUO to each concept and merge them to use as violence unlearning LoRA. We do not train the cross-attention layers to minimize the dependency on the text prompt for unlearning [12] . Our experiments are conducted on Stable Diffusion version 1.4 to compare with other baseline methods. We use ESD [12] , UCE [13] , SPM [33] as baselines for the concept unlearning category of the safety mechanism. For implementation details, please refer to the Appendix.Red teaming. For black box red teaming methods that do not access model weights, we use Ring-A-Bell [56] and SneakyPrompt [60] . Ring-A-Bell uses a genetic algorithm to find prompts that have similar embeddings to unsafe prompts defined by the attacker. We used 95 pre-trained prompts for nudity and 250 prompts for violence provided by the authors. SneakyPrompt slightly perturbs undesirable prompts that are blocked by the safety filter, e.g., "A naked man riding a bike.", to bypass the safety filter, e.g., "A grponypui man riding a bike". We used 200 prompts trained by the authors on SD1.4v for evaluating SneakyPrompt on nudity.For white-box attacks that access model weights, we use Concept Inversion [42] . The concept inversion protocol is as follows. First, malicious images are generated using prompts provided by the i2p benchmark [38] . For nude images, prompts in the sexual category are used, and for violence, prompts with a toxicity probability of 0.95 or higher are used. For the images obtained through this process, a special token <c> is trained using textual inversion with a model that has a safety mechanism applied. The trained special token is then used as a prefix for the sexual/toxic prompts from the i2p benchmark.Evaluation. To evaluate whether we have defended against red teaming, we use external models. If these models determine that the image does not contain the unsafe concept, we consider the defense successful and report this ratio as the defense success rate. To classify whether images contain nudity, we use the NudeNet detector [2] . Following Tsai et al. [56] , if the detector detects one of the following labels in the image: "FEMALE BREAST EXPOSED, FEMALE GENITALIA EXPOSED, ANUS EXPOSED, MALE GENITALIA EXPOSED", we consider the image to contain nudity.To classify whether images contain violence, we used GPT-4o, a state-of-the-art multimodal LLM [58, 61, 1] . We use the user prompt, "Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? Please answer yes or no." [49] along with the image. If the model's response is "yes", we determine that it contains violence, while "no" indicated unlearning success. If neither token is present, we regenerate the model output.To evaluate model performance unrelated to unlearned concept, we measure the FID [19] and CLIP scores [18] using MS COCO 30k validation dataset [30] . These metric provide insights into the capabilities of unlearned models. Additionally, we report the LPIPS [65] between images from the original SD1.4v model with those from the unlearned model using identical noise. This quantifies how the unlearned model's distribution differs from the original. Unlearning is a task to satisfy the trade-off between unlearning performance and prior preservation simultaneously. We plot the Pareto curve by adjusting the magnitude of KL regularization. Specifically, each point in the figures corresponds to β ∈ {100, 250, 500, 1000, 2000}.Nudity. Figure 4 shows the quantitative results for nudity detection. Our method maintains model performance comparable to the current state-of-the-art SPM in terms of prior preservation while achieving a defense success rate (DSR) of nearly 90% against all red teaming methods. A notable characteristic of the Pareto curve is the sharp bend where the increase in unlearning performance diminishes and the model rapidly loses its prior knowledge. Based on this observation, we recommend choosing β = 500 and β = 250 for defending against black box and white box red teaming methods, respectively.Figure 5 presents the qualitative results from Ring-A-Bell and concept inversion attacks. While previous prompt-based unlearning methods are easily circumvented with red teaming, our method demonstrates robust defense against these attacks. This result highlights the effectiveness of our image-based unlearning approach, which is independent of the prompt and robust against promptbased red teaming. Interestingly, in the case of concept inversion, completely corrupted images are generated, unlike in the Ring-A-Bell attack. Through experimentation, we observed that it is challenging to robustly block textual inversion unless the generated result deviates significantly from the data manifold. This suggests that to defend against white box red teaming, which leverages gradient information, the internal features of the model must be completely removed rather than merely bypassed. Analyzing the different mechanisms of black box and white box red teaming presents an interesting direction for future research.In Table 1 , we report the FID and CLIP scores for MS COCO 30k. These scores indicate how well the unlearned model preserves its ability to generate images for unrelated prompts. Unlike other methods, DUO does not modify text embeddings or prompt-dependent weights; instead, it removes visual features directly from the model, making prior preservation quite challenging. Nonetheless, in terms of FID and CLIP scores, DUO shows performance comparable to other unlearning methods. Thus, for nudity, DUO is robust against prompt-based adversarial attacks while also preserving the original model's generation capabilities for unrelated concepts. Figure 6 qualitatively confirms that the generation results do not significantly differ between the prior model and the model after unlearning.Violence. Figure 7 demonstrates the red teaming result on violence. Similar to nudity, DUO shows remarkable robustness against attacks while maintaining the same prior preservation score. We observe that the defense success rate values are generally not completely blocked compared to nudity. We believe is due to the fact that violence is a more abstract and complex concept compared to nudity, making unlearning more difficult. Figure 8 show qualitative results. We present the outcomes using β = 500 for ring-a-bell and β = 1000 for concept inversion. Table 1 shows the FID and CLIP scores for MS COCO 30k. Similar to the nudity concept removal experiment, our method maintains comparable prior preservation performance to existing methods. Unlike the nudity experiment, in the violence concept removal experiment, we observe that the trade-off between prior preservation and defense success rate varies more steeply with the parameter β.Defence success rate Prior preservation Output preservation regularization. We study the effect of output preservation regularization. As shown in Figure 9 , lambda significantly enhances prior preservation performance at a similar level of DSR. This effect is more dramatic in the small β regime. Without L prior , the COCO generation result is severely degraded to the point of being unrecognizable. On the other hand, the unlearning result using L prior still produces plausible images at the same β. This demonstrates that output preservation regularization is effective in unlearning, where only the desired concept is removed. To assess the effectiveness of choosing the DPO method, we conducted an ablation study to evaluate the impact of KL divergence regularization. Please refer to Appendix A for ablation study for KL divergence regularization. 
conclusion and limitation: 
 We addressed the vulnerability of existing unlearning T2I methods to prompt-based adversarial attacks. To eliminate dependency on prompts, we conducted image-based unlearning and solved it using preference optimization methods. To address the challenge of prior preservation, we curated the dataset using SDEdit and proposed additional output-preservation regularization methods. As a result, our method remained robust against adversarial prompt attacks while maintaining image generation capabilities for unrelated concepts. Since our method involves unlearning visual features, unrelated concepts that share excessively similar visual features may be influenced by unlearning. We anticipate that this issue could be addressed by curating paired datasets that include similar concepts, but we leave this as future work.Impact statement While our method is designed to improve the safety of text-to-Image models, it could potentially be misused by malicious actors. For example, attackers might attempt to manipulate the unlearning process to create more harmful content. Therefore, in an open-source scenario, our method should be applied to the model before releasing the model weights. Additionally, our method may still be vulnerable to new adversarial attacks. Therefore, it is necessary to compensate our method with other safety mechanisms such as dataset filtering or safety checkers. A Ablation study KL-constrained optimization In this section, we study the effect of KL-constrained optimization. When β → 0, the sigmoid function can be approximated as a linear function, then DUO approaches a naïve combination of gradient ascent for preferred samples and gradient descent for de-preferred samples [66] . As shown in Figure 10 , when β → 0, the images are somewhat degraded, and the generation results for unrelated concepts also change. On the other hand, when KL divergence regularization is present, we can confirm that the generation results for unrelated concepts hardly change at all. shows that when the number of pairs is less than 64, there is a noticeable improvement in the Defense Success Rate. However, increasing the number beyond 64 pairs does not yield significant changes in the Pareto curve. This analysis suggests that 64 pairs provide a good balance between performance and computational efficiency for our method.Reproducibility We have conducted experiments using different random seeds (1 to 4) to demonstrate the stability and reproducibility of our results. Figure 13 shows that DUO's Pareto curve is not sensitive to random seed variation and consistently outperforms baseline methods. Figure 14 provides qualitative evidence of similar unlearning results across different seeds. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: In our paper, we plan to release a model with not-safe-for-work (NSFW) concepts removed, so there is no high risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Prior UnlearnDiffAtk [68] We conducte additional experiments using UnlearnDiffAtk [68] , a state-of-the-art white-box attack method designed for assessing the robustness of unlearned diffusion models.As illustrated in Figure 12 , DUO achieves Paretooptimal performance compared to existing baselines when subjected to UnlearnDiffAtk. This means that DUO provides the best trade-off between maintaining model performance and resisting attacks, outperforming other methods in both aspects simultaneously.To further validate prior preservation performance of our approach, we evaluated the model's ability to generate visually similar but safe concepts. For example, we compare the generation of red images (e.g., ketchup, strawberry jam) after removing the Violence concept, which is closely related to "Blood". Table 3 below presents mean and standard devidation of LPIPS scores between 128 images generated from the unlearned model and the pretrained model.Lower scores indicate less impact on unrelated features. These results demonstrate that DUO effectively maintains the capability to generate visually similar but unrelated concepts compared to the existing methods. Figure 15 shows qualitative results of prior preservation for visually similar concepts.Stable-Diffusion 3 In Figure 16 , and 17, we successfully applied DUO to Stable Diffusion 3 (SD3) [10] , which uses the transformer-based mmDiT architecture. The evaluation process for SD3 was identical to that used for SD1.4v. In the Pareto curve, each point from left to right represents β ∈ {100, 250, 500, 1000, 2000}, with a consistent learning rate of 3e-5 across all experiments. Additionally, Table 4 shows FID scores on MS COCO. It demonstrates that applying DUO to SD3 maintains comparable fidelity to the pretrained model.Nudity For generating the unsafe images x -, we use "naked" as prompt with a guidance strength of 7.5. For x + , we use "dressed" as with a strength of 7.5, and use "naked" as a negative prompt. We generate 256 images. We use nudenet [2] to regenerate images if there was no nudity in x -or if there was nudity in x + .Violence To generate violence images we use four sub-concepts: Blood, Suffering, Gun, and Horror. To generate unsafe images x -, we use the prompt: "a man, bleeding", "a man, suffering", "a man, with a gun", "horror". We generate 64 image per each prompt, total 256 images. For safe images x + , we use the prompt: "a man, in comfort", "a man, enjoying", "a man, with peace", "comfort", with using unsafe prompt as a negative prompt.SDEdit When we use SDEdit, the magnitude of the added noise is t = 0.75T , where T is the maximum diffusion timesteps, and the guidance scale used is 7.5.DUO Since β linearly multiplies the learning rate at the time near initialization, we rescale the learning rate, dividing it by the same amount each time we increase β. Using β = 100 as a baseline, we use a learning rate of 3e-4, batch size 4, and LoRA rank of 32. When conducting violence unlearning, we generally observe that the training is more unstable compare to nudity, and we mitigate this by training LoRA to unlearn each concept and combine them as violence LoRA. Exploring the relationship between dataset characteristics and optimization stability in DUO is an interesting avenue for future work.For Textual Inversion training, we used the Adam optimizer with a learning rate of lr=5.0e-03, batch size 4, and 3000 gradient steps. We used the same hyperparameters for all unlearning models.Eq. ( 10) → Eq. ( 17) Start from Eq. (10) .EQUATIONSince -log σ(•) is a convex function, we can utilize Jensen's inequality.EQUATION= -E x + 0 ,x - 0 ,q(x + 1:T |x + 0 ),q(x -EQUATIONIn (15b), the intractable p θ (x + 1:T |x + 0 ) and p θ (x - 1:T |x - 0 ) are approximated by the diffusion forward process q(x 1:T |x 0 ). Now, let's apply the product rule, i.e., p(x 0:T ) = p(x T ) T t=1 p(x t-1 | x t ), to p θ (x 0:T ) and p ϕ (x 0:T ).L Diffusion-DPO ≤ -E x + 0 ,x - 0 ,q(x + 1:T |x + 0 ),q(x - 1:T |x - 0 ) log σ([β T t=1 log p * θ (x + t-1 |x t ) p ϕ (x + t-1 |x t ) -log p * θ (x - t-1 |x t ) p ϕ (x - t-1 |x t ) ]) (16a) = -E x + 0 ,x - 0 ,q(x + 1:T |x + 0 ),q(x - 1:T |x - 0 ) log σ([β T t=1 (log p * θ (x + t-1 |x t ) q(x + t-1 |x t ) -log p * ϕ (x + t-1 |x t ) q(x + t-1 |x t ) ) -(log p * θ (x - t-1 |x t ) q(x - t-1 |x t ) -log p * ϕ (x - t-1 |x t ) q(x - t-1 |x t ) )]) (16b) = -E x + 0 ,x - 0 ,q(x + t |x + 0 ),q(x - t |x - 0 ) log σ([β T t=1 D KL q(x + t-1 |x t ) p * θ (x + t-1 |x t ) -D KL q(x + t-1 |x t ) p * ϕ (x + t-1 |x t ) -D KL q(x - t-1 |x t ) p * θ (x - t-1 |x t ) -D KL q(x - t-1 |x t ) p * ϕ (x - t-1 |x t ) ])Finally, by applying the variational lower bound of the diffusion loss, we obtain the upper bound of the above equation as follows [24] :L Diffusion-DPO ≤ -E (x + t ,x - t )∼D,x + t ∼q(x + t |x + 0 ),x - 0 ∼q(x - t |x - 0 ) log σ -βT ω(λ t ) ∥ϵ -ϵ θ (x + t , t)∥ 2 2 -∥ϵ -ϵ ϕ (x + t , t)∥ 2 2 -∥ϵ -ϵ θ (x - t , t)∥ 2 2 + ∥ϵ -ϵ ϕ (x - t , t)∥ 2NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.(17) 


Decision: This paper was evaluated by four experts who generally praised its novel approach and robust experimental results. Although the initial review scores were mixed and several concerns were raised, the authors effectively addressed these issues during the rebuttal phase. Consequently, reviewers who had initially provided lower scores revised their assessments in favor of acceptance. With unanimous support for acceptance, AC deems the paper a strong candidate for acceptance.

Reviews: review: paper_summary: paper_summary: The authors address the issue of adversarial attack to diffusion model generating inappropriate image contents in this paper, critiquing the previous work on unlearning technique unlearns harmful prompt but making themself vulnerable to adversarial prompt. Instead they propose the image-based unlearning technique to remove unsafe visual features while retain the performance to generate image for safe concepts. During the training, they propose the method to generate a pair of safe and unsafe content and employ preference optimization to encourage diffusion model to generate the safe counterpart. The present the result in the evaluation to show that their model is better at defending adversarial attacks such as SneakyPrompt, Ring-A-Bell and Concept Inversion.
strength_weakness: strength_weakness: - There is a missing detail in experiments setup. The authors mention in line 194 that they decompose the violence concept into four categories and apply DUO to each of them and then merge the final models. If I understand correctly, the authors apply DUO independently for the four concepts resulting in four final models with LoRA. Please clarify how merging works to combine the four models.
questions: questions: Please clarify the missing detail
limitations: limitations: The authors discuss the limitation in the conclusion.
paper_summary: paper_summary: The authors introduce synthesized image data and preference optimization for concept unlearning in diffusion models. Additionally, they consider the regularization of model preservation performance to ensure a balanced approach.
strength_weakness: strength_weakness: 1. The efficacy of the proposed method is heavily dependent on the quality and diversity of the synthesized image pairs. The utilization of a small dataset, consisting of only 64 pairs, raises significant concerns about the potential for overfitting. To enhance the credibility of the findings, it is imperative for the authors to conduct comprehensive ablation studies. These studies should explore the impact of different sets of synthesized image pairs on model performance, potentially revealing critical insights into the method's robustness and generalizability.

2. The current robustness assessments of the study are limited by the employment of relatively weak attack scenarios. For a more robust evaluation of the unlearned models, it is crucial to incorporate a stronger, white-box attack methodology. The use of UnlearnDiffAtk [1], a commonly applied tool for assessing the robustness of unlearned diffusion models, is recommended. This approach would not only adhere to contemporary research standards but also significantly bolster the validity of the results. Furthermore, detailed reporting of the attack success rate (ASR) associated with UnlearnDiffAtk would provide a more precise quantification of the models' resilience against sophisticated attacks.


[1] "To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy to Generate Unsafe Images ... For Now", ECCV 2024.
questions: questions: Check the weaknesses section for more details.
limitations: limitations: Effectiveness of the proposed method heavily relies on the synthesized image pairs and different synthesized image pairs might cause high performance variance.
paper_summary: paper_summary: This paper proposes a diffusion unlearning optimization framework to achieve NSFW visual content removal in T2I models. Specifically, the authors develop an image-based unlearning method that utilizes curated paired image data (unsafe images and their corresponding safe images generated by the SDEdit model) for preference optimization. Additionally, they introduce a regularization term to preserve the denoising capability of the diffusion model. This work claims to be the first to apply preference optimization to the unlearning problem. Experimental results demonstrate that their method effectively removes unsafe visual concepts without significant performance degradation on unrelated topics.
strength_weakness: strength_weakness: 1. The authors only fine-tuned Stable Diffusion v1.4 for evaluation, which employs U-Net for denoising. However, other popular text-to-image (T2I) diffusion models, such as PixArt[1] and Stable Diffusion v3, which use Transformers for denoising, are not mentioned. It is somewhat limited in discussing the proposed method on only one T2I model. Can the proposed method be applied to the above models?

2. The unsafe concept may introduce some unrelated visual features when generating images, such as image details. How can the authors ensure that the proposed method does not affect these unrelated features when removing unsafe content?

[1] PixArt-$\alpha$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. ICLR 2024
questions: questions: Please refer to the above weakness section.
limitations: limitations: Refer to previous sections.
paper_summary: paper_summary: The paper proposes a new method for unlearning diffusion-based generative models. The proposed method uses the basic idea of reinforcement learning with human feedback. To collect pairs of images to be unlearned and corrected for preference optimization, the author uses SDEdit to modify the NSFW content. Experimental results show that the proposed model successfully unlearns harmful content from the generated images.
strength_weakness: strength_weakness: - Although this paper shows some promising results about diffusion model unlearning, the main technical contribution is limited to a combination of SDEdit and DiffDPO.
- It is clearly an overstatement that the preference optimization does not affect the quality of the unrelated concepts (in line 149).
- I have concerns about the reproducibility of this work.
- Minor
    - The main text and appendix have multiple typos and require careful proofreading.
    - Typo: Eq 17 → Eq 11 in the main text
    - Type: line 275: lambda?
    - Is DCO in line 285 a typo?
questions: questions: - The reason why the gradient ascent term reduces the model's denoising ability, even in the presence of the KL term, is unclear. Could you provide more details on how adding a L_prior loss, which minimizes the difference between denoising directions at time T, can address this issue? The explanation is insufficient to grasp the logic behind the loss.
- What does ‘prior’ mean in ‘prior preservation performance’? If LPIPS measures the perceptual similarity between images, can we just use perceptual similarity? Since the prior in ‘prior preservation performance’ and the prior in ‘L_prior’ are different, it’s a bit confusing to follow section 4.3.
limitations: limitations: Limitations are well-addressed.

Comments: title: Read the rebuttal and discuss with the authors
Reply to Authors
comment: We are pleased that we have been able to address the reviewer's concerns. We sincerely appreciate the time and effort the reviewer dedicated to reviewing our paper and providing valuable feedback. We will incorporate these insights, including the new experiments presented in this rebuttal, into our final manuscript. Thank you for helping us improve the quality of our work.
We are pleased that we have been able to address the reviewer's concerns. We sincerely appreciate the time and effort the reviewer dedicated to reviewing our paper and providing valuable feedback. We will incorporate these insights, including the new experiments presented in this rebuttal, into our final manuscript. Thank you for helping us improve the quality of our work.
Thanks for the detailed responses with additional experiments. After reading the other reviews and rebuttal, I have decided to raise my score to 6. I hope a future revision will address the potential limitations and include additional experiments.
Hi Reviewers, 

The authors have submitted their rebuttal responses addressing the concerns and feedback you provided. We kindly request that you review their responses and assess whether the issues you raised have been satisfactorily addressed.

If there are any areas where you believe additional clarification is needed, please do not hesitate to engage in further discussion with the authors. Your insights are invaluable to ensuring a thorough and constructive review process.

Best
AC
The authors' response has addressed my concerns, and I am raising my rating to 6. I recommend including the newly conducted experiments in the revision to enhance readers' understanding and provide additional insights.
We are glad to see that the reviewer's concerns have been addressed. We appreciate the time and effort the reviewer dedicated to reviewing our paper and offering valuable feedback.
Thanks for your reply. The authors have addressed my concerns.  After considering the comments from other reviewers and your reply, I plan to keep my rating.