Title: Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Models

AI Safety

Model Personalization
Abstract: Text-to-image (T2I) diffusion models have achieved impressive image generation quality and are increasingly fine-tuned for personalized applications. However, these models often inherit unsafe behaviors from toxic pretraining data, raising growing safety concerns. While recent safety-driven unlearning methods have made promising progress in suppressing model toxicity, they are found to be fragile to downstream fine-tuning, as we reveal that state-of-the-art methods largely fail to retain their effectiveness even when fine-tuned on entirely benign datasets. To mitigate this problem, in this paper, we propose ResAlign, a safety-driven unlearning framework with enhanced resilience against downstream fine-tuning. By modeling downstream fine-tuning as an implicit optimization problem with a Moreau envelope-based reformulation, ResAlign enables efficient gradient estimation to minimize the recovery of harmful behaviors. Additionally, a meta-learning strategy is proposed to simulate a diverse distribution of fine-tuning scenarios to improve generalization. Extensive experiments across a wide range of datasets, fine-tuning methods, and configurations demonstrate that ResAlign consistently outperforms prior unlearning approaches in retaining safety, while effectively preserving benign generation capability. Our code and pretrained models are publicly available at https://github.com/AntigoneRandy/ResAlign.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes ResAlign, a safety-driven unlearning framework aimed at enhancing the resilience of text-to-image (T2I) diffusion models against downstream fine-tuning. It addresses a critical limitation of existing unlearning methods: their fragility to fine-tuning, even on benign datasets, which can inadvertently recover harmful behaviors. ResAlign introduces a novel optimization objective that penalizes the resurgence of harmful behaviors post-fine-tuning, approximated efficiently using a Moreau Envelope formulation and enhanced with a meta-learning strategy to generalize across diverse fine-tuning scenarios. Extensive experiments on Stable Diffusion v1.4 demonstrate that ResAlign outperforms state-of-the-art baselines in maintaining safety after fine-tuning while preserving benign generation capabilities.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n\n- The paper tackles an underexplored yet significant issue—the vulnerability of safety-driven unlearning to downstream fine-tuning. This is a timely and practical contribution, especially given the increasing personalization of T2I models.\n- The use of a Moreau Envelope-based approximation to model fine-tuning as an implicit optimization problem is clever and computationally efficient, avoiding the need to track multi-step fine-tuning trajectories. The integration of meta-learning to simulate diverse fine-tuning configurations further enhances generalization.\n- Proposition 1 links the additional optimization term to the curvature of the loss landscape, providing a plausible explanation for improved resilience.\n- The experiments span multiple datasets, fine-tuning methods (e.g., DreamBooth, LoRA), and configurations, convincingly showing ResAlign’s superiority in safety retention and its minimal impact on benign generation quality.\n\n### Weaknesses\n\n- Figure 1 shows that ResAlign-generated images appear as mosaics post-unlearning, raising concerns about practical usability. While safety is improved, this severe degradation could limit real-world applicability, a point insufficiently addressed. \n- The reliance on Stable Diffusion v1.4, an older model by 2025 standards, questions the method’s relevance to state-of-the-art T2I architectures (e.g., SDXL). This limits the generalizability of the findings.\n- The paper compares ResAlign only to a few methods (e.g., ESD, SafeGen, AdvUnlearn, LCFDSD), neglecting recent advancements such as RECE (ECCV24), Receler (ECCV24) and RACE (ECCV24). This omission limits the evaluation’s comprehensiveness.\n- The Moreau Envelope and Proposition 1 assume small perturbations and isotropic randomness, which may not hold under extensive fine-tuning or large learning rates. The paper lacks discussion on these assumptions’ real-world validity.\n\nQUESTIONS:\nQ1: Figure 1 shows ResAlign-generated images as mosaics. What causes this degradation, and how could it be mitigated without compromising safety?\n\nQ2: Experiments use Stable Diffusion v1.4. How does ResAlign perform on newer models like SDXL?\n\nQ3: The paper compares ResAlign to a limited set of baselines (e.g., ESD, SafeGen, AdvUnlearn, LCFDSD), neglecting recent methods such as RECE (ECCV24), Receler (ECCV24) and RACE (ECCV24). How does ResAlign perform relative to these newer approaches, and what implications might their inclusion have for the evaluation’s comprehensiveness?\n\nQ4: Beyond the $\\mathcal{L}_{\\text{adaptive}}$ fine-tuning attack in Section C.2, what other adaptive fine-tuning strategies (e.g., alternative loss modifications or multi-stage optimization) could challenge ResAlign’s safety constraints? How can future evaluations enhance ResAlign’s robustness against such strategies?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for unlearning in diffusion models, aiming to ensure that a concept is completely forgotten—even after downstream fine-tuning. The authors observe that previous unlearning methods tend to lose their effectiveness once the model undergoes fine-tuning, even on data different from the target dataset. Motivated by this, the paper introduces a framework that explicitly incorporates the fine-tuning process into the unlearning formulation. Through mathematical induction and a meta-learning-based configuration, the proposed method achieves robust unlearning that is resistant to downstream fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The observation that fine-tuning—even on non-target data—can compromise unlearning is insightful and could inspire future research.\n2. The proposed method is well-motivated and theoretically grounded.\n3. The experimental results are generally strong and supportive of the method's effectiveness.\n\nWeaknesses:\n\n1. The paper omits a key related work [1], which is only briefly mentioned in the introduction but not discussed in the related work section nor empirically compared. This reference also uses a meta-learning approach and is highly relevant. The paper should clarify how its proposed formulation improves upon [1] and include direct empirical comparisons.\n2. The experiments are limited to older diffusion model versions (e.g., Stable Diffusion v1.4 and v1.5). Extending the evaluation to newer models like SD v2.0 or SD XL would enhance the paper’s applicability, especially since many fine-tuned checkpoints are available for these versions on platforms like HuggingFace and Civitai.\n3. The paper lacks a comparison of computational costs against baseline, including both VRAM usage and training speed. Such information would be valuable for understanding the practical feasibility of the proposed method in real-world deployments.\n\n[1] Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, and Min Lin. Meta-unlearning on diffusion models: Preventing relearning unlearned concepts. arXiv preprint arXiv:2410.12777, 2024.\n\nQUESTIONS:\nThe proposed formulation accounts for the fine-tuning process, so the optimization is not solely over the original model but also considers its fine-tuned variants. However, as shown in Tab. 1, the proposed method also performs better than baselines in scenarios without any downstream fine-tuning, which seems counter-intuitive. Could the authors explain why the method still improves unlearning effectiveness even in the absence of fine-tuning?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe target of this paper is to develop a defense method against fine-tuning attacks to expose unsafe generations on text-to-image diffusion models. The basic idea is to add an additional penality to ensure a low loss term on finetuned parameters. This also enforces the optimum to be located at a non-sharp region.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written and easy to follow. The background & related work section reminds readers of the related knowledge very effectively. The Motivation & Problem Formulation paragraph also introduces the high-level idea and motivation very clearly.\n\n2. The research question itself (defend against diffusion finetuning attacks) is interesting and the paper is one of the first works to try to address it.\n\nWeakness:\n\n1. There is limited details on ME approximation, which is the major technique involved in the method part of this paper. It is insightful to provide some viable approximations you may consider during the progress and why ME approximation is adopted. Also, does ME approximation have any insightful implications on this specific scenario of finetuning attack defenses.\n\n2. There is a lack of related work on optimization methods to achieve flat-region optimum and specification of differentiation to this line of work. For instance, is there any work that directly optimization the hessian trace (with approximation)?\n\n3. The results in Tab.1 is not very convincing. It seems that the model before fine-tuning is more safe than the baselines, then it is not known that whether the safety after fine-tuning comes from this better initial point or the additional penalty on curvature of loss landscape.\n\nQUESTIONS:\n1. Since the parameter space is high-dimensional, the effectiveness of the method can depend on how good the approximation region \\theta_{FT}^* lies in compared to the true effective fine-tuning attack region. In other words, the fine-tuning datasets and configurations all make a difference here. Then, do you have any criteria or tips to select the fine-tuning datasets and configurations?\n\n2. What is the result of changing $\\beta$?\n\nLIMITATIONS:\nThe scope of this paper is about instance-wise unlearning. In other words, the model is trained to unlearn certain unsafe instances. It is essential to include discussions on whether such instance-level unlearning will imply concept-level unlearning (e.g., sexual content). Also, the discussion of instance-level full unlearning against recovery and theconcept-level one.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the challenge of safety of diffusion  model, in particular, the safety of unlearning methods for diffusion models. Although unlearning could help block unsafe generation of diffusion models, the authors found that existing diffusion models with unlearning methods still face the safety challenge after fine-tuning, even with benign images. As a result, the author presents an simple yet effective regularization to the unlearning framework, that considers model parameters after fine-tuning. The author conducted extensive experimental across many configuration, and the experimental results justify the safe-driven and benign generation  abilities of the proposed unlearning framework.\n\nSTRENGTHS AND WEAKNESSES:\nThe strengths:  \n- First of all, the motivation of this paper is attractive and interesting. The authors consider the safety of diffusion models after fine-tuning. This problem is more challenging and realistic than the safety of training diffusion model. \n- Second, the authors presented an intuitive method by using a simple yet effective regularization, which considers the model parameters after fine-tuning, not just the current state. \n- Third, for consideration of computation efficiency, the authors use Implicit Differentiation to calculate the gradients. \n- Fourth, I found the experiments sufficient and extensive. The authors conducted extensive experiments across diverse configurations to justify the safe-driven and benign generation  abilities of the proposed unlearning framework. \n\nThe weaknesses: \n- On the meta-learning part: The authors used meta-learning to simulate different fine-tuning situation. However, I believe there two issues with the meta-learning part. First, the configurations for the meta-learning is still limited and the authors haven't presented strong ablation study to prove its usefulness. Second, the meta-learning would increase the running time of these method. The author haven't discussed about this. \n- On the Effectiveness on Contaminated Data: The proposed even showed robustness on harmful data. However, the authors haven't discussed about this.\n\nQUESTIONS:\nOverall, I believe this work is interesting and solid. But I still have two concerns, as in the \"Strengths And Weaknesses\" part. The authors could refer to this part, and consider my concerns.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}