Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning
Authors: Boheng Li
Renjie Gu
Junjie Wang
Leyi Qi
Yiming Li
Run Wang
Zhan Qin
Tianwei Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
AI Safety
Model Personalization
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose a novel safety-driven unlearning framework for diffusion models that can better maintain unlearning performance after downstream fine-tuning.

Abstract: Text-to-image (T2I) diffusion models have achieved impressive image generation quality and are increasingly fine-tuned for personalized applications. However, these models often inherit unsafe behaviors from toxic pretraining data, raising growing safety concerns. While recent safety-driven unlearning methods have made promising progress in suppressing model toxicity, they are found to be fragile to downstream fine-tuning, as we reveal that state-of-the-art methods largely fail to retain their effectiveness even when fine-tuned on entirely benign datasets. To mitigate this problem, in this paper, we propose ResAlign, a safety-driven unlearning framework with enhanced resilience against downstream fine-tuning. By modeling downstream fine-tuning as an implicit optimization problem with a Moreau envelope-based reformulation, ResAlign enables efficient gradient estimation to minimize the recovery of harmful behaviors. Additionally, a meta-learning strategy is proposed to simulate a diverse distribution of fine-tuning scenarios to improve generalization. Extensive experiments across a wide range of datasets, fine-tuning methods, and configurations demonstrate that ResAlign consistently outperforms prior unlearning approaches in retaining safety, while effectively preserving benign generation capability. Our code and pretrained models are publicly available at https://github.com/AntigoneRandy/ResAlign.

Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning
Boheng Li', Renjie G Wang", Leyi Qi', Yiming Li
Run Wang", Zhan Qin', Tianwei Zhang! Nanyang Technological University, Singapore *Central South University, China "Key Laboratory of Aerospace Information Security and Trusted Computing,
Ministry of Education, School of Cyber Science and Engineering, Wuhan University, China State Key Laboratory of Blockchain and Data Security, Zhejiang University, China
Abstract
Text-to-image (T2I) diffusion models have achieved impressive image generation quality and are increasingly fine-tuned for personalized applications. However, these models often inherit unsafe behaviors from toxic pretraining data, raising growing safety concerns. While recent safety-driven unlearning methods have made promising progress in suppressing model toxicity, they are found to be fragile downstream fine-tuning, as we reveal that state-of-the-art methods largely fail retain their effectiveness even when fine-tuned on entirely benign datasets. To miti- gate this problem, in this paper, we propose ResAllig, safety- unlearning framework with enhanced resilience against downstream fine-tuning. By model- ing downstream fine-tuning as an implicit optimization problem with a Moreau envelope-based reformulation, ResAlign enables efficient gradient estimation to minimize the recovery of harmful behaviors. Additionally, a meta-learning strategy is proposed to simulate a diverse distribution of fine-tuning scenarios to improve generalization. Extensive experiments across a wide range of datasets, fine-tuning methods, and configurations demonstrate that ResAlign consistently outperforms prior unlearning approaches retaining safety, while effectively preserving benign generation capability. Our code and pretrained models are publicly available here. Disclaimer: This paper includes Al generated images containing partially nude human figures and other sensitive content, shown only for research purposes.
Introduction
Text-to-image (T2I) diffusion models have emerged as a dominant class of generative AI due to their unprecedented ability to synthesize high-quality, diverse, and aesthetically compelling general images from natural language descriptions [67, 58]. Beyond synthesizing general images, there is also a growing interest in customizing pretrained models for personalized generation, e.g., generating images of specific facial identities or artistic styles that are underrepresented in the original training data [45]. This is typically achieved by fine-tuning the pretrained base model on a small reference dataset for few steps [69, 39, 76]. The development of several advanced fine-tuning methods [24, 69, 20] as well as the rapid proliferation of "fine-tuning-asee -atee-tuuinn------eviieee a-s-vieeeee"" platforms [5] has made personalization widely accessible, fueling a surge in applications such as stylized avatars, fan art, and thematic illustrations, which are becoming increasingly popular especially among younger users [74].
Yet alongside the rapid advancements of diffusion models, growing concerns have emerged regarding their potential to generate inappropriate or harmful content (e.g., sexually explicit imagery) [13, 70].
"Corresponding Author: Yiming Li (e-mail: liyiming tech@gmail com).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Due to the lagg--saale and web-crawe wreberraweed nature of their training datasets, modern T2I models inevitably ingest amounts of harmful material during pretraining [71]. As result, these models can repro- duce such content either when explicitly prompted or inadvertently triggered. For example, recent studies [70, 61, 40] based on real-world user generations demonstrate that widely-deployed models like Stable Diffusion [67] are particularly prone to producing unsafe content, even though many of the prompts that lead to unsafe outputs appear benign and may not be intended to generate harmful results. These vulnerabilities not only allow malicious exploitation through direct or adversarial prompting [79, 84], but also increase the risk of harmful unintended exposure for ordinary, benign users [40], raising serious ethical concerns for worldworld deployment. In response to these concerns, a variety of safety-driven unlearning methods [13, 43, 83] have recently been proposed to modify the pretrained models' parameters in order to suppress their capacity for unsafe generation.
While existing methods have shown encouraging results in reducing model toxicity and the resulting unlearned models are promising to be used as "safe" base models for downstream fine-tuning in practical workflows, one natural yet largely unexplored question whether the safety of unlearned models remains resilient after downstream fownstream.fin--tunnngg fine-tuning. Unfortunately, recent studies [56, 15] have shown that many existing methods can be easily reversed, where fine-tuning on harmful samples for as few as 20 steps can largely recover model's s unsafe capability. More strikingly, our empirical results reveal that even when fine-tuned on purely benign data, state-of ootte-of-the-rr unlearning methods can regress, with the model's harmfulness approaching its pre-unlearning state. In other words, even entirely benign users without any malicious intent or harmful data may inadvertently trigger recovery of unsafe behaviors, posing unforeseen safety risks in real-world use. These findings suggest that current methods may be significantly more brittle than previously assumed and are largely unprepared to serve as reliably safe base models for downstream fine-tuning, underscoring the urgent need for more resilient approaches that can withstand post-unlearning adaptation.
Towards this end, in this paper, we propose resilient safety-driven unlearning framework dubbed ResAlign to mitigate the aforementioned problem. The intuition behind our method is that unlearning should not only suppress harmfulness al the current model state, but also explicitly minimize the degree to which harmful behaviors can be regained after (simulated) downstream fine-mfiiint fine-tuning. While conceptually simple, it is particularly challenging to develop a principled and efficient optimization framework to realize this objective. This is because tine-tuning itself is a multi-step optimization process, making it non-trivial to predict which update direction on the original parameters helps minimize the regained harmfulness after downstream fine-tuning. To address this, we approximate fine-tuning an implicit optimization problem with Moreau envelope formulation [52, 63], which enables efficient gradient estimation via implicit differentiation. Besides, to ensure generalizability against the wide variability in real-world downstream fine-tuning procedures (e.g., different datasets, fine-tuning methods, and hyperparameters) we design a meta-learning approach that simulates a distribution of plausible fine-tuning configurations during training, allowing the model to generalize its resilience across broad range of downstream adaptation scenarios. We also provide insights from the theoretical perspective to explain the empirical effectiveness of our method.
In conclusion, our main contributions are threefold. (1) We empirically reveal that existing safety- driven unlearning methods largely fail to retain their effectiveness after downstream fine-tuning, even when the data does not contain unsafe content. (2) We propose ResAlign, a resilient safety- driven unlearning framework for T2I diffusion models. By leveraging a Moreau envelope- approximation and a meta-learning strategy over diverse adaptation scenarios, ResAlign explicitly accounts for and minimizes post-unlearning degradation due downstream fine-tuning efficiently with high generalizability. We further provide theoretical insights to help understand the empirical effectiveness of our method. (3) Through extensive experiments, we show that ResAlign consistently outperforms baselines in maintaining safety after tine-tunin, and generalizes well to wide range of advanced fine-tuning methods, datasets, and hyperparameters. It also preserves both general and personalized generation quality well, generalizes across various diffusion models and loss functions, and remains effective even under harmful data contamination or adaptive attacks.
2 Background & Related Work
Text-to-Image (T2I) Diffusion Models. Diffusion models are probabilistic generative models that learn the data distribution by reversing a predefined forward noising process through iterative denoising [22, 67]. Given clean sample a and a predefined noise schedule (ax, ( the forward
2

---

diffusion process constructs a noisy version at timestep t as = + osC, where ~ (0, I). Then, a denoising neural network Eo parameterized by is trained to predict the added noise e from 24 and the timestep l. In the context of text-to-image generation [67]. the prediction is further conditioned on a natural language prompt p. The model thus learns a conditional denoising function (o(Te, i), and its training objective is to minimize the following denoising score matching loss:
Ldenoise (0, Duain) (x,,(( We
||Ea + ore, f) - -e12
where x is the ground-truth image, p is the text condition, I is uniformly sampled from (I...., TT, and wy is a weighting factor that balances noise levels. This objective trains the model to accurately predict how to remove noise from noisy images across different timesteps, allowing it to generate realistic images from pure noise when conditioned on text during inference. Currently, the Stable Diffusion series [67, 58], which operate the diffusion process in a learned latent space and pretrained on large-scale datasets, stands as the most widely adopted open-source model for T21 generation. They are also increasingly fine-tuned on downstream datasets learn and generate concepts that are absent in their pretraining datasets for customized generation purposes [69, 57, 36).
Mitigating Unsafe Generation in T21 Models. To mitigate unsafe generation in T2I models, several strategies have been developed recently [60, 78, 48, 13), which can be broadly categorized into detection-based and wnlearning- -based. Detection-bused strategies deploy external safety filters (e.g.. DNN-based harmful prompt/image detectors) to inspect and block unsafe requests [7, 48]. While effective, they do not inherently "detoxify" the diffusion model, often introducing non- negligible additional computational and memory overhead during inference [43], and can be easily removed once the model is open-sourced [64]. On the other hand, iitearning-based approaches internalize safety constraints by directly modifying model parameters to reduce unsafe generation its source [13, 35, 43]. At a high level, given a pretrained T2I diffusion model 0 IR existing safety-driven unlearning approaches typically aim to optimize the model with the following objective:
0" E arg min Cramnful (0) + aR(()), BED
where Charmfun( behaviors, and R()) is a utility-preserving regularization term that maintains the model's benign generation capabilities. The hyperparameter a >> 0 is weighting coefficient. Since the seminal work of CA [35] and ESD [13]. safety-driven unlearning has rapidly gained traction, with a growing line of subsequent works expanding the design space across multiple dimensions including loss function design [35, 13), sets of updated parameters [43, 13], optimization strategies [14, 83, 18], and others.
(0) (0) is a harmful loss such that minimizing it encourages the model to unlearn harmful
Despite this progress, a recent pioneering work [56] has identified malicious fine-tuning issue, where unlearned models can quickly regain harmful generative abilities after fine-tuning on a small set of unsafe images, producing diverse outputs that resemble the quality and variety of the original pre-unlearned model. To the best of our knowledge, the most advanced attempt to address this issue is LCFDSD [56], which encourages separation in the latent space between clean and harmful data distributions to increase the difficulty of recovering unsafe behaviors. However, this comes with notable degradation in the benign utility of the model and only brings slight improvements over existing unlcarning methods, as validated in our experiments (Sec. 4.2). To summarize, how to effectively mitigate the post-finettuning resurgence of harmful capabilities in unlearned models remains an important yet largely unaddressed problem and is worth further exploration.
3 Methodology
Motivation & Problem Formulation. Despite recent progress in safety-driven unlearning, our experiments (Sec. 4.2) reveal that many statc-of-the-arr methods fail to retain their effectiveness after downstream fine-tuning, even when the adaptation data is entirely benign (instead of solely with harmful data). We speculate that this fragility stems from their objective formulation (Eq. (2)): they primarily focus on suppressing harmful behaviors at the current parameter state, without accounting for nearby regions in the parameter space that may be reached through downstream fine-tuning. As result, the local neighborhood of the unlearned model may still be vulnerable, such that even benign gradient signals during fine-tuning can inadvertently push the model into toxic regions and erode its safety. Motivated by this observation, we aim to learn model parameters that are not only safe in their current state, but also lie in regions of the parameter space where safety is preserved under downstream
3

---

fine-tuning, In other words, we seek to explicitly penalize the increase in harmfulness loss caused by (simulated) downstream fine-tuning during unlearning. Formally, given pretrained T2I diffusion
model parameterized by 0 E RR,, we propose the following resilient unlearning objective:
0" E arg min Charmful (0) + (0) + B [[harrmful (OFT) - Charmful (0)] DEG
(3)
where fJr = ADAPT(B, Drr.C) represents the parameters obtained by adapting @ on dataset DFr with configuration C. For example, if C is the standard gradient descent for T steps with learn- ing rate of 17, then we have 0Tr = - n Ei= standard diffusion denoising loss and o indicates the model parameters at the i-th step with g(o) 0. a and B are hyperparameters that balance the three terms. In this paper, we follow previous works [85, 83] and instantiate Ctarmfu))) as the negative denoising score matching loss on a set of inappropriate prompt-image pairs and (0) as the distillation loss that minimizes the difference of noise prediction between the current model and the original model éo, on set of preservation prompts, i.c., (0)
DFr), where LFT represents the
-B,,,, Daumge [wo ||ē + p, <l < and E, (xto [R0- [ p. t) êe, (=e. p. t)][)] Note that our method is also com- patible with other designs of these losses, as long as they offer similar utility (see results in Sec. 5).
R(0) =
Efficient Hypergradient Approximation via Implicit Differentiation. To optimize the resilient unlearning objective in Eq. (3), one natural idea is to gradient-based optimization. Thus, take gradients with respect to and rearrange terms. This yields the following expression:
(1
(annntaa + (0) + VV Chammful (0frr)
The first two terms in Eq. (4) are standard and can be easily computed via backpropagation. The difficulty lies in the last term, i.c., the hypergradient V Vaaammtuul (fFT). Because f.r is an
implicit function of 0, according to the chain rule, we have Va (fF7) = (06F-/00) T Crarnfuull (FFr). which involves the Jacobian 00./ Since Ofr is typically obtained through T-step iterative gradient descent, this derivative expands into product of Jacobians /05r V (B()) )rF)), where (F DFr) is the Hessian matrix of the fine- tuning loss w.r.t. g(t), As such, direct computation requires storing and backpropagating through the entire trajectory and their Hessian matrices, which is computationally and memory-prohibitte prohibitive.
(I IFFo0
To address this issue, inspired by recent work in bilevel optimization [47, 16, 81, 63), we approximate fine-tuning as implicit optimization over the Moreau envelope [52] of the loss. Specifically, approximately regard the solution of downstream fine-tuning (i.e., the resulting parameters Ofr) as the minimizer of the following Moreau envelope (ME) [52, 16, 63], i.e.
feF E arg min CF (FT) + _ 110 - 011
1
(5)
or
27
where y (set as 1 in this paper) is a proximity coefficient [63]. Intuitively, when fine-tuning diffusion models, we typically start from a strong pretrained base model and only take a few gradient steps. This implicitly restricts the solution OFr to remain near 0, making it appl-approx by the minimizer of the proximal objective in Eq. (5). As a result, OF naturally satisfies the first-order optimality condition, i.c., Vo,,(0) (or,(O)CFT(B (or( DFT) +)++++ + 0) = 0 according to the Karush-Kuhn- Since O is implicitly defined as a function of 0 through this optimality condition, we apply the implicit function theorem to differentiate both sides with respect to 0. Then, by right-multiplinng both sides by Voj, Lharmful (OFr), we can derive the following equation (details in Appendix A):
thcorem [34, 63].
U LIT (OfT: DFT) + 21) Vo (OFr) = V or Charmfal (OFT)
V
(6)
Let A : V2 C 11, V La (0rr), and b = , Vo oF. Cnarntul- we observe from Eq. (6) if
that Lo (BFT) is essentially the solution to the linear system Ar b, which can be efficiently
solved using the Richardson iteration method [66] (setting the relaxation parameter as w = = 1):
(k+1) = ?b - 7 V2 (er(OEr: DFr) (+).
(7)
with initialization (0))) 0. Lastly, we take the final iterate ((K) as the implicit gradient V V (0Fr) and plug it into Eq. (3), which enables end-to-end gradient-based optimization.

---

Our approach has the following advantages. First, computing the implicit gradient V Crammtul (frT) only depends on the final fine-tuned parameters fer and the local Hessian- -eector product V This product can be efficiently computed as a Hessian-vector product (HVP), which is readily sup- ported in modern autodiff frameworks such as Py PyTorch (via backward-mode automatic differentiation), without explicitly forming or inverting any second-order Hessian matrices. Besides, as the gradient is entirely determined by Or.. there is no need to store the intermediate models during fine-tuning, which signifi- cantly reduces memory overhead. Finally,
8yy LFT x.
Algorithm 1 GETHYPERGRAD
1: Input: Base parameters 8, downstream fine-tuned model 0FT, loss functions LFT, Charmiul, proximity coefficient y number of steps
2: Output: Hypergradient Vo (PFT)
observe that the Richardson iteration used to approximate the implicit gradient (i.e., Eq. (7)) converges quickly within few steps (5 in this paper), enabling effi- cient training for large-scale diffusion mod- els. The algorithm procedure for obtaining the hypergradient is placed in Alg. 1.
3: x00) <- b < 7,- Vo Charntul (frT)
4: for l 0 to K- 1 do
Compute Hessiannvector product Vig (0r(0)r) (1k) via iterative solver and reverse- autodiff
6:
x g(&+1) + V Cer(((r)) -x
7:
end for
8: Return (K)
Cross-cone Cross-configuration Generalization via Meta Learning. So far, we have successfully derived dejjectory-independee hypergradient based on Moreau envelope approximation, which can be efficiently computed as long as ffr is available. However, this requires access to the downstream fine-tuning configuration C and dataset Dpr during unlearming time, which is typically infcasible. A natural workaround is to use a fixed proxy configuration to simulate downstream fine-tuning during training. Yet, such models carry risk of overfitting to the fixed simulated configuration [49], which may hinder generalization to other settings, especially those unseen during training.
To this end, we draw inspiration from meta-learning [10, 63, 53] to improve the crosscoonfiuuaation generalization of ResAlign. Dif- ferent from conventional meta- learning that aims to enable fast few-shot adaptation, we aim to train a model whose resilience gen- eralizes across distribution of downstream fine-tuning datasets
Algorithm ResAAlign
1: Input: Initial model parameters Oo. number of outer loop iterations
I, number of inner loop iterations J, dataset D, distribution over configurations x rC), loss functions Lyr; Lramtut, R, learning rate
n) hyperparameters a, 3, Y, K
2: Output: Final unlearned model parameters 0₃
3: for 0 to I do
Outer Loop
4:
g. 4 (1 o, Crumiua (0:) + V R(,).)
5:
for j= to J do
b Inner Loop
6:
Sample data DeT ~ D and configuration C ~ =m((C) Ofr., ADAPT Dri, C)
and configurations. Specifically, we treat the fine-tuning configura-
7:
8:
9t.; + GETHYYERGRAD((., lir,,y LFT, Cramnua K) end for
tion C (including loss function, op- timizer, learning rate, and number of steps) and data sampling Dpr as meta-variables. During each inner
9:
10:
gi + 9: + $ Σ3 g.s
b Aggregate hypergradients
J
11: 0,+1 < 19-
12:
end for
loop iteration, we sample a batch
13:
Return 0
of data DFT ~ 'D and configuration (C) from the pool of these meta-variable choices. Then, we run an adaptation process to obtain the fine-tuned model OFr ADAPT(O, Drr, and use Alg. to compute hypergradient with respect to This process is repeated for J iterations, which gives us pool of hypergradients reflecting the model's unlearning dynamics under diverse fine-tuning regimes. Then, these hyper- gradients are aggregated and used to update the base model parameters in our outer loop iterations. Following previous works [10, 49]. a first-order approximation is used for efficiency. Overall, this meta- cetta-ennraaizzttoon mechanism encourages the base model to resist re-acquiring harmful capabil- ities under diverse set of plausible fine-tuning conditions while avoiding overfitting to specific simulation configuration. The overall algorithm procedure is summarized in Alg. 2.
Theoretical Insights. As we will show in experiments, despite introducing only a conceptually simple term, our ResAlign consistently improves safety resilience across wide range of downstream fine-tuning scenarios. To better understand the (potential) underlying mechanism behind ResAlign's empirical effectiveness, in this section, we provide a qualitative theoretical analysis, as follows:
Let 0 E &Rd and OFr E &Rd denote the parameters of the base model and the fine-tuned model, respectively, Assume the harmful loss Lharmful is twice differentiable around 0 the parameter
Proposition 1.
5

---

difference of the two models & E I is small and can be regarded as a scaled version of a unit random variable R with scaling factor a E R. i.e. {= 02. Taylor expansion around model parameters and ignore higher-onder terms, we have:
=2. If we exploit the second-order
arg min El [[hhormfal
Charmfui (0)] arg min Tr( V2 Charmgua (0))
(8)
where Tr(V 2a = Ei 8 Crongal is trace of the Hessian matrix of Ltarmju W.r.t. O.
30;
In general, Proposition 1 shows that our additional term can be regarded as implicit penalty on the trace of the Hessian of the harmfulness loss with respect to The trace of the Hessian a well-known indicator of the overall curvature of the loss landscape [29, 11], where larger values reflect sharper minima that are highly sensitive to parameter perturbations, while smaller values correspond to flatter and more stable regions. Interestingly, prior works in generalization and robustness [32. 11] have shown that SGD-style optimizers tend to lead models to sharp minima, i.c., regions in the loss landscape with high curvature that are sensitive to parameter perturbations. This connection offers potential explanation for the fragility of existing unlearning methods: since they do not explicitly regularize the curvature of the harmfulness loss, the resulting models, while appearing safe at their current parameter state, may lie in locally sharp and unstable regions of the loss landscape. In such regions, even ordinary parameter updates, like those induced by benign fine-tuning, can result in disproportionately large shifts in harmful loss, thereby inadvertently reactivating harmful capabilities. In contrast, our ResAlign introduces an implicit penalty loss, encouraging convergence to flatter regions of the loss surface. This may help reduce the model's sensitivity downstream updates and thereby improves its resilience to post-unlearning fine-tuning. We provide proof of Proposition I and further discussion in Appendix A.
the trace of the Hessian of the harmfulness
4 Experiments
4.1 Experimental Setup
Baselines & Diffusion Models. We compare our method with 6 text-to-image diffusion models, including SD v1.4 [67], ESD [13], SafeGen [43]. Adv [83], and two variants of LCFDSD [56] (i.e., LCFDSD-NG and LCFDSD-LT). We directly use their provided checkpoints or run their official code with default hyperparameters to obtain their unlearned models (see more details in Appendix B) Note that the baselines are mostly implemented based on SD v1. 1.4 and mainly focus on the unsafe concept of sexual, and many of them do not provide extensions to other types of unsafe content or to other model architectures. Thus, we conduct our main experiments under the same setting to ensure a fair comparison. We also validate ResAlign on other diffusion models in See. 4.2.
Fine-tuning Methods & Datasets. Both standard fine-tuning and advanced personalization-tuiing methods are considered in our experiment. For standard fine-tuning, we adopt a selected subset of two datasets: DreamBench++ [57] and DiffusionDB [75], which contain high-quality images of human characters, artistic styles, etc. We use these datasets to serve as representatives for benign fine-tuning. In addition, we follow [56] and use the Harmful-Img
dataset, which consists of a diverse set of sexually explicit prompt-image pairs, to understand the resilience of ResAlign when the mie-tuning dataset (partially) contains inappropriate data. Beyond these, we further evaluate the generalizability of our method under more advanced personalization-uum -unniii methods, including LoRA [24], DreamBooth [69]. SVDiff [20], and CustomDiffusion [36], with more personalization datasets including Pokémon [65], Dog [57], ArtBench [46], and VGGFace2-HQ [4]. Finally, we employ the I2P [70] and Unsafe [61] datasets for measuring harmful generation capabilities, and the
COCO [3] dataset for assessing benign generation capability. More details are in Appendix B.2.
Evaluation Metrics. We assess model performance from both safety and generation quality perspec- tives. For harmful generation capability, we report the Inappropriate Rate (IP) and Unsafe Score (US). IP is computed by generating images from 12P dataset prompts [70] and measuring the proportion of outputs identified as harmful, where sexual content is detected by the NudeNet detector [54]. Similarly, US is computed on images generated from the Unsafe dataset [61] prompts with another pretrained NSFW classifier (i.e., MHSC [61]). A lower IP (L) and US (4) indicate weaker unsafe tendencies, meaning the model poses less risk of misuse and is less likely to expose benign users to inappropriate content [40, 70]. Note that, unlike Pan et al. [56] who evaluate model safety on
the full set of unsafe prompts, we focus on the sexual category subset in both datasets to ensure

---

Tab. 1: Evaluation across different fine-tuning settings. The results averaged across 3 independent runs.
Harmful Generation
Renign Generation
Before
Fine-tuned on DreamBench++
Fine-tuned on DiffusionDB
Model
FID CLPP Score
Aesthetics Score t
IP L
US
IP 4
US
IP L
US
v1.4 [67]
0.3598
0.1850
16.90 / 31.17 66004
ESD [13]
0.0677
0.0100
0.1661
0.0467
0.2209
0.0817
16.88 30.26 6600
SafeGen [43]
0.1199
0.0650
0.3154
0.1167
0.3344
0.1333
17.11 / 31.11 /5.94
AdvUnleam [83]
0.0183
0.0033
0.1038
0.0317
0.2975
0.1233
18.31 / 29.01 75990
LCFDSD-NG [56]
0.0788
0.0150
0.2238
0.0867
0,2474
0.0950
47.21 / 30.09 / 5.23
LCFDSD-LT [56]
0.1833
0.0467
0.2467
0.0917
0.2832
0.1117
31.69 / 30.72 5.60
Ours
0.0014
0.0033
0.0186
0.0050
0.0687
0.0550
18.18 / 31.03 5,99
more targeted and fair comparison. For benign generation, we use FID [21]. CLIP score [62], and Aestheties Score [37], which are widely used by previous works [56, 83), to measure the ability of the unlearned models to generate benign concepts. Finally, we also follow previous work [20, 69] to use CLIP-I and CLIP-T [69], as well as DINO score [55] to mcasure the performance of our model in generating personalized concepts after fine-tuning. More details are presented in Appendix B.3.
Implementation Details. By default, we initialize our model using SD v1.4 and train it following Alg. 2. For our meta-learning the distribution of configurations x (C) includes the learning rates of [1 x 10-4, 1 x 10 5 1 x 10 10-%, the steps of [5, 10, 20, 30), the fine- fine-tuning loss (i.e., Lr)) of both standard denoising loss and the prior-presere parameter fine-tuning and LoRA [24], and the optimizer of both SGD and Adam. For each adaptation simulation, we select a random combination of the above configurations to form C. Some abnormal configurations (e.g., large learning rates combined with large training steps) are empirically excluded to ensure training stability. The outer loop learning rate is set to 2 x 10 4. Training is performed on single NVIDIA RTX 100 GPU until convergence, which typically requires ~ 1 GPU hour. During training. the peak and average memory consumption are ~ 56 GB and ~ 24 GB, respectively. In our evaluation, all fine-tuning is full- cull-parameter fine-tuning on the respective dataset for 200 steps with a batch size of 1 and learning rate of x 10-5 by default (more details in Appendix B.4). To reduce randomness, all main fine-tuning experiments are repeated three times with different random seeds (i.e., three independent rums), and report the averaged results. Besides, also report the standard error of the mean in our figures as the error area to help understand the statistical significance of our results. More details on dataset selection and implementation are available in Appendix B.
denoising loss [69], the algorithm of both full-
4.2 Experimental Results
Main Results. We begin by evaluating the resilience of unlearned models under standard fine-tuning. As shown in Tab. 1, before downstream adapta-
$D
ESD
Adeliniam
LCEDSD-LT
Ouze
tion, all methods are able to significantly reduce
the inherent toxicity of the pretrained SD vl.4
Befoe FT
model, as reflected by their lower IP and US scores. However, after fine-tuning on both be- nign datasets, all baselines universally encounter notable resurgence of harmful behaviors, even matching the levels of the original unaligned SD v1.4 (e.g.. AdyUnlearn on DiffusionDB and SafeGen both datasets). We also provide some visualization examples showing how dif- ferent models react to inappropriate prompts be- fore and after fine-tuning in Fig. 1. These facts indicate that existing unlearning techniques of- fer limited robustness when subjected to down- stream adaptation, even when the dataset does not explicitly contain harmful samples. In contrast, our ResAlign consistently outperforms all base- lines by notable margin. Beyond safety, we also compare the impact of unlearning methods on benign content generation, with quantitative results reported in the rightmost column of Tab. 1 and qualitative results in Fig. 3. While some baselines incur noticeable degradation FID, CLIP score, or aesthetic quality (c.g., LCFDSD), our method introduces minimal performance drop and remains
Diffastabi DheanBesscbee
Fig. 1: Visualization of harmful generation. Baseline methods largely lose their effectiveness after fine-tuning while our method retains safety. The black blocks are added by the authors to avoid disturbing readers.
7

---

Tab. 2: Evaluation on different personalization settings.
The results are averaged across 3 independent runs.
Harmful Gen.
Personalized Generation
Setting (Porsonalization Method Dataset)
Method
IP
US 4
CLIP-T
CLIP-I1
DINO
DreamBooth [69] + Dog [69]
SD v1.4
0.3645
0,1883
0.2671
0.9532
0.8763
Ours
0.0021
0.0000
0.2631
0.9466
0.8605
CustomDiiflusion [36] + ArtBench [46]
SD v1.4 Ours
0.3416
0.1900
0,2668
0.6024
0.1611
0.0011
0.0000
0,2702
0.6171
0.1785
SD v1.4 Ours
0.3269
0.1683
0.3178
0.6544
0.4549
LoRA [24] Pokémon [65]
0.0032
0.0033
0,3175
0,6512
0.4332
SD v1.4
0.3244
0.1567
0.2491
0.6108
0.5672
SVDiff [20] + VGGFace2-HO [4]
Ours
0.0050
0.0033
0,2451
0,5826
0,5510
competitive with the original SD v1.4 and most baselines. This indicates that ResAlign not only ensures stronger safety resilience but also preserves the benign generative capability of the model.
We further analyze the learning dynamics of different methods across various fine-tuning steps. As can be seen in Fig. 2, baseline methods show notable fluctuations and a rapid in- crease in IP with only small number of fine-tuning steps, suggesting that their un- learning is brittle in the weight space. Re- sAlign, on the other hand, maintains a con- sistently low level of IP even after exten- sive fine-tuning (e.g., up to 500 steps), with remarkably smaller fluctuations. These re- sults collectively confirm that our method not only suppresses harmful behavior effec- tively but also allows the unlearned model to better withstand re-acquisition of unsafe capabilities under realistic downstream fine-tuning steps.
AdvUnleam
LCFDSO4NG
Durs
0.25
2:00
9:35
02
0.30
0.1
0:66
300
400
502
300
400
500
(a) DreamBenchh++
(b) DiffusionDB
Fig. 2: Evaluation across different fine-tuning steps.
Effectiveness across More Fine-tuning Methods & Datasets. Beyond standard full-paraman parameter fine- tuning, we further evaluate whether ResAlign remains resilient under more advanced personalization tuning methods, and whether it continues to support benign fine-tuning for downstream tasks.
As shown in Tab. 2, our method consis- tently maintains strong resilience across all four fine-tuning techniques and datasets. Specifically, the IP and US scores remain close to pre-fine-tuning levels, demonstrat- ing that ResAlign effectively prevents the re-emergence of harmful behavior under these advanced specialized adaptation set- tings, including those unseen during train- ing (c.g.. SVDiff and CustomDiftusion). At the same time, our model achieves com- parable, and in some cases even slightly higher (e.g., on ArtBench), CLIP-I, CLIP- T, and DINO scores compared to the original SD v1.4. These quantitative results, well as the qualitative visualization results in Fig. 3 suggest that our method well preserves the personalization capability of the unlearned model, i.e., ResAlign does not interfere with benign downstream tasks.
General Generation
Pensonalized Generation
LCFDSD NG
Ours
Ours
Dog
VGGiaee2-00
Fig. 3: Visualization results on benign generation. Our un- learned model maintains both general and personalized gener- ation capability similar to the original SD v1.4.
Effectiveness across More Fine-tuning Configurations. We also assess whether our method remains effective under other fine-tuning hyperparameters. Specifically, we test two commonly varied settings: learning rate and optimizer. We still use the DreamBench++ and DiffusionDB dataset and apply LoRA fine-tuning with different learning rate and optimizer settings. As shown in Tab. 3 and 4, ResAlign continues to perform well across these configurations, maintaining low harmfulness across different settings. This be attributed to our meta-learning strategy, which exposes the model to distribution of configurations, thereby improving generalization to a broad range of potential configs.
Effectiveness on Contaminated Data. We further consider a challenging scenario where the fine- tuning data either intentionally or inadvertently contaminated with unsafe content. To evaluate this, mix our datascts with the Harmful- Imgs dataset [56] at varying contamination ratios, and use
8

---

Tab. 3: Effectiveness of ResAlign under various linettunig learning rates. Adapt. refers to an adaptive learning rate scheduler that initiates at 5 x 10 and gradually decays using cosine annealing down to the minimum learning rate of x 10 6 The reported metrie is IP after fine-tuning.
Tab. 4: Effectiveness of Re- sAlign under various fine- tuning optimizers. The met- ric IP after fine-tuning.
Dataset
10-4
10"4
10
10m
xx 10
5x 10
Adape.
Optimier
DB..
DreamBenchas DiffusionDB
0.001
0.022
00111
0.001
0.0004
00887
0.059
0.083
0.043
0.084
0.069
0.044
0.059
Adam
0.0026
0O475
this contaminated data to fine-tune the unlearned models. As shown in Fig. 4, all methods exhibit increasing levels of harmfulness as the proportion of unsafe data rises. Notably, most baselines experience a steep degradation, with their IP approaching that of the original SD v1.4 when 20% or more of the data unsafe. In contrast, ResAlign maintains significantly lower harmfulness scores across all contamination levels, demonstrating better resilience even in this challenging scenario.
Overall, we argue perfect resilience against harmful data is inherently difficult-if not impossible- as, an adversary can, after all, treat the recovery of unsafe behavior as new learning task [8], which pretrained diffusion models are adept due to their strong generalization and few-shot learning capability [69,36]. 36]. Despite this, we believe the mitigation and insights provided by ResAlign are non-trivial and meaningful. It not only achieves notably better suppression of harmfulness recovery under benign fine-tuning, which domi- nant in real-world use cases, but also raises the cost of adversaries by making the re- acquisition of unsafe behaviors harder. We hope our work motivates further investiga- tion into this underexplored yet impactful
AdvUlteam
LGNDSI-NG
0.3
6.5
0.25
0.00
0.15
2:30
06
1.0
0.8
contamtoerinn San (50
Contemination Batto (S)
(a) DreahBenchh+*
(b) DiffusionDD
Fig. 4: Evaluation on contaminated data.
regime and inspires future strategies that
enhance robustness against regaining unlearned harmful capabilities during downstream adaptation.
Generalizability across More Diffusion Models. By design, our method model -agnotic and can be readily applied to diverse diffusion architectures. To further verify the applicability of ResAlign across wider range of models, we evaluate it on both official diffusion models (Stable Diffusion v2.0 [1] and SDXL. [58]) and two widely adopted community variants (AnythingXI. [80] and PonyDiffusion [59]). Specifically, we first initialize with its corresponding checkpoint and apply ResAlign to unlearn sexual harmful concepts. We then evaluate the safety of each unlearned model (No FT) and subsequently perform LoRA fine-tuning on Dream- Bench++ and DiffusionDB to assess its safety after downstream fine-tuning. As summarized Tab. 5, ResAlign consistently achieves low harmful generation rates across all architectures and fine-tuning datasets, demonstrating its strong generality and effectiveness across different diffusion models.
Tab. 5: Effectiveness ResAlign across more diffuusio models. The metric is IP.
Model
No FT
DB++
DifI.
V2.
0.004
0.031
0.078
SDXL
0.033
0.044
0.059
AnythingXL
0.015
0.062
0.087
PonyDiffusion
0.023
0.045
0.067
Ablation Study. We further con- duct an ablation study and hyper- parameter analysis on the Dream- Bench++ dataset. From Tab. 6, we can see that every compo- nent contributes to our ResAlign, with hypergradient approxima-
Tab, 6: Effect of components.
0.05
Component
Metric
0.04
Hyper.
Meta (D)
Meta (C)
IP 4
FID
0.03
0.2266
18.24
0.1826
18.07
0.0322
18:35
Regularication Strength
0.0186
18.18
Fig. 5: Effect of y.
tion (Hyper.) helping reduce regained harmfulness and meta-learning on dataset (Meta. (D)) and configurations (Meta. (C)) notably mitigates overfitting and enhance generalization.
Hyperparameter Analyses. Finally, we analyze the effects of some key hyperparameters intro- duced in our experiments. The first hyperparameter is the number of sampled configurations J used in the meta-learning process. As shown in Tab. 7, the overall safety after fine-tuning im- proves the number of sampled configurations increases. This aligns with intuition, since more meta-learned configurations provide a better estimate of the hypergradient and avoids overfitting. albeit at the cost of longer training time. Interestingly, we observe that increasing J from 1 to 3 provides a notable boost in IP, while further increasing to 5 yields only marginal additional gains, possibly because J = 3 already provides sufficiently generalizable hypergradients. Besides,

---

controls the degree of proximity to the base model. To study its impact, we train ResAlign using different ? and evaluate the corresponding model's IP after being fine-tuned on DreamBench++. A larger 7 can better approximate the ground-truth fine-tuned model, yet also leads to higher insta- bility due to larger variance in hypergradient approximation, as shown in Fig. 5. Overall, our method is stable when 7 is in a reasonable range (i.e., 0 ~ 1). Thus, a relatively wide range of y can be selected for ResAlign.
Tab. 7: Effect the number of sampled configurations J. The metric is IP after fine-tuned on DiffusionDB.
3
Performance (IP)
0.002
0.068
0.054
Training time (min)
43,5
58.2
73.9
5 Discussion
Comparison with More Baselines. In addition to the baselines discussed in Tab. we further compare ResAlign with two additional relevant methods. The first is IMMA [85], which employs bi-level optimization framework prevent the model from learning certain concepts. Although originally designed to prevent unauthorized personal- ized learning, it can also be adapted to our safety-driven unlearning setting for evaluation. The second is the con- current work Meta-Unlearning [15], which introduces a meta-objective that penalizes the norm of harmful gradients and encourages conflicts between harmful and benign gradients, thereby slowing down relcarning harmful concepts while also encouraging the model to "deconstruct" its benign utility when fine-tuned on harmful data. As shown in Tab. 8, while both methods achieve better results than the baselines in Tab. 1, they remain less effective than ResAlign. We attribute this to two main factors. First, both IMMA and Meta-Unlearning adopt first-order approximation, which essentially treats the hypergradient's Jacobian as an identity matrix. This simplification may be less accurate than our ME-based approximation. Second, both methods use fixed datasets and configuration when simulating fine-tuning, which may be less generalizable than cross-cone conss-coffiuurttion meta learning design. We provide further discussions more related work in Appendix and empirical comparisons with additional basclines in Appendix C.2.
Tab. 8: Comparison with other related & con- current works. The metric is IP after fine-tuning.
Method
DB++
Diff.DB
1MMA [85]
0.0752
0.1321
Mcta- Unleaming [15]
0.1128
0.1515
Ours
0.0186
0.0687
Alternative Design. In our main experiments, we adopt the negative denoising score matching loss as Coummuul (i.e., performing gradient ascent on harmful data), owing to its simplicity and strong empirical effectiveness. However, as observed in prior studies [82, 85, 41] and our experiments (e.g.. Fig. 1), this GA-style loss may cause the model generate distorted, mosaic-like output images for harmful or semantically related prompts, which may negatively affect user experience to some extent. Fortunately, ResAlign is a loss-agnostic framework. Users who prefer generating safe yet semantically meaningful images under unsafe prompts can easily employ alternative unlearning objectives. To validate this hypothesis, we replace Charmful with the unlearn- ing losses from ESD and CA train the correspond- ing unlearned models, and evaluate their safety both immediately after unlearning (No FT) and after LoRA fine-tuning DreamBench++ and Diffu- BionDB. As shown in Tab. 9, ResAlign can still mitigate the post-fine-tuning safety rebound for these methods. Notably, the IP before and after unlearning is higher than that of the gradient ascent version, possibly because the ESD and CA losses are more difficult to optimize. We provide further discussions in Appendix E.5 and encourage future work to explore more advanced unlearning losses.
Tab, 9: Effectiveness on ResAlign when combined with unlearning loss from ESD [13] and CA [35].
Method
No FT
DB++
Diff.DB
ResAlign + ESD
0.0610
0.0820
0.1182
ResAlign + CA
0.1230
0.1611
0.1665
6 Conclusion
This paper proposes ResAlign, a novel unlearning framework designed to enhance the resilience of unlearned diffusion models against downstream fine-tuning. Through a Moreau envelope- based reformulation and a meta-learming strategy, ResAlign effectively suppresses the recovery of harmful behaviors while maintaining benign generation capabilities. The effectiveness of ResAlign is validated through extensive experiments across various datasets and fine-tuning setups, and also qualitatively explained from a theoretical perspective. We hope our work can raise community's 's attention to the resilience of unlcarning methods and encourage further research into more robust strategies.
10

---

Acknowledgments
We sincerely thank all anonymous reviewers and our area chair for their comprehensive feedback, helpful suggestions, and the time they invested, which significantly improved the quality of this paper. We also thank Kaifeng Zhang from Wuhan University for his valuable assistance with some preliminary validation experiments the early stage of this project. This work was supported in part by National Research Foundation, Singapore and DSO National Laboratories under its AI Singapore Programme (AISG Award No: Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. also supported in part by the National Natural Science Foundation of China (NSFC) under Grants No. 62202340, No. 62576255, No. 62441238, and No. U/441 'Leading Goose' R&D Program of Zhejiang (2024C01169). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore, DSO National Laboratories, Infocomm Media Development Authority, NSFC, or any other agencies.
and National Research Foundation,
as well as the 'Pioneer' and
References
[1] Stability AI
Stable diffusion v2. stable-diffusion 2, 2022.
cttpss//augingaaa..cc/taaaall
[2] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Chris- tian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):12400) 2019.
[3] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1rX1v:1504.00325, 2015.
[4] Xuanhong Chen, Bingbing Ni, Yutian Liu. Naiyuan Liu, Zhilin Zeng, and Hang Wang. Sim- swapt+: Towards faster and high-quality identity swapping. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(1):576-58) 2023.
[5] Civitai. Civitai: A platform for sharing and discovering ai-generated art models, 2025. Accessed: 2025-05-15.
[6] CompVis. Stable diffusion v1-4.
http:///uugginggaaec.o////aoppp stable-diffusion-ve Accessed: 2025-05-14.
[7] CompVis. Stable diffusion safety checker.
cttp:/ccccccoompmcccsaaaaa wtw.:ffuuiionssftt-eheeeeee www.saacceeeeeeeeeeeeeeae 2022. Accessed: 2025-03-21.
[8] Jiangyi Deng, Shengyuan Pang, Yanjiao Chen, Liangming Xia, Yijie Bai, Haiqin Weng, and Wenyuan Xu. Sophon: con-fine-tunable learning to restrain task transferability for pre-trained models. In 2024 IEEE Symposiun on Security and Privacy (SP), pages 2553-2571. IEFE, 2024.
[9] Prathamesh Dharangutte and Christopher Musco. Dynamic trace estimation. Advances in Neural Information Processing Systems, 34:30088-30099 2021.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. International conference on machine learning, pages 1126-1135. PMLR, 2017.
[11] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.

---

[13] Rohit Gandikota, Joanna Materzynska. Jaden from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2426-2443 2436, 2023.
and David Bau. Erasing concepts
[14] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúska, and David Bau. Unified concept editing diffusion models. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 5111-5124 2024.
[15] Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, and Min Lin. Meta- unlearning on diffusion models: Preventing relearning unlearned concepts. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2131-2141, 2025.
[16) Lucy I. Gao, Jane Ye, Haian Yin, Shangzhi Zeng. and Jin Zhang. Moreau envelope based -eeeernno----wwaalll--ooooooe reformulation and algorithm for bilevel programs. arXiv preprint 1.Kv:2300.16761. 2023.
[17] Naveen George, Karthik Nandan Dasaraju, Rutheesh Reddy Chittepu, and Konda Reddy Mopuri. The illusion of unlearning: The unstable nature of machine unlearning in text-to-image diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13393-13402, 2025.
[18] Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Reliable and efficient concept erasure of text-to-image diffusion models. In European Conference Computer Vision, pages 73-88. Springer, 2024.
[19] Riccardo Grazzi, Luca Franceschi, Massimiliano Pontil, and Saverio Salzo. On the iteration complexity of hypergradient computation. In International Conference Machine Learning, pages 3748-3758. PMLR, 2020.
[20] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine- fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7323-7334, 2023.
[21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. NeurIPS, 2017.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[23] Chia-Yi Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. Safe lora: The silver lining of reducing safety risks when finetuning large language models. Advances in Neural Information Processing Systems, 37:65072-65094, 2024.
[24] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022
[25] Chi-Pin Huang, Kai-Po Chang, Chung-Ting Tsai, Yung-Hsuan Lai, Fu-En Yang, and Yu- Chiang Frank Wang. Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. In European Conference on Computer Vision pages 360-376. Springer, 2024.
[26] Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Joshua Kimball, and Ling Liu. Antidote: Post-fine-tuiing safety alignment for large language models against harmful fine-tuning attack. In Forty-second International Conference on Machine Learning. 2025.
[27] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409.18 18169, 2024.
[28] Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In The Thirteenth International Conference on Learning Representations, 2025.
12

---

[29] Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communicaatins in Statistics-Simuaatiinn Simulation and Computation. 18(3): 1059 1076, 1989.
[30] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[31] Kaiyi Ji, Junjie Yang, and Yingbin Liang, Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning, pages 4882-4892. PMLR, 2021.
[32] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On larce-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint (elt::600044888 2016.
[33] Changhoon Kim, Kyle Min, and Yezhou Yang. Race: Robust adversarial concept erasure for secure text-to-image diffusion model. In European Conference on Computer Vision, pages 461-478. Springer, 2024.
[34] Harold W Kuhn and Albert W Tucker. Nonlinear programming. In Traces and emergence of nonlinear programming. pages 247-258. Springer, 2013.
[35] Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun- -an Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22691-22702, 2023.
[36] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1931-1941, 2023.
[37] LAION-AI.
Laion-asstheticc--redioorr v1. aesthetic- prreeiiccceeedcooo.r.r. 2022. Accessed: 2025-05-14.
https:// //gtth////thtt.../// wwwithub.commmteee Com/LAION-ATI
[38] Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
[39] Boheng Li, Yanhao Wei, Yankai Fu, Zhenting Wang. Yiming Li, Jie Zhang, Run Wang, and Tianwei Zhang. Towards reliable verification of unauthorized data usage in personalized text-to- image diffusion models. In 2025 IEEE Symposium on Security and Privacy (SP), pages 73-73. IEEE Computer Society, 2024.
[40] Guanlin Li, Kangjie Chen, Shudong Zhang, Jie Zhang, and Tianwei Zhang. Art: Automatic red-teaming for text-to-image models protect benign users. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[41] Jiaqi Li, Chuanyi Zhang, Miaozeng Du, Hui Zhang, Yongrui Chen, Qianshan Wei, Junfeng Fang, Ruipeng Wang, Sheng Bi, and Guilin Qi. Forget the token and pixel: Rethinking gradient ascent for concept unlearning in multimodal generative models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 12179- 2025.
[42] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730 19742. PMLR, 2023.
[43] Xinfeng Li. Yuchen Yang, Jiangyi Deng. Chen Yan, Yanjiao Chen, Xiaoyu Ji, and Wenyuan Xu. Safegen: Mitigating sexually explicit content generation in text-to-image models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 4807 4821, 2024.
[44] Yiming Li, Yang Bai, Yong Jiang, Yong Yang, Shu-Tao Xia, and Bo Li. Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection, Advances in Neural Information Processing Systems, 35:13238 2022.
13

---

[45] Yiming Li, Shuo Shao, Yu He, Junfeng Guo, Tianwei Zhang, Zhan Qin, Pin- Yu Chen, Michael Backes, Phillip Torr. Dacheng Tao, and Kui Ren. Rethinking data protection in the (generative) artificial intelligence era. arXiv preprint arXiv:2507.03034, 2025.
[46] Peiyuan Liao, Xiuyu Li, Xihui Liu, and Kurt Keutzer. The artbench dataset: Benchmarking generative models with artworks. arXiv preprint arXiv:2206. 1404, 2022.
[47] Risheng Liu, Zhu Liu, Wei Yao, Shangzhi Zeng, and Jin Zhang. Moreau envelope for non- convex bi-level optimization: A single-loop and hessian-free solution strategy. In Forty-first International Conference on Machine Learning, 2024.
[48] Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. Latent guard: a safety framework for text-to-image generation. In European Conference on Computer Vision, pages 93-109. Springer, 2024.
[49] Yixin Liu, Chenrui Fan, Yutong Dai, Xun Chen, Pan Zhou, and Lichao Sun. Metacloak: Pre- venting unauthorized subject-driven text-to-image diffusion-based synthesis via meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24219-2222 242288, 2024.
[50] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXi:: 2017.
[51] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- based hyperparameter opti- mization through reversible learning. In International conference on machine learning, pages 2113-2122 PMLR, 2015.
[52] Jean-Jacques Moreau. Proximité et dualité dans un espace hilbertien. Bulletin de la Société mathématique de France, 93:273-299, 1965.
[53] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1rXiv:1803.022 180380893909... 2018
[54] nochLtecc. Nudenet. attps://ee //github. http:://ggttuuu.ooooootttitteete www./oott-tecch//neeicctoea 2023. Accessed: 2025-
03-21.
[55] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, To-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
[56] Jiadong Pan, Hongcheng Gao, Zongyu Wu, Taihang Hu, Li Su, Qingming Huang, and Liang Li. Leveraging catastrophic forgetting to develop safe diffusion models against malicious finetuning. Advances in Neural Information Processing Systems, 37:1 11108-111111111 115232, 2024.
[57] Yuang Peng. Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, and Shu-Tao Xia. Dreambench++: A human-aligned benchmark for personalized image generation. In The Thirteenth International Conference on Learning Representations, 2025.
[58] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[59] PurpleSmanAI. Pony diffusion v6 xl. / pony-diffusion-vue
2025.
[60] Wenjie Qu, Wengrui Zheng, Tianyang Tao, Dong Yin, Yanze Jiang, Zhihua Tian, Wei Zou, Jinyuan Jia, and Jiaheng Zhang. Provably robust multi-bit watermarking for ai-generated text. arXiv preprint arXiv:2401.1682) 2024.
14

---

[61] Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. In Proceedings of the 2023 ACM SIGSAC conference on computer and communications security, pages 3403-3417 2023.
[62] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[63] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances neural information processing systems, 32, 2019.
[64] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramer. Red-teaming the stable diffusion safety filter. NeurIPS ML Safety Workshop, 2022.
[65] reach vb. Pokémon blip captions. https: https://huggte http:///gggggg.ccccccaaaaaaaaaae cocpee/eeeeectttttteerhch--bbbba
pokemon-bile sision-blip-caee blip-captions, 2023. Accessed: 2025-05-14.
[66] Lewis Fry Richardson. Ix. the approximate arithmetical solution by finite differences of physical problems involving differential equations, with application to the stresses in masonry dam. Philosophical Transactions of the Royal Society of London. Series A, containing papers of mathematical or physical character, 21 210(459-470):377--3
1911.
[67] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[68] Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszeze, Hassan Sajjad, and Frank Rudzicz. Immunization against harmful fine-tuning attacks. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5234 5247, 2024.
[69] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-drive generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 22500-223 10, 2023.
[70] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 03322222331,2222 2023.
[71] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278-25225 2022.
[72] Shashanka Ubaru, Jie Chen, and Yousef Saad. Fast estimation of tr(f(a)) via stochastic lanczos quadrature. SIAM Journal on Matrix Analysis and Applications, 38(4): 1075-1099, 2017.
[73] Yibo Wang, Tiansheng Huang. Li Shen, Huanjin Yao, Haotian Luo, Rui Liu, Naiqiang Tan, Jiaxing Huang, and Dacheng Tao. Panacea: Mitigating harmful fine-tuning for large language models via post-finettning perturbation. Advances in Neural Information Processing Systems, 2025.
[74] Yiyang Wang and Weining Zhang. Factors influencing the adoption of generative ai for art designing among chinese generation A structural equation modeling approach. IEEE Access, 11:143272-143284, 2023.
[75] Zijie Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image genera- tive models. Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 893-911, 2023.
15

---

[76] Zilan Wang, Junfeng Guo, Jiacheng Zhu, Yiming Li, Heng Huang, Muhao Chen, and Zhengzhong Tu. Sleepermark: Towards robust watermark against fine-tuning text-to-image diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 8213-8224, 2025.
[77] Yijun Yang, Ruiyuan Gao, Xiaosen Wang. Tsung- Yi Ho, Nan Xu, and Qiang Xu. Mma- diffusion: Multimodal attack diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7737-7746, 2024.
[78] Yijun Yang, Ruiyuan Gao, Xiao Yang, Jianyuan Zhong, and Qiang Xu. Guardt2i: Defending text-to-image models from adversarial prompts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[79] Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt: Jailbreaking text-to-image generative models. 2024 IEEE symposium on security and privacy (SP), pages 897-912. IEEE, 2024.
[80] Yuno779.
Anything xl. modd:Verrsiondd-90854, 2025. Community model on CivitAl. Accessed: 2025-08-22.
[81] Kaibo Zhang, Yunjuan Wang, and Raman Arora. Stability and generalization of adversarial training for shallow neural networks with smooth activation. Advances in Neural Information Processing Systems, 37:16160-16193 2024.
[82] Qingjie Zhang, Haoting Qian, Zhicong Huang, Cheng Hong, Minlie Huang, Ke Xu, Chao Zhang, and Han Qiu. Understanding the dilemma of unlearning for large language models. arXiv preprint arXiv:250..2467, 2025.
[83] Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. Advances in Neural Information Processing Systems, 97:36748-36776. 2024.
[84] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang. Jiancheng Liu, Ke Ding, and Sijia Liu. To generate not? driven unlearned diffusion models are still casy to generate unsafe images... for now. European Conference on Computer Vision, pages 385-403 Springer, 2024.
[85] Amber Yijia Zheng and Raymond A Yeh. Imma: Immunizing text-to-image models against malicious adaptation. In European Conference on Computer Vision, pages 458-475. Springer, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and scopes.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have claimed the limitations of our work in Appendix E.6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

We have included the assumptions along with our proposition and included the complete proof as well more discussions in Appendix A.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have described the detailed configurations as well as the methodology in our paper and Appendix to reproduce the claims and results. Moreover, our code as well as the checkpoints for reproducing our main results are made public.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: our paper, datasets used this paper (e.g., DiffusionDB and Dream- Bench++) are all opensourced and available for public access.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wwtpccccppccc.oec.cc...ccc.cctle g/gucdg Cooeeuumiis wwabbbmmsssson wwwiiisssospliooooeooooooieeiiee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have introduced the detailed experimental settings and implementation details in our main paper and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In our paper, all fine-tuning experiments are independently repeated for 3 times with different random seeds and report the avaraged results. Moreover, in Figures, also provide the standard error of the mean as the error area, which a widely-used method illustrate statistical significance of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have introduced our used resources and the expected training time in our implementation details section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttps://nerrr wwtpurips...ppppppeeeeee cctpublic//eieecccccccccceeceeee Beciiic///hiiittieeeeeeeee csGuidelines?
Answer: [Yes]
Justification: We confirm results follow the NeurIPS Code of Ethics https://neurip:: cc/ cc/ppblic/Bthicctiiieee
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We included the ethics statement and discussions on social impact in the Appendix E.7.
Guidelines:
20

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks, as the paper aims to defend against harmful generation in diffusion models instead of introducing safety risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We confirm all used assets are properly cited or credited, and the licenses are properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Please refer to our code repository for details.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... wcc//onfeeeecce/222//11111 for what should or should not be described.
23

---

A Omitted Proofs & Derivations
A.1 Omitted Derivation for Eq. (6)
Recall that we approximate the parameters obtained via finite-step fine-tuning from pretrained model using the following Moreau envelope (ME) formulation:
Prr arg min Crr(d', DFr) +
10' - 012
27
As fUr is the minimizer of the above optimization problem, it satisfies the following first-order optimality condition according to the KKT theorem:
1 - - 0) V CPT(FTT DFT) +
(10)
7
Define function F(O', V ( DFT) - ( 0). The optimality condition implies that F( 0) = O. Since F implicitly defined as a function the Implicit Function Theorem (IFT) to compute how ffr changes with respect to 9. Specifically, IFT states that if F( and gg F is invertible at ffr. then:
through this equation, we can apply
00
0F
aF
(11)
00
00'
08
We compute the Jacobians of F as follows:
OF
1
OF
Li DFT)
I,
(12)
00'
00
Substituting into Eq. (11), we obtain:
00FT
Var Drr)
1 7
-1
1 ワ
(13)
00
Now consider the loss function Lnamiuul (hamlia(fFF)... By the chain rule:
00FT
V Traamnuul (0FT)
V opp (0fam
(14)
00
Then, since the Jacobian matrix is symmetric (as it involves the inverse of symmetric positive definite matrix), we simplify:
V hhammful (OFT) -VV2) DrT) + 1
V oi harmnaal (OFT)
(15)
Finally, multiplying both sides by the inverse term's denominator yields:
C DAT) )1) V Lahhannta = V or, Charmfal (O (OFT)
(16)
Noting that 8' is evaluated its optimal value lir, equivalently write V2 Crr Drr) as V2 (1r(0 Drr) for clarity. This concludes the derivation.
A derivation similar to ours was presented by Rajeswaran et al. [63] in the context of meta-learning Specifically, they model the lower-level transfer (or adaptation) step in meta-learning as a proximal regularized optimization process, which is conceptually similar to our Moreau envelope- based formulation. This allows them to compute the meta- gradients with respect to the meta-parameters through implicit differentiation, technique that shares structural similarity with our derivation above. However, while the underlying mathematical tools overlap, our goals and problem settings are fundamentally distinct. We provide more detailed discussions in Appendix D.

---

A.2 Omitted Proof for Proposition
Proof. Prior works have shown that fine-tuning typically induces only small parameter changes to the pretrained parameters [36]. Therefore, we regard the fine-tuned parameters as the pretrained parameters plus sufficiently small additive perturbation, i.e., ffr + ç. where & = oz By the second-order Taylor expansion around and dropping higher-order terms:
Charmful (frr) Chamnful (0 E)
Charmial (0) + V VoLnummuul (0) 25
(17)
Because z a unit random variable, we have [[[2] = 0. Then, taking expectation over 2, we obtain:
σ²
EB hhrrmful (frT) Charmful (0)] oV Chammtu (0) E[2]
+ 2
VV
ha (0)=
Chamtuu (0)z]
(18)
To compute this expectation, the fact that for any symmetric matrix H and random vector with RB[=]]] -,, we have:
E| Hz] = Tr(E(2z [1) - Tr (
=
1
(19)
Applying this to V C harmlul (0), we obtain:
1
E Vi Chammfull (0)=]
ā (V2 hammtuul (0))
(20)
Hence, substituting into Eq. (18), we obtain the following expression:
o²
EE Charmful (0pr) Charmful (0)
Tr
(21)
2d
2² Since the term 22 is constant independent of 0, minimizing the expected regained harmful loss is
equivalent to minimizing the trace of the Hessian matrix:
arg min E [hurmtull (6FT)
Lharmtul (0) = arg min Tr (1ammtu
(22)
Thus, we finished the proof.
Remarks & Discussions. Proposition I suggests that our proposed additive term implicitly penalizes the trace of the Hessian of the harmful loss near the unlearned model parameters. doing so, it encourages the optimization process not only minimize the harmful loss, but also to favor solutions with flatter local geometry, i.e., regions with lower curvature. This intuition is illustrated in Fig. 6. While prior unlearning methods aim to reduce Charmfuls they typically rely standard SGD-based optimizers, which are known to prefer sharp local minima [32]. As result, these methods may converge to parameter gions such as @ that, although locally optimal, are highly sensitive downstream perturbations. As a result, even benign fine-tuning, which aims to optimize only the benign loss Lrr (i.e., aim to push 6% U to O, may inad- vertently push the model back into regions of high harmful loss, reactivating undesired behaviors.
Cparmtul Eyr
0
unldorn
13n learm
67
0 FT
Fig. 6: Illustration of the impact of (benign) fine-tuning on harmful loss for flat (left) and sharp (right) minima.
"Here, unlike many previous works that restrictively assume z follow a specific distribution (c.g., Gaussian or uniform on the sphere), or confining analysis to first-order approximations like the Neural Tangent Kernel (NTK) [38], we only assume the additive perturbation is scaled unit random variable, i.c., any distribution satisfying E[z] 0 and 2[ '] 1. which is milder and more flexible assumption.
25

---

In contrast, our ResAlign method implicitly promotes flatter optima" (e.g., G in the figure). These flatter solutions are more resilient to subsequent fine-tuning: local updates to minimize LFT induce smaller changes in harmiul thereby reducing the risk of unlearning failure. This perspective suggests that the fragility of prior unlearning methods may arise not from the nature of the downstream data itself, but from the sharpness of the solutions they converge to.
With that being said, we acknowledge that our proposition, like other analyses based on Taylor expansions and additive perturbation models [30, 11 may become less accurate in scenarios involving large parameter shifts (e.g., very large learning rates or very long tine-tuning schedules), or when the perturbation lacks well-behaved distribution. Nonetheless, in practice, particularly in the diffusion model fine-tuning setting, small learning rates and moderate step sizes are commonly used to preserve generative quality and prevent overfitting [69], which supports the validity of the small perturbation assumption. Moreover, since our meta-learning process performed over diverse data sampling and configurations modeling & as an isotropic random variable is a reasonable abstraction. As one of the earliest attempts to theoretically understand the interplay between unlearning and subsequent fine-tuning, our analysis is meant provide qualitative, approximate explanation rather than definitive conclusions. We hope our work can inspire the community to pursue deeper and more precise theoretical investigations into the robustness of unlearning under realistic deployment conditions.
B Detailed Experimental Settings
B.1 More Details on Unlearned Models
In our main experiments, we evaluate our method against 6 text-to-image diffusion models, including the vanilla Stable Diffusion v1.4 [67] and 5 state-of-the-ar baselines for unsafe concept erasure: ESD [13], SafeGen [43]. AdvUnlearn [83], and two variants of LCFDSD [56] (i.e., LCFDSD-NG and LCFDSD-LTT. Below are brief introductions and implementation details of these unlearned models:
SD v1.4 [6]: Stable Diffusion (SD) v1.4 is classical text-to-image latent diffusion model trained on the LAION-2B- dataset, large-scale web-scraped collection of image-text pairs. While this dataset enables the model to learn broad range of visual concepts, it has only undergone rudimentary filtering. As a result, it contains a significant amount of inappropriate content, which lead the model reproduce amplify harmful undesired concepts during generation [70], making it suitable to serve as a valuable baseline for evaluating exx--to-mmage models' inherent toxicity [83, 70]. In our experiments, we directly use the model checkpoint provided by CompVis".
ESD [13]: ESD is concept erasure method that fine-tunes the text-to-image diffusion model's weights to unlearn undesired concepts by matching the noise of the target concept to that of the negative guidance noise. In main experiments, we directly use its provided pretrained "diffusers- Sefffusser-nudity-BBee Soddtt--EDD-UUNET.PP checkpoint', which have unlearned the nudity concept from the SD v1.4 model, for evaluation.
SafeGen [43]: SafeGen is a vision-only unlearning method that specifically targets the sexual category. It works by fine-tuning the diffusion model's self-attention layers using an unsafe-blurre image dataset to mosaic unsafe generation. In work, we directly load their officially released -ereeee---eeeraned----aaee
model hosted on HuggingFace via Diffusion Pipeline for evaluationS.
Ady Unlearn [83]: AdVUnlaam is currently the state-of-ther -ft-e--rtttttttt- safety-driven unlearning method to defend against adversarial prompts. It is built upon the unlcarning loss of ESD and additionally in- corporate an adversarial training paradigm to discover and defend against ad adversarially perturbed un- safe concepts. In experiments, directly load the "AdvUnleam _udity fexx__encodeee full.pt"
"Note that the trace Proposition represents sum the eigenvalues of the Hessian of the harmful loss. Therefore, the minimizer of this trace does not necessarily indicate the curvature to be uniformly small across all directions. It could also some eigenvalues significantly positive or negative as long as their sum remains small. However, since our meta-learinng process simulates and optimizes over multiple directions of fine-tuning updates, we expect that the obtained solution corresponds a region where the harmful gradients are relatively small, at least along the meta-lcamed directions.
"https//huggingffceccoo www..cooooiiiiissaeeeeeitttteeee
hhttss//erraiigg bwwlah haulab.iinfoweeigghhsssd http:///eesddmdddde///////// models/NSF/
wtw.://nggeeeoooetttteeoooooaaaa Pretraaiedd Wieiteneeettttteeeeeeeettst
26

---

checkpoint released by the authors", which erases sexual-related concepts by fine-tuning the text encoder of the Stable Diffusion v1.4 model for evaluation.
LCFDSD [56]: LCFDSD is safety-driven unlearning method designed to defend against harmful fine-tuning, with the goal of mitigating the resurgence of unsafe behaviors when a model is subsequently fine-tuned on harmful data. The method is motivated by the insight that increasing the separation between the latent distributions of clean and harmful data can make it more difficult for the model to learn unsafe content. To achieve this, LCFDSD fine-tunes the original diffusion model using distribution separation loss, coupled with an additional KL regularization loss. The original paper proposes two variants for implementing the distribution separation loss: Noise Guidance (NG) and Latent Transformation (LT). We reproduced the results using the official code provided by the authors Specifically, we adopted the "Safety Reinforcement" setting described in their paper, which initializes the model with an unlearned checkpoint provided by ESD ("diffusers-mudit--SSll Senuddtt-EDD--UNETTTT... -UNEEUUU""") and then applies NG and LT for further training.
B.2 More Details on Fine-tuning Datasets & Methods
Both standard fine-tuning and advanced irssonaiizationttnigg methods are considered our ex- periment. For standard fine-tuning, three datasets are involved: DreamBench+++ [57], DiffusionDB [75], and Harmful-Imgs [56]. Besides, for advanced personalization-tniin Pokémon [65], Dog [57], ArtBench [46], and VGGFace2-HQQ [4] are utilized. Below are their details:
Dream Bench++ [57]] DreamBench++ is a recently introduced dataset specifically designed for evaluating personalized generation. It consists of 360 high-quality, novel text-image pairs collected from various real-world platforms, such as objects, living subjects, and artistic styles that did not barely appeared in the pretraining dataset of SD. These pairs can be used for downstream fine-tuning to enable text-to-image diffusion models to learn and generate novel concepts (i.e., customized generation), such as new objects and unique styles. In our experiments, we aim to simulate a general, benign novel concept learning scenario, so we randomly sample 100 images from the "human" and "style" categories, respectively. The DreamBench++ dataset is certified to contain no harmful images (e.g., sexually explicit content) by the publisher [57].
DiffusionDB [75]: DiffusionDB is large-scale dataset constructed from real-world user in- teractions on public Stable Diffusion Discord channels. It comprises 14 million high-quality, human-authored text-image pairs that reflect authentic user preferences and cover broad spectrum of content, including photorealistic portraits, real-world objects, and stylized, dreamlike imagery. We select 100 text-image pairs from the dataset for our fine-tuning evaluation.
Harmful-Imgs [56]: Harmful-Imgs is dataset composed of 452 sexually explicit text-image pairs. Introduced in prior work [56], it is primarily used to simulate harmful fine-tuning scenarios in diffusion models, where the downstream dataset is contaminated with inappropriate data. In our experiments (Fig. 4), we randomly sample sexually explicit images from this dataset and use them replace clean images in the first two datasets (i.e., DiffusionDB and DreamBench+++) at varying contamination ratios ranging from 0% to 100%. This setup allows us to simulate different levels of harmful data injection, enabling controlled evaluation of model's resilience and safety under compromised fine-tuning conditions.
Besides standard fine-tuning, we also evaluate ResAlign on advanced personalization datasets and methods, introduced as follows:
Pokémon [65]: Pokémon contains 833 high-quality images of Pokémon characters, and each image has corresponding text caption generated by caption model BLIP [42]. In our experiments, we use LoRA [24] to fine-tune the diffusion model to learn the styles from the Pokémon dataset. The configurations follow the official script provided by HuggingFace'. After fine-tuning, the personalized model can generate images in Pokémon anime style.
Dog [69]: Dog contains a set of 5 images of dogs in a specific breed that is not present in the original SD's training dataset. In our experiments, we use DreamBooth [69] to fine-tune the
imagge/rraninnnn text to Amage_lora,p
27

---

diffusion model and learn the characteristics of this specific dog. The configurations follow the official script provided by HuggingFace 10 After personalization learning on this dataset, the model is able to generate this specific dog with special identifier (e.g.. "a [V*] dog").
ArtBench [46]: ArtBench is dataset consisting of 60,000 artworks spanning 10 distinct artistic styles and genres. It is widely used to evaluate a model's ability to learn and generate images in specific artistic styles. In our experiments, we use CustomDiffusion [36] to fine-tune the diffusion model on ArtBench. The training configurations follow the official script provided by HuggingFace" Following the recommendations, we randomly sample 50 images from the same genre to serve as the training data. We repeat the process with three randomly selected genres and report the average performance across them. After learning on this dataset, the model is able to generate images in the same artistic style with a special identifier (e.g.. "an artwork in [V*] style").
VGGFace2-HQ [4]: VGGFace2-HQ is high-quality human face dataset derived from the original VGGFace2. It contains diverse set of identities with high-resolution facial images and is commonly used to evaluate personalized face generation and identity preservation capabilities in generative models. In our experiments, we use SVDiff [20] to fine-tune the diffusion model on VGGFace2- HQ. The training configurations follow the official script provided by the paper's 's authors 2 The resulting model learns to generate identity-specific faces, i.e., "a [V*] face" will generate face that closely resembles particular identity in the training dataset. The reported results are averaged across three randomly selected identities.
B.3 More Details on Evaluation Metrics
In our experiments, adopt comprehensive set of metrics to evaluate the performance of unlearned models across three key dimensions: safety, benign generation quality, and personalized generation capability. First, assess model's ' unsafe generation tendency, we use two metrics: Inappropriate Rate (IP) and Unsafe Score (US). Below are their introduction and implementation details:
IP [70]: Inappropriate Rate (IP) is calculated by generating images using prompts from the I2P dataset [70]. I2P consists of prompts collected from real-world online forums, where users shared prompts that happened to generate harmful images. Notably, only about 1.5% of the full set of 4702 prompts are explicitly labeled as toxic as analyzed in [70], indicating that the harmfulness often stems from the model's 's own unsafe generalization rather than from obviously unsafe inputs. In our main paper, we focus the sexual category within I2P, selecting all 931 relevant prompts. The IP score is then defined as the average proportion of images flagged as inappropriate across all generated images. A lower IP indicates lower risk of generating harmful content.
US [61]] Unsafe Score (US), on the other hand, is evaluated using prompts from the Unsafe dataset [61], with outputs assessed by a pretrained NSFW classifier MHSC [61]. The overall procedure is similar to the evaluation of IP. Note that the original Unsafe dataset is not classified into different categories. To focus on the sexual category, we filter the dataset by generating images for each prompt and select subset of 200 prompts that have the highest averaged sexually unsafe probability as rated by MHSC. This subset is used in our experiments to evaluate the unsafe score.
Second, we evaluate the model's ability to generate general benign content using three widely adopted metrics computed on the COCO dataset [3]. To ensure reproducibility and reduce computational overhead, we follow Zhang et al. [83] and adopt their publicly released subset of 10,000 randomly sampled text-image pairs from COCO for evaluation.
FID [21]: Fréchet Inception Distance (FID) measures the distributional distance between generated and real images. A lower FID score indicates higher realism and visual quality. For each text-image pair in the evaluation set, we use the caption to generate an image with the evaluated model, and compute the FID between the generated and corresponding ground-truth images. This metric quantifies how well the unlearned model retains its ability to produce realistic, benign content.
CLIP Score [62]: The CLIP score assesses semantic alignment between generated images and their text prompts using the CLIP model [62]. A higher score indicates stronger text-image consistency.
http::/wh...mmooooooooo////////e drrprmbonndrabbthh..yy http://www...tttooooocccceeeoooe wwiffusion//rriinnnnnommm cuustm sattiiusssoooffffssiioo.omm diffusion.py
28

---

We use the same 10,000 captions as in the FID computation, generate corresponding images, and calculate the average cosine similarity between CLIP embeddings of each image-capting pair.
Aesthetic Score [37]: The aesthetic score evaluates visual appeal using a pretrained neural aesthetic predictor trained on human-labeled preferences. Higher scores correspond to greater visual quality and human alignment. Following the same evaluation protocol, we generate 10,000 images and compute their aesthetic scores. The final score is obtained by averaging over all generated images.
Finally, to evaluate the model's performance in generating personalized content after fine-tuning, we adopt the following metries:
CLIP-I and CLIP-T [69]: These two metrics are widely used to evaluate the quality of personalized generation. CLIP-I measures the semantic similarity between the generated image and the reference images (i.e., the training set images), while CLIP-T assesses the alignment between the generated image and its associated textual prompt. A higher CLIP-I score indicates better identity/featture preservation, whereas a higher CLIP-T score reflects stronger text-image alignment, suggesting better text controllability and reduced overfitting to visual appearance alone. The evaluation prompts and protocols follow the DreamBooth paper [69].
DINO Score [55]: DINO score is recent metric derived from self-supervised vision transformers. It measures feature-level similarity between generated and reference images, and is particularly useful for capturing fine-grained structural and identity details in personalized generation tasks.
These metrics offer a comprehensive evaluation of the unlearned model's behavior across safety, general generation quality, and personalized fine-tuning effectiveness. Overall, model with higher CLIP Score, Aesthetic Score, CLIP-I, CLIP-T, and DINO Score, and lower IP, US, and FID is favored, as it indicates better ability in generating stronger personalized generation capability, and reduced risk of producing harmful or unsafe content.
human preference-alinnee general images,
B.4 More Implementation Details
We used three datasets for training ResAlign, each serving a distinct objective. First, the harmful dataset D Phammuu is used compute the harmful loss naamful We constructed this dataset by selecting 150 unsafe prompts from existing unsafe datasets (c.g.. NSFN-SSWWSE 56k [43]). Second, the preservation dataset preserve used compute the regularization term R0). For this dataset, we selected 140 benign prompts from COCO-Objects and CelebA-HQ. Finally, to simulate downstream fine-tuning data, we constructed the fine-tuning simulation dataset Dyr by randomly sampling 100 prompts from prompt pool selected from DiffusionDB. NSFW-56K, and Dharmful- For all datasets, we generated corresponding images for each prompt to form text- image pairs. We manually refined and reformatted some prompts ensure consistency across all datasets. Notably, DiffusionDB is utilized in multiple contexts throughout our work. To avoid data leakage and overfitting, we manually verified that there is no overlap between any of the subsets used for different purposes. Additionally, gradient clipping and adaptive hyperparameter scheduling are also used to stabilize training.
For evaluation, unless otherwise stated, our fine-tuning is based on the official script provided by diffusers whose default configuration is full-pan hallmparrmeter fine-tuning on the UNet parameters with learning rate of 1 x 10-5, a batch size of 1, and training step of 200, using Adam W [50] optimizer with default hyperparameters. For methods such as AdyUnleam and Receler, where the unlearning is performed on modules (e.g., text encoder or auxiliary adapters) rather than on the UNet, we include these modules during downstream fine-tuning as well. This ensures fair and comprehensive assessment of each method's worst-case resilience.
C More Experimental Results
C.1 Results on Adversarial Attacks & Potential Adaptive Attacks
While defending against adversarial attacks and adaptive attacks is not the primary focus of this work, we evaluate ResAlinn's 's effectiveness when faced with several well-established attack strategies for unlearned diffusion models and explore a potential adaptive attack strategy to understand whether ResAlign can be easily bypassed or compromised with minimal effort.
httpss/stheeeooiinneooootttoooos to_imae
29

---

Results on Existing Adversarial Attacks. We first evaluate the robustness of ResAlign against several existing adversarial attacks, including SneakyProma [79]. MMA-Diffusion [77]. and Unlearn- DiffAtk [84], which is specifically designed against unlearned diffusion models. To comprehensively assess robustness under different threat models,
Tab. 10: Evaluation on existing adversarial attacks.
we consider both black-box and white-box sce-
narios. In the black-box setting, we directly utilize the adversarial prompt benchmarks from Prompt and MMA-Diffusion. For each ad- versarial prompt, we generate images using the target model. A prompt is considered successful if at least one of the generated images is flagged as unsafe by the NudeNet detector. We follow prior work [77] and report the metric ASR-3, defined as the percentage of prompts for which at least one unsafe image is produced. In the white-box setting, adopt the evaluation protocol of UnleamDiiffAr [84], which involves using gradient information from the unlearned model to iteratively optimize adversarial prompts for maximum reactivation assess ResAlign's robustness under strong, white-box
Attack
ESD
AdvUnlear
Ours
SneakyPrompi
13.78%
3.57%
0.51%
MMA MMA-Diffusion
26.00%
1.60%
1.50%
UnleamDiffAAk
83.05%
24.58%
33.90%
of harmful behaviors. This allows adversarial setup, and we report the ASR metric, evaluated as the ratio of final successful optimized prompts as rated by NudeNet detector. As shown in Tab. 10, although ResAlign is not specifically designed to defend against adversarial prompts, it achieves high level of robustness. Our method significantly outperforms ESD and is even comparable AdvUnlearn, the state-of- of-the-art approach explicitly targeting adversarial prompt defense. Moreover, it is feasible to combine Adv Unlearn with our ResAlign unlearned model, which we verify can reduce the attack success rate of UnleamDiffAttk to 8.47%. We will extend our method to defend against adversarial prompts in our future work.
Discussion on Adaptive Attacks. We further investigate whether ResAlign can be bypassed by adaptive strategies. Intuitively, ResAlign constrains the local loss landscape around the current unlearned parameters (see Eq. 5 and Proposition 1), which weakens harmful-gran -gradeet signals under fine-tuning, We thus evaluate two representative adaptive strategies that aim to escape this subspace.
First, we study parameter-space escape attack, which augments the downstream fine-tuning ob- jective with a term that actively pushes parameters away from the current solution: Ladaptive (8') = C
X1|0' pllp. where we consider p {1,2} and set À 0.5. The intuition is that by encouraging the optimizer to move far from 0. the model may escape the resilient region induced by ResAlign and thus recover harmful capability. However, shown in Tab. 11, both variants fail to meaningfully bypass ResAlign, as their IP scores remain largely comparable to those obtained from standard fine-tuning.
Tab. 11: Results of the parameter- -panc escape attack.
The reported metric is IP after fine-tuning.
Setting
DreamBench++
DiffusionDB
Harmful-Imgs
0.0294 0.0283
0.0523 0.0512
0.2381
p=
0.2402
Second, evaluate a progressive relearning attack inspired by [17], which hypothesizes that gradu- ally exposing the model from clean to increasingly unsafe data may reopen memory pathways and strengthen recovery. To simulate this, we split the 200-step fine- tuning into 4 equal stages: the first two use benign datasets (Dream- Bench++ and CelebA-HQ), the third stage uses relatively safer subset of Harmful- Imgs (bottom 50% as ranked by NudeNet). and the final stage uses the full Harmful-Imgs set. As shown in Fig. 7, we observe that while this strategy brings better IP increase than single datasets, the IP does not substantially increase until the final stage where explicit harmful images are introduced, indicating the pro- gressive schedule alone cannot meaningfully circumvent ResAlign without access to sufficient harmful data. Overall, we conclude that it is non-trivial effectively bypass ResAlign through simple adaptive modifications to the loss function or data exposure schedule in the absence of sufficient harmful data.
02
0.1
0.0
50
100
Fine-tuning Steps
Fig. 7: Evaluation results of the progressive relearing attack.
C.2 Comparison with Additional Baselines.
To further strengthen our evaluation, we additionally compare ResAlign with recent unlearning
methods: RACE [33], RECE [18], and Receler [25]. As summarized in Tab. 12, while these
methods achieve strong pre-fine-tuning safety performance, they consistently exhibit noticeable rebounds in toxicity after downstream fine-tuning on both DreamBench++ and DiffusionDB. Two key
30

---

observations can be drawn from these results. First, the post- fine-tunmen tuning safety degradation remains universal issue across all evaluated methods, including these newly proposed ones, suggesting that current unlearning methods generally lack resilience under downstream fine-tuning. Second, stronger pre-fine finn-nunttng safety does not necessarily imply better post-fine-tuning resilience. For example, RACE achieves low initial IP but suffers notable safety deterio- ration after fine-tuning. These findings further corroborate our central conclusion: existing unlearning approaches primarily optimize for immediate safety rather than long- term resilience, underscoring the importance of developing frameworks like ResAlign that explicitly address this gap.
Tab. 12: Comparison with additional base- lines. We report the results averaged over three independent runs. The metric is IP.
Method
No FT
DreamBench++
DiflusionDB
RACE RECE
0.031
0.127
0.217
0.043
0.074
0.116
Receler
0.063
0.142
0.183
We hope these results can inspire future work toward more resilient safety alignment methods.
C.3 More Ablation Study & Hyperparameter Analysis Results
Effect of Meta Learning. To further verify the effectiveness of our meta-learning design. we conduct ablation study by varying the diversity of fine-tuning algorithms used during meta-training.
Specifically, we train ResAlign with and without meta learning. yet with different parameter selection schemes. The results are summarized in Tab. 13. From these results, two insights can be drawn. First, without meta-learning. ResAlign tends overfit to the specific fine-tuning algorithm seen during training. For example, although LoRA and LyCORIS share similar structures, a model trained only on LoRA performs poorly when applied to LyCORIS. consistent with prior findings that local settings [49, 44]. Second, incorporating multiple configurations during meta-learning (e.g.. Full Param. LoRA) not only improves in-distribution performance but also enhances cross-algorithm generalization, even to unseen fine-tuning methods such as SVDiff. These results confirm that meta-learning effectively mitigates overfitting and yields more generalizable hypergradients.
Tab. 13: Effect of meta-learning. The metric is IP.
Training Config
Full Param.
LoRA
LyCORIS
SVDiiT
Full Param. only (wlo ML)
0.028
0.084
0.074
0.013
LoRA only twlo ML)
0.096
0.045
0.069
0.047
Full Param. + (w ML)
0.030
0.054
hypergradients can overfit to
Effect of B. We further study the effect of the regularization weight 3, which controls the relative strength of the resilience regularization term in ResAlign. As shown in Tab. 14, a very small /3 leads to insufficient regu- larization, whereas an excessively large B (close to 1.0) may cause the model to over- look current harmfulness signals and lead to unstable training. Nevertheless, ResAlign remains robust within a wide and practical range (e.g., B E [0.3, 0.9]), consistently achieving strong safety performance on both +reamBench++ and DiffusionDB. This demonstrates that ResAlign is not overly sensitive to the choice of B.
Tab. 14: Effect of B. The metric is IP after fine-tuning.
B
0.5
0.7
0.9
1.0
DreamBench++
0.094
0.038
0.046
0.018
0.017
0.160
DiffusionDB
0.181
0.096
0.069
0.083
0.061
0.241
Effect of y on DiffusionDB dataset. In the main paper, we evaluated the impact of different values of on our ResAlign's performance using the DreamBench++ dataset. In this section, we also conduct the ablation study on the DiffusionDB dataset. As shown in Fig. 8, a similar trend is observed on the DiffusionDB dataset. This supports our conclusion that ResAlign achieves stable resilience to downstream fine-tuning across a wide range of y values, indicating that ResAlign is not overly sensitive to the choice of this hyperparameter.
0.10
0.05
Regularzation Stroggh
Fig. 8: Effect of y.
C.4 Computational Efficiency
We further compare the computational cost of ResAlign with representative baselines in terms of both training time and average GPU mem- ory usage. As shown in Tab. 15, our method achieves a favorable balance between efficiency and performance. Specifically, ResAlign re- quires comparable average memory to most full- parameter fine-tuning baselines (e.g., LCFDSD-NG). Notably, although peak GPU memory
Tab. 15: Comparison of computational cost.
Method
Avg. VRAM (GB)
Training Time (min)
ESD
12.4
14.3
AdyUnleam LCFDSD-NG Mltt--Ulleaming Res (Ours)
29.4
210.3
20.3
38.4
35.7
725. 4
25.4
58.2
31

---

usage temporarily doubles (compared to average GPU memory usage) during Hessian-vectts product computation, this overhead occurs only briefly. Overall, ResAlign remains computationally practical and well-suited for real-world unlearning applications.
D
More Related Work
D.1 Flat-Minima Optimization and Hessian Trace Regularization
As discussed in Proposition 1, our ResAlign implicitly imposes regularization on the trace of the Hessian of the harmful loss, possibly encouraging the model to converge to flatter regions of the loss landscape. This connects our work to the broader literature on Hessian-based regularization and flat-minima optimization, which has been broadly studied in optimization and generalization theories.
Classical approaches such as Hutchinson's 's estimator [29], Lanczos approximation [72], and Cheby- shev polynomial methods [9] have been proposed approximate minimize the trace of the Hessian. These techniques have found applications in diverse areas including convex optimization and phys- ical simulation. However, they typically rely on stochastic numerical estimators that are either non-differentiable or computationally intensive to differentiate through, making direct optimization of the Hessian trace largely impractical for modern large-scale neural networks.
In deep learning, several works have explored indirect flatness optimization, such as Sharpness-Aware Minimization [1]] and Entropy-SGD [2], which aim to improve generalization by favoring flat minima in the loss landscape. Yet, they are primarily designed for classification tasks and have not been extensively studied for diffusion models or investigated from the perspective of safety resilience.
Our work differs in both motivation and mechanism. Rather than explicitly minimizing the Hessian trace or relying on costly curvature estimation, ResAlign achieves a similar flatnes-innuucin effect implicitly through a simple resilient unlearning objective. This design efficiently regularizes the harmful loss landscape toward flatter regions without introducing direct Hessian computations, making it particularly well-suited for large-scale diffusion models and safety-driven unlearning.
D.2 Meta Learning and Bi-level Optimization
Meta-learnin, or "learning to learn," " is classical paradigm in machine learning that aims to train a meta-model capable of rapidly adapting to new tasks with limited data [10, 633. The high- level idea is to learn across multiple tasks such that the resulting meta meta-parameters encode general learning dynamics that transfer efficiently to unseen scenarios. Unlike conventional meta- meta-learning, whose primary goal to improve few-shot adaptation, our work leverages meta-learning to enhance the generalizability of hypergradient estimation and to mitigate overfitting to a specific simulated fine-tuning configuration.
Meta-learning can also be formulated as a bi-level optimization problem, where the outer optimization updates the vneta-aaaaetters to achieve better generalization across tasks, and the inner optimization represents task-specific adaptation. From this perspective, our framework is closely related to bi-level optimization and its variants that estimate the hypergradient (or "meta-gre "eett-grrdient") connecting the two levels. A central challenge in such formulations lies in efficiently computing this hypergradient. To address this, prior works have developed approximate implicit differentiation (AID) methods that estimate gradients through the optimality condition of the inner problem [19, 31]. Our method can be viewed as special instance of AID derived under the Moreau envelope- based reformulation, which enables efficient hypergradient estimation without unrolling fine-tuning trajectories.
As acknowledged in Appendix A, iMAML [63] is, the best of our knowledge, the most technically related prior work to ours, which also introduces a proximity term in the inner optimization to regularize the update w.r.t. the base parameters. However, our work differs in two key aspects. First, the optimization objective fundamentally diverges: iMAML aims to improve few-shot generalization, whereas we aim to mitigate the recovery of harmful behaviors during downstream fine-tuning. Second, iMAML requires explicitly enforcing the proximity term during the inner optimization of each fine- tuning task, which inherently constrains the optimization dynamics the lower-level task. In contrast, our method does not require any modification to the downstream fine-tuning loss. Instead, we treat the fine-tuned model as the minimizer of an ME objective when estimating the hypergradient. This design, facilitated by the properties of diffusion models (see more analyses in Appendix I) allows
32

---

ResAlign to impose no constraints on the choice of downstream loss, making it broadly applicable. Beyond these key aspects, our work also differs in the technique for solving the linear system, and has proposed a cross-configuration meta learning technique.
D.3 Defending against Harmful Fine-tuning
The degradation of safety alignment after downstream fine-tuning has recently emerged as an important and actively studied phenomenon. Most existing research [26, 23, 68, 73, 28, 27] focuses on large language models, where researchers have explored various strategies to mitigate such degradation, including interventions during pretraining, safety-aligned fine-tuning, and controlled data filtering or objective design. Among them, Booster [28] is most relevant our work, which works by estimating the effect of a single-step harmful fine-tuning mhe-tuning update on model safety and employs a first-order approximation to compute defensive gradient update.
In contrast, research on defending against harmful fine-tuning in diffusion models remains largely underexplored. To the best of our knowledge, besides the methods discussed in Sec. (e.g.. LCFDSD), the concurrent work Meta-Unlearning [15] is also directly related to our scenario. Additionally, while works such as IMMA [85] and SOPHON [8] were originally developed for preventing diffusion models from learning specific tasks or concepts, their underlying mechanisms can also be adapted to enhance resilience against safety degradation after fine-tuning.
The key distinction between our approach and these existing methods lies in how the hypergradient is estimated and how the downstream fine-tuning configurations selected. Specifically, our frame- work introduces (i) a principled implicit differentiation based on the Moreau envelope approximation, and (ii) cross-configuration generalization via meta-learning. These two components jointly enable ResAlign to more accurately capture the higher-order interactions between model parameters and downstream fine-tuning dynamics, leading to improved resilience against harmful behavior recovery. We provide further discussion and experimental comparisons in Appendix E. I and Sec. 5.
E
More Discussion
E.1
Discussion on Other Approximations
In this section, we discuss our choice of the Moreau envelope- based approximation for estimating the hypergradient. In our early trials, we also explored several other viable alternatives. The first the first-order approximation widely adopted by previous works [85, 28, 15, 8], which directly
assumes the Jacobian satisfies 0% a0 ~ I, thus simplifying the hypergradient as VeCaamiuul Vez (0 Although computationally efficient, this approach yields substantially degraded performance compared to our ME method (IP: 0.16 vs. 0.07 on DiffusionDBB, as it neglects the higher-order interactions between base and fine-tuned parameters. Besides, we also considered unrolled differentiation [51], which computes the hypergradient by backpropagating through the full fine-tuning trajectory. While theoretically accurate, it is computationally prohibitive for large diffusion models. For instance, on SD v1.4, single A A100 GPU can only unroll up to 3 steps, far too few to capture meaningful fine-tuning dynamics. Finally, our ME-based approximation models the fine-tuned parameters the minimizer of ME objective and applies implicit differentiation to estimate the hypergradient by solving a linear system via Richardson iteration. This approach only and highly scalable.
s
requires access to the final fine-tuned model, making it ieriiuuratiio-agnossee Empirically, is nearly as fast as the first-order method (approximately 58 min vs. 50 min) but delivers much better alignment performance. We thus finally adopt ME-based approximation.
One key underlying insight behind our ME-based approximation that pretrained diffusion models typically require only small parameter updates to adapt to downstream tasks [36], and in some cases (e.g., Textual Inversion [12]), updating even a single embedding can suffice. Hence, viewing the fine-tuned model as the minimizer of a task-specific objective regularized by proximity to the base model aligns closely with the real fine-tuning behavior of modern diffusion models. This makes our approach both practical and theoretically well-suited for this defense scenario.
33

---

E.2 Discussion on the Diversity of Meta Learning Configurations
A key principle underlying our design is to maintain both the dataset and configuration pool as diverse and representative as possible, which is crucial for the meta-leamed update to generalize to unseen fine-tuning scenarios. This motivates us to propose the meta-lcarning technique. Currently, our implementation of ResAlign serves as prototype to validate the effectiveness of the proposed meta-learning mechanism, and we adopt several representative fine-tuning configurations widely used in the diffusion community, such standard LoRA-based and Dream setups. This already provides strong generalization across real-world downstream tasks without evident overfitting (e.g.. anime style transfer, object and face personalization, and art style learning, as shown in Tab. 1-4). Moreover, our framework is flexible-users can readily customize the dataset and configuration pool to better match their target domain. For instance, in avatar personalization tasks that primarily involve human faces and small learning rates, incorporating more face-relater data and sampling smaller leamiig-rate configurations during meta-learning would likely yield better results.
To validate whether enriching configurations enhance our generalization, we have expanded our configuration pool to include broader range of fine-tuning algorithms (e.g., LoRA with varying hyperparameters, LyCORIS, and QLoRA) and objective functions (e.g.. SVDiff and CustomDiffu- sion losses). This integration is straightforward since our ME-based estimation only requires the final fine-tuned model, imposing no assumptions on the underlying optimization or configuration. Preliminary results indicate that this extension can further improve robustness-for instance, on DoRA+DiffusionDD the updated model achieves an improved post-finettuning IP of 0.048 compared 0.069 in the original setup. Future work can further expand the meta-leaming pool and systemati- cally investigate the interactions among different fine-tuning configurations, which may lead to even more effective and broadly generalizable meta-learned updates in ResAlign.
E.3 Discussion on ResAlign's Effectiveness on Contaminated Data
Recall that our experiments in Fig. 4 show that ResAlign can also mitigate harmfulness rebound even when exposed to contaminated or harmful data during downstream fine-tuning. This phenomenon can possibly be theoretically explained by Proposition I, which shows that ResAlign effectively imposes penalty on the trace of the Hessian (i.e.. the second-order derivatives) of the harmful loss with respect to model parameters. Intuitively, this encourages the model to converge to locally flatter region of the loss landscape for the harmful objective. Ideally, around the optimized parameters, the first-order gradients of the harmful loss would be small in most directions. Consequently, even when the model encounters harmful samples, the gradient signals that could re-enable harmful behaviors remain weak, making it difficult for downstream fine-tuning to substantially increase harmfulness. In practice, achieving perfectly flat region where all harmful gradients vanish is largely unrealistic due to the stochasticity of diffusion training and the high-dimensional landscape. Nonetheless, Res ResAlign empirically suppresses the harmful recovery to some extent, as reflected by the consistently lower IP observed even under contaminated conditions.
E,4
On Instance-level and Concept-level Unlearning
Our work focuses on safety-driven unlearning, which aims to reduce model toxicity and unsafe generations. the existing literature, such unlearning can be achieved through two complementary paradigms: (i) misee unlearning [56, 43], which directly unlearns certain unsafe instances from the model (e.g., through gradient ascent or reinforcement learning-based preference optimization on a collected dataset), and (ii) concept-level unlearning [13, 14], which suppresses specific semantic concepts (c.g., "nudity" or "violence") by aligning or perturbing their corresponding representations. While these two strategies differ in implementation, both have been shown to effectively mitigate harmful behaviors and can generalize to unseen unsafe prompts. For example, our evaluation metrics (i.e., IP and US) are measured prompts completely disjoint from the training set, yet both instance-level and concept-level approaches substantially reduce unsafe generations, suggesting that instance-level unlearning can also imply broader concept-level mitigation.
Importantly, our proposed ResAlign framework is loss-agnostic and can seamlessly incorporate either instance-level or concept-level unlearning objectives. As reported in Tab. 9, applying ResAlign on concept-level losses can also improve their post-fine- fint-innttuungg safety, demonstrating that our framework generalizes across unlearning paradigms. Moreover, our experiments reveal that all existing unleam-
34

---

ing baselines suffer from a significant safety drop after downstream fine-tuning, regardless of the paradigm of their unlearning objectives. This observation highlights a fundamental and underexplored challenge in safety-driven unlearning, i.e., the lack of resilience to downstream adaptation, which ResAlign takes an early yet meaningful step toward addressing.
E.5 Discussion on Side Effects and Artifact Patterns
While ResAlign achieves strong safety resilience, we also observe several characteristic failure patterns associated with the use of the GA loss for unlearning. Specifically, GA may induce over- rejection behaviors, where the model rejects prompts that are safe in intent but semantically similar to unsafe ones seen during unlearning. Importantly, this phenomenon is not necessarily detrimental. In some cases, even semantically safe prompts can inadvertently yield unsafe generations, and such conservative refusal can effectively prevent these failures from occurring. Besides possible over-rejection, during repeated experiments, we find that even under identical configurations, the rejection patterns may vary across train- ing runs. As shown in Fig. 9, besides the global wavy or scale-like mosaic artifacts across the entire image as shown in Fig. I, typical artifact patterns also include (but are not limited to): (a)-(b) muscle-like distortions on human bod- ies, (c)-(d) incomplete global denoising. (c)-(f) wave- or grid-shaped destructive streaks, (g)-(h) localized wavy or scale-like mosaic patterns human bodies. While these artifacts effectively suppress unsafe content, we ac- knowledge that some of them may negatively impact user experience or even cause discomfort (e.g.. for individuals with trypophobia). Thus, we strongly encourage practitioners consider this side effect brought by GA, and carefully evaluate model behaviors before real-world deployment. User discretion is also recommended.
(al-() micce-ikke distaction
[e]-(]) globally incomplete denoising
on human bodies
(e-(I glotal grid-lupel destructive streaks
(g>(Bh) localized wavy scale-like mosni putterns
Fig. 9: Visualization results on rejection ar- tifact patterns observed in our experiments.
For users who prefer safer yet more visually coherent outputs, substituting the GA-based loss with milder unlearning objective (c.g., ESD's loss) can alleviate such over-rejection behaviors, as even rejected attempts will be given safe image instead. We encourage future research to systematically study these patterns and develop better loss designs that balance safety with user experience.
E.6 Limitations & Future Work
Our work still has the following limitations, which we aim to address in future work. First, our approach requires simulating fine-tuning during the unlearning process, which introduces additional computational overhead. However, the total cost remains within a practical range (less than one GPU hour for full run in our main experiments), which is significantly more efficient than retraining a model from scratch. In real-world scenarios, users can choose methods that best balance their needs for resilience and safety given their computational budget. Second, our Moreau envelope-based based approximation is inherently an approximation. Nevertheless, our experiments show that it is robust to long fine-tuning steps and wide range of learning rates, and it outperforms methods based first- order approximations. As discussed in the main paper, obtaining fully accurate hypergradients would require tracking and computing full Hessian matrix products, which is computationally prohibitive for diffusion models. Future work could explore more accurate yet efficient estimation strategies. Third, we acknowledge that when the downstream fine-tuning dataset is entirely composed of toxic data, both our method and existing baselines inevitably experience an increase in harmfulness. We believe that perfect resilience under such adversarial conditions is intrinsically difficult, given the strong generalization and few-shot capabilities of modern diffusion models. Nonetheless, our method demonstrates stronger resilience compared to baselines, maintaining relatively low IP even under fully toxic fine-tuning Fourth, similar to previous works, the NSFW classifiers used in our experiments are not perfectly accurate, and there might be false negatives or false positives. However, as all methods are evaluated under the same setting, the results are still fair and comparable to large extent. We hope future work can build more accurate classification models. Finally, while we designed and evaluated adversarial & adaptive attacks and demonstrated ResAlign's robustness against them, we recognize that stronger or more sophisticated attacks may emerge in the future. Developing robust defenses against such potential threats remains an important direction for future work.
35

---

E.7 Ethics Statement & Broader Impact
This work aims to address the safety challenges of text-to-image diffusion models, particularly their vulnerability to unsafe behavior inherited from toxic pretraining data and re-emerging during downstream fine-tuning. Our proposed method, ResAlign, is designed specifically improve the resilience of safety-driven unlearning techniques, helping to mitigate the unintended recovery of harmful behaviors when models are fine-tuned on downstream data. We have carefully reviewed and ensured that our research adheres to the NeurIPS Ethics Guidelines for Authors All experiments are conducted using publicly available datasets and established benchmarks under their original licenses, or are constructed from filtered or recombined subsets of these datasets, ensuring new unsafe content is introduced. All illustrated potentially sensitive outputs in this paper have sensitive regions carefully masked for viewer protection. We do not release any attack-related tools or data; any derived resources will be made available through gated access upon request. We will responsibly notify relevant developers if future findings reveal potential vulnerabilities. This work is intended solely for defensive and safety-enhancing purposes; we do not endorse or support any offensive unethical applications. We believe that our contributions will help advance the trustworthy and responsible development of generative Al technologies.
36


Reviews:
[{"review_id": "k5gsDNvY3V", "reviewer": "Reviewer_ioTR", "summary": "This paper studies the challenge of safety of diffusion  model, in particular, the safety of unlearning methods for diffusion models. Although unlearning could help block unsafe generation of diffusion models, the authors found that existing diffusion models with unlearning methods still face the safety challenge after fine-tuning, even with benign images. As a result, the author presents an simple yet effective regularization to the unlearning framework, that considers model parameters after fine-tuning. The author conducted extensive experimental across many configuration, and the experimental results justify the safe-driven and benign generation  abilities of the proposed unlearning framework.", "questions": "Overall, I believe this work is interesting and solid. But I still have two concerns, as in the \"Strengths And Weaknesses\" part. The authors could refer to this part, and consider my concerns.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths:  \n- First of all, the motivation of this paper is attractive and interesting. The authors consider the safety of diffusion models after fine-tuning. This problem is more challenging and realistic than the safety of training diffusion model. \n- Second, the authors presented an intuitive method by using a simple yet effective regularization, which considers the model parameters after fine-tuning, not just the current state. \n- Third, for consideration of computation efficiency, the authors use Implicit Differentiation to calculate the gradients. \n- Fourth, I found the experiments sufficient and extensive. The authors conducted extensive experiments across diverse configurations to justify the safe-driven and benign generation  abilities of the proposed unlearning framework. \n\nThe weaknesses: \n- On the meta-learning part: The authors used meta-learning to simulate different fine-tuning situation. However, I believe there two issues with the meta-learning part. First, the configurations for the meta-learning is still limited and the authors haven't presented strong ablation study to prove its usefulness. Second, the meta-learning would increase the running time of these method. The author haven't discussed about this. \n- On the Effectiveness on Contaminated Data: The proposed even showed robustness on harmful data. However, the authors haven't discussed about this.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ioTR, we sincerely thank you for your precious time and valuable comments. We are deeply encouraged by your positive recognition of our **attractive and interesting motivation, intuitive and solid method, and extensive experiments**. We sincerely hope the following clarifications and new analyses can address your concerns.\n\n---\n\n**W1.1:** On the meta-learning part: The authors used meta-learning to simulate different fine-tuning situation. However, I believe there are two issues with the meta-learning part. First, the configurations for the meta-learning are still limited.\n\n**A1.1:** Thank you very much for your careful reading and insightful comments! We hope the following clarifications and additional results can address your concern:\n- We appreciate your point regarding the diversity of our current configuration set. Recall that, our meta-learning's objective is to mitigate the risk of overfitting to a single simulated configuration. By training across a variety of fine-tuning settings and aggregating them, meta-learning acquire a hypergradient that generalizes well to diverse downstream configurations. **At this stage, our implementation is intended as a prototype to verify the effectiveness of this meta-learning mechanism.** Therefore, in our current manuscript, we selected mainstream fine-tuning settings widely adopted in the community as a representative. **Our results show that even under this initial setup, ResAlign already generalizes well to unseen configurations without evident signs of overfitting (e.g., Tab. 2, 4, and 5).** \n- However, we fully agree that covering a more diverse set of configurations would undoubtedly further improve generalizability. **Following your suggestion, we have initiated the extention of our meta-learning configurations during the rebuttal period.** Specifically, we have extended our configuration pool to further include more fine-tuning algorithms (e.g., LoRA with different hyperparameters, LyCORIS, and QLoRA) and losses (e.g., SVDiff and CustomDiffusion's fine-tuning losses). Note that this intergation is seamless and implementation-friendly, as our ME-based estimation only requires the final fine-tuned model and does not impose any assumptions on the downstream fine-tuning algorithm or configuration. **Our preliminary results show that this can indeed further improve performance** (e.g., on DoRA+DiffusionDB, our new model has an improved post-finetuning IP of 0.048, compared with 0.069 of the original model). \n \nWe will continue our evaluation and add the corresponding results and more discussions in our revision. If you have further suggestions on specific configurations or real-world fine-tuning scenarios that you would like us to incorporate, we are very happy to follow up. Sincerely thank you again for your valueable comment!\n\n\n---\n\n**W1.2:** Currently, the authors haven't presented strong ablation study to prove meta-learning's usefulness. \n\n**A1.2:** Thank you for the helpful comment! **Following your suggestion, we conducted ablations on the diversity of fine-tuning algorithms within the meta-learning process as a representative example.** Specifically, we trained ResAlign under different meta-learning sampling schemes on learning algorithm and evaluated their performance on various downstream fine-tuning algorithms. The reported metric is IP after fine-tuned on DiffusionDB.\n  | Config (during training) | Full Param. | LoRA |   LyCORIS  |  SVDiff  |\n  | :---------------------------: | :---------: | :--: | :------: | :------: |\n  |        Full Param. only  (w/o meta learning)      |     0.028    | 0.084 |   0.074   |   0.043   |\n  |           LoRA only      (w/o meta learning)     |     0.096    | 0.045 | 0.069 |   0.047   |\n  |       Full Param. + LoRA  (w/ meta learning)    |     0.030    | 0.051 |   0.054   | 0.036 |\n  \nFrom these results, we draw two key insights: \n- **Without meta learning, there is a high risk of overfitting.** For example, although LyCORIS and LoRA are similar in paradigm, training ResAlign only on LoRA still performs poorly on LyCORIS. This aligns with previous findings [1,2] that hypergradient estimation over a single setting may overfit.\n-  **Our meta learning (Full Param. + LoRA) not only improves in-distribution performance, but also enhances performance on similar (LyCORIS) and even completely unseen (SVDiff) methods.** This indicates that our meta-learning indeed mitigates overfitting and learns a hypergradient that is more generalizable. \n  \nWe will add more ablation studies and related discussion in the revision. \n\nRef:\n\n[1]: Liu et al. Metacloak: Preventing unauthorized subject-driven text-to-image diffusion-based synthesis via meta-learning. CVPR 2024.\n\n[2]: Li et al. Untargeted backdoor watermark: Towards harmless and stealthy dataset copyright protection. NeurIPS 2022.\n\n---\n\n**W1.3:** Second, the meta-learning would increase the running time of these methods. The authors haven't discussed about this.\n\n**A1.3:** Thank you very much for your insightful comment! We fully agree that the computational cost introduced by meta-learning is an important consideration for real-world deployment. Please find our clarifications and additional experiments below:\n\n- We admit that the meta-learning process does increase the training time, as we estimate and aggregate hypergradients across multiple simulated fine-tuning configurations at each update step. **The main hyperparameter controlling this is the number of sampled configurations per iteration, i.e., $J$.** Increasing it improves the generalizability of the learned update direction but also proportionally increases training time. Thus, **$J$ reflects a direct trade-off between computational cost and effectiveness.**\n-  To better understand this trade-off, we conducted an ablation study varying $J$:\n\n| J (number of configurations) | 1     | 3 (default) | 5      |\n| ---------------------------- | ----- | ----------- | ------ |\n| Performance (IP)            | 0.092  | 0.068        | 0.054  |\n| Training time (minutes)      | 43.5 | 58.2    | 73.9 |\n\nWe observe that increasing $J$ from 1 to 3 provides a noteble boost in performance (lower IP), while further increasing to 5 yields only marginal additional improvement, possibly because $J=3$ already achieves sufficiently generalizable hypergradients.\n\n- We believe that this computational overhead is acceptable for two reasons. **First, all the cost of ResAlign with meta-learning is  only a *one-time cost* incurred during the model preparation (training) stage**, regardless of how many times the model is deployed or downstream fine-tuned. There is no additional runtime overhead at all during downstream fine-tuning or inference, i.e., the user-side deployment cost remains unchanged. **Second, setting $J=3$ (our default) results in a training time of less than 1 hour on a single A100 GPU**, which is efficient compared to retraining a diffusion model from scratch with filtered data.  It is worth noting that expanding the diversity of configurations in meta-learning does not further increase the per-iteration complexity, since we only sample $J$ configurations per update. \n\nWe will add further discussion and quantitative analysis of runtime trade-offs in the revision. Thank you again for this valuable suggestion!\n\n---\n\n**W2:** On the effectiveness on contaminated data: The proposed method even showed robustness on harmful data. However, the authors haven't discussed about this.\n\nR2: Thank you for your careful reading and insightful suggestion! This is really a good point. We hope the following explanation can address your concern:\n- Recall that Proposition 1 shows our ResAlign method essentially imposes a penalty on the trace of the Hessian (second-order derivatives) of the harmful loss with respect to the model parameters. **Intuitively, this encourages the model to converge to a region of the loss landscape for $\\mathcal{L}_{\\text{harmful}}$ that is locally flat.**\n- More precisely, “flatness” in this context indicates that, around the optimized parameters, the first-order derivative (the gradient) of the harmful loss is minimized in all directions (ideally, this would approach zero, indicating a perfect local minimum). **In such a situation, even if the model is directly exposed to harmful samples, the gradient signal enabling harmful recovery would still be weak**, making it difficult for fine-tuning to substantially increase harmfulness. \n- Of course, due to the high complexity of diffusion models and the stochastic nature of SGD, it is generally unrealistic to reach such a perfect minimum where the gradient is strictly zero everywhere. **Therefore, empirically, our method can suppress the recovery of harmful capabilities to a certain extent**, but cannot guarantee that the harmfulness will not increase at all during downstream updates, as also reflected by the fact that IP can still be optimized in our experiments.\n\nWe will add more discussion in our revision and further explore the underlying mechanism in our future work. Thank you again for this valuable point!"}, {"review_id": "YXsslqpf2l", "reviewer": "Reviewer_Q5a3", "summary": "The target of this paper is to develop a defense method against fine-tuning attacks to expose unsafe generations on text-to-image diffusion models. The basic idea is to add an additional penality to ensure a low loss term on finetuned parameters. This also enforces the optimum to be located at a non-sharp region.", "questions": "1. Since the parameter space is high-dimensional, the effectiveness of the method can depend on how good the approximation region \\theta_{FT}^* lies in compared to the true effective fine-tuning attack region. In other words, the fine-tuning datasets and configurations all make a difference here. Then, do you have any criteria or tips to select the fine-tuning datasets and configurations?\n\n2. What is the result of changing $\\beta$?", "limitations": "The scope of this paper is about instance-wise unlearning. In other words, the model is trained to unlearn certain unsafe instances. It is essential to include discussions on whether such instance-level unlearning will imply concept-level unlearning (e.g., sexual content). Also, the discussion of instance-level full unlearning against recovery and theconcept-level one.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow. The background & related work section reminds readers of the related knowledge very effectively. The Motivation & Problem Formulation paragraph also introduces the high-level idea and motivation very clearly.\n\n2. The research question itself (defend against diffusion finetuning attacks) is interesting and the paper is one of the first works to try to address it.\n\nWeakness:\n\n1. There is limited details on ME approximation, which is the major technique involved in the method part of this paper. It is insightful to provide some viable approximations you may consider during the progress and why ME approximation is adopted. Also, does ME approximation have any insightful implications on this specific scenario of finetuning attack defenses.\n\n2. There is a lack of related work on optimization methods to achieve flat-region optimum and specification of differentiation to this line of work. For instance, is there any work that directly optimization the hessian trace (with approximation)?\n\n3. The results in Tab.1 is not very convincing. It seems that the model before fine-tuning is more safe than the baselines, then it is not known that whether the safety after fine-tuning comes from this better initial point or the additional penalty on curvature of loss landscape.", "was_revised": true, "final_justification": "I appreciate the additional analysis of why using ME by comparing two other trials, and the related work on optimization methods to achieve flat-region optimum. The paper would be more insightful with these two improvement. Based on the additional comments and analysis, I raise my score to acceptance.", "rebuttal": "Dear Reviewer Q5a3, we sincerely thank you for your detailed and constructive comments. We greatly appreciate your recognition of our **clear writing, interesting problem setting, and novelty**. We sincerely hope the following clarifications and new experiments can address your valuable concerns.\n\n---\n\n**W1:** There is limited detail on ME approximation. It is insightful to provide some viable approximations you may consider and why ME is adopted. Also, does ME have any insightful implications on this specific scenario of finetuning attack defenses.\n\n**A1:** Thanks so much for this insightful comment!\n- **In our early trials, we have indeed considered several possible alternatives.** Below are the considered strategies:\n    - **First-order Approx.:** it directly assume the Jacobian satisfies $\\partial\\theta^\\*\\_{\\text{FT}}/\\partial\\theta\\approx I$. Thus, $\\nabla\\_{\\theta} \\mathcal{L}\\_{\\text{harmful}}(\\theta^\\*\\_{\\text{FT}}) \\approx \\nabla\\_{\\theta^\\*_{\\text{FT}}} \\mathcal{L}\\_{\\text{harmful}}(\\theta^\\*\\_{\\text{FT}})$. **This strategy is efficient but inaccurate.** Empirically, **it results in notably poorer performance** compared to our ME (IP: 0.16 vs. 0.07 on DiffusionDB).\n    - **Unrolled Differentiation:** it computes the hypergradient by unrolling the entire fine-tuning trajectory and backpropagate through them all. **While theoretically more precise, it is memory- and computation-prohibitive** as both scale linearly with the number of fine-tuning steps. On SD v1.4, one A100 GPU can only support unrolling up to 3 steps—far too few for meaningful downstream fine-tuning. Therefore, we consider it impractical in our setting.\n    - **Our ME-based Approx.:** We approximate the fine-tuned solution as the minimizer of a ME objective and use implicit differentiation to estimate the hypergradient by solving a linear system via Richardson iteration. It only requires the final fine-tuned model, allowing seamless integration with any configuration. **It is highly efficient and scalable for large diffusion models.** Empirically, **our method is nearly as fast as the first-order method**, taking only about 58 minutes (vs. 50 mins for the former), **while delivering much better results**.\n- **A key insight under why our ME works well** for fine-tuning defense is that large-scale pre-trained diffusion models are already very strong and typically require only small updates to reach the optimum for most downstream tasks. Specifically, recent studies have shown that, after fine-tuning, model parameters would undergo only minor changes [1]. In some cases (e.g., Textual Inversion [2]), even tuning a single embedding suffices for successful personalization. Thus, modeling the fine-tuned model as the minimizer of a FT loss plus a proximity term to the base (i.e., our ME approx.) is highly realistic. This makes our approach both practical and theoretically well-suited for this defense scenario.\n\nWe will add more details on ME and further discuss it in our revision. Thank you again for this insightful suggestion!\n\nRef:\n\n[1]: Kumari et al. Multi-Concept Customization of Text-to-Image Diffusion. CVPR 2023.\n\n[2]: Gal et al. An image is worth one word: Personalizing text-to-image generation using textual inversion. ICLR 2023.\n\n---\n\n**W2:** There is a lack of related work on optimization methods to achieve flat-region optimum and specification of differentiation to this line of work. For instance, is there any work that directly optimization the hessian trace (with approximation)?\n\n**A2:** Thank you very much for your careful reading and for raising such a deep and thoughtful question!\n- **Indeed, there has been research on optimizing for flat minima or minimizing the Hessian trace in optimization theory**, such as Hutchinson’s Estimator, Lanczos Approximation, and Chebyshev Approximation. However, these methods typically rely on stochastic or numerical methods that are either non-differentiable or very costly to differentiate. Thus, **directly optimizing the Hessian trace (even with these approximations) is usually impractical for large-scale neural networks.**\n- In recent years, there has been work that indirectly optimizes flatness for DNNs (e.g., Sharpness-Aware Minimization, Entropy-SGD). **However, these approaches primarily target classifiers, aiming to improve generalization rather than safety resilience.** To our knowledge, these methods have not been extended to diffusion models or linked to safety resilience, and **our work is the first to explore this area**. ResAlign avoids the expensive estimation of Hessian; instead, the model automatically regularizes flatness in an efficient and practical way, making it well-suited for large-scale diffusion models.\n\nWe will conduct a more comprehensive literature survey and add a more detailed discussion of related work in our revision. Sincerely thanks again for raising this insightful point! \n\n---\n\n**W3:** The results in Tab.1 are not very convincing. It seems that the model before fine-tuning is more safe than the baselines, then it is not known whether the safety after fine-tuning comes from this better initial point or the additional penalty on curvature of loss landscape.\n\n**A3:** Thanks so much for this thoughtful question! We conducted a comprehensive ablation study to confirm that our better post-fine-tuning safety inherents from our proposed regularization term. **Due to space limit, please kindly refer to the last response (Q1 & R4) of Reviewer #ev88 for details.** We are sorry for the inconvenience and thanks very much for your understanding!\n\n---\n\n**Q1:** The effectiveness of ResAlign can depend on how good the approximation region lies in compared to the true effective fine-tuning attack region. In other words, the fine-tuning datasets and configurations all make a difference here. Then, do you have any criteria or tips to select the fine-tuning datasets and configurations?\n\n**A4:** Thank you for this insightful question! \n- **Our key principle is to keep the data and configuration pool as diverse and representative as possible, which underpins our meta-learning method.** In practice, we use the large and diverse DiffusionDB plus a broad meta-learning configuration pool (Sec. 4.1). As shown in our experiments, this unified setup already yields strong performance across a range of real-world tasks (Tab. 1-2) and configs, including unseen ones (Tab. 4-5, Fig. 2-4). **Thus, our current implementation already provides strong practical value for most downstream applications** (e.g., anime style transfer, object & face personalization, and art style learning).\n- Further, we suggest that **users may customize the dataset and configuration pool to better match their specific application**. For example, for avatar personalization models (which mostly involves human faces and smaller learning rates), it would be beneficial to include more face-related data and sampling more small learning rate configs during meta learning. We will add more discussion in our revision. Thank you again for this insightful question!\n\n---\n\n**Q2:** What is the result of changing $\\beta$?\n\n**A5:** Thanks for this valuable question! $\\beta$ controls the relative strength of our resilience regularization. A very small $\\beta$ leads to insufficient regularization, while a very large $\\beta$ (close to $1$) may cause the model to ignore current harmfulness, resulting in training instability. As shown below, **ResAlign is not sensitive to the specific value of $\\beta$ when in a reasonable range (e.g., $[0.3, 0.9]$) and consistently performs well.** \n\n|         | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.0 |\n| - | - | - | - | - | - | - |\n| DreamBench++  | 0.094 | 0.038  | 0.046  | 0.018  |  0.017      | 0.160        |\n| DiffusionDB  | 0.181  | 0.096  | 0.069  |   0.083  | 0.061   | 0.241        |\n\nWe will add these results and more discussion in our revision. Thanks again for the constructive question!\n\n\n\n---\n\n**L1:** The scope of this paper is about instance-wise unlearning. In other words, the model is trained to unlearn certain unsafe instances. It is essential to include discussions on whether such instance-level unlearning will imply concept-level unlearning (e.g., sexual content). Also, the discussion of instance-level full unlearning against recovery and the concept-level one.\n\n\n**A6:** Thanks so much for this constructive comment!\n\n- **We respectfully clarify that the scope and goal of our work is safety-driven unlearning, aiming at reducing model toxicity and unsafe generations.** In current literature, this can be achieved either by directly unlearning collected unsafe instances (i.e., instance-wise unlearning), or by aligning certain concept keywords' noise/activations (i.e., concept-level unlearning). **While these two strategies differ in implementation, both have proven effective at reducing toxicity and generalizing to unseen unsafe prompts.** For example, our toxicity metrics (i.e., IP and US) are evaluated on unsafe prompts completely disjoint from train set, yet both types of approaches mitigate unsafe generation, demonstrating their capability for global mitigation.\n- Furthermore, while our main experiments use an instance-level loss, **our framework is loss-agnostic and fully compatible with concept-level unlearning losses**. As shown in Appendix Table 10, plugging concept-level losses into our resilient framework also boosts post-finetuning safety, confirming our method's generality.\n- Finally, regarding safety resilience, **our results show that all current baselines universally suffer from a drop in safety after downstream finetuning, regardless of whether they use instance- or concept-level losses**. We believe this is a fundamental and underexplored challenge that deserves further attention, and our framework offers an early yet valuable step towards understanding and mitigating this challenge.\n\nWe will more discussion in our revision. Thanks again for this thoughtful feedback!"}, {"review_id": "tTky4Wuop3", "reviewer": "Reviewer_ev88", "summary": "This paper proposes a new method for unlearning in diffusion models, aiming to ensure that a concept is completely forgotten—even after downstream fine-tuning. The authors observe that previous unlearning methods tend to lose their effectiveness once the model undergoes fine-tuning, even on data different from the target dataset. Motivated by this, the paper introduces a framework that explicitly incorporates the fine-tuning process into the unlearning formulation. Through mathematical induction and a meta-learning-based configuration, the proposed method achieves robust unlearning that is resistant to downstream fine-tuning.", "questions": "The proposed formulation accounts for the fine-tuning process, so the optimization is not solely over the original model but also considers its fine-tuned variants. However, as shown in Tab. 1, the proposed method also performs better than baselines in scenarios without any downstream fine-tuning, which seems counter-intuitive. Could the authors explain why the method still improves unlearning effectiveness even in the absence of fine-tuning?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The observation that fine-tuning—even on non-target data—can compromise unlearning is insightful and could inspire future research.\n2. The proposed method is well-motivated and theoretically grounded.\n3. The experimental results are generally strong and supportive of the method's effectiveness.\n\nWeaknesses:\n\n1. The paper omits a key related work [1], which is only briefly mentioned in the introduction but not discussed in the related work section nor empirically compared. This reference also uses a meta-learning approach and is highly relevant. The paper should clarify how its proposed formulation improves upon [1] and include direct empirical comparisons.\n2. The experiments are limited to older diffusion model versions (e.g., Stable Diffusion v1.4 and v1.5). Extending the evaluation to newer models like SD v2.0 or SD XL would enhance the paper’s applicability, especially since many fine-tuned checkpoints are available for these versions on platforms like HuggingFace and Civitai.\n3. The paper lacks a comparison of computational costs against baseline, including both VRAM usage and training speed. Such information would be valuable for understanding the practical feasibility of the proposed method in real-world deployments.\n\n[1] Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, and Min Lin. Meta-unlearning on diffusion models: Preventing relearning unlearned concepts. arXiv preprint arXiv:2410.12777, 2024.", "was_revised": true, "final_justification": "In my original review, I was mainly concerned about the differences between this paper and a specific prior work that also uses meta-learning, in terms of both performance and methodology, as well as the generalization ability of the proposed method. These concerns have been well addressed in the rebuttal, which I find convincing. I therefore recommend accepting the paper.", "rebuttal": "Dear Reviewer #ev88, we sincerely thank you for your precious time and valuable comments. We are deeply encouraged by your positive recognition of our inspiring motivation, good motivation, and grounded method, and extensive experiments. We sincerely hope the following clarifications and new analyses can address your concerns.\n\n---\n\n**W1:** The paper omits a key related work [1], which is only briefly mentioned in the introduction but not discussed in the related work section nor empirically compared. This reference also uses a meta-learning approach and is highly relevant. The paper should clarify how its proposed formulation improves upon [1] and include direct empirical comparisons.\n\n**A1:** Thank you so much for this constructive feedback! We genuinely appreciate your attention to [1]. It is absolutely a good and relevant work that we also value. We hope the following clarifications address your concerns.\n\n- **Technical Differences.** While both works reference “meta-learning” in names, **our technical motivations and implementations are fundamentally different from [1]**.\n    - **[1] draws inspiration from meta-learning literature to introduce a meta-objective** that penalizes harmful gradient’s norm and makes harmful and benign gradients conflict, enabling the model to “deconstruct” its benign utility when fine-tuned on harmful data. The commonly used first-order approximation trick is leveraged to make it tractable. **This meta-objective directly improves model safety after fine-tuning.**\n    - In contrast, **ResAlign leverages meta-learning to sample from a broad distribution** of downstream fine-tuning configurations and datasets. **The primary motivation is to promote cross-configuration generalization and prevent overfitting.** Our core safety resilience, however, comes from our ME-based implicit differentiation technique, which allows faithful modeling of long-term adaptation dynamics. \n    - Therefore, we respectfully emphasize that, **although both works coincidentally use the term “meta-learning” in names, the underlying motivations, techniques, and principles are fundamentally different. The appearance of [1] does not diminish the novelty of our approach.**\n\n- **Empirical Comparisons.** We fully agree that a direct empirical comparison is valuable. We respectfully note that **we have already included such comparisons in Appendix D.2 of our original submission (see Tab. 11)**, benchmarking ResAlign against [1] and several other recent preprints or possible baselines from other less related domains. As shown in Tab. 11, while does enhance resilience over traditional unlearning methods, **[1] is still less resilient than our ResAlign** (e.g., DiffusionDB: IP 0.1515 vs. 0.0687, see full results in Tab.11 of Appendix D.2). We attribute this advantage to: (1) [1] adopts a first-order approximation to compute the meta-objective, which in effect treats the Jacobian as an identity matrix. This is less accurate than our ME-based approximation, which thus somewhat limits the method’s effectiveness; and (2) our cross-configuration meta-learning design, which offers better generalization than the fixed data/config setup in [1].\n\n- **Further Explanation.** Finally, we would like to provide some clarification regarding our handling of [1]. **At the time of submission (NeurIPS'25 deadline), [1] was presented as an arXiv preprint and not formally published.** Considering the possibility of further technical or implementation improvements of preprint papers, we focused our main paper comparisons on formally published works, while placing comparisons with [1] in the appendix. We sincerely apologize for not highlighting this comparison more clearly up front. Recently, we have noted that [1] has been officially accepted at ICCV. **In our revision, we will provide a detailed discussion in the related work section and update our empirical comparisons in the main paper.** Thank you again for this valuable feedback!\n\n---\n\n**W2:** The experiments are limited to older diffusion model versions (e.g., Stable Diffusion v1.4 and v1.5). Extending the evaluation to newer models like SD v2.0 or SD XL would enhance the paper’s applicability, especially since many fine-tuned checkpoints are available for these versions on platforms like HuggingFace and Civitai.\n\n**A2:** Thanks for this valuable and practical suggestion!\n- By design, our method is model-agnostic and readily applicable to different diffusion architectures. We respectfully note that **our originally submission have already included additional results on SD v1.5 and SDXL** (as in Appendix C.1, Tab. 6), which have shown that ResAlign remains effective and greatly improves safety resilience after fine-tuning on these models.\n- We fully agree with your point that many popular fine-tuned checkpoints are widely used on platforms like CivitAI. To address this, **in addition to SD v2.0, we have also included new experiments during the rebuttal period** on two widely adopted fine-tuned models: AnythingXL and PonyDiffusion:\n\n| Model         | Pre-ft | DreamBench++ | DiffusionDB |\n| ------------- | ------ | ------------ | ----------- |\n| SD v2.0       | 0.004 | 0.031       | 0.078      |\n| SDXL          | 0.033 | 0.044       | 0.059      |\n| AnythingXL    | 0.015   | 0.062         | 0.087        |\n| PonyDiffusion | 0.023   |  0.045         | 0.067      |\n\n- As shown, **ResAlign is still highly effective on these models, demonstrating the generality of our framework**. We will include these new results in our revision. Thank you again for this constructive suggestion!\n\n---\n\n**W3:** The paper lacks a comparison of computational costs against baseline, including both VRAM usage and training speed. Such information would be valuable for understanding the practical feasibility of the proposed method in real-world deployments.\n\n**A3:** Thank you for raising this insightful question!\nFollowing your insightful comment, we provide a direct comparison of training time and average VRAM usage (all methods are run with batchsize=1 for fairness):\n\n| Method            | ESD  | AdvUnlearn | LCFDSD-NG | Meta-Unlearning | **ResAlign (Ours)** |\n| ----------------- | ---- | ---------- | --------- | --------------- | ------------------- |\n| Avg. VRAM (GB)    | 12.4 | 29.4       | 20.28      | 35.7            | 25.4                |\n| Training Time (minutes) | 14.3  | 210.3       | 38.4       | 725.4            | 58.2              |\n\nIn general, **our avg VRAM usage is similar to most other baselines that fine-tune all parameters.** (Note: ESD is lower because it only updates cross-attention layers.)  **Besides, our training time is in within a practical range for real-world use, and substantially more efficient than Meta-Unlearning.** Note that our peak VRAM would be ~2x during HVP computation, but this only occurs briefly and can be easily mitigated using off-the-shelf gradient checkpointing technique. In practice, popular diffusion frameworks like HuggingFace Diffusers provide this feature with only minimal changes to the training script. We will add the results in our revision.\n\n---\n\n**Q1:** As shown in Tab. 1, the proposed method also performs better than baselines in scenarios without any downstream fine-tuning. Why?\n\n**A4:** Thank you very much for this thoughtful question! \n- Compared with classical baselines (e.g., ESD), ResAlign has 3 major technical differences. **(1)** We adopt the negative denoising score matching loss (i.e., gradient ascent, GA), which has been shown to yield better unlearning as it is easier to optimize. **(2)** We train on NSFW-56K, a large, diverse dataset (vs. baselines’ single concept keyword prompt). **(3)** Our additional resilient unlearning objective term, which is our core contribution.\n- To understand the effect of each difference, we conducted an ablation study with the results shown below:\n\n| Method                     | IP (pre-ft) | DreamBench++ | DiffusionDB |\n| -------------------------- | ----------- | ------------ | ----------- |\n| ESD (baseline)             | 0.0677      | 0.1661       | 0.2209      |\n| GA                         | 0.0347      | 0.2324       | 0.2474      |\n| GA + NSFW-56K              | 0.0029      | 0.2270       | 0.2350      |\n| GA + NSFW-56K + Reg (Ours) | 0.0014      | 0.0186       | 0.0687      |\n\n- From the results, we draw that **(1)** **Using GA and NSFW-56K contributes to safety before fine-tuning**, consistent with previous conclusions.  **(2)** **However, GA and NSFW-56K differences does not provide resilience after fine-tuning.** In contrast, these methods also degrade sharply after downstream updates. **(3)** **Only with our resilient unlearning objective do we observe significantly improved post-finetuning safety.** This aligns with our results in the main paper (Tab.3 first row), where w/o our resilient unlearning objective term the model cannot retain performance after fine-tuning.\n\nTherefore, we can conclude that **the improved pre-fine-tune safety comes from our loss and dataset choice**. However, the safety resilience  after fine-tuning (i.e., our main contribution) comes specifically from our proposed objective (i.e., the additional penalty on the curvature of the loss landscape), not merely from a better initial point before fine-tuning. We will add the results and more discussion in our revision."}, {"review_id": "ajlxxr160g", "reviewer": "Reviewer_bbmS", "summary": "The paper proposes ResAlign, a safety-driven unlearning framework aimed at enhancing the resilience of text-to-image (T2I) diffusion models against downstream fine-tuning. It addresses a critical limitation of existing unlearning methods: their fragility to fine-tuning, even on benign datasets, which can inadvertently recover harmful behaviors. ResAlign introduces a novel optimization objective that penalizes the resurgence of harmful behaviors post-fine-tuning, approximated efficiently using a Moreau Envelope formulation and enhanced with a meta-learning strategy to generalize across diverse fine-tuning scenarios. Extensive experiments on Stable Diffusion v1.4 demonstrate that ResAlign outperforms state-of-the-art baselines in maintaining safety after fine-tuning while preserving benign generation capabilities.", "questions": "Q1: Figure 1 shows ResAlign-generated images as mosaics. What causes this degradation, and how could it be mitigated without compromising safety?\n\nQ2: Experiments use Stable Diffusion v1.4. How does ResAlign perform on newer models like SDXL?\n\nQ3: The paper compares ResAlign to a limited set of baselines (e.g., ESD, SafeGen, AdvUnlearn, LCFDSD), neglecting recent methods such as RECE (ECCV24), Receler (ECCV24) and RACE (ECCV24). How does ResAlign perform relative to these newer approaches, and what implications might their inclusion have for the evaluation’s comprehensiveness?\n\nQ4: Beyond the $\\mathcal{L}_{\\text{adaptive}}$ fine-tuning attack in Section C.2, what other adaptive fine-tuning strategies (e.g., alternative loss modifications or multi-stage optimization) could challenge ResAlign’s safety constraints? How can future evaluations enhance ResAlign’s robustness against such strategies?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths \n\n- The paper tackles an underexplored yet significant issue—the vulnerability of safety-driven unlearning to downstream fine-tuning. This is a timely and practical contribution, especially given the increasing personalization of T2I models.\n- The use of a Moreau Envelope-based approximation to model fine-tuning as an implicit optimization problem is clever and computationally efficient, avoiding the need to track multi-step fine-tuning trajectories. The integration of meta-learning to simulate diverse fine-tuning configurations further enhances generalization.\n- Proposition 1 links the additional optimization term to the curvature of the loss landscape, providing a plausible explanation for improved resilience.\n- The experiments span multiple datasets, fine-tuning methods (e.g., DreamBooth, LoRA), and configurations, convincingly showing ResAlign’s superiority in safety retention and its minimal impact on benign generation quality.\n\n### Weaknesses\n\n- Figure 1 shows that ResAlign-generated images appear as mosaics post-unlearning, raising concerns about practical usability. While safety is improved, this severe degradation could limit real-world applicability, a point insufficiently addressed. \n- The reliance on Stable Diffusion v1.4, an older model by 2025 standards, questions the method’s relevance to state-of-the-art T2I architectures (e.g., SDXL). This limits the generalizability of the findings.\n- The paper compares ResAlign only to a few methods (e.g., ESD, SafeGen, AdvUnlearn, LCFDSD), neglecting recent advancements such as RECE (ECCV24), Receler (ECCV24) and RACE (ECCV24). This omission limits the evaluation’s comprehensiveness.\n- The Moreau Envelope and Proposition 1 assume small perturbations and isotropic randomness, which may not hold under extensive fine-tuning or large learning rates. The paper lacks discussion on these assumptions’ real-world validity.", "was_revised": true, "final_justification": "Initially, I had concerns regarding the mosaic effect observed in some of the harmful generations and the lack of certain experimental evaluations, specifically the generative model used and some baseline comparisons. The authors' rebuttal effectively addressed these issues by providing clear explanations and supplementary experiments that resolved my initial concerns. \nMoreover, the other reviewers are also generally favorable towards acceptance. Therefore, I decide to increase my rating from 4 to 5.", "rebuttal": "Dear Reviewer #bbmS, we sincerely thank you for your precious time and valuable comments. We are deeply encouraged by your positive recognition of our **important issue, timely and practical contribution, grounded method and theoretical explanation, and extensive experiments**. We sincerely hope the following clarifications and new experiments can address your concerns.\n\n---\n\n**W1 & Q1:** Figure 1 shows that ResAlign-generated images appear as mosaics, raising concerns about practical usability. While safety is improved, this severe degradation could limit real-world applicability. What causes this degradation, and how could it be mitigated?\n\n**A1:** Thank you very much for this exceptionally thoughtful question! We appreciate your close reading and insight.\n\n- **The mosaic effect is primarily caused by the choice of unlearning loss**. Specifically, our main implementation uses negative denoising score matching loss (i.e., gradient ascent), which maximizes the difference between the predicted and ground-truth noise for unsafe prompts. As a result, the model learns to predict \"incorrect\" noise for unsafe prompts, leading to failed denoising and thus the observed mosaic-like outputs.\n- Respectfully, we believe **this does not strongly diminish practical utility** for two reasons:\n    - **The mosaic occurs only for unsafe generations.** For safe prompts and benign downstream fine-tuning, the utility is unaffected, as shown by comparable the FID/CLIP/Aesthetics scores with baselines in Tab. 1 and the strong Personalized Generation Utility in Tab. 2.\n    - **The primary goal of safety-driven unlearning is to *prevent unsafe generation.*** In this sense, **generating an useless image (mosaiced) for malicious prompts is also viable** (similar to direct filtering or blocking). This phenomenon is also observed in several other SOTA unlearning methods (e.g., SafeGen, LCFDSD), which also produce blurred or meaningless outputs for unsafe prompts.\n- Of course, we recognize that user requirements can vary in the real-world. **Fortunately, ResAlign is a loss-agnostic framework: users who prefer returning a safe yet meaningful image can simply adopt a different unlearning loss** (e.g., ESD's), which would return a clothed woman when prompted by \"a nude woman\". In fact, we have already explored such variants in the Appendix C.4 of our original submission (Tab. 10). **Results show that we can achieve safe and meaningful generation under unsafe prompts (see Fig. 8) while still maintaining good post-fine-tuning safety  (Tab. 10).**\n\nWe will expand this discussion in our revision. Thank you again for highlighting this important practical point!\n\n---\n\n**W2 & Q2:** The reliance on Stable Diffusion v1.4, an older model by 2025 standards, questions the method’s relevance to state-of-the-art T2I architectures (e.g., SDXL). This limits the generalizability of the findings. How does ResAlign perform on newer models like SDXL?\n\n**A2:** Thanks for this valuable and practical suggestion!\n- By design, our method is model-agnostic and readily applicable to different diffusion architectures. **In fact, we have already included results on SD v1.5 and SDXL in Appendix C.1 of our original submission, which have shown that ResAlign remains effective on these models (see Tab. 6).**\n- To further demonstrate the broad applicability of ResAlign, **during the rebuttal period, we have also included new experiments during on additional recent popular models** in diffusion forums (e.g., CivitAI):\n\n| Model         | Pre-ft | DreamBench++ | DiffusionDB |\n| ------------- | ------ | ------------ | ----------- |\n| SD v2.0       | 0.004 | 0.031       | 0.078      |\n| SDXL          | 0.033 | 0.044       | 0.059      |\n| AnythingXL    | 0.015   | 0.062         | 0.087        |\n| PonyDiffusion | 0.023   |  0.045         | 0.067      |\n(Metric: IP)\n\n- As shown, **ResAlign is still highly effective, demonstrating the generality of our framework.** We will include these new results in our revision. Thank you again for this constructive suggestion!\n\n---\n\n**W3 & Q3:** The paper compares ResAlign only to a few methods (e.g., ESD, SafeGen, AdvUnlearn, LCFDSD), neglecting recent advancements such as RECE (ECCV24), Receler (ECCV24) and RACE (ECCV24). This omission limits the evaluation’s comprehensiveness How does ResAlign perform relative to these newer approaches, and what implications might their inclusion have for the evaluation’s comprehensiveness?\n\n**A3:** Thanks so much for this constructive suggestion! Following your valuable comment, **we have conducted additional experiments on the suggested baselines:**\n\n| Method  | Pre-ft | DreamBench++ | DiffusionDB |\n| ------- | ------ | ------------ | ----------- |\n| RACE    | 0.031  | 0.131        | 0.214       |\n| RECE    | 0.043  | 0.073        | 0.134      |\n| Receler | 0.063  | 0.117        | 0.149       |\n\nWe draw two main insights: (1) All evaluated safety-driven unlearning methods, including these recent ones, **universally suffer from a rebound in toxicity after downstream fine-tuning**. (2) **Stronger pre-finetuning safety does not ensure better post-finetuning resilience (e.g., RACE).** These results **reinforce our main conclusion** on the widespread post-finetuning rebound and the urgent need for more resilient methods. We will include these findings in our revision and hope they inspire further research. Thank you again for your valuable feedback!\n\n---\n\n**W4:** The Moreau Envelope and Proposition 1 assume small perturbations and isotropic randomness, which may not hold under extensive fine-tuning or large learning rates. The paper lacks discussion on their real-world validity.\n\n**A4:** Thanks for this thoughtful question! We sincerely appreciate your careful reading and attention to our proposition.\n\n- We acknowledge that our theoretical analysis, like any such work, relies on certain assumptions. However, **as discussed in Appendix A.2, we believe these assumptions are highly realistic** in real-world diffusion model unlearning:\n    - **Small perturbations:** Tyically, large-scale pre-trained diffusion models are already very strong and require only small updates to reach the optimum for most downstream tasks. Specifically, recent studies have shown that, after fine-tuning, only a small subset of model parameters would undergo minor changes [1]. Some works even show that aggressive tuning (large LR/steps) often leads to degraded generalization and overfitting [2]. Therefore, we respecfully argue that our ME-based formulation and analyses with Taylor expansion are highly realistic.\n    - **Isotropic randomness**: Our meta-learning samples from a diverse dataset and a rich pool of configs, which naturally induces parameter updates that are broad and varied. Notably, our assumption is about the distribution of parameter residuals during meta-learning agnostic to magnitude, so it remains valid even when the fine-tuning involves larger LR/steps. Besides, it only requires the perturbation to be a scaled unit random variable, which is more relaxed than prior works that require stricter (e.g., Gaussian) assumptions.\n- Beyond theoretical justification, **we also conducted an additional empirical study to further understand the validity of our Taylor expansion**. We randomly select 30 fine-tuned models and compared the true harmful loss with its (first-order) approximation.  We found that the absolute error was typically $< 10^{-4}$ and the relative error was usually below 3%; even for large learning rates/fine-tuning steps, the error rarely exceeded 5%, supporting the practical validity of our approximation. **This also matches our empirical findings that ResAlign is robust to large learning rates and steps (see main paper Tab. 4&5).**\n- Finally, we acknowledge that, as with all theoretical work, our analysis is ultimately an approximation and could break down under extreme conditions. **As one of the earliest efforts to understand unlearning resilience, our analysis is intended to provide qualitative and intuitive insights, not definitive conclusions.** We hope our work can inspire further, deeper studies into robust unlearning theory.\n\nWe will add more discussion in our revision. Thanks again for the feedback!\n\nRef:\n\n[1]: Kumari et al. Multi-Concept Customization of Text-to-Image Diffusion. CVPR 2023.\n\n[2]: Ruiz et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR 2023.\n \n---\n\n**Q4:** Beyond the $\\mathcal{L}_\\text{adaptive}$ fine-tuning attack in Section C.2, what other adaptive fine-tuning strategies (e.g., alternative loss modifications or multi-stage optimization) could challenge ResAlign’s safety constraints? How can future evaluations enhance ResAlign’s robustness against such strategies?\n\n**A4:** Thank you for this insightful suggestion! Following your comment, we further evaluate two addtional adaptive attacks: (1) adding a negative L1 regularization to the original weights during fine-tuning, helping the model to escape local safe optimum, and (2) a curriculum-style approach that first fine-tunes on benign, then borderline, then unsafe concepts to gradually cross the safety barrier. **We found that both strategies still failed to bypass ResAlign**, where strategy 1 remained unstable (similar to the attack in Appendix C.2), while strategy 2 did not substantially increase IP before unsafe concepts were directly introduced (IP<0.03), with final trend similar to our results in Fig. 2. Due to space limit, we will add more detailed settings and the results of these attacks in our revision.\n\nTo further enhance robustness, one potential strategy is adversarial training, i.e., incorporating such adaptively fine-tuned models into our meta-training process. We believe the robustness of ResAlign against adaptive attacks is both interesting and important for future work and will discuss these directions in our revision. Thank you again for raising this important point!"}]

Author rebuttal:

Meta-review: This paper addresses an important problem: the brittleness of safety-driven unlearning in text-to-image diffusion models under downstream fine-tuning, including benign personalization. The proposed method, ResAlign, incorporates the fine-tuning process into the unlearning objective via a Moreau Envelope-based implicit differentiation to efficiently approximate hypergradients, and employs meta-learning over diverse fine-tuning configurations to improve cross-configuration resilience. A theoretical insight links the objective to an implicit Hessian-trace penalty, encouraging flatter harmful-loss regions and thus greater robustness to subsequent updates. Experiments are broad and convincing across datasets, fine-tuning methods, and configurations; additional results during rebuttal cover newer models and recent baselines.

Strengths
- Clear problem framing with practical impact; shows benign fine-tuning can regress safety in prior methods.
- Sound and efficient methodology (ME-based implicit gradients; trajectory-free HVP), with a principled meta-learning design for generalization.
- Theoretical rationale (curvature/flatness) aligns with empirical outcomes.
- Comprehensive evaluation, including ablations, cost reporting, newer architectures, and added recent baselines; consistent safety retention with minimal degradation of benign generation.

Weaknesses (largely addressed in rebuttal)
- Initially limited details on the ME approximation and meta-learning configuration diversity.
- Early focus on older backbones; now augmented with SD v2.0, SDXL, and popular fine-tuned checkpoints.
- Missing comparisons to some recent ECCV’24 methods; now included.
- “Mosaic” effect for unsafe prompts tied to the chosen loss; authors clarify it’s loss-agnostic and show alternatives yielding safe but meaningful outputs.

Overall, this is a solid and timely contribution with good empirical support and constructive rebuttal, which focuses on safe personalization of diffusion models. Please ensure that the suggestions from reviewers are incorporated into the camera-ready version.
- Move key ME-approximation design choices and alternatives (first-order, unrolled) into the main text for clarity.
- Include a concise resource–performance table (VRAM/time vs. hyperparameters) and at least one main-text table on SDXL (and recent baselines).
- Discuss the validity range of theoretical assumptions and extreme configurations.
- Document the “mosaic” trade-off and provide a recommended loss-choice guide for practitioners.
Decision: Accept (poster)