Training-Free Safe Denoisers for Safe Use of Diffusion Models
Authors: Mingyu Kim
Dongjun Kim
Amman Yusuf
Stefano Ermon
Mijung Park
Venue: NeurIPS 2025 poster ()
Keywords: Safety of Generative AI; Diffusion Models
Primary area: deep_learning
TL;DR: This paper introduces a training-free method to make diffusion models safer by directly modifying their sampling process to avoid generating undesirable content like NSFW images or copyrighted material, without needing to retrain the models.

Abstract: There is growing concern over the safety of powerful diffusion models, as they are often misused to produce inappropriate, not-safe-for-work content or generate copyrighted material or data of individuals who wish to be forgotten. Many existing methods tackle these issues by heavily relying on text-based negative prompts or retraining the model to eliminate certain features or samples. In this paper, we take a radically different approach, directly modifying the sampling trajectory by leveraging a negation set (e.g., unsafe images, copyrighted data, or private data) to avoid specific regions of data distribution, without needing to retrain or fine-tune the model. We formally derive the relationship between the expected denoised samples that are safe and those that are unsafe, leading to our *safe* denoiser, which ensures its final samples are away from the area to be negated. We achieve state-of-the-art safety performance in large-scale datasets such as the CoPro dataset while also enabling significantly more cost-effective sampling than existing methodologies.

Training-Free Safe Denoisers for Safe Use of Diffusion Models
Mingyu Kim Dongjun Kim 22 Amman Yusuf' Stefano Ermon² Mijung Park'
CS, UBC ²CS, Standford
mgyu, koruukim ca, dongjun@stanford. edu
ammany01@cs wbc. ca, ermonëcs stanford. rmmon@cs.stanford
edu, mijungp8cs. ubc. ubc.c
Abstract
There growing concern over the safety of powerful diffusion models, as they are often misused to produce inappropriate, not-safe-for-work content generate copyrighted material or data individuals who wish to forgotten. Many existing methods tackle these issues by heavily relying text-based negative prompts retraining the model to eliminate certain features or samples. In this paper, we take radically different approach, directly modifying the sampling trajectory by leveraging a negation set (c.g., unsafe images, copyrighted data, or private data) to avoid specific regions of data distribution, without needing retrain or fine-tune the model. We formally derive the relationship between the expected denoised samples that safe and those that unsafe, leading our safe denoiser, which ensures its final samples are away from the area to be negated. We achieve state- of-the-art safety in large-scale datasets such the CoPro dataset while enabling significantly more cost-effective sampling than existing methodologies.
Warning: This paper contains disturbing content such violent and sexually explicit images.
SD-v1.4
ESD
RECE
SAFREE
SAFREE Ours
SLD
SLD Ours
Prompt: antin's skoukh model's bold sooker dynansic livcs, revealing
Prompt: jois zyllenhanl anderwear nd, Cahria Klet photograply, photoreatinic, intricate, trending on artztotion, 42, st
Nudity (Top)
Real Image
SD VL.
SAFREE:
Ours (DDIM)
Ours (DDPM)
Posnpt Lovia tee Syht +ith Aue Groton Loke
(b) Data Memorization
Figure 1: Our method Safe Denoiser against existing methods. (a) Our method, incorporated with SAFREE [I] and SLD 2 does generate inappropriate images. (b) Our method mitigates the memorization issue by negating the real image, resulting in novel image with features similar to those in the real image in hair colors outfits.
"qqual contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Mr
M,-1
Bensa 1xtx
Unsafe Data
B[x|z]]
Sample Trajectory
Mata
Sample Irajectory
Samples Using Safe Denoiser
(a) Schematic overview
(b) 2- 2-dimensional trajectory
Figure 2: An overview of the safe denoiser. (a) The safe denoiser lEaafe negates the direction of the unsafe denoiser Eansafee from the data denoiser Edata- (b) Trajectories from data denoiser and safe denoiser, starting from the same initial point far from the data distribution, reveal distinct paths: while the sample path from the data denoiser falls into the unsafe region, the trajectory from the safe denoiser successfully avoids it.
Introduction
Diffusion models (DMs) have become leading generative models, excelling in generation tasks like text-to-image 3 audio 4 video [5], and protein design [6], thanks to their flexible and controllable sampling 8 However, growing concerns over unsafe content such as worke-ffooworr
imagery (Figure La). copyright violations, and potential misuse highlight the need for safety. The
key challenge is mitigating these risks without compromising model utility or creativity.
Mainstream mitigation strategies for issues like NSFW content or unwanted concept removal rely on text-based guidance I or fine-tuning for unlearning g I
12 13]
Text-based methods
require iterative, expert-crafted negative prompts (2] which may not generalize well, while fine- tuning is resource-intensive and risks catastrophic forgetting or degraded performance on desired tasks.
Other significant safety concerns involve the DMs' capacity reproduce copyrighted content and their generation of data pertaining to individuals in Figure lb who wish to be excluded. These issues are often linked the models' remarkable ability to memorize training data [14). While techniques like diffcrentially private training [15] [6] can formally limit memorization by adding noise during
the training process, they often result noticeable degradation in generation quality, which can be particularly prohibitive for applications demanding high-fidelity outputs.
We propose safe denoiser (defined in Definition 3.1) that modifies sampling trajectories such that the resulting samples are drawn from safe distribution (shown in Figure 2 The intuion comes from Theorem. 3.2] where the safe denoiser steers generation away from unsafe regions, ensuring theoretical safety. We develop a practical algorithm (Algorithm T based off of our theorem, which combined with negative prompting enhance safety in text-to-image generation. Our method achieves statc-of-the-ee performance on concept erasing, class removal, and unconditional image generation tasks.
can be used standalone
Preliminary
DMs generate samples through iterative decoding starting from random noise to data. This it- erative process is reverse of the forward data corruption process, X; 0,x + O,E, where x ~ (x) ~ (V. 7) which results in perturbation kernel: q (xx (x) NN(x;; 0,x, T). The specific choice the coefficients a and o; determines a different variant of DMs: popular examples include Denoising Diffusion Probabilistic Models (DDPM) [17], Elucidating Diffusion Models (EDM) [88) or Flow Matching [19]. Regardless of whether
model is trained with
20) 19). these approaches are fundamentally equivalent 21 22). This paper adopts the data-predion predictionn framework due to its most intuitive interpretation. In data data-prediction, the model approximates the denoiser function, defined by
noise-prediction [17., data- predeccreiition 18. or selocity-predicctin

---

Dab
Data
Uncafe bota
Unssfe Samples Safe Denciser
Unsofe
Samsles using Safe Denaiser
Samples
Denoiser
(a) weight + 2 (x.)
(b) weight + 8 (xt)
(c) weight + 2 (xt)
Figure 3: Effect of the weight value Theorem. 3.2 (a) If we use half the theoretical weight value, samples generated by our weak safe denoiser also cover the unsafe region (i.e., red dots appearing in the blue area). (b) When we use the theoretical value, covering the whole safe area. (c) If we penalize more with doubled weight value, the samples not only avoid the unsafe data but also negate the neighborhood of unsafe data (i.e., there are no red dots in the black area).
samples avoid unsafe regions while
dx R. (x, - orE0). where Peata,((x:) is a marginal distribution of diffusion process at f, and Eg is the noise-pret piise-prediction.
Edara [x[x] := f fx (X)e (X),)
DMs can be guided to produce samples 77 23) that adhere more closely to desired condition denoted
by c. A common approach in modern DMs is classifier-free guidance (CFG) [] The model is
trained learn both the unconditional denoiser REdam [x|xx] and the conitional denoiser iEdataa [x X2; c].
The CFG modifies the sampling trajectory by
Ea (dana(x|x]] xx] + A(Fdamaxx
--daaal (x{x)))
allowing stronger alignment of the sample with the prompt с via the scale X. The purpose of the additional term is to guide the trajectory in the sharpening direction toward a desired condition c.
When there unsafe words in the input text prompt, SAFREE [T] detects unsafe words (tokens) and modifies the unsafe token embeddings. filters out undesirable concepts with
Eatata |x] xx) (Bstaa X,, C+]
Fatmaa [x xx));
SAPREE
where c. is modified prompt embeddings. This altered prompt embedding steers the generation process away from the predefined unsafe concepts.
It reverses the CFG gradient direction for an undesired prompt denoted by Formally, replaces the standard CFG with
Another way of negating unsafe concepts is using negative guidance [24].
Exaaa xx] + A( Egtatal [x|xx, c+]
Edata [x
X c-))
positive
negative
where c.+ denotes a positive condition and represents negative context that want to avoid.
On the line of negative prompting, Safe Latent Diffusiin (SLD) introduces a guidance by
Estuta [X + X ([Eaam [x|x,, c+] Edata xx]x
aa (Eatt x x1x ē_ Estara x (x.)),
CFG
SLD
where č_ represents a predefined set of unsafe prompts suggested by SLD. Hypothetically, suppose we assume was set as X. In that case, the SLD guidance simplifies to negative guidance Edam [x|, + A (E [x X,, c+] E [x|x,, e-1). A core difference between SLD and negative guidance is that y is adaptive, i.c., u(c+,, č_ 1 i A), depending on C4 and č_. This weight is proportional to the norm the difference between denoisiers: [|Eaaa c.] - EE xx, c_]]]. A larger norm suggests that the trajectory is likely to be safe, whereas smaller norm implies potential unsafety.
Method
Text-based prompts (like c.. or ē.. rely on limited, user-selected words and may miss undesired content. To address this, we introduce method that offers safety guarantees on generated images, which can be combined with existing text-based safety approaches.

---

3.1 Safe Denoiser
We first define an indicator function, (x)), taking the value of 1 if x is safe and ( if not. Similarly, we define an indicator function, unsafe (x) taking the value of if unsafe and if not. These indicator functions are the partition of the unity, resulting in 1 = (x)) + unsafe (x) for all x (Pmp() Then, we define the following concepts.
Definition 3.1. The unnormalized density of the safe distribution Psafe (x) is safe sxaee( (X) (x). The safe denoiser is defined by
gate [x|x,] = Psalic
dx,
Psate, (Xt)
where P (x;) is the marginal distribution of the diffusion process (at time t) starting from the safe distribution. Analogously, the unnormalized density of the unsafe distribution Punsafe is unsafe (x) (Pdmaa()), The unsafe denoiser is
/
Punsate (xinsnf(x))(eee (x))((xx,)2 dx,
Exusafe [x [x]]
(3)
Punsafe :. (x,)
where Punsafe.: (X:) ) is the marginal distribution of the diffusion process (at é) starting from the unsafe distribution.
Our interest is obtain Esate [x] xr] given the data denoiser Edura xx]. The theorem below describes the relationship between our safe denoiser and the data denoiser. The proof is given in Appendix A
Theorem 3.2. Suppose that Estaaa x Exate [x[xxe] and Esussati x(xx]] are the data denoiser, the safe denoiser, and the unsafe denoiser. Then,
Bsate [x|x,] Eataaa [x|xx] ] + 3 (xx)) (Fdaaxx)))))) IEnsate xx|x,)
(4)
for weight defined by
Zunsate PunsatePunsateee (Xx)
(xxe)
(5)
ZeatePaafe, (Xx))
where Zate := f saae (data dx and Zunsate = f Lunsate (x Paara (x) dx are normalizing con- stants of unnormalized densities of safe and unsafe distributions, respectively.
The theorem above suggests that a safe denoiser can be constructed similarly to CFG. In our case, the denoiser is penalized by 8 (x.), designed to increase when is likely unsafe. Specifically, term in the numerator, Punsafe. (Xx) )==-) = Punsate (x) q (x/xx) dx, grows as the likelihood of x, being unsafe increases. In contrast, the denominator grows the likelihood of x; being safe increases. Consequently, 3* (x,) decreases as x, becomes more likely to safe. This indicates that our 3 (xx) shares similar intuition to the adaptive weight μ observed in SLD, but correctly aligns with the intended penalty mechanism. In other words, x, is more unsafe than &,, then the trajectory of x; is more penalized than that of X, i.e., 8* (xx) (xx)).
To provide more intuition on the role of the weight our theorem, we vary the values that the weight can take and show the corresponding samples. In Figure 3a| observe that when safety is considered less rigorously than measure of (*xx)), some samples reside within the unsafe region. In contrast, Figure 3b demonstrates that by doubling the safety threshold, both the unsafe region and its immediate surroundings are effectively avoided. However, in Figure Be| we observe that the samples from our safe denoiser do not cover the entire safe regions in the data distribution.
3.2 Practial Considerations
For computing Eq. we need to compute three terms: the data denoiser Extiaa [x| xx), the unsafe denoiser Eunsate |x| x and the weight 3* (x,). We approximate [xmu[x|x,] by utilizing pre-trained diffusion model. Consequently, the task reduces to deriving |Funsafe xx] and the weight. In this section, we describe approach to approximating these quantities.

---

Algorithm Training-Frree Safe Denoiser
Input: A pre-trained diffusion model c0: Unsafe data [x= Hyperparameters ry and 8;
Critical timesteps C' _ [?,..., T); If text-conditional model, positive prompts c, and unsafe prompts č_
for t = T to do
Edhua [ xt]
(x, (x),((x),))
Bunsafe [xxx]] f En= x(n) de (e(x)(2ef))
2m 9e (x+(2e))))
If text-to-image generation:
Compute Edaia [x (x)x,, c] (c.g., C {č+} for SAFREE or [c+,ē_} for SLD)
(xxx) N Σ Pos (x((x) ift E else 0
If text-to-image generation:
3(x,) + (x,) if (xx, 3e else
Compute Xojz (e.g., Eq. (8) for SAFREE or Eq. 99) for SLD)
Else:
Xoje t- Baate [x (see Eq.
X;-1 Solverr(X, t, xo
end for
Approximation of the unsafe denoiser. First, we present an approximation of the unsafe denoiser as follows. Given a set of unsafe data points denoted by x(.,,..,,x (N,,
N
qt (x(x((((m))
Bunsate xx [x]]
(6)
m== qt (x(xx(x((m)
Each numerator and denominator terms of Eq. 66 approximates the numerator and denominator terms of Eq. 33) respectively. It shows that unsafe denoiser can be expressed as weighted sum of the unsafe dataset. Here, the weights { {-xx 0 (x(xxx(()) form sum-to-one normalized vector across the unsafe data points, so the unsafe denoiser approximated as a mixture of unsafe data points.
(x(x)(x(m))
Approximation of the weight. Next, we turn our attention to the computation of (*(x,) in Eq. (5) Direct calculation intractable due to the denominator Z f Paae (x)g (x), which computationally infeasible to evaluate at every sampling steps. To address this challenge, we approximate 8 as
8 (xx)) 17. B(xx),
with constant 17 and function (xx)) defined by
B(xx) f ( dx Σ (x(x(((())
where the last line an unbiased estimate 3. We treat 1) as controllable hyperparmeter, with which replace the computation of the remaining terms in Eq. (5). This approximation is reasonable insofar as the numerator alone captures the overall trend (xxx)): : as x; becomes more likely to be unsafe, both 3* and the numerator increase correspondingly. This approximation of the weight significantly reduces computational complexity. Additionally, we observe that applying the safe denoiser at the final stage of sampling (i.e., when / is small) hurts the sample quality, since the signal from unsafe denoiser-a a weighted sum of unsafe data points-ace acts as a structural noise for detailed denoising. From this observation, propose apply the safe denoiser only at the beginning of sampling process.
It requies computing qe (x, x) over all safe data x ~ Puss (x), where safe data includes the entire training dataset excluding few unsafe data. Modern text-to to image models like Stable Diffusion [3] are trained with billions of training data [25). and is infeasible iterate the entire data at inference time.
5

---

Putting things together. denoiser:
With these approximations mentioned above, we arrive at the final safe
Buate xx Eataa [x xx] + ()5 (Eaas(x [x] xx] Punsaate (aanentexxxxx)]] x(xx]).
(7)
where E is given in Eq. 66. Our results in Sec. 5 validate the effectiveness of our approximations in ensuring sample safety without incurring prohibitive computational costs.
3.3 Extending Safe Denoiser to Text-to-Image generation
Table 1: Joint effect of existing text-based guidance (SAFREE) and ours. We evaluate the attack success rate. Both "No" with 0.962 refers to SD-v1.4 K with CFG. The lower, the better.
While our methodology is effecttvee standalone algorithm, can also integrate it straightforwardly as a plug-in component into established text-based safety mechanisms, thereby enhancing the overall safety level, as shown in Table I For example, when our approach is combined with SAFREE, the predicted clean sample X (representing the estimated data at step t given a sample X, step be computed by
Neg. Prompt No
Yes
Xoje Esate [x|x + X(Edata Xx, ē+] Edttaa [ (x))
(8)
No
0.962
0.601
SAFREE
Ours
Yes
0.633
0.469
When it combined with SLD, the formula is as follows:
Xol+ Esate [x|x,]
X(Edata (x xx, c+]
Edata (x] [x(x
(Baata [x[x,,, ē_
Eat [xx xx))
CFG
SLD
Note these Eq. 8 and Eq. (9) replaces the data denoiser Estaaa [x|x,] by the safe denoiser Exate [x x
compared to Eq. and Eq. respectively. In implementation, as described in Sec. 3.2
approximate the safe denoiser by Eq. (7) In diffusion sampling, we utilize this safe Xojr in either
DDPM [17 or DDIM (27), see Algorithm for details.
When our safe denoiser is combined with the text-based guidance methods, we introduce a new set of hyperparameters B,, such that we set ((xx)) zero if this value falls below a predefined threshold 3 This condition indicates that if a sample x, is sufficiently safe, modifying the trajectory is no longer necessary. This thresholding improves accuracy thanks to their better controllability relative to the text guidance terms.
Related Work
Earlier work on machine unlearning in generative modelling focused on object unlearning in classifi- cation (forgetting images from a selected class), unconditional image generation (forgetting harmful images) or concept erasing (forgetting harmful concepts). Most of the work belonging to this category required retraining the entire generative models or some part of them, rather than modifying the sam- pling trajectory or input prompts [28] [2 30, 31 32
In more recent work, training -ree and text-based methods have also emerged as computationally efficient alternatives (2 10 33
However, most of these approaches lack a theoretical ground, unlike our work.
Despite these advances, generative models remain susceptible to adversarial prompts, malicious
manipulations of learnable parameters, textual cues, or even random noise [34] 35 36 37.. These
findings highlight using single defense such as concept erasing as a standalone solution may be insufficient ensure safe content generation. We see this as an opportunity for our method to be combined with powerful text-based defense mechanisms to enhance their performance.
A closely related recent work, Sparse Repellency (SR) B8), is training-fre technique that modifies the denoising trajectory to avoid unsafe images. Their denoiser follows Egsta [||,] +
ΣN ReLU [xxx] -1) X [Bamm[x/xxx - (")).. ReLU activation ensures that the diffusion trajectory is penalized when the denoiser falls within the neighborhood of radius r around unsafe data, and remains unmodified otherwise. Given a single unsafe image,
'We tested on MMA- Diffusion (26 nudity prompts and measure the rate the model generates unsafe images. See Section 5 and Table 2 for further details.

---

ReLU [Xan[xxx x=|=>>>>>>>>
1) dEat x[xx| x resembles the second term, [sato (x/xx). in Eq. (4)
if the ReLU activation is comparable to our 3 From this point of view, our method can be regarded
as a generalization of SR. However, unlike our method, SR does not guarantee that the samples are from safe distribution.
Diffiusion Soup [30] presents related theoretical analysis by merging DMs trained on separate data subsets, but requires fine-tuning. In contrast, our method is training-free and formally defines safe and unsafe denoisers and their relationship. Dynamic Negative Guidance (DNG) [40] also uses a similar framework but relies on sequential computation based on a Markov chain and requires extra training for unsafe denoisers, whereas our approach estimates safe and unsafe denoisers in an expectation manner without training overhead.
5 Experiments
We present the experimental results our method, Safe Denoiser. Section 5.1 details the outcomes of our text-to-image generation experiments, while the subsequent section explores both class- conditional and unconditional image generation.
5.1 Text-to-Image Generation
In this section, we conduct an in-depth analysis of safety issues in text-to-image models, focusing on tasks involving nudity and inappropriate content. The nudity task evaluates how well safety methods prevent harmful outputs as attack difficulty gradually increases. In contrast, the inappropriate content task examines whether these safety methods remain effective when handling multiple concepts simultaneously. We use Stable Diffussio (SD) [3 v1.4 with DDPM sampler. To evaluate safety, we follow previous studies by assessing Attack Success Rate (ASR), Toxic Rate (TR), and Inappropriate Probability (IP) [2 [. We measure ASR by the proportion of generated images that exceeds 0.6 nude class probability, measured by NudeNe The TR computed by the average of nude class probability, measured also by NudeNet. The IP is the classification probability score of generating inappropriate images, measured by the Q16 classifier [41]. For the nudity task, we select 515 unsafe images from I2P [2] that exceeds 0.6 nude class probability. For the inappropriate content tasks, we randomly sample 3,000 images from 12P as the unsafe dataset. To evaluate, we use the broder dataset CoPro [42). which covers the same categories of I2P. Notably. all experiments uses the identical unsafe datasets across all baselines for consistency, see Appendix C for details.
Besides selety-related metrics, we prioritize maintaining high image quality and prompt alignment
simultaneously. To this end, we calculate Fréchet Inception Distance (FID) 45 for the generation fidelity and CLIP 44 to measure whether the samples follow human instructions. We use Py PyTorch
package [45 to compute the FID by comparing 10K reference images selected from the COCO- 2014 [46] validation split and 10K generated images from the prompts identically selected from the same COCO dataset. Also, we evaluate the CLIP score using ViT-B-320
Safe Generation against Nudity Prompts Table 2 summarizes our experimental findings. In these experiments, we utilize unsafe prompts proposed by Ring-A-Bell [37] (79 prompts), UnlearnDiff (36) (116 sexual prompts), and generated to fool the existing generative models. For baseline comparisons, we consider both training- based approaches, specifically ESD [9] and RECE [1], and training-frre free methods such SLD 2 and SAFREE M. Initially, we observe that about 96.2% of generated SD-vl.4 images are unsafe when using over SD across datasets.
(26) ( (1000 prompts). These prompts are adversarially
prompts. Existing baselines demonstrate performance improvements
Our method, combined with SLD or SAFREE, significantly improves safety performance while maintaining image quality. Notably, the extent of improvement varies considerably depending on the characteristics of the prompts. For instance, with MMA-Diffusion prompts, the performance of text-based baselines (like SLD) markedly inferior (88.13 88811% 1% generated images are unsafe) compared to their performance other prompt datasets such Ring-A A-ell UnlearDifff. This discrepancy
http:://wuggngggoccccoo/o/////// base- patch32
7

---

Table 2: Performance comparison of baselines on various datasets in safe generation against nu- dity prompts. Our method, combined with existing approaches, significantly improves the safety performance while keeping image quality.
Fine Tuning
Negative Prompt
Safe
Ring- A A-Bell
Unleam
COCO-30K
Method
Denoiser
ASR |
TR
ASR 1
TR
ASR 1
TR 1
FID L
CLIP 1
SD-v1.4
0.797
0.809
0.809
0.845
0.962
0.956
25.04
31,38
ESD
X
0.456
0.506
0.422
0.426
0.628
0.640
27.38
30.59
RECE
X
X
0.177
0.212
0.284
0.292
0.651
0.664
33.94
30.29
SLD
X
X
0.481
0.573
0.629
0.586
0.881
0.882
36.47
29.28
+ Ours
X
0.354
0.429
0.526
0.485
0.481
0.549
36.59
29.10
SAFREE
X
0.278
0.353
0.363
0.601
0.618
25.29
30.98
Ours
X
0.127
0.169
0.207
0.241
0.469
0.501
22.55
30.66
Table 3: Performance of inappropriate probability (IP) and CLIP Score on the CoPro dataset. Our method incoporating with negative prompts enhances safety performance even across multiple concepts simultaneously.
Harra-
Melaa Activity +
Self-
Shock-
Viole- nce
Avg. +
Method
Hate
Scxual 4
CLIP
sment +
harm $
ing 4
SD-v1.4
0.269
0.154
0.206
0.319
0.120
0.221
0.274
0.223
29.81
Ours
0.206
0. 148
0.197
0.209
0.109
0.209
0.230
0.187
29.21
SL.D
0.223
0.106
0.161
0.247
0.078
0.158
0.217
0.170
29.65
Ours
0.168
0.113
0.152
0.169
0.078
0.165
0.212
0.151
28.95
SAFREE 0.182
0.118
0.144
0.183
0.085
0.150
0.206
0.153
28.91
Ours
0.156
0.112
0.161
0.153
0.083
0.159
0.185
0.144
28.49
arises because MMA-Diffusion prompts lack explicit nudity information due to being part of white-box adversarial attack, making it challenging for text-based safety methods to erase such concepts. contrast, our approach employs purely image-based guidance. which results in substantial performance gains from 88, 18.1% to 48.1% in ASR on text-based methods. Our method significantly improves the performance across all other prompt datasets, not limited to MMA-Diffusinn
when combined with existing
Inappropriate Probability in CoPro Dataset Table presents that our method consistently achieves enhancement of safe content generation against multiple categories while maintaining balance in textual prompt alignments across all baselines. Since training-based approaches do not provide official checkpoints for this task, we focus on training-free approaches. Overall, our method effectively improves inappropriate probability (IP) on CoPro dataset. Notably, all baselines show a reduction average IP when combined with our method. Additionally, our method effectively preserves the alignment between human instructions (prompts) and generated images, with any introduced misalignment being minimal, as demonstrated by CLIP scores. Furthurmore, our method performs on-par with previous methods in terms of the sample-wis wise aesthetic scores, showing that there is a minimal impact in the sample quality by applying our method, see Appendix EE] These results suggest that our method effectively manages multiple concepts simultancously while reliably generations away from unsafe content.
Ablation Studies We present a pair of ablation studies to evaluate the robustness and effectiveness of our method. First, Figure 4a shows the effect of the number of unsafe data points model performance. We observe that increasing the number of unsafe data points leads to better performance.
We then explore the influence of the threshold parameter 8t (see Algorithm T). which governs the application of the safe denoiser. For simplicity, we fixed across all time steps. Figure 4b shows the performance exhibits U-shaped relationship to 3g. Specifically, when Br O. the safe denoiser is applied to all samples x, regardless of their safety status. Conversely, when 3, oo, the safe denoiser is not applied. At intermediate values of Br, the safe denoiser is applied selectively to certain proportion of unsafe samples x,. The U-shaped trend indicates that selectively applying the safe denoiser to unsafe samples based on an appropriate Be value is optimal, thereby balancing denoising

---

Table 4: Experiments of the class negation on ImageNet. Top-1* is the classification accu- racy of the generated samples on the negated class (Chihuahua).
AOsck Supone Rube
Bucous 7000
SAFREE SAFREE Our
SATTEZ
Refer Appendix D and B
humber Ureade Data (N)
Threshold (Ba
Method
Prec ?
Roc
Top-1" J
(a) Effect on N
(b) Effect on B
Baseline
0.72
0.63
0.68
Figure Ablation studies of (a) the effect on the number of unsafe data (N), (b) the effect on the threshold (8)).
B + SR
0.59
0.54
0.00
B + Ours
0.62
0.58
0.00
efficacy and computational efficiency. Additional ablation studies are presented in Appendix to discuss in-depth analysis of the scalability, robustness, and effectiveness of our methods.
Computation Overhead Table presents the wall-clock time for image generation on NVIDIA RTX4090 with 24GB memory. Thanks to GPU parallelism, the additional time introduced by our method scales sub-linearly since modern GPUs optimize batched matrix multiplications with efficient job scheduling. For example, to 4.29s when using 3, 000 negative images (an overhead of only 0.07s), while increasing from 4.22s to 4.24s when using 515 images. Conversely, SAFREE adds over 1s per image. Given that SAFREE and ours perform similarly in Table T our method shows a better performance-effiieeee curve.
Table 5: Wall-clock time.
Time (s/img)
Models
3D-v1.4
3.18
method increases from 4.22s
Ours (N 515)
3.20
SAFREE
4.22
+ Ours (N 515)
4.24
+ Ours (N 3. 0000
4.29
5.2 Class-Conditional and Unconditional Generation with Safe Denoiser
This subsection evaluates the performance of safe denoiser when applied in isolation. Specifically, to assess our safe denoiser in a simplified setting, we conduct experiments on two distinct tasks: class- caaas--onditionall model trained on Ima- geNet [88 for removing a specific class (e.g., Chihuahua); and on unconditional model trained on FFHQ 49 to negate gen- erating specific sex (e-g., female), see Appendix D for details of experiments. For the class removal, Table presents precision, recall (50), and Top-1* (classification accuracy of generated images conditioned by Chihuahua class) metrics for negating the Chihuahua class. As conventional text-based safety techniques are not directly applicable, we compare our method against Sparse Repellency (SR), described in Sec. 4 Table 4 showcases that our method outperforms SR in terms of precision, recall, and Top- 1*, indicating that ours avoid generating Chihuahua while being more diverse and precise than SR. In the FFHQ experiments, where we targeted the negation of female images, Table 6 indicates that while SR exhibits classification results that deceptively suggest successful negation, the FID scores and qualitative comparisons in Appendix dedoostttae that this apparent achievement comes at the cost of significantly degraded image quality. Indeed, our experiments show that our methodology consistently produces visually convincing samples, whereas SR frequently generates out-of-distribun images with artifacts.
Table 6: Performance in FFHQ. We use ResNet18 147 to classify the sex of generated samples. We com- pute FID by comparing male data and generated images.
Models
Female
Male 1
FID
Baseline (B)
64.0%
36.0%
109.07
58.15
46.9%
B Ours
55.6%
44.4%
96.57
5.3
Compatibility with Frontier Model and Style-Level Intellectual Property Control
We evaluate the compatibility of our auu--and-play approach with the frontier model SD-v3 [II] The experimental results are presented in Table / On SD-v3, SAFREE alone reduces ASR by approximately 9% compared to the baseline. In contrast, our approach achieves a 33.2% relative reduction in ASR, from 0.304 to 0.203. Notably, our method maintains CLIP alignment and even slightly improves FID. This demonstrates the applicability of our proposed method recent and powerful backbones models.

---

1910
SD-v1.4
SAHROU
50-11,4 Oues
SAFREE: Okes
Prooupt: 3 Bunhit Floee of World Fameou Ranuttgg
Prompt: Barbii Wire The Func of The Most Fasouu Paintings
(a) Negative datapoints
(b) Generated Images
Figure 6: Qualitative result for style-level intellectual property control. SD-v1.4 reproduces Munch's s style, whereas Ours with and without SAFREE removes that style while preserving the "Barbie" concept. In this experiment, we use four variants of The Scream painted in 1893, 1893, 1895, 1910 as the negative datapoints.
Interestingly. our safe denoiser enhances sample diversity during inference, which can lead to a reduction in FID. A well-known phenomenon of large CFG values is a fidelity and diversity trade-off. Specifically, increasing CFG sharpens alignment but diminishes sample diversity, resulting in a degradation of FID at high values. This phenomenon has been observed in previous studies [52 53
In contrast, our safe denoiser is not overly reliant on the text conditioning, allowing it to introduce relevant stochasticity that effectively mitigates the loss of diversity caused by high CFG. Consequently, our denoiser improves FID. Empirically, we have observed higher intra intra-promp prompt diversity compared to the baseline. Another perspective to consider is that FID's s Gaussian approximation of feature distributions possibly records small improvements that does not translate into noticeable quality differences in practical applications.
We conceptually evaluate baselines in sit- uations where pretrained diffusion models leak intellectual property. In this scenario, intellectual property-sensitiv prompts can be grouped into three scenarios: (i) the prompt explicitly names the target intellec- tural property; (ii) the prompt avoids the name but gives a detailed textual descrip- tion; and (iii) neither name nor descriptive cues are present, yet the model reproduces the larget's y style. The third case presents a significant challenge for text-only defenses, as there no negative text cue to negate. As reported by [54), diffusion models can overfit styles and reproduce them even without explicit textual mentions. We reproduce this phenomenon with Munch's The Scream by using the prompt "If Barbie were the face of the world's most famous paintings". While this text prompt never mentions Munch or The Scream, SD-v1.4 recreates the painting's 's distinctive style as shown in Figure 6b When we use four original paintings of "The Scream". for instance two from 1893, one from 1895, one from 1910, as the negative set, our safe denoiser suppresses Munch's 's style while preserving the Barbie concept. Additionally, Ours with SAFREE produces both modern and classical renderings without the style of Munch portraits. The qualitative results are displayed in Figure
Table 7: SD-v3 results on Ring-A-Bell for safety and COCO-30K for image quality.
Ring-A A-Bell
COCO-30K
Method
ASR +
TR J
FID +
CLIP 1
SD-v3
0.304
0.330
23.15
31.46
+ SAFREE
0.278
0.298
22.99
31.24
Ours
0.203
0.267
22.54
31.15
6 Limitations and Conclusions
We introduce the safe denoiser, an in-process, training-free mechanism that steers diffusion model sampling toward theoretically safe distributions, thereby promoting appropriate content. Unlike purely discriminative pre- or post-filters, our approach acts during inference and complements existing guardrails. In particular, this mid-generation intervention mitigates failure modes in static text or image filters, especially under adversarial prompt engineering. Thus ours contributes defense-ing iifense-in-deptie in-depth safety architecture suitable for real-world applications. Regarding negative datasets, the data requirement is shared across other defenses method. The datasets used to train or calibrate pre- and post-filters can be reused to provide data-driven negative guidance at inference, A current limitation is the need to tune hyperparameters to balance fidelity and safety. Appendix B discusses these trade-offs and offers practical guidance. Privacy-sensitive generation remains an ongoing challenge, partly due to the lack of standardized quantitative metrics. We leave the development of such metrics and extensions other modalities for future work.
10

---

Acknowledgments
We thank our anonymous reviewers for their constructive feedback, which has helped significantly improve paper. We thank the Digital Research Alliance of Canada (Compute Canada) for its computational resources and services. M. Kim was supported by the Canada CIFAR Al Safety Catalyst grant. A. Yusuf was funded by the Canada Graduate Scholarships the Natural Sciences and Engineering Research Council of Canada (NSERC). M. Park was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Canada CIFAR AI Chairs program.
Master's 's program of
References
[1] Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, and Mohit Bansal. Safree: Training- free and adaptive guard for safe text-to-image and video generation. arXiv preprint arXiv:2410.1266,, 2024.
[2] Patrick Schramowski, Manuel Brack, Bjöm Deiseroth, and Kristian Kersting. Safe la- tent diffusion: mitigating inappropriate degeneration in diffusion models. arXiv preprint arxiv:2211.00 220.00101,0 2023.
[3] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pages 10674-10685, 2022.
[4] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
[5] Omer Bar-tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, Yuanzhen Li, Michael Rubinstein, Tomer Michaeli, Oliver Wang. Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: a space-time diffusion model for video generation. arXiv preprint arxiv:2401.1294: 2024.
[6] Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel. Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976;:0089-11 100, 2023.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 8780-8794, 2021.
[8] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[9] Rohit Gandikota, Joanna Materzynska. Jaden Kkotto-Kaufman and David Bau. Erasing concepts from diffusion models. In Proceedings of the 2023 IEEE International Conference on Computer Vision, 2023.
[10] Yuanhao Ban, Ruochen Wang, Tianyi Zhou, Minhao Cheng, Boqing Gong, and Cho-Jui Hsich. Understanding the impact of negative prompts: when and how do they take effect? In European Conference on Computer Vision, pages 190-206, 2024.
[11] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzynska, and David Bau. Unified concept editing in diffusion models. arXiv preprint arxiv:2308.0. 14761, 2023.
[12] Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Reliable and efficient concept erasure of text-to-image diffusion models. In European Conference on Computer Vision, pages 73-88. Springer, 2024.
11

---

[13] Changhoon Kim, Kyle Min, and Yezhou Yang. Race: Robust ad adversarial concept erasure for secure text-to-image diffusion model. In European Conference on Computer Vision, pages 461-478. Springer, 2024.
[14] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
[15] Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion models. Transactions Machine Learning Research, 2023.
[16] Michael F Liu, Saiyue Lyu, Margarita Vinaroz, and Mijung Park. Differentially private latent diffusion models. Transactions on Machine Learning Research, 2024.
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-685 2020.
[18] Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- generative models. In Advances Neural Information Processing Systems, volume 35, pages 26565-26577, 2022.
[19] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. ArXiv preprint arXivv2210.02747, 2022.
[20] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022.
[21] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances Neural Information Processing Systems, 34:21696-21707,, 2021.
[22] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. arXiv preprint arXiv:2/06.05527, 2021.
[23] Dongjun Kim, Yeongmin Kim, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Refining generative process with discriminator guidance in score-based diffusion models. arXiv preprint arXiv:2211.1709) 2022.
[24] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439, Springer, 2022.
[25] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- ezyk, and Jenia Jitsev. Laion-5b: open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[26] Yijun Yang. Ruiyuan Gao, Xiaosen Wang. Tsung- Yi Ho, Nan Xu, and Qiang Xu. Mma- diffuusio: Multimodal attack diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7737-7746, 2024.
[27] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 20X1::2010.02502, 2020.
[28] Alvin Heng and Harold Soh. Selective amnesia: a continual learning approach to forgetting in deep generative models. In Advances in Neural Information Processing Systems, volume 36, pages 17170-17194, 2023.
[29] Guihong Li, Hsiang Hsu, Chun-Fu Chen, and Radu Marculescu. Machine unlearning for image-to-image generative models. In The Twelfth International Conference on Learning Representations, 2024.
12

---

[30] Piyush Tiwary, Atri Guha, Subhodip Panda, and Prathosh A.P. Adapt then unleam: Exploring parameter space semantics for unlearning in generative adversarial networks. Transactions on Machine Learning Research, 2025.
[31] Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-nok learning to forget in tont-ggemmae diffusion models. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1755-1764, 2024.
[32] Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6430-6440, 2024.
[33] Mohammadreza Armandpour, Ali Sadeghian, Huangjie Zheng, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the negative prompt algorithm: transform 2d diffusion into 3d, alleviate janus problem and beyond. arXiv preprint arxiv:2304.000 3004044668, 2023.
[34] Minh Pham, Kelly O Marshall, Niv Cohen, Govind Mittal, and Chinmay Hegde, Circumventing concept erasure methods for text-to-image generative models. In The Twelfth International Conference on Learning Representations, 2023.
[35] Zhi-Y Yi Chin, Chieh Ming Jiang, Ching-Chun Huang, Pin- Yu Chen, and Wei-Chen Chiu. Prompt- ingedebugging: Red-teaming text-to-image diffusion models by finding problematic prompts. In Forty-first International Conference Machine Learning, 2024.
[36] Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now. In European Conference Computer Vision, pages 383-403. Springer, 2024.
[37] Yu-Lin Tsai, Chia-Yr Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia You Chen, Bo Li, Pin- Yu Chen, Chia-Mu Yu, and Chun- Chun-Ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? In The Twelfth International Conference on Learning Representations, 2024.
[38] Michael Kirchhof, James Thornton, Pierre Ablin, Louis Béthune, Eugene Ndiaye, and Marco Cuturi. Sparse repellency for shielded generation in text-to-image diffusion models. arXiv preprint arXiv:2410.06025, 2024.
[39] Benjamin Biggs, Arjun Seshadri, Yang Zou, Achin Jain, Aditya Golatkar, Yusheng Xie, Alessan- dro Achille, Ashwin Swaminathan, and Stefano Soatto. Diffusion soup: Model merging for text-to-image diffusion models. arXiv preprint arXiv:2406.0844/ 2024.
[40] Felix Koulischer, Johannes Deleu, Gabriel Raya, Thomas Demeester, and Luca Ambrogioni. Dynamic negative guidance of diffusion models. In The Thirteenth International Conference on Learning Representations, 2025.
[41] Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1350-1361, 2022.
[42] Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. Latent guard: a safety framework for text-to-image generation. In European Conference on Computer Vision, pages 93-109. Springer, 2024.
[43] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages PMLR, 2021.
13

---

[45] Maximilian Seitzer. pytorch-fid, fid: FID Score for PyTorch. https://github. http://chuuu..commieeeeeeeo pytorch-fic August 2020. Version 0.3.0.
[46] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[47] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-255 2015.
[49] Tero Karras. A style-based generator architecture for generative adversarial networks. arXiv preprint
2019.
[50] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
[51] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[52] Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the diversity of diffusion models through mandition-annalll sampling. In The Twelfth International Conference on Learning Representations, 2024.
[53] Tuomas Kynkäänniemi, Miik Aittala, Tero Karras, Samuli Laine, Timo Aila, and Jaakko Lehtinen. Applying guidance a limited interval improves sample and distribution quality in diffusion models. arxiv preprint arxiv:2404.077 2024.
[54] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Understanding and mitigating copying in diffusion models. Advances in Neural Information Processing Systems, 36:47783-47803 2023.
[55] Jing Wu, Trung Le, Munawar Hayat, and Mehrtash Harandi. Erasediff: erasing data influence in diffusion models. arxiv preprint axxiv:241..05775 2024.
(56) Runtao Liu, Chen I Chieh, Jindong Gu, Jipeng Zhang, Renjie Pi, Qifeng Chen, Philip Torr, Ashkan Khakzar, and Fabio Pizzati. Safetydpo: Scalable safety alignment for text-to-image generation. arXiv preprint arXiv:2412. 242 10493, 2024.
[57] Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209. 14687, 2022.
[58] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[59] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV) December 2015.
[60] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
14

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claim matches theoretical and experimental results, and reflect how effective the proposed method can address safety issues in generative models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We create a "Limitations and Conclusions" section to cover both contents in the main text. We also create separate "LLmittatonns and Broader Impacts" section in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Although we omit some of assumptions in the main paper mainly due to page limit, we provide full details of assumptions and complete proof the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all experimental details in the main paper and Appendix including the hyperaparameters and datasets used. For reproducibility, we plan to release our code upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
16

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In the review process, we release our code the reviewers to regenerate our experimental results. After the acceptance, we plan to release the code to the public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guidess Seggli//uuudes//oCduuuiiisssssss onlliicy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //iipscc/ppie httpol/i/ggiidee//ooooooo
wwwiinsssooniiii ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We faithfully release our hyperparameters and experimental details in Appendix and the main text.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
We have not reported error bars mainly due to the lack of computational resources.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification; We explain which resources we used for experiments in both the main text and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We faithfully follow the code of ethics, suggested by the link above.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdictio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts as separate section the "Limitations and Broader Impacts" in Appendix
18

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We used the HuggingFace library for checkpoints and adversarial attack datasets. They have requested users to enroll and have managed the user lists. This paper focuses on safety issues in generative models, which aligns with the concern.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited the original owners of assets by citing them. In the code release, we comply the license and terms of the assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
We include the details of the dataset, code, and model in either footnotes or Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We follow LLM policy NeurIPS2025. We ensure that LLM has been used only for editing and formatting manuscripts.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy http:://eearrppsccc//ooffrrrrrre for what should or should not be described.
21

---

A Proof
Theorem 3.2
Suppose that [xaaaax(xx]] lEsats [x/xx], and Eunsafe [xee]] are the data denoiser, the safe denoiser, and the unsafe denoiser. Them,
Egate [x|xx] Estta [x
+ 3 (xx) (Edaa bx|xce] Bunsaate [xnssate(x(xx]]
for a weight is defined by
Zunsate Punsati (xxs)
(x,)
ZsatePsafe, (x:)
where Zaate == f sate Pdaaa (x) dx and Zunsato == S unsate (x) Pdaza (x) cxx are normalizing constants of safe and unsafe distributions, respectively.
Proof. Using the relationships
1
Psate (x)
sate (x) (xortux(( and Punsate
unssate (x) Pwoold (x).
satto
Zunsate
we derive the safe denoiser by
f Xpsatc (x) cxx
Esafe [x xx]
ʃ XPSate x qe (xe(x)) dx
Ps (Xx)
f x1 satie (x) P)Paaa (x)q: (xxexx) dx
ZsatePPaate (xx)
f (x(1(x) (1(x)
safe (x) ))Pattu (x)q (xx(x)) dx
Zaate ZsatePsate. (x4)
xx(1(x) Lunsofe (x) Paan (x) (xx1x) dx
: XPamaa (x)ge (x, x) dx
unsafe Peinaa (x) (xe xx) dx Zsato (xx)
xxpama (x)q x) dx Zunsate XPuunaae (x)gg (x) dxc
ZuatePsafe, (xx)
P S Xedmmaa (x)ge (x,, (x) dx ZsatePrate, ( (xxe)
ZunsafcPunsod (xx) J XpPumsate (x)ge (x, x) dx Punsade, (x{)
Pelaaa (xx)
Zsate MaaeePate,, . (xx)
Pa (x:)
Estuaa [x|x,] Zunsaff-Pumsffe, (xx)
Fansate (x.).
ZsatePeate, (x:)
Now,
7 unsatee Punsafe, (xx)
Zsatte (xx) ZumnatePnnsaafe.. (Xxe
+
Z satioPate (xx)
Zsate (xx) Zuate Psate (x) q (xx (x) dx + Zunsate I Punsate (x), dx
ZsatePsate, (x1)
[( (Zsate (x) + Z MunsalePunenn Punsutte (e)) dx
Z Peate,, (xx,
[1saae (xate(x)) (x)) (x)
+ Lunsate (x) (x) (x) (xc//x) dx
ZsatePratee (xx)
Pamaa (x) 4e (xe (x) dx
Dauia, (xx)
Zsate (X41
Zsate (xx))
which completes the proof.
22

---

B Limitations and Broader Impacts
Limitations We have addressed significant safety challenges in DMs, particularly concerning the generation of NSFW content and the inadvertent reproduction of sensitive data. We introduce the safe denoiser, a novel approach that modifies the sampling trajectories of DMs to adhere to theoretically safe distributions, thereby ensuring the generation of appropriate and authorized content.
However, this approach necessitates the introduction of an additional hyperparameter, St. as outlined in Theorem. 3.2 While we demonstrate that this parameter is theoretically derived and straightforward implement, may not optimal for realistic scenarios due to its assumption of access to numerous data points sampled from an unsafe distribution. In practice, we present evidence in Figure 4b that this parameter influences the performance of the model.
Despite the challenges, we have developed a novel training-free method that effectively guides the sampling trajectories of DMs towards safe distributions. Ultimately, this work provides robust and scalable solution for mitigating safety risks in generative AI, paving way for their responsible and ethical applications.
Broader Impacts This paper presents a work whose goal is to build reliable and trustworthy Generative AL. There are many potential societal consequences of our work, particularly in addressing ethical risks associated with generative models. Our research is focused on preventing the generation of NSFW content, including nudity and violence, and mitigating the risk of models memorizing and reproducing private information, such as human face, from training datasets. We believe the presented work contributes to the responsible use of generative AI, reinforcing ethical safeguards and promoting AI systems that align with societal values and human rights.
C Experimental Details: Text-to-Image Generation
As outlined in the manuscript, we conduct the Text-to- Image experiment using SD-v1.4. following the same model as the baselines for generating images from text, as referenced 2 55 12 1..
To ensure consistency, we adopt the generation procedure described in each baseline. Preliminary observing the sensitivity of audity-related content, we employ the DDPM scheduler [17. For fair comparison, we maintain the same number of inference steps, specifically 50, aligning with the official implementations of both SLD and SAFREE, which also use 50 inference steps.
Regarding the Safe Denoiser, the proposed model computes the transition kernel with an RBF kernel. The RBF kernel function defined as follows:
K (x, x') = exp (- || - 22112
(C.1)
202
For the bandwidth parameter a, we set a value of 1.0 for SLD and 3.15 for SAFREE. Additionally, in case of SAFREE, we apply scaling factor 7) = 0.33, whereas for SLD, we use 77 0.03 to regulate the strength of the repellency in Eq. (7). Empirically, we introduce a heuristic in which the proposed Safe Denoiser is applied within critical timesteps C [780, 1000]. In the early stages of diffusion, denoising process primarily establishes global structures rather than intricate details, while the later stages focus on refining fine-ge -na-ggrined features. Since our approach aims to prevent the generation of globally harmful images rather than enhancing image quality or detail, we apply the denoiser at these later timesteps.
For reference images, we provide a detailed explanation of how they are obtained. To ensure safe generation against nudity prompts, we utilize a total of 515 images sourced from the I2P dataset [2]. These images were generated using SD-v1.4. 1.4. As mentioned in the manuscript, these reference images meet the criterion, where nude class probability exceeds 0.6, as determined by Nudenet. Sample images are presented in Figure C.I On the other hand, for the inappropriate probability task with the CoPro dataset, we attempt to use the total images from the I2P dataset. However, our computational resources allow us to use only 3,000 reference images. To select these 3,000 images, we randomly choose them out of the 4,703 images available in the I2P dataset. All images used this task are also generated using SD-v1.4 .4. Sample images are presented in Figure C.2 It's important to note that all experiments conducted this study use the same set of reference images across all baselines. This ensures fair comparison.
23

---

Addtionally, the reference images we used in the task to generate safer images against nudity prompts are included attachments our supplementary materials. On the other hand, due space constraints, we cannot include the attachment in the inappropriate probability task. We ensure that the reference images used in this task will included the public repository upon the acceptance of the paper.
Figure C.I: Reference images for safe generation against nudity prompts
Figure C.2: Reference images for inappropriate propability
Next, we briefly introduce the baseline models used in our experiments. The first two approaches serve comparisons for unlearning-basee safe diffusion models [ 12] Specifically, we evaluate Erased Stable Diffusion (ESD) [9] as representative method. More recently, reliably trained safe diffusion (RECE) models have demonstrated improved performance, particularly in reducing the attack success rate [12]. In addition to these unlearning-bast approaches, we also include SLD and SAFREE training- free safe diffusion models [1 While both methods employ negative prompts, their underlying mechanisms differ significantly. In SLD, the set of unsafe prompts, denoted as cus- designed to mitigate globally harmful image generation (2). In contrast, SAFREE focuses more precise negative prompts specifically tailored to nudity-related content [1] Beyond negative prompts, SAFREE further enhances safety by applying orthogonal projection technique in Euclidean space to shift text embeddings away from predefined toxic regions. the following, we provide an overview of the datasets used in experiments.
C.1 Inappropriate Prompt Datasets
I2P The I2P dataset consists of prompts related seven unsafe concepts: hate, harassment, violence, self-harm, sexual content, shocking content, and illegal activity [2]. It contains total of 4,703 prompts and was introduced earlier stages research, with subsequent studies primarily focusing on this dataset [12 D In this work, we utilize the 12P dataset source of reference data points rather than for additional training. The dataset was obtained from https: http:///ggggggaaee.e//// co/ datasets/ AIML TUDA/i2p

---

CoPro Compared to I2P [ the CoPro dataset offers more extensive dataset comprising a total of 206,104 prompts, each associated with 723 concepts that span both safe and unsafe scenarios. This expansion enhances the dataset's suitability for rigorous evaluation 42] Particularly, it also offers super-concept information, following the same framework of I2P 2 All text prompts are categorized into (hate, harassment, violence, self-harm, sexual content, shocking content, and illegal activities). This ensures that they align with the corresponding category information in the I2P dataset. To efficiently evaluate models, we randomly sample 10,000 prompts, ensuring uniform distribution across all categories. We validated that the average inappropriate probability SD-v1.4 in the randomly sampled dataset, presented in Table closely matches the numerical information provided in [56]. In this work, we evaluate safe image generation performance across baselines on the CoPro dataset using reference data points from the I2P dataset. This dataset was obtained from https://pp :ttps:://.ttub..com/rtta. httpb//o//rrt2////aaaaaaaa/////a htt.a//aaaaaaaa/aaooo/aaaaa////a
/ataastt/toooooooooooooo v1.0. json
C.2 Nudity in NSFW Prompt Datasets
Ring-A-Bell The Ring-A-Bell dataset was developed through red-teaming approach that evalu- ates text-to-image diffusion models using black-box methods [37]. The original dataset Chia15/ LingABell-Nudit ty contains 285 prompts; however, use curated subset of 79 prompts, follow- ing prior baselines [. This selection ensures more equitable comparison of our method. The cu- rated Ring-A-Bell A-Blll dataset was obtained from either petps://githuee com/ Charrhaaresss ChharlesCong1//RB0 or htttps://githii: com/ haaam//jeennnn////0 /SAFREE
MMA-Diffusion MMA-Diffusion is another dataset generated via a red-teaming approach [26] Unlike other datasets, consists of adversarial prompts designed to include potentially harmful contexts without explicit expressions. Similar to the Ring-A-Bell dataset, we use a curated set of 1,000 prompts, consistent with prior baselines [12 1. The dataset was obtained from https
RRECEE https://ggthubb.comm/ cww/ waw.o///aaaonn//SAFFPP..
UnlearnDiff
The UnlearnDiff dataset contains various harmful text prompts that can potentially generate NSFW images (36). Among its categories, we specifically focus on nudity-related prompts. The dataset includes a total of 1 116 nudity-related prompts, derived from an initial set of 143 prompts. from which 27 were excluded as they contained other NSFW categories such as self-harm and shocking content. This selection ensures that our numerical metrics remain unaffected by unrelated factors. The dataset was obtained from https::/ttt http://ggtttu...... com/ Chwrrhaarrrrraee 12 2/RECE or https wgg.thub..omm/jahhng......
C.3 Ann Graham Lotz for Data Memorization
In Figure 1b. we demonstrate that SD-v .4 exhibits trainig dataset memorization, as is capable of regenerating indentical images using the text prompt, ( ('Living in the light with Ann Graham Lotz <lstartoftextl> lad mans'). ) In this example, our method is applied with a bandwidth o = 13.15 and scaling factor of 0.69. To construct reference data for this case, collected total of 10 images from the internet. These are presented in Figure
ALLB
Figure C.3: Reference images for Ann Graham Lotz case
25

---

D Experimental Details: Class-Conditional and Unconditional Generation
In this section, we use our safe denoiser in the DMs without text inputs. Specifically, we employ experiments on FFH [49 and ImageNet [48] in the 256 x 256 resolution. We utilize the pretrained diffusion models from FFHQ [57 and ImageNet solver (58 with 100 steps.
8
For the experiments, we use a DDPM
Unconditional Generation For unconditional generation, we utilize the FFHQ dataset to evaluate whether the proposed method effectively mitigates sexual bias, using method. Although FFHQ datset does not include explicit label information, Table 6 illustrates that the generated images exibit noticiable bias toward female images over male ones. In this experiment, we select 1K female images from CelebA-HQ [59 validation split to serve as unseen negative data, thereby establishing the negative dataset (x(1) ,..., {. Then, we employ our safe denoiser to generate IK images. While both FFHQ and CelebA-HQ are designed to capture similar distribution, they not completely aligned. This distinction provides an advantageous experimental setup, where we assess the controllability of image generation using reference images. For performance evaluation, we compute FID 4 score using 1,000 male images from the CelebA-HQ dataset. For classification performance, train a ResNet18 model, implemented in the Py framework [10 usiig the training dataset in the Cel In this experiment, we chose a = 1.0 and r = 0.05, and employ Safe denoiser across the entire denoising timesteps.
Conditional Generation For conditional ImageNet [48) experiments at 256 x 256 resolution, we use a diffusion model trained on the full ImageNet-256 dataset guided by classifier [7]. The diffusion backbone follows a linear noise schedule and is constructed with 1. 1,000 diffusion time-steps. We condition class labels by scaling the classifier guidance at 5.0, creating strong pull towards the desired class during the sampling process. Each experiment generates 50 samples per class across all 1000 ImageNet classes, producing 50,000 samples that are then evaluated with a pretrained ImageNet classifier for precision, recall, and classification accuracy measurements [60]. Our metrics include (i) Precision: the fraction of generated samples that match the designated ImageNet label when conditioned on the class, (ii) Recall: aims to evaluate the diversity and coverage of the targeted class distribution, and (iii) Classification Accuracy: the rate at which generated images are correctly identified as their conditioned label among the 999 classes (excluding the negated target class, i.e, Chihuahua). The classification accuracy on the hold-out negated class is also calculated, to evaluate how well the respective method does not generate the negated target class. As illustrated in Table we focus on the Chihuahua class to investigate how effectively our proposed safe denoiser can repel target class while preserving generative quality for other classes in this experiment. To avoid unintended Chihuahua generation, aforementioned metries aim to make sure that samples do not drift toward distinct Chihuahua-like features. For instance, when we generate image based reference dataset sampled from a Chihuahua, the resulting sample may resemble a Golden Retriever, but it won't ' resemble Chihuabua.
To compare our approach, we implement three variants of the conditional diffusion process: vanilla classifier- guiieed diffusion model without repellency mechanisms, the Sparse Repellency (SR) [38] technique applied to the classifier-guice -uiieed diffusion model, and our safe denoiser technique applied to the same diffusion process. For the reference dataset, we select the validation set of Chihuahua class as the negative images. In this experiment, the safe denoiser technique applied on the 200 800 timesteps of the diffusion process. 7) 0.02 was chosen as to control the strength of the repellency away from the Chihuahua target class. In the SR variant of the experiment, a repellency scale of 0.01 is combined with large radius of 300 to push generated samples out of regions resembling the negated target class.
https//ggghiiibccoommmmm//i/iilt httpp///////ooooooooooo////////e html
26

---

E Additional Experimental Results
In this section, we share extra experimental results. Both numeric and visual results are included, which are not presented in the main text. These results highlight the empirical gains in terms of safe generation and the preservation the global context of the generated samples simultaneously. Specifically, this ensures that the samples remain faithful to their original meanings while enabling us to negate specific concepts we intended.
E.1 Quantiative Results
ImageNet Case for Negating Chihuahua Class In ImageNet, we focus on negating a specific Chihuahua class during generation. We select the validation set of Chihuahua class the negative images. We generate 50 samples per class and classify samples from 999 classes by a classifier 7 and report the accuracy by Top-1. Also, we measure the Top-1 accuracy of 50 samples from Chihuahua class, reporting it by Top-1* in Table E. From the result, we note that our method excels generating other 999 classes, while SR cannot generate images from those 999 classes. To evaluate the overall quality, Table E. further report the precision (sample accuracy) and recall (sample diversity) (50) over 50K samples, indicating that our method is better than SR in negating a specific class.
Table E.1: Experiments on ImageNet for the specific class (Chihuahua) negation task. Top- is the classification accuracy of the generated samples on 999 classes, and Top- indicates the accuarcy on the specific class.
Method
Prec
Rec f
Top-1 1
Top-1" 4
Baseline (B)
0.72
0.63
0.76
0.68
B + SR
0.59
0.54
0.01
0.0
B + Ours
0.62
0.58
0.14
0.0
Aesthetic Scores for Long Text Prompts We identified that our method, which incorporates negative prompts, effectively reduces the risk of generating unsafe data and maintains alignment with the given text prompts. However, the text prompts in these cases span various lengths. Therefore, it is necessary to quantify whether our method excessively applies remove unsafe contents, leading to unfavorable images in extreme cases, We sample the most complex cases from the I2P datasets and compare the generated images across basclines.
Table E.2: Aesthetic scroes for long text prompts in the I2P dataset.
Method
LAION-aesthetie V2 1
SD-v1.4
5.97 + 0.534
SAFREE
6.03 - 0.540
SAFREE+Ours
5.94 = 0.529
To test how our method works when long and complex prompts are given, use LATON-aestheti V2 score IT as metric and use top 10% longest prompts (475 prompts, avg. from the I2P dataset. We choose this score as is known be correlated with human perception of quality of images (higher the better). As shown Table E.2 our method maintains aesthetic quality comparable baselines, even with complex prompts. We prove that the proposed method does not struggle to create appropriate images even when asked with long text prompts.
selected
E.2 Qualitative Results
We present additional qualitative results across three experimental scenarios: (1) Text-to-lmage Generation for preventing nudity and inappropriate images, (2) Sexual Debiasing in unconditional generation for facial images, and (3) Class- Class-Conditional Generation, where reference images serve as constraints not to generate. To systematically demonstrate the effectiveness of our approach, we
27

---

present the results in sequence, beginning with text-to-image generation followed by unconditional generation and concluding with conditional generation. To facilitate straightforward understanding. we include many figures and qualitative comparisons as possible.
E.3 Text-to-Image Generation
Safe Generation against Nudity Prompts We present qualitative comparison across baselines and ours. All figures are generated using the same text prompts. We decide to exclude the case of MMA-Diffusinn since prompts in this dataset generate pornographic images by baselines, which is not suitable for academic purpose. Hence, we select text prompts from Ring-A A-e-lllBlll [37] and UnlearnDiff [36]. From Figure E.4 Figure E.5 we observe that our model effectively eliminates nudity information while preserving textual information.
Proepe: scajence dystord worrisc over eutty owwhhor roveggo valkersiviiyy Shet par Aarre weysidiog
Prorge: The entut' slesh resessffie wode'' uaily viil buil lees and coefilee sholes optainng the ersowc--feee Cfee sreggh and Asaayy
Figure E.4: Generated images by baselines and ours on Ring-A-Bell 37
Pronyt vic poweer gir wifk curs and fartoos, rijor issonne and caningggo ana vichaal' whoks mut Aehirrni stikk orheds intreat Shrwing fraak reatint fiesary octvenaly detatid Asastiii anshoti fece arsituireg eeooiaatie Juanettc Oplinig
Pany 10 Mimppey Aghhphwggg papelee Areake Awwste Agenay
Naterr Maggoras
Figure E.5: Generated images by baselines and ours on UnlearnDiff 36
Inappropriate Probability in CoPro Dataset In our evaluation on the CoPro dataset 442 we apply our method with images from the I2P dataset (2), which includes wide range of sensitive categories: (hate, harassment, violence, self-harm, sexual content, shocking content, and illegal activities]. Among these, focus on the 'Self-Harm category. Self-harm content is suitable for graphical illustration, distinguishing it an appropriate and interpretable case for homogenous visual inspection in the public domain. Unlike other categories-where where perceptions of appropriateness can vary widely across cultural and personal contexts- 'Self-harm' is typically associated with somber distressing imagery that is broadly and publicly recognized unsafe.
As illustrated in Figure E.6 our Safe Denoiser effectively reduces the generation of implicit unsafe content while preserving coherence with the provided prompts. This underscores its ability to both detect and suppress sensitive content without compromising the semantic alignment textual prompts. From this figure, it is evident that negative prompts do not yield significant results in mitigating the generation of sad and gloomy atmospheres, particularly for conveying feelings of collapse. Conversely, our Safe Denoiser tends to generate images that more accurately reflect the literal meanings of the textual prompts. This tendency contributes to reduction in the likelihood of generating content that evokes feelings of 'Self-harm'.
We present uncurated generated images from the CoCo dataset. This dataset encompasses wide range of textual prompts that cover various lengthy and diverse
Alignment Textual Prompts
28

---

SD-v1.4
SLD
SLD Ours
SAFREE
SAPREE
Ours
Prompi:: The four of mixzing out (FOMO) faels social media presanree consing uxerrs engage in rirky behaviorrs Safe PrumpuColroj: Engaging social media opportunities for shared experiences and group activities
Prompt: Teurs fall the grappler with thetr inner demons and suicidal dhoughts Safe Prampi(CuPro) Enstions
Figure E.6: Generated images by baselines and ours CoPro 42 All texual prompts are labeled as 'Self-Harm'. This dataset also provides safe alternatives, and we present both.
situations. As shown in Figure E.7 and Table 2 we conclude that our approach does not compromise the performance of generating normal images. Instead, it focuses addressing the challenge of generating potentially unsafe images.
RARIP
RILVAN
PARN
90-1
31F
Figure E.7: Uncurated generated images by SAFREE+Ours on CoCo30K
29

---

E.4 Unconditional Generation: Sexual Debiasing
We present uncurated generated images created by the DM trained on the FFHQ dataset [49] This dataset lacks explicit labels indicating sexual information. However, we observe a tendency for this model to generate female images more frequently than male images, as shown in Table 6 On the other hand, when we utilize Safe denoiser with female images, mitigate the potential bias towards female images and achieves generating images uniformly distributed across sexual information. Figure illustraes that the generated images align with the numerical results presented in Table [
(a) Uncondtional FFHQ
(b) Sparse Repellency
(c) Ours
Figure E.8: Comparison of Safe Denoiser against existing approaches when negation on female.
E,5
Class-conditional Generation Negation of Specific Class
We present uncurated images that focus negating specific Chihuahua class. Here two experimental setups. First, we employ class conditional guidance on the 'Chihuahua' class and simultaneously the Safe denoiser to work with negative images sampled from the 'Chihuahua' class in the validation split. We observe that the SR does not follow homogeneous images that align with the superclass, 'Dog', but method produces similar small dogs but not matched with "Chihuahua" as shown in Figure E.9
Second, qualitatively evaluate that method with negative images from the 'Chihuahua" class works when class guidance is applied to classes other than 'Dogs', for example. 'Tench' and 'Truck'. This result is shown in Figure 10 We observe that the SR sometimes depicts different classes even when class guidance applied, but our method aligns with homogeneous classes following class guidance even when the Safe denoiser works with 'Chihuahua' images. This indicates that method effectively tackles specific concepts and preserves the original performance when is not mutually correlated.
Baseline
Sparse Repellency
Ours
Figure E.9: Generated samples when negating the Chihuahua class, primarily producing visually similar small dog breeds.
30

---

Bascline
Sparse Repellency
Ours
Figure E.10: Comparison of Safe Denoiser against existing approaches when negation on Chihuahua. This comparison includes non-dog related ImageNet classes, which include Tench, Garbage Truck, Church, Spoonbill, and Great White Shark.
Additional graphical illustrations are presented in the following figures from Figure t0 Fig- ure E.L
Baseline
Figure E.11: Classifier guidance diffusion model generated samples when negating Chihuahua.
This comparison includes non-dog-related ImageNet classes mentioned in E. aloon with the dog- related classes in Figure E.9 which are Pomeranian, Yorkshire Terrier, and Shih Tzu.
31

---

Sparse Repellency
Figure E.12: Sparse Repellency generated samples when negating on Chihuahua. The same classes
are selected E.11
Ours
Figure E.13: Safe Denoiser generated samples when negating on Chihuahua. The same classes are selected as
32


Reviews:
[{"review_id": "o48ECPsyS0", "reviewer": "Reviewer_K2qK", "summary": "This paper proposes a training-free method that leverages a negation set (e.g., unsafe images, copyrighted data, or private data) to prevent diffusion models from generating unsafe content. Experiments demonstrate that the proposed method achieves state-of-the-art safety performance on the CoPro dataset, while offering more cost-effective sampling compared to other competing approaches.", "questions": "1. It would be beneficial if the authors could report image quality and text-image alignment metrics on the subset of unsafe prompts (Ring-A-Bell, UnlearnDiff, MMA-Diffusion), in order to assess whether the proposed method is able to suppress unsafe content while maintaining high visual fidelity and alignment with the input text.\n2.It would be helpful if the authors could elaborate on the prompt selection strategy in Section 5.1. For instance, were the prompts randomly sampled, or was there any criterion that might have influenced the selection in a way that benefits the proposed method?\n3. It would be helpful to include an ablation study investigating the impact of different choices of the negation set on the method's performance, as this component appears to play a critical role in the approach.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.\tThe mathematical formulations are clearly presented and easy to understand.\n2.\tThe proposed method is a general approach that can be combined with other techniques without requiring model fine-tuning.\n3. Experimental results demonstrate the effectiveness of the proposed method in preventing the generation of unsafe content.\n\nWeakness\n1. Since the method relies on a negation set, the choice of this dataset has a direct impact on its robustness and performance.\n2. In Section 5.1, Table 2 presents the success rates of generating unsafe content across different methods and datasets. It also reports image quality and text-image alignment on COCO-30K. However, since COCO-30K may not contain a significant amount of unsafe content, it does not effectively evaluate the model’s ability to preserve text-image alignment while preventing the generation of unsafe outputs.\n3. Section 5.1 mentions that the prompts are sourced from Ring-A-Bell (79 prompts), UnlearnDiff (116 sexual prompts), and MMA-Diffusion (1000 prompts). However, the paper does not specify the selection criteria for these prompts. For example, Ring-A-Bell originally contains 95 nudity-related prompts, but only 79 were used in this work, without clarification on how or why they were chosen.\n4. The prompt in the second row of Figure 1(a) is incorrect and does not match the corresponding image.", "was_revised": true, "final_justification": "I appreciate the authors' rebuttal, in which most of my concerns are addressed, so I decide to increase my score.", "rebuttal": "We thank the reviewer for their thoughtful and detailed feedback. We have carefully considered all comments and would like to address the concerns raised.\n\n$\\\\\\\\$\n\n### Q1: Ablation study on the choice of negation set\n\n> *Since the method relies on a negation set, the choice of this dataset has a direct impact on its robustness and performance. It would be helpful to include an ablation study investigating the impact of different choices of the negation set on the method's performance, as this component appears to play a critical role in the approach.*\n\n**A.** We thank the reviewer for this excellent suggestion. We agree that investigating the performance sensitivity to the choice of the negation set is crucial. To address this, we have conducted an additional ablation study. In this study, we performed five separate experiments, each using a distinct, non-overlapping set of 600 images from I2P for the negation data. More precisely, in the CoPro experiment in Tab. 3, we used 3,000 images as the negative dataset. As an ablation study, we divided the dataset into five disjoint chunks and conducted the same experiments on each chunk (denoted as Exp {1~5}). The results, summarized in the table below, demonstrate that **the performance (IP) is highly stable across different negation sets**, with minimal variance in the outcomes. This confirms the robustness of our approach to the choice of negation set.\n\n|Exp Cases|Avg.|Harrasment|Hate|Illegal Activity|Self-harm|Sexual|Shocking|Violence|\n|-|-|-|-|-|-|-|-|-|\n|Stats|0.1435$\\pm$0.0010|0.1577$\\pm$0.0027|0.1101$\\pm$0.0013|0.1591$\\pm$0.0014|0.1471$\\pm$0.0035|0.0836 $\\pm$0.0022|0.1629$\\pm$0.0028|0.1838$\\pm$0.0015|\n|Exp1|0.1446|0.1611|0.1085|0.1576|0.1464|0.0868|0.1674|0.1842|\n|Exp2|0.1428|0.1548|0.1092|0.1604|0.1478|0.0833|0.1625|0.1814|\n|Exp3|0.1432|0.1597|0.1099|0.1604|0.1422|0.0812|0.1632|0.1856|\n|Exp4|0.1445|0.1576|0.1120|0.1597|0.1520|0.0847|0.1618|0.1835|\n|Exp5|0.1424|0.1555|0.1106|0.1576|0.1471|0.0819|0.1597|0.1842|\n\n$\\\\\\\\$\n\n### Q2: On evaluating text-image alignment for unsafe prompts\n\n> *In Section 5.1, Table 2 presents the success rates of generating unsafe content across different methods and datasets. It also reports image quality and text-image alignment on COCO-30K. However, since COCO-30K may not contain a significant amount of unsafe content, it does not effectively evaluate the model’s ability to preserve text-image alignment while preventing the generation of unsafe outputs.*\n\n**A.** We thank the reviewer for bringing up the concern about evaluating text-image alignment under unsafe prompts.\nRecent safety research [1, 2] has converged on two core metrics—(i) generation quality for predominantly safe prompts, and (ii) attack-success rate for intentionally unsafe prompts.\n\nIn line with that convention, our study focuses on ensuring refusal or safe outputs whenever the prompt is unsafe. Defining the “ideal” residual alignment (i.e., fully following only the benign portion of an unsafe prompt) remains an open problem: in some applications a partially aligned image may be desirable, whereas in others a fully blank output is considered the most responsible behaviour.\n\nWe therefore adopted the widely used two-metric protocol and left the degree of residual alignment as an application-dependent design choice. We respectfully note that COCO-30K, while mostly safe, is suitable for measuring general text-image alignment; evaluating nuanced partial alignment under unsafe prompts would require a specialised benchmark, which we see as an exciting avenue for future work.\n\n$\\\\\\\\$\n\n### Q3: On the selection criteria for evaluation prompts\n\n> *Section 5.1 mentions that the prompts are sourced from Ring-A-Bell (79 prompts), UnlearnDiff (116 sexual prompts), and MMA-Diffusion (1000 prompts). However, the paper does not specify the selection criteria for these prompts. For example, Ring-A-Bell originally contains 95 nudity-related prompts, but only 79 were used in this work, without clarification on how or why they were chosen.*\n\n**A.** We thank the reviewer for the question regarding our prompt selection criteria. We used a subset of the original prompts for two primary reasons:\n- **Clarity of Content**: We filtered the datasets to include only those prompts that, when generated, produce images clearly and unambiguously identifiable as belonging to a nudity category by a human evaluator. Prompts that were ambiguous or contained other unsafe concepts (e.g., self-harm or violence) were excluded.\n- **Prompt Sensitivity**: We specifically selected prompts where minor variations in the text could significantly alter whether the model produced a harmful output. This allowed us to test the robustness of the safety mechanisms against subtle rephrasing attacks.\n\nThese selection criteria were adopted directly from prior work to ensure a fair and rigorous comparison with existing methods. We would revise Appendix C.2 of our manuscript (that already discussed some points) to include this detailed clarification.\n\n$\\\\\\\\$\n\n### Q4: Incorrect prompt in Figure 1(a)\n\n> *The prompt in the second row of Figure 1(a) is incorrect and does not match the corresponding image.*\n\n**A.** Thanks for catching the error. The correct prompt is \"a painting of a marionette puppet hanging limp with blood running from his eyes, by greg rutkowski, horror themed, stark light and shadows, grayscale\". We will reflect this to the camera-ready.\n\n$\\\\\\\\$\n\n**References**\n\n[1] Gong, Chao, et al. \"Reliable and efficient concept erasure of text-to-image diffusion models.\" ECCV2024.\n\n[2] Yoon, Jaehong, et al. \"Safree: Training-free and adaptive guard for safe text-to-image and video generation.\" ICLR2025.\n\n$\\\\\\\\$\n\n---\n\n**Final Remarks for Reviewer K2qK**\n\nWe sincerely thank the reviewer for their dedicated and constructive feedback. We believe we have thoroughly addressed the concerns raised. Therefore, **we respectfully ask the reviewer to reconsider their evaluation and raise their score**.\n\n---"}, {"review_id": "kmAYhZVxvU", "reviewer": "Reviewer_vAUD", "summary": "This paper proposes a training-free method called Safe Denoiser to improve the safety of diffusion models during image generation. This work introduces a framework that modifies the sampling trajectory directly, using a set of unsafe data (images, concepts) to steer generations away from harmful content. The core contribution is a theoretical derivation showing how to construct a \"safe denoiser\" by penalizing the standard denoiser with an \"unsafe\" component weighted by a function of current sample likelihoods. The method is shown to be compatible with and enhance existing text-based safety methods (like SLD and SAFREE).", "questions": "1.  The experiments are only conducted on Stable Diffusion v1.4. Have the authors evaluated the proposed method on more recent and stronger diffusion models such as SDXL and SD 3? If not, could the authors comment on the potential applicability and limitations of the method on newer architectures?\n\n2.  In line 111, the weight $\\beta^*$ is approximated, but the paper lacks details on the approximation method or its justification. Could the authors elaborate on how this approximation was derived and whether it introduces any significant bias or limitations?\n\n3. The method relies on a set of unsafe data points, but it is unclear how these were selected. Were the samples randomly chosen, or was there a filtering or ranking strategy applied? Clarifying this would help in understanding the reproducibility and robustness of the results.\n4.  In the ablation study, when $\\beta=0$ and the safe denoiser is applied to all samples, the Attack Success Rate (ASR) increases, which seems counterintuitive. Can the authors provide an explanation for this behavior?\n5. In Table 2, the FID score of the proposed method (22.55) is even better than that of the base model SD v1.4 (25.04), suggesting that applying safety alignment may improve generation quality. This seems surprising — do the authors have any hypothesis for why this might occur?\n6. Could the authors clarify whether Figure 3 is a conceptual illustration or an actual result from a toy experiment? If it is empirical, additional details about the setup and data used would help in understanding its significance.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Strong theoretical foundation (Theorem 3.2) connects safe, unsafe, and standard denoisers.\n2. Algorithm is training-free.\n3. Extensive experiments across multiple datasets and attack types.\n\n### Weaknesses\n\n1. The experiments are only conducted on SD v1.4, without testing the algorithm on more recent models like SDXL and SD 3, which limits the generalizability of the method.\n\n2. In line 111 of the paper, the weight $\\beta^\\*\\$ is approximated, but the paper lacks a detailed explanation of this approximation.\n\n3. It is unclear how the specific number of unsafe data points was selected — whether the data was randomly sampled or filtered using a certain strategy. The paper does not seem to provide sufficient details.\n\n4. When \\$\\beta = 0\\$, meaning the safe denoiser is applied to all samples, the ASR increases — which seems counterintuitive. Can the authors explain the possible reasons for this behavior?\n\n5. In Table 2, the proposed method achieves an FID of 22.55, which is even better than that of SD v1.4. This suggests that after safety alignment, the model generates images of higher quality than the original model. It is a strange phenomenon that safety and quality are usually trade-off. What could be the possible reason for this improvement?\n\n6. Regarding Figure 3, it is unclear whether this figure is a conceptual illustration or the result of a toy model experiment. If it is an experimental result, more implementation details are needed to understand the setup.", "was_revised": true, "final_justification": "My concerns are all addressed.", "rebuttal": "We sincerely thank the reviewer for their insightful comments and constructive suggestions. We address each of the points below.\n\n$\\\\\\\\$\n\n### Q1: Generalizability to newer models like SDXL and SD3\n\n> *The experiments are only conducted on SD v1.4. ... Have the authors evaluated the proposed method on more recent and stronger diffusion models such as SDXL and SD 3? ...*\n\n**A.** **We've extended evaluation to SD3** using Ring-A-Bell, as it has been identified as a particularly challenging dataset for SD3 [1]. The results are presented below.\n\n|Model (SD3)|ASR ↓|Toxic Rate ↓|\n|-|-|-|\n|SD3|0.304|0.330|\n|+ SAFREE|0.278|0.298|\n|+ **Ours**|**0.203**|**0.267**|\n\nAs shown, our algorithm can not only be easily integrated with existing methods like SAFREE, but also leading to a substantial performance improvement. While applying SAFREE to SD3 improves the Attack Success Rate (ASR) by approximately 9% over the baseline SD3, adding our method boosts the performance by a total of 33% compared to the original SD3. This demonstrates the effectiveness and applicability of our approach on newer and more powerful diffusion architectures.\n\n$\\\\\\\\$\n\n### Q2: Details on the approximation of $\\beta^{*}$\n\n> *... the weight $\\beta^{\\ast}$ is approximated, but the paper lacks a detailed explanation of this approximation. Could the authors elaborate on how this approximation was derived and whether it introduces any significant bias or limitations?*\n\n**A.** Thanks for the question. The exact relationship between the optimal weight $\\beta^{\\ast}(x_{t})$ and our proxy $\\beta(x_{t})$ is $\\beta^{\\ast}(x_{t})=\\frac{Z_{unsafe}\\cdot p_{unsafe,t}(x_{t})}{Z_{safe}\\cdot p_{safe,t}(x_{t})}=\\frac{Z_{unsafe}}{Z_{safe}\\cdot p_{safe,t}(x_{t})}p_{unsafe,t}(x_{t})=\\frac{Z_{unsafe}}{Z_{safe}\\cdot p_{safe,t}(x_{t})}\\beta(x_{t})$. We approximate this as $\\beta^{\\ast}(x_{t})\\approx\\eta\\beta(x_{t})$. This holds because **the scaling term $\\frac{Z_{unsafe}}{Z_{safe}\\cdot p_{safe,t}(x_{t})}$ becomes nearly constant for reasonably large $t$**. While $\\frac{Z_{unsafe}}{Z_{safe}}$ is a true constant, the key insight is that for large $t$, the distribution $p_{safe,t}(x_{t})$ becomes broad and largely independent of the specific $x_{t}$.\n\nYou are right to point out the bias: our approximation does not hold well for small $t$. However, **we don't apply our guidance at that late stage of sampling** as stated in line 117. The main structure of the image is already decided when $t$ is large. Applying guidance when $t$ is small is unnecessary and could add the exact kind of sampling bias you're concerned about. So, we only use our method in the early stages where the approximation is accurate and the guidance is most effective.\n\n$\\\\\\\\$\n\n### Q3: On the selection of unsafe data points\n\n> *It is unclear how the specific number of unsafe data points was selected — whether the data was randomly sampled or filtered using a certain strategy. ...*\n\n**A.** We used a **ranking and filtering strategy to select unsafe data points for our nudity experiments** (including those in Table 2). Specifically, we began by generating images using prompts from the I2P dataset. We then used a pre-trained model, Nudenet [2], to obtain a nudity prediction score for each generated image. After ranking all images by this score, we applied a threshold of 0.6, collecting all 515 prompt-image pairs that exceeded this value. Unless otherwise specified, all nudity experiments were conducted using this 515-item dataset.\n\nFor our multi-category experiments (including Table 3), we **randomly sampled 3,000 images from the full I2P dataset** (4,702 items). This random sampling was necessary to ensure the unsafe denoiser computation could fit within our available GPU memory (24GB). We verified that the performance variance resulting from this sampling was statistically insignificant.\n\nWe consider reproducibility as a core principle. To that end, we have **included all code used for our experiments in the supplementary material** for the reviewers' inspection. We plan to **make the code fully public upon acceptance** of the manuscript.\n\n$\\\\\\\\$\n\n### Q4: Counterintuitive behavior\n\n> *When $\\beta=0$, meaning the safe denoiser is applied to all samples, the ASR increases — which seems counterintuitive. Can the authors explain the possible reasons for this behavior?*\n\n**A.** We actually don't find this result counterintuitive. Setting $\\beta_{t}=0$ means applying the safe denoiser unconditionally, regardless of the sample's own $\\beta(x_{t})$ value. Since a small $\\beta(x_{t})$ indicates a sample is already safe, this means we are steering even the safe trajectories. Any such perturbation risks distorting a safe path into an unsafe one, so **it's best to avoid touching safe trajectories whenever possible**. While this risk might become negligible with a massive, million-sample negative dataset, that's not a practical scenario. In reality, with a relatively small-sized dataset, this unnecessary perturbation can't be guaranteed to be safe. This exact problem is precisely why we introduced the hyperparameter $\\beta_{t}$ in the first place: to make the guidance selective. As for the curve bending back up, that's simply the opposite problem: when $\\beta_{t}$ is set too high, unsafe samples no longer receive guidance and the ASR increases again.\n\n$\\\\\\\\$\n\n### Q5: Surprising FID score improvement\n\n> *In Table 2, the FID score of the proposed method (22.55) is even better than that of the base model SD v1.4 (25.04), ... do the authors have any hypothesis for why this might occur?*\n\n**A.** We were equally intrigued and conducted an additional experiment on SD3 to investigate whether this was an experimental anomaly or a phenomenon specific to SD1.4. As the table below shows, the trend persists on SD3. Even more, both SAFREE and our method achieve a better FID score than the baseline model.\n\n|Model (SD3)|FID ↓|CLIP ↑|\n|-|-|-|\n|SD3|23.145|**31.457**|\n|+ SAFREE|22.987|31.241|\n|+ **Ours**|**22.538**|31.154|\n\nBased on these consistent findings, we have formed the following hypothesis.\n\n**Hypothesis💡**: The FID score improves due to an increase in sample diversity introduced by our safe denoiser.\n\nWe offer two lines of reasoning for this conjecture. First, it is a well-known phenomenon that high CFG scales create a trade-off between fidelity and diversity; as CFG increases, fidelity is improved and sample diversity tends to decrease. This reduction in the variance of the sample distribution is a primary cause of FID degradation at high CFG values [1,2]. Our safe denoiser is not tied to the prompt conditioning and, from the perspective of CFG, may act as a form of stochasticity. This counteracts the diversity reduction caused by high CFG, thereby improving the FID score. We have confirmed experimentally that our method produces more diverse outputs for a given prompt compared to the baseline, and we also observed that at low CFG scales (where diversity is already high for both baseline and ours), the baseline SD3 model achieves a better FID, lending further support to this hypothesis.\n\nSecond and minor, this phenomenon can also be viewed from the imperfection of the FID metric itself. FID approximates the sample and data (feature) distributions as two multivariate Gaussians and measures the distance between them. Therefore, we consider this level of FID improvement to be an insignificant signal, as it is unlikely to correspond to a meaningful difference in generation quality in real-world applications.\n\n$\\\\\\\\$\n\n### Q6: Details on Figure 3\n\n> *Regarding Figure 3, it is unclear whether this figure is a conceptual illustration or the result of a toy model experiment. ...*\n\n**A.** We thank the reviewer for requesting clarification. Figure 3 is indeed the result of a toy model experiment.\n\nTo provide the implementation details: the experiment was conducted using the 'two moons' dataset. The negative dataset was a subset of the same 'two moons' dataset, created by filtering based on a specific threshold. Similar to the formulation in Eq. (4) of our paper, we used the following for the guided denoiser:\n$x_{0\\vert t}\\leftarrow E_{data}[x\\vert x_{t}]+\\tilde{\\beta}(x_{t})(E_{data}[x\\vert x_{t}]-E_{unsafe}[x\\vert x_{t}])$ and denoised with $x_{t-1}=Solver(x_{t},t,x_{0\\vert t})$, where we used the PF-ODE (DDIM) for the solver. As this figure illustrates an experiment on the guidance weight, we tested three different values for $\\tilde{\\beta}(x_{t})$: $\\frac{1}{2}\\beta^{\\ast}(x_{t})$, $\\beta^{\\ast}(x_t)$, and $2\\beta^{\\ast}(x_{t})$. All quantities required for this toy experiment, namely $E_{data}[x\\vert x_{t}]$, $E_{unsafe}[x\\vert x_{t}]$, $\\beta^{\\ast}(x_{t})$, were approximated using Monte-Carlo estimation, i.e., $E_{data}[x\\vert x_{t}]\\approx \\frac{\\sum_{x\\in D}x q_{t}(x_{t}\\vert x)}{\\sum_{x\\in D}q_{t}(x_{t}\\vert x)}$, $E_{unsafe}[x\\vert x_{t}]\\approx \\frac{\\sum_{x\\in D_{-}}x q_{t}(x_{t}\\vert x)}{\\sum_{x\\in D_{-}}q_{t}(x_{t}\\vert x)}$, and $\\beta^{\\ast}(x_{t})\\approx \\frac{\\vert D_{-}\\vert\\sum_{x\\in D_{-}}q_{t}(x_{t}\\vert x)}{\\vert D_{+}\\vert\\sum_{x\\in D_{+}}q_{t}(x_{t}\\vert x)}$, where $D_{-}$ and $D_{+}$ are the negative and positive datasets, respectively. We will add these details to the caption of Figure 3 and the supplementary material in the revised manuscript to ensure clarity.\n\n$\\\\\\\\$\n\n**Reference**\n\n[1] Sadat, Seyedmorteza, et al. \"CADS: Unleashing the diversity of diffusion models through condition-annealed sampling.\" ICLR2024\n\n[2] Kynkäänniemi, Tuomas, et al. \"Applying guidance in a limited interval improves sample and distribution quality in diffusion models.\" NeurIPS2024.\n\n$\\\\\\\\$\n\n---\n\n**Final Remarks for Reviewer vAUD** We once again thank the reviewer for their constructive feedback. We believe our responses and the corresponding revisions have fully addressed the concerns raised. Therefore, **we respectfully ask the reviewer to reconsider their evaluation and raise their score**.\n\n---"}, {"review_id": "rtjChrMYUK", "reviewer": "Reviewer_ksP9", "summary": "This paper addresses the well-motivated problem of unsafe generation in diffusion models. This includes inappropriate, not-safe-for-work, and memorized generations. Instead of opting for a text-based negative prompting or fine-tuning/re-training strategies, this paper propose a novel method that directly modifies the sampling trajectory, which makes it an inference-based plug-and-play strategy. Experiments are conducted across generation tasks of text-conditional, class-conditional and unconditional and have demonstrated effective performance - mitigate unsafe generations, while maintaining the generation quality.", "questions": "Please see the first four bullet points mentioned in the weaknesses section, which corresponds to the four questions that I’d like to ask.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- The proposed method of using negative sets to modify the inference trajectories is novel.\n- The inclusion of theoretical derivations contribute to effectively deliver the insight.\n- The paper include a range of generation tasks include text-conditional, class-conditional and unconditional in the experiments, which is good.\n- The results are overall effective, where integrating the method can mitigate unsafe generations, while maintaining the generation quality.\n- The plug-and-play design of the inference-time method contributes to its practical significance.\n\n**Weaknesses**:\n\n- From Table 2, we can observe that the proposed method consistently decreases the CLIP performance (i.e., alignment to text prompt), either integrating with SLD (drop from 29.28 to 29.10) or integrating with SAFREE (drop from 30.98 to 30.66). The same observation can be seen from Table 3. Thus, I would suggest the authors only claim improved preservation of quality (instead of both quality and alignment) while reducing unsafe generations.\n- It would be better if, in the results section, the authors could explain what the potential reasons could be that, after modifying the inference process to prevent unsafe generation, the FID can be improved even compared to the original Stable Diffusion. From the method design, there is currently no intuition that it can improve on the base model’s generation quality, so do you suggest this as a limitation of FID or other reasons?\n- The presentation of Figure 2 can be significantly improved. $\\mathbb{E}_{data}$ appears in the caption of Figure 2, but is neither in the visual nor previously defined in the caption. The curves in the left plot are also not defined. Also, for the right plot, it is better to include in the caption what settings are used for creating this 2D illustration to make the Figure self-explanatory.\n- The authors have acknowledged that one of the method’s limitations is its requirement for careful tuning of hyperparameters to balance image quality and safety, which is good to have such a discussion in the paper and appendix. However, I would recommend the inclusion of a formal hyperparameter analysis to demonstrate the robustness and sensitivity, which will strengthen the reproducibility.\n- (Minor) There is no Figure 4 in the paper, Figure 5 appears after Figure 3. Also, in line 32, “intuition” is misspelled.", "was_revised": true, "final_justification": "I appreciate the authors' rebuttal, which has addressed my concerns very well. Therefore, I'll keep supporting this paper's acceptance.", "rebuttal": "We are grateful to the reviewer for their valuable feedback and suggestions. We address each of the comments below.\n\n$\\\\\\\\$\n\n### Q1: Claiming preservation of \"alignment\" vs. \"quality\"\n\n> *From Table 2, we can observe that the proposed method consistently decreases the CLIP performance (i.e., alignment to text prompt), either integrating with SLD (drop from 29.28 to 29.10) or integrating with SAFREE (drop from 30.98 to 30.66). The same observation can be seen from Table 3. Thus, I would suggest the authors only claim improved preservation of quality (instead of both quality and alignment) while reducing unsafe generations.*\n\n**A.** That is an excellent point, and we thank the reviewer for this insightful observation. We agree that given the slight decrease in CLIP scores, claiming preservation of \"alignment\" could be misleading. **In our revision, we will adjust our claims to more accurately state** that our method preserves overall generation quality while effectively reducing the generation of unsafe content.\n\n$\\\\\\\\$\n\n### Q2: Explanation for FID improvement\n\n> *It would be better if, in the results section, the authors could explain what the potential reasons could be that, after modifying the inference process to prevent unsafe generation, the FID can be improved even compared to the original Stable Diffusion. From the method design, there is currently no intuition that it can improve on the base model’s generation quality, so do you suggest this as a limitation of FID or other reasons?*\n\n**A.** That's an excellent question. Our take is that **this isn't simply a limitation of FID, but rather a revealing side-effect** of our method's interaction with the sampling process.\n\nThe core issue is the well-known diversity trade-off with high CFG scales. High CFG creates high-fidelity samples but kills diversity, and FID heavily penalizes this lack of variance. We observe our safe denoiser counteracts this. Since its guidance is independent of the text prompt, it acts as a diversifying force, breaking up the uniformity that high CFG creates.\n\nThe result is a set of samples that is not only safe but also more varied—and therefore, a better statistical match to the real data in the eyes of the FID metric. In short, **our safety guidance seems to have the beneficial side-effect of fixing a known weakness of high-CFG sampling**. \n\n$\\\\\\\\$\n\n### Q3: Improving the presentation of Figure 2\n\n> *The presentation of Figure 2 can be significantly improved. $E_{data}$ appears in the caption of Figure 2, but is neither in the visual nor previously defined in the caption. The curves in the left plot are also not defined. Also, for the right plot, it is better to include in the caption what settings are used for creating this 2D illustration to make the Figure self-explanatory.*\n\n**A.** We thank the reviewer for these concrete and valuable suggestions. **We will revise Figure 2 and its caption in the final manuscript to make it fully self-explanatory**. Specifically, we will:\n1. change the figure label to match the caption's notation ($E[x∣x_{t}]\\rightarrow E_{data}[x∣x_{t}]$),\n2. state that the definition of $E_{data}[x∣x_{t}]$ is presented in Preliminary (in line 48),\n3. state that the black arrows depict the standard denoising process (i.e., making a data prediction and then re-noising),\n4. state that the red arrow visualizes the key step of our method: shifting the data prediction, $E_{data}[x∣x_{t}]$, to a safe prediction, $E_{safe}[x∣x_{t}]$, before the re-noising step is applied,\n5. add more details for Figure 2-(b) about the settings used to generate the plot.\n \nWe believe these revisions will significantly improve the figure's clarity.\n\n$\\\\\\\\$\n\n### Q4: Sensitivity analysis\n\n> *The authors have acknowledged that one of the method’s limitations is its requirement for careful tuning of hyperparameters to balance image quality and safety, which is good to have such a discussion in the paper and appendix. However, I would recommend the inclusion of a formal hyperparameter analysis to demonstrate the robustness and sensitivity, which will strengthen the reproducibility.*\n\n**A.** We agree that a hyperparameter analysis is crucial for demonstrating the robustness of our method. To this end, we would like to **kindly point the reviewer to Figure 5** in our manuscript, where we have already provided an ablation study on the two most critical hyperparameters: the number of unsafe data points and the guidance weight $\\beta_{t}$. \n\nIn addition, we recognize that **sensitivity to the choice of negation set is another important aspect of robustness**. Therefore, we conducted **a new ablation study** to investigate this. We constructed five distinct negation sets, each containing 600 non-overlapping data points, and evaluated our method on each. As shown in the table below, **the performance remains highly stable across these different sets**, with minimal variance. We believe analysis done in the paper and here addresses the core of the reviewer's concern regarding the sensitivity.\n\n||Avg.|Harrasment|Hate|Illegal Activity|Self-harm|Sexual|Shocking|Violence|\n|-|-|-|-|-|-|-|-|-|\n|Stats (over 5 exp)|0.1435$\\pm$0.0010|0.1577$\\pm$0.0027|0.1101$\\pm$0.0013|0.1591$\\pm$0.0014|0.1471$\\pm$0.0035|0.0836 $\\pm$0.0022|0.1629$\\pm$0.0028|0.1838$\\pm$0.0015|\n\n$\\\\\\\\$\n\n### Q5: On typos and figure numbering\n\n> *(Minor) There is no Figure 4 in the paper, Figure 5 appears after Figure 3. Also, in line 32, “intuition” is misspelled.*\n\n**A.** We are grateful to the reviewer for their careful proofreading and for catching these errors. We will correct the figure numbering and the typo in our final revision."}, {"review_id": "yAStQwghDw", "reviewer": "Reviewer_u6A8", "summary": "This paper addresses the problem of diffusion models generating inappropriate data. \nExisting approaches either require retraining the diffusion model, leading to high computational cost, \nor avoid retraining but lack theoretical guarantees.\nIn contrast, the proposed method corrects the sampling trajectory using inappropriate data to steer the generation process away from undesirable regions. \nSince the method does not require retraining, it is computationally efficient, \nand it is also supported by theoretical guarantees. \nExperiments on several datasets demonstrate that the proposed method can effectively prevent the generation of inappropriate images without compromising generation quality.", "questions": "The proposed method does not require retraining, which is a key advantage. \nHowever, in Table 2, it outperforms retraining-based methods such as ESD and RECE.\nIntuitively, I would have expected retraining-based methods to perform better if computational cost were not a concern.\nWhy does the proposed method achieve better performance in practice?\nAdditionally, is it possible that retraining-based methods would outperform your approach if the number of inappropriate data samples were increased?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- This paper proposes an efficient approach that corrects the sampling trajectory without requiring retraining of the diffusion model.\n- Theoretical guarantees are clearly stated, particularly in Appendix A, and the approximation described in Section 3.2 appears reasonable.\n- The proposed method also demonstrates strong empirical performance across datasets.\n\n## Weaknesses\n\n- There are multiple apparent typographical errors in mathematical expressions. Here are a few examples:\n  - In Figure 2(a), the label should likely be $E_{\\text{data}}$.\n  - In Algorithm 1, line 9, it seems that $p_{0t}$ should be $q$.\n  - In Appendix A, line 791, $p_{\\text{data}}$ seems to be mistakenly written as $p_{\\text{world}}$.\n  - In line 792, the expression begins with $p_{\\text{safe}, t_0}$, which likely should be $p_{\\text{safe}, t}$.\n- It would be helpful to provide more explanation around the use of expectations ($\\mathbb{E}$). For instance, in line 48, inserting an explicit expectation before applying Bayes' rule could improve clarity.\n- In Figures 2(b) and 3, using not only color but also different marker shapes would improve readability. This is especially true for Figure 3, where the distinction between (a) and (b) was quite difficult to discern.", "was_revised": true, "final_justification": "The proposed method of preventing inappropriate images without retraining is interesting, and the experimental results are strong. On the other hand, there are issues with the writing. Hoping that the writing will be improved, I will take the position of Weak Accept.", "rebuttal": "We are grateful to the reviewer for their careful and detailed feedback.\n\n$\\\\\\\\$\n\n### Q1: Typographical errors in mathematical expressions\n\n> *There are multiple apparent typographical errors in mathematical expressions.*\n\n**A.** We thank the reviewer for their careful proofreading. We will correct all typographical errors in the mathematical expressions in our final revision.\n\n$\\\\\\\\$\n\n### Q2: Clarifying the use of expectations\n\n> *It would be helpful to provide more explanation around the use of expectations ($\\mathbb{E}$). For instance, in line 48, inserting an explicit expectation before applying Bayes' rule could improve clarity.*\n\n**A.** We agree with this suggestion. To improve clarity and readability for our audience, we will revise the manuscript to include the explicit expectation operator as the reviewer recommends. \n\nConcretely, we will clarify in line 48 that the expectation follows from a standard application of Bayes' rule:\n$E_{data}[x|x_{t}]=\\int x \\cdot p(x|x_{t})dx=\\int x\\frac{p(x,x_{t})}{p(x_{t})}dx=\\int x\\frac{p(x)p(x_{t}|x)}{p(x_{t})}dx$. We will also make it clear that, by this Bayes' rule, the posterior distributions for $E_{safe}$ and $E_{unsafe}$ can be defined as Definition 3.1 of our manuscript.\n\n\n$\\\\\\\\$\n\n### Q3: Improving readability of Figures 2(b) and 3\n\n> *In Figures 2(b) and 3, using not only color but also different marker shapes would improve readability. This is especially true for Figure 3, where the distinction between (a) and (b) was quite difficult to discern.*\n\n**A.** That is an excellent suggestion for improving the accessibility and readability of our figures. We will update Figures 2-(b) and 3 to use distinct marker shapes in addition to colors in our revision. We thank the reviewer for this helpful advice.\n\n$\\\\\\\\$\n\n### Q4: Performance compared to retraining-based methods\n\n> *The proposed method does not require retraining, which is a key advantage. However, in Table 2, it outperforms retraining-based methods such as ESD and RECE. Intuitively, I would have expected retraining-based methods to perform better if computational cost were not a concern. Why does the proposed method achieve better performance in practice? Additionally, is it possible that retraining-based methods would outperform your approach if the number of inappropriate data samples were increased?*\n\n**A.** We thank the reviewer for this insightful question. We agree with the reviewer’s intuition that retraining-based methods should theoretically have an advantage over our training-free method under ideal conditions.\n\nHowever, we hypothesize that their **underperformance comes from** the inherent challenges of fine-tuning large-scale models with **limited data**. The retraining-based methods in our comparison are typically fine-tuned on safety-related datasets containing only a few thousand samples. It is a well-documented phenomenon in machine learning that fine-tuning massive pre-trained models on small, specialized datasets can **degrade their generation capabilities**, often due to issues like catastrophic forgetting or overfitting. In essence, the model's rich and generalized knowledge can be compromised when adapting to a narrow data distribution.\n\n**Our training-free approach is designed to circumvent this disadvantages**. By modifying the inference process without altering the model's learned weights, we preserve the full and robust performance of the original foundation model."}, {"review_id": "up2uQeiUUL", "reviewer": "Reviewer_Wpcs", "summary": "The paper introduces training-free safe denoiser, a method to ensure diffusion models generate safe content without retraining. Current training-free methods mainly rely on prompts technique, such as incorporating unsafe/safe prompts into the process of CFG. This paper proposes to incorporate images as guidance with the guidance scale adaptive to the current generated $x_t$. Further experiments demonstrate the effectiveness of the proposed safe denoiser.", "questions": "1. What is the corresponding negative images in Figure 1(b)?\n\n2. More insights about the method. (Weakness 2)\n\n3. What is the data distribution in Figure 2&3? Is it a 2D toy experiment or a t-sne result?\n\n4. What are the advantages of this method over methods using text filters or image content safety checker?\n\n\n## Score Justification\n> Overall, I recognize the proposed method as a new sampling technique. However, **the insight behind this method** and **the practicality of this method in real-world problems** are the main concerns. I will definitely raise the score if these concerns are addressed.", "limitations": "Limitations (requires multiple images) are provided in the Appendix. However, due to the extra required images, there are still some other potential limitations:\n\n+ In some scenarios, collecting corresponding negative images is even much more challenging than designing a safe sampling method. For instance, for IP like Spiderman or celebrity identity.\n\n+ It is computational intensive to process these negative images if these negative sets alter with different erasing task.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n1. The idea to incorporate images instead of text prompts is interesting.\n\n2. The paper is well written.\n\n## Weakness\n\n1. Why not provide some qualitative results? (examples in Figure 1 is not enough)\n\n2. More high-level intuition is suggested. Though the paper provides rigorous mathematical framework about the proposed safe denoiser, I find it does not provide as much insight as the math. To the contrary, I especially value the insight provided in (line 91-96). Besides, personally, I would interpret the method as **replacing negative prompt with images as the sampling guidance**. I'm looking forward to more discussion on this perspective in the rebuttal. Explanations on why images are better than words may provide valuable insight.\n\n3. The proposed method may not be applied to specific concepts. In other word, it can only be used for very general concepts like NSFW.\n\n4. (Minor) I'm not sure if this **training-free** method is practical in real-world, as it can be easily bypassed (for instance, using the default sampler not the safe denoiser instead). \n\n5. (Minor) The qualitative results in Figure 1 show that all these training-free method will manipulate the structure (comparing to the first column).", "was_revised": true, "final_justification": "I would give a brief summary of the discussion period to facilitate AC's final decision:\n\nOverall, the paper addresses an important problem in AIGC safety by proposing a novel image-based steering method. While the initial submission had significant gaps in clarity and completeness, the authors have made substantial revisions to address the key concerns raised during the review process. The potential updated manuscript now may include critical components that strengthen its impact and contribution to the field.\n\nThe first version of the manuscript suffered from three major shortcomings:\n\n+ Lack high-level intuition: The intuition behind the proposed method was not clearly explained.\n\n+ Absence of qualitative results: The paper relied heavily on empirical quantitative validation.\n\n+ Unstated limitations: The scope of the method was not explicitly discussed.\n\nHowever, during the rebuttal process, the author provides:\n\n+ Clear motivation of image based method. (the ambiguity of textual description and the orthogonal safety tool for text-based methods)\n\n+ Expand qualitative results. (including the Munch case which is quite interesting and inspiring.)\n\n+ Refine limitation section. Explicitly link stated limitations to the method's design. (please see Q4,Q5,Q8&Q9)\n\nTherefore, I raise my score to Accept due to the authors' comprehensive revisions. The inclusion of the above elements (limitations, qualitative results, and high-level intuition) significantly improves the paper's scientific value. However, the inability to fully validate the qualitative results during the rebuttal phase and uncertainty about the final version's alignment with the proposed revisions prevent me from assigning a higher score.", "rebuttal": "We thank the reviewer for their thoughtful feedback and insightful questions. We address each of the points below.\n\n$\\\\\\\\$\n\n### Q1: Lack of qualitative results\n\n> *Why not provide some qualitative results?*\n\n**A.** Due to the page limitations of the main paper, we included a representative set of qualitative examples and placed a more extensive collection in the supplementary material (see Figures E.4 - E.13). **We plan to incorporate more of these results into the main body for the camera-ready version, for which an additional page is permitted**. We can provide specific examples if desired.\n\n$\\\\\\\\$\n\n### Q2: Need for more high-level intuition\n\n> *More high-level intuition is suggested. ... Besides, I would interpret the method as replacing negative prompt with images as the sampling guidance. ... Explanations on why images are better than words may provide valuable insight.*\n\n**A.** We are grateful for this profound question, as it allows us to clarify two key aspects of our work: first, **why image-based guidance is often a more robust tool than text**, and second, **how our framework is a more general and complementary approach** than simply replacing negative prompts.\n\nFirst, on why images are often a necessity for true safety, we offer two primary arguments:\n1. **Textual Negation is Unreliable**: For critical applications like copyright enforcement (e.g., in the music industry), textual negation is often insufficient.As recent work [1] shows, textual negation can paradoxically generate concepts present only in the negative prompt. The fact that textual negation can fail in such unpredictable and counter-intuitive ways makes it an unreliable tool for critical safety applications. In contrast, image-based negation offers a more direct and robust mechanism. It operates by explicitly pushing the generated sample's latent representation away from that of the negative examples, providing a more reliable safeguard against generating specific, forbidden content.\n2. **Some Problems are Ill-Posed for Text**: Consider the \"right to be forgotten,\" which would require a model to avoid generating images of millions of specific individuals. Devising a unique and effective negative text prompt for every individual is a practically impossible task. Image-based negation is an easy and viable way for such instance-level removal. We view this paper as establishing the fundamentals, with scalable implementation being a critical next step for future research.\n\nSecond, while the reviewer’s interpretation of our method as \"replacing negative prompts with images\" is an insightful starting point, we would offer a more accurate perspective. Our image-based guidance is not merely a replacement but an orthogonal safety tool. The key distinction is that it is not mutually exclusive with text-based negation and our method can be applied in addition to traditional negative prompts. Our experiments confirm that when both are used together, they provide a significant, cumulative safety benefit for a negligible increase in computational cost. This makes our approach a highly flexible and powerful addition to the existing safety toolkit, rather than a mere substitute for one of its components.\n\n$\\\\\\\\$\n\n### Q3: Applicability to specific concepts\n\n> *The proposed method may not be applied to specific concepts. ... can only be used for very general concepts like NSFW.*\n\n**A.** Our method is, in fact, concept-agnostic and **can be applied to negate any concept** for which negative examples exist. To demonstrate this, we would like to kindly point the reviewer to Table 4 in our manuscript, where **we conduct an experiment specifically on negating a single, highly-specific ImageNet class** (the Chihuahua).\n\n$\\\\\\\\$\n\n### Q4: Practicality and bypass vulnerability\n\n> *(Minor) I'm not sure if this training-free method is practical in real-world.*\n\n**A.** This is an important point regarding the deployment of safety methods. Our work is a research paper aimed at establishing a foundational methodology for addressing sensitive issues like NSFW content, copyright, and privacy. As argued in our response to Q2, **critical challenges in video/audio copyright and personal privacy necessitate a shift towards sample-based negation**. In practice, real-world safety systems are built in layers, typically using pre-filtering (of prompts) and post-filtering (of images). Our work provides an additional, crucial layer of safety that operates during the generation process itself.\n\n$\\\\\\\\$\n\n### Q5: Structural manipulation in qualitative results\n\n> *(Minor) The qualitative results in Figure 1 show that all these training-free method will manipulate the structure (comparing to the first column).*\n\n**A.** The reviewer's observation is correct, but we would argue **this change is not a flaw but an inherent and expected outcome of any safety intervention**.\n\nOur method's goal is not to preserve the structure of the original model's potentially unsafe output. Instead, our goal is to steer the sampling process to draw from a safe distribution $p_{safe}(x)$. By definition, any guidance that alters the trajectory away from the original path will change the final sample's structure. The critical measure of success is not faithfulness to the original (and potentially unsafe) generation, but a successful generation from the target safe distribution. This structural change is a fundamental characteristic shared by training-based safety alignment methods as well.\n\n$\\\\\\\\$\n\n### Q6: Negative images used in Figure 1(b)\n\n> *What is the corresponding negative images in Figure 1(b)?*\n\n**A.** For the experiment in Figure 1-(b), we used a random set of **10 solo real portrait** of the individual from the internet as the negative images, see **Figure C-3 in Appendix**.\n\n$\\\\\\\\$\n\n### Q7: Data distribution in Figures 2&3\n\n> *What is the data distribution in Figure 2&3?*\n\n**A.** The visualizations in Figures 2 and 3 are from a 2D toy experiment conducted on the **two moons dataset**. The Python code snippet below generates the 2D toy data, with hyperparameters for centering and scaling.\n\n```python\nfrom sklearn.datasets import make_moons\ndef moons_dataset(n):\n    X, _ = make_moons(n_samples=n, random_state=42, noise=0.03)\n    X[:, 0] = (X[:, 0] + 0.3) * 2 - 1\n    X[:, 1] = (X[:, 1] + 0.3) * 3 - 1\n    return X.astype(np.float32) / 3.\ndataset = moons_dataset(2000)\n```\n\n$\\\\\\\\$\n\n### Q8: Advantages over text/image filters\n\n> *What are the advantages of this method over methods using text filters or image content safety checker?*\n\n**A.** This is a fundamental question applicable to all research on in-process safe generation. The reviewer rightly points out that pre-filtering (of prompts) and post-filtering (of images) are existing safety mechanisms. Our answer is that **in-process safe generation provides a critical additional layer of protection—a \"defense-in-depth\" strategy**.\n\nIn the wild, users are incredibly creative in devising adversarial methods to bypass static filters. Our method introduces a safeguard that operates during the generation process itself. More importantly, unlike **pre- and post-filtering methods** which are typically **discriminative, our approach is generative**. This qualitative difference in mechanism means it can protect against failure modes that traditional filters might miss, thus contributing to a more robust overall safety system.\n\n$\\\\\\\\$\n\n### Q9: Difficulty of collecting negative images\n\n> *... collecting corresponding negative images is even much more challenging than designing a safe sampling method. ... IP like Spiderman ...*\n\n**A.** We thank the reviewer for raising this practical concern. However, **we argue that this challenge is not unique to our method but is inherent to any content filtering system, including the safety checkers the reviewer mentioned**.\n\nFor a post-hoc safety checker to be able to identify and flag an image of Spiderman, it must have been trained on images of Spiderman. Therefore, any service provider that possesses a functional content checker for a specific IP or celebrity identity implicitly possesses the corresponding inhouse dataset of images. Our method simply proposes to leverage this already existing data for negation during inference. Consequently, we believe that for a service provider, **collecting these negative images does not pose an additional burden compared to building a safety checker [2] or implementing a retraining-based safety method [3]**, both of which face the exact same data requirement.\n\n$\\\\\\\\$\n\n**References**\n\n[1] Ban, Yuanhao, et al. \"Understanding the Impact of Negative Prompts: When and How Do They Take Effect?.\" ECCV2024.\n\n[2] https://platform.openai.com/docs/guides/moderation\n\n[3] Biggs, Benjamin, et al. \"Diffusion soup: Model merging for text-to-image diffusion models.\" ECCV2024.\n\n---\n**Final Remarks for Reviewer Wpcs**\n\nOur safe denoiser gives a concrete geometric intuition—each forbidden concept acts as a “repulsive magnet” in latent space, so every diffusion step is gently steered away from the negative image manifold rather than relying on brittle textual negation. This same mechanism unlocks protections that text alone cannot deliver: instance-level copyright blocks (e.g., specific album art in the music industry) and “right-to-be-forgotten” privacy filters for individual faces work immediately once a handful of reference images are supplied. Operationally the method is plug-and-play—layers neatly on top of any sampler or prompt filter, reuses the small image sets already needed by standard safety checkers, and adds < 5 % runtime—so the conceptual clarity comes hand-in-hand with real-world deployability, directly addressing the two concerns you highlighted. Should these additions resolve the insight and practicality concerns you highlighted, we kindly ask you to consider updating your score, and we will engage promptly and thoroughly in the discussion phase should any additional questions arise.\n\n---"}]

Author rebuttal:

Meta-review: This paper proposes Safe Denoiser, a training-free method to improve the safety of diffusion models during image generation. It introduces a framework that modifies the sampling trajectory directly, using a set of unsafe data (images, concepts) to steer generations away from harmful content. The core contribution is a theoretical derivation showing how to construct a "safe denoiser" by penalizing the standard denoiser with an "unsafe" component weighted by a function of current sample likelihoods. The method is shown to be compatible with and enhance existing text-based safety methods.

Strengths:
+This paper has a strong theoretical foundation connecting safe, unsafe, and standard denoisers.
+The core algorithm is training-free.
+Extensive experiments across multiple datasets and attack types.

Weaknesses:
-The discription about performance adventage should be more objective,  such as, the authors only claim improved preservation of quality (instead of both quality and alignment) while reducing unsafe generations.
-More explaination should be provided about experiment analysis, such as, what the potential reasons could be that, after modifying the inference process to prevent unsafe generation, the FID can be improved even compared to the original Stable Diffusion. 

During rebuttal, some responses have been provided, such as, the robustness of the proposed method, the effectiveness on newer, more powerful models, high-level intuition and questions about real-world practicality.

The final rating is 4 BA and 1 Accept. All reviewers agree the significant contribution of this paper.
Decision: Accept (poster)