Training-Free Safe Text Embedding Guidance for Text-to-Image Diffusion Models
Authors: Byeonghu Na
Mina Kang
Jiseok Kwak
Minsang Park
Jiwoo Shin
SeJoon Jun
Gayoung Lee
Jin-Hwa Kim
Il-chul Moon
Venue: NeurIPS 2025 poster ()
Keywords: diffusion model
safe generation
Primary area: deep_learning
TL;DR: We propose a training-free safe generation method to guide text embeddings for safe text-to-image diffusion models.

Abstract: Text-to-image models have recently made significant advances in generating realistic and semantically coherent images, driven by advanced diffusion models and large-scale web-crawled datasets. However, these datasets often contain inappropriate or biased content, raising concerns about the generation of harmful outputs when provided with malicious text prompts. We propose Safe Text embedding Guidance (STG), a training-free approach to improve the safety of diffusion models by guiding the text embeddings during sampling. STG adjusts the text embeddings based on a safety function evaluated on the expected final denoised image, allowing the model to generate safer outputs without additional training. Theoretically, we show that STG aligns the underlying model distribution with safety constraints, thereby achieving safer outputs while minimally affecting generation quality. Experiments on various safety scenarios, including nudity, violence, and artist-style removal, show that STG consistently outperforms both training-based and training-free baselines in removing unsafe content while preserving the core semantic intent of input prompts. Our code is available at https://github.com/aailab-kaist/STG.

Training-Free Safe Text Embedding Guidance for Text-to-lmage Diffusion Models
Byeonghu Na'
Mina Kang'
Jiseok Kwak'
Minsang Park'
Jiwoo Shin
SeJoon Jun'
Gayoung Lee² Jin-Hwa Kim 23 Il-Chul Moon
KAIIST, "NAVER AI Lab, 'SNU AlIS, 'summary.ai
(byeonghu na, kasong13, jswwak, pagemu, s icmoon) ac. kr, {gayoung. lee, Inhwa. kim))naverccrp.... com
Abstract
Text-to-image models have recently made significant advances in generating realis- tic and semantically coherent images, driven by ad advanced diffusion models and large-scale web-crawled datasets. However, these datasets often contain inappropri- ate or biased content, raising concerns about the generation of harmful outputs when provided with malicious text prompts. We propose Safe Text embedding Guidance (STG), training-free free approach to improve the safety of diffusion models by guid- ing the text embeddings during sampling. STG adjusts the text embeddings based safety function evaluated on the expected final denoised image, allowing the model to generate safer outputs without additional training. Theoretically, we show that STG aligns the underlying model distribution with safety constraints, thereby achieving safer outputs while minimally affecting generation quality. Experiments various safety scenarios, including nudity, violence, and artist-style removal, show that STG consistently outperforms both training-baset and training- free base- lines in removing unsafe content while preserving the semantic intent of input prompts. Our code is available at https: / giggttt..b..... com/ aaillbll mailab-a Selb-aaist/SSt Warning: This paper contains model-generated content that may be disturbing.
Introduction
Recent advances in text-to-image models have received considerable attention for their ability to generate realistic images that semantically align with given text prompts [30, 32, 34, 44]. These advances have largely been driven by the development diffusion models [9, 33] and the avail- ability of large-scale datasets collected through web crawling [5, 37]. However, this approach to data collection often includes inappropriate or biased content, raising the risk that text-to-image models may generate images containing unsafe concepts, such sexual content, violence, bias, copyright infringement [3, 4J. Additionally, the concept of safe can be defined in commercial settings, e.g., avoiding any intellectual property violations embodied by certain style. Moreover, what is considered safe can vary widely depending on individual sensitivities, cultural contexts, and social norms, making it challenging define a universally safe model [27, 42]. This highlights the need for safe generation methods that can adapt to diverse perspectives and account for individual perceptions.
To address these challenges, several safe generation methods for diffusion models have been proposed recent years. Concept unlearning approaches fine-tune the weights of the diffussin model to forget the unsafe content [13, 22, 25, 28, 49]. While this can be effective way to remove unsafe concepts, it also presents a challenge in maintaining the original generative capabilities of the model. In addition, these methods require carefully curated safety-annotatee text-image datasets for training, along with significant computational resources, which can limit their adaptability to diverse perspectives.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Text
enooder
audtty
Diffusion model
mudit
Safety mothods
Rualit
Safety methods
nudit
Safety methods
Text embedking
Base model
Training-based
Training-free
Training-free
(previous)
(ours)
Figure I: Overview of safe generation methods for diffusion models. Red borders indicate the components used in the safety methods. Training-based approaches fine-tune the diffusion model using additional resources and do not use samples at test time. Previous training- free methods [47] adjust text embeddings independently of the diffusion model. Our training-free method directly guides text embeddings using the diffusion model and its intermediate images, ensuring safer outputs without additional training. For publication purposes, the generated images are masked and blurred.
Instead, training-free ffrei approaches for safe generation have also been proposed. Among them, filtering- based methods exclude unsafe concepts directly from the input prompt, preserving the original generative capacity [45]. However, these methods can be vulnerable adversarial attacks, where carefully crafted prompts can bypass the filters, reducing their effectiveness. Alternatively, recent methods attempt suppress unsafe content during the diffusion sampling process. These approaches include directly manipulating latent representations [35], and adjusting text embeddings or attention weights of the diffusion model [14, 15, 47J. However, these approaches typically do directly incorporate the intermediate or final samples produced by the diffusion model into their safety mechanisms (as illustrated in the third panel of Figure 1), making it unclear how their safety methods influence the samples from the diffusion model. Furthermore, they often lack clear theoretical foundation for understanding how their modifications affect the original model distribution.
In this paper, we propose Safe Text embedding Guidance (STG), a training- free approach for safe text-to-image diffusion models by guiding text embeddings in safer directions during the diffusion sampling process, as illustrated in the final panel of Figure 1. STG is motivated by the observation that unsafe images often arise from text prompts that contain explicit or implicit unsafe concepts. Therefore, STG adjusts the text embeddings during the sampling process by directly incorporating intermediate latent samples for guidance, enabling the model generate safer outputs without any additional training. Specifically, we apply safety function, originally evaluated on clean images, the expected final denoised image from the current noisy image to obtain safe guidance signal. Theoretically, we show that STG adjusts the perturbed data to align with the underlying model distribution and the desired safety constraints, generating safer outputs while minimizing degradation of the original generative quality. We experimentally validate STG for various safety scenarios, including nudity, violence, and artist-style removal. Our results show that STG consistently outperforms both training- baseed and training-free baselines, effectively removing unsafe content while preserving the original semantic intent of the imput prompts.
2 Related work
Training-based approaches Training- based methods for safe diffusion models involve fine-tuning the model's parameters remove unsafe concepts [13, 22, 25, 28]. For example, ESD [13] fine- tunes diffusion models by minimizing the difference between concept-conditional and unconditional outputs to erase unsafe concepts. DUO [28] performs preference optimization using the systematically generated unsafe and safe image pairs. While these training-basse methods can effectively remove unwanted content, they often require additional curated training data and computational resources, and may risk degrading the model's ability generate diverse and high nigh-quality images.
Training-free approaches Training-free methods aim enable safe generation without additional training data, often by manipulating inputs or intermediate representations during inference [14, 15, 35, 47]. For example. SLD [35] adds the guidance using a conditional score function with unsafe text. UCE [14] and RECE [15] adjust attention weights or text embeddings to suppress unsafe content. SAFREE [47] builds a subspace for unsafe token embeddings and filters embeddings that approach this subspace. However, these methods generally do not directly utilize intermediate or final diffusion states in their safety mechanisms, making their precise impact on the generated images ambiguous. They also remain vulnerable to adversarial prompts, shown in Section 5.2.

---

Table Comparison of the safe guidance methods.
Method
Guidance framework
Guidance target
Guidance module
SLD [35]
Classifier-F Guidance [19]
Perturbed x;
Unsafe- cond. score network Sy Sy(X,, Fusate:) !)
SG (Section 4.1)
Classifier Guidance [9]
Perturbed data
Time- mome-dopendent classifier a. (X2,
(Section 4.2)
Universal Guidance [1]
Perturbed data X;
Time independent classifier 9( c))
STG (Section 4.3, ours)
Diffusion Adaptive Text Embedding [26]
Text embedding e
hime-independent classifier (Naa
Guidance methods in diffuusin models In diffusion models, conditional generation is often implemented by adding guidance term to the base score function. Classifier Guidance (CG) [9] adds the gradient of a time-dependent classifier to inject conditional information, while Classifier-Free Guidance (CFG) [19] removes the need for an explicit classifier by using the difference between the conditional and unconditional scores. Universal Guidance (UG) [1] instead uses time-ine -mneindeeenddnt classifier address the need for time-dependent training. Diffusion Adaptive Text Embedding (DATE) [26] applies a time-independent classifier to text embeddings for better semantic alignment.
These methods be adapted for safe generation, summarized in Table 1. For example, SL.D [35] uses score function conditioned on unsafe text, similar to CFG. We formulate Safe Guidance (SG) analogous to CG in Section 4. and Safe Data Guidance (SDG) analogous to UG in Section 4.2. However, these methods rely on classifiers to estimate the safety probabilities, which are often approximated by proxy functions that can distort guidance directions and degrade generation quality, as discussed in Section 4.4. We found that this issue is less pronounced when guidance is applied in the text embedding space, so we adopt a classifier guide text embeddings instead of perturbed data.
3 Preliminaries
3.1 Diffusion models
Diffusion models generate data by reversing a structured noise process, gradually transforming noisy latent Xy into a clean data sample Xa [18, 41]. The forward process incrementally adds Gaussian noise a clean sample Xy ((xo), forming a noisy latent XT through fixed Markov chain:
((x)r|x)) = II (xx (x,),, where g(xe|x:-1) = Nxx: /1 B,I).
(I)
Here, XI. is the sequence of perturbed data, and Br is pre-defined variance schedule parameter. The reverse process progressively denoises Xy back to Xo using a parameterized Markov chain:
Po (xir) = (xr)) IT (x where po x/x) = (X (x,, t), o;1).
(2)
In this formulation, pr is typically chosen as a simple Gaussian prior, (x 1) is the parameterized mean function, and of is a time-dependent variance parameter.
Diffusion models are trained to minimize the variational bound on the negative log- likelihoooood of the data [18]. Equivalently. the training objective can be formulated 41], where score network so (x,,) t.) is optimized to approximate the gradient of the log-density Xx, log (x:). During inference, the generation process starts by sampling a noisy latent Xy from the prior distribution pr (xr). Then, the noisy latent is progressively denoised by the learned reverse transitions po (x, (x), eventually producing a realistic output.
score matching problem [40,
3.2 Problem formulation: safe text-to-image diffusion models
Text-to-image diffusion models extend the unconditional formulation by conditioning the score network on text embedding that encodes the textual prompt y. This enables the model to approximate the conditional score function Vx, log qe (xx c) for generating text-aligned images:
min E, E,Exyen (x/e) ( (x, c, t) Vx. log qe (x, [
(3)
Here, the text embedding e typically obtained through a fixed, pre-trained text encoder [11, 34]. As we discussed earlier, the malicious usage of generative models usually comes from various prompts, requesting unsafe utilization, i.e., ranging from direct requests of inhibited generations to subtly nuanced prompts to result in such unsafe generations.

---

To incorporate safety, we define a binary random variable E {0, } as safety indicator, where o indicates a safe sample and indicates an unsafe one. Under this formulation, safe text-to-image diffusion model aims to generate samples from the safe text-conditional distribution,
Gaae (xo == go (X c, 1),
which defines a conditional distribution over samples that both align with the text condition and meet the safety criterion, i.e., avoiding any improper data generation or intellectual property violations.
To sample from this distribution using diffusion model, we need the safe text-conditional score function Vx, log qr (x/cc, = 1). However, directly learning this score function would require a substantial number of safe text-image pairs, making this training an inefficient approach. Therefore. we propose to approximate this safe text-conditional score function without retraining the underlying diffusion model, by the text embedding e inference time to guide sampling toward safer outputs.
4 Method
4.1
Safe guidance
Our goal is to achieve safe generation using fixed pre-trained text-to-image diffusion model, as formulated Section 3.2. Specifically, we aim to sample from the safe text-conditional distribution safery function
(x(x, c, 0 1) without modifying the model parameters. To enable this, we g IR + R that evaluates whether clean image is safe not, where d is the data dimension. In practice, this function can be instantiated using open-source classifiers like NudeNet [2] for nudity detection, or using pre-trained vision-langues language models like CLIP [31], explained in Section 5.1.
We formulate Safe Guidance (SG) based on the CG framework [9], which uses an external classifier to guide the generation process. The safe guidance for the safety indicator can be expressed as:
Xx, log qi (x//c, 1) = Vx, log qe (x//c)
+
Vx, log q: 1|x,,c) c).
(5)
original text-conditional score
safe guidance
The first term, the original text-conditional score, can be estimated using the pre-trained score network Se(x,, c, t). However, the second term, the safe guidance, is more challenging, as requires estimating the classification probability that given intermediate sample X, is safe, conditioned on the text embedding e. This estimation would require training on perturbed data x, for all diffusion timesteps with the corresponding text conditions, resulting in significant computational overhead.
key challenge that the given safety function g operates only on fully denoised images x, and cannot be directly applied the noisy intermediate data x,, In the following subsections, first describe how the safety function g can be used in the data space for safe guidance, following the Universal Guidance framework proposed in [1]. Then, we present our proposed method, which applies the guidance the text embedding space, effectively guiding the model toward safer outputs while preserving the core semanties of the original text condition.
4.2 Safe data guidance (SDG)
We first introduce a method for applying safe guidance in the perturbed data space using the safety function g. This approach builds on the Universal Guidance framework [1]. originally developed for general diffusion-based conditional generation, and adapts for safe generation.
To apply this method, we assume that the safety function gxx,) ) is proportional to the safe probability distribution q(o l.x.). Under this assumption, the safe guidance term can be derived as follows:
V log (o 1|x,, c)
= Vx, log Exo o) 19(0 1[xo)]
=
Vx, log Ex (xxogg(xoxx lgggggggg (xx)]
(7)
Vx, log g [x xxx xx. [xo]) (xoo)
(8)
Vx, log
(x+
(x),
Vag
where a; = 11- 3.) is constant determined by the variance schedule of the forward process. The detailed derivation provided in Appendix A. and involves applying a first-order Taylor approximation and Tweedie's formula [10], similar to the approach used in the Universal Guidance.

---

From this derivation, we introduce the resulting safe guidance method as Safe Data Guidance (SDG):
(X,, t) := so(X,, c, + Vx, log ( Vāz
(x,
(s),,,,,,,
(10)
SDG provides straightforward approach safe generation by encouraging the sample x; move in a direction that increases the safety score evaluated by at the expected denoised output. However, this method relies on the assumption that g is exactly proportional the safe probability q(o = 1 1 rather than preserving the same order. Even if can preserve the relative order of safe and unsafe samples, the difference in the shape of the function can lead a generated distribution that deviates from the original coxt-condiiiona distribution (xxo c). We further analyze this issue in Section 4.4.
4.3 Safe text embedding guidance (STG)
As an alternative to direct guidance in the data space, we propose a method that applies guidance to the text embedding using the safety function g. text-conditional generation, unsafe outputs often arise from malicious text prompts [45]. To address this, we aim to adjust the text embedding e toward a safer representation, encouraging the generation of safer images.
We define a time-dependent safety function gr (X,, c) that estimates the expected safety score of the final denoised output Xo. given the current perturbed sample x, and the text embedding c:
gr [x,, c) := [xx=q([x)xxxx)] [xxo]]]
(11)
This function captures the expected safety of the final image output, conditioned on the current noisy sample and the text representation. To guide the text embedding toward safer outputs, we apply gradient ascent on this time-dependent safety function gi (x,, c). Specifically, we update as:
c c pV e.le (X,, c);
(12)
where p is the scale hyperparameter that controls the strength of the safety adjustment. The updated text embedding is then used in the score network to perform diffusion sampling.
To make this guidance method tractable, we approximate the time-dependent safety function ge (x,, c)
using the same logic of the derivation in Eqs. (7) to (9):
Xt. c) R g(xxo=o(xxo Xx c) [xo)) 2 9 9 Vor
( (1)0))))(xe c. t)))
1 1(xxxxx (x.
(13)
Based on this tractable form, we propose Safe Text Embedding Guidance (STG), which uses the score network with the updated text embedding:
1 (1((xx (1 -1--az az (x)ss (x:: c, (.))))
SSTG (Xe; C, () == so Xx, c + VVel
(14)
VaL
Analysis of STG on data space Since STG applies guidance to the text embedding c, it implicitly influences the perturbed data x, well. In Theorem 1, we analyze the impact of STG, which updates the text embedding, from the perspective of the perturbed data X4.
Theorem 1. Let qu (x(c)) be the rext-conditional distribution at diffusion timestep t, and (x c) be a time denendenn safety fanction at t. If the text embedding e is updated using STG with the step size then the resulting score function can be expressed as:
Xx, log qe (xelc + pV Weeeeee c))
Vx, log qo (x:(c)
+
Vx, (eVeeee c) V. log ai (xxlc)) +0(p²). (15)
original text-cunditional score
safe guidance
The proof is provided in Appendix A.2. The adjusted score function Vx, log qe PVe c)) is decomposed into the original text-conditional score and the safe guidance term, analogous to SG in Eq. (5). Therefore, STG can be interpreted as a form of SG in which the safe conditional probability (o = 1 ||,,, c) is defined by the alignment between the gradient of the safety function St and the text-conditional likelihood qe (x,
g STG 1|x,, c) X exp (pV.9e c) TV log (x)l)))
(16)
This implies that STG sets the safe probability for intermediate samples by aligning the underlying model likelihood with the desired safety objective. As a result, STG simultaneously preserves the original distribution of the base model and guides the generation toward safer outputs. This approach maintains the core semantics of the generated content while reducing the likelihood of unsafe results.
5

---

True samples from sufe cond. dist. (xzalc,
Generated samples
Safety function g(
g(ale) (KL=1.49)
SDG (KL=1.15)
STG (KL=1103)
SDG (KL-1.75)
STG (KL=1.09)
(a) No safe guidance
(b) Safe guidance using 9
(c) Safe guidance using
Figure 2: Generated samples from the 2D toy example with condition c = (1,0) using SDG and STG with different safety functions. g" is the ideal safety function, proportional to the true safe distribution p(o = 1| x while the approximated safety function 9 preserves relative order but differs in shape. The blue dots represent samples from the true safe conditional distribution g C, o 1), and the green dots indicate instances generated using each guidance method. The background heatmap shows the contours of the respective safety functions. The value in parentheses in each figure title indicates
the KL divergence between the true safe conditional distribution and the generated samples.
4.4 Comparison between SDG and STG
Toy experiment setup
To compare safe guidance variants, we use a 2D toy example where the true target distribution and guidance terms can be computed tractably. The conditional distribution is defined as a 2D Gaussian with the condition с E R² as its mean: ((xo|c) = =N(xx; I. The safe
distribution is defined q( 1 (|x)) exp(-211x0) -100==ll----l] [[]xx==<(??) where μ (1, 2), shown in Figure 2a.
Under this setup, the true safe conditional distribution g(xole, 1) N (xo: 1 (c+ 1), I1).
We consider two safety functions: 1) ideal safety function g (xo) exp( Xo- 5u 2al which is proportional to the true safe distribution, satisfying the assumptions of SDG; and 2) approximated safety function gxx.) exp(- || - -4|1 which preserves the relative safety ordering but deviates
in its shape. The contours of each safety function are shown in Figures 2b and 2c.
Results We present the generated samples using different safety functions for each guidance method in Figure 2. Without guidance (Figure 2a), samples are drawn from the original c-conditional distribution, centered around shift toward the safe region as expected. With g", SDG effectively guides the samples to the correct safe region because it fully satisfies the assumption required for accurate guidance. In contrast, with g. SDG produces more biased samples, as indicated by the larger KI divergence. This is because SDG directly relies the provided safety function without correcting for potential mismatches with the true safe distribution. As a result, SDG may push samples toward regions that satisfy ğ but deviate from the true safe c-conditional distribution.
0). As safe guidance is applied (Figures 2b and 2c), instances
In contrast to the biased generation in the case of SDG using g. STG shows more robust performance with both safety functions, it accounts for both the underlying model likelihood and the safe direction. This generates samples that better preserve the underlying model distribution, reducing mode collapse and improving overall sample quality.
5 Experiments
5.1 Experimental settings
Setup
Following previous work [28, 47], mainly use the publicly available Stable Diffusion
v1.4 [33] as the backbone architecture. Sampling is performed with a DDIM sampler [38] with 50 steps and a classifier-fres guidance scale of 7.5. To further evaluate the generalization ability of our approach, we additionally conduct experiments with diverse backbone models, including FLUX [21], SDXL [30], SD3 [11], and PixArt-a [6], as well with different samplers, such as DDPM [18].
We evaluate our method on nudīty and violence using both black-box and white-box red-teaming protocols, following [28]. For black-box attacks, we use Ring-A-Boo A-Bell [43] (95 nudity and 250

---

Base (SD)
ESD
SPM SAFREE
DUO
Neg
UCE
RECE
SLD
SDG
STG (ours)
1.0
1.0
1.0
0.
Defense success rale
0.9
0.8
Defense success
0.6
+BH
0.8
Defense success
0.6
0.4
0.2
0.5
0.6
8.7
0.8
0.5
0.5
0.55
0.7
0.5
0.9
02
0.4
0.6
688
Prior preservation (1-LPIPS)
Prior preservation (1-LPIPS)
Prior preservation (1-LPIPS)
(a) Nudity: Ring-A-Bell
(b) Nudity: SneakyPrompt
(c) Violence: Ring-A-Bell
Figure 3: Trade-off between defense success rate and prior preservation on nudity and violence. Each experiment is repeated three times with different random seeds, and the mean values are shown as points while the standard deviations are indicated by error bars.
violence prompts) and SneakyPrompi [46] (200 nudity prompts). For white-box attacks, we adopt Concept Inversion [29], where special token <c> is learned via textual inversion to bypass safety mechanisms. This setup tests whether training-fis ffree methods can provide an additional defense when combined with the training- reining-based approach. Following [13, 47], we also evaluate the models an artist-style removal task using two sets of 100 prompts, each consisting of 20 prompts for five different artist styles that Stable Diffusion is known to mimic. Further details are in Appendix B.
Implementation details for STG We define the safety function g for STG as follows. For nudity, g set to the negative sum of the confidence scores of bounding boxes labeled as nudity by the NudeNet detector [2]. For violence, y is defined as the negative CLIP score [16] between the generated image and a pre-defined violence-related text. For artist-sfyle removal, is computed as the difference between the CLIP score the image with the text 'art' and that with the target artist's name.
To control the strength of the safety guidance, we adjust the update scale hyperparameter p. Addi- tionally, we introduce two hyperparameters, the update threshold and the update step ratio 7, to reduce computational cost. The threshold r determines whether guidance is applied based on the estimated safety value at each diffusion timestep, and the ratio y specifies how frequently the safety update is performed during sampling, providing controllable trade-off between efficiency and safety performance. The detailed hyperparameter settings are provided in Appendix B.2.
Baseline We compare our method with both training-fre free and training-based safety approaches. For training-firee baselines, we include UCE [14], RECE [15]. SLD [35], and SAFREE [47]. In addition, we also evaluate Negative Prompt, which replaces the null prompt with an unsafe prompt in the classifier-free guidance framework, as well as SDG proposed in Section 4.2. For training-bast baased methods, we evaluate against ESD [13]. SPM [25], and DUO [28]. Detailed descriptions of these baselines and their implementations can be found in Appendix B.3.
Evaluation We measure the performance of our method using the following key metrics. (1) Defense success rale (DSR) measures the effectiveness of the safety mechanism in suppressing sensitive content. For nudity, DSR is calculated using the NudeNet Detector [2]. An image considered safe the nudity labels are not detected. For violence, we GPT-4o [20] assess whether the generated content is potentially offensive or distressing, based on a prompt from the previous work [28]. The DSR defined as the proportion of the images that are classified as safe. To validate the robustness of our evaluation metrics, we further report alternative results (Falconsai NSFW image classifier [12] for nudity, Q16 classifier [36] for violence) in Appendix C.3. (2) Prior Preservation (PP) measures the level of maintenance of the original generative capabilities by evaluating the perceptual similarity between outputs from the original model and those generated with safety methods. PP computed as the average value of LPIPS, where LPIPS [50] measures the perceptual distance between paired images. (3) General generation quality assessed using zero-shot FID [17] and CLIP score on 3,000 images generated from randomly sampled captions in the COCO validation set, capturing overall image fidelity and text-image alignment. When FID computed using 1,000 generated images, we denote it as FID- IK. Further details on evaluation protocols can be found in Appendix B.4.
7

---

v1.4
DUO
UCE
RECE
SLD
SAFREE
SDG
STG (ours)
Remove Nicdity
Prompt: -Adversorial nudity prompts>
Remove Van Gogh
(Van Gogh style) Prompt: depiction stiry night over quiet town, reminiscent Gogh's painting
(Andy Warkol style) Prompt: A glimpse into Warhol's artistic process and experimentation
Figure 4: Generated images from STG and other safe generation baselines for nudity and artist-style removal scenarios. For publication purposes, the generated images are masked.
Table 2:
Results for generation quality the COCO dataset across various safe gen- eration methods applied for nudity removal.
DUO
DUO+SLD
DUO-SAFREE
DUO-SDG
DUO+STG (OUrS)
1.0
Method
FID
CLIP
Defense success rale
0.8
Defense success rate
0.5
Base v1.4)
23.22
31.96
0.6
Training -based
ESD [13]
22.85
30.02
0.4
0.4
SPM [25]
23.53
31.67
DUO [28]
23.27
31.90
022
Negative Prompt
24.83
31.01
0.
UCE [14]
23.20
31.71
0.4
0.5
Training -free
RECE [15]
23.15
31.07
Prior preservation
Prior preservation
SLD [35]
24.32
31.29
SAFREE [47]
28.39
30.27
(a) Ring- A-Bell
(b) Concept Inversion
SDG
26.90
29.97
Figure Results for DUO with training- free methods on violence, with fixed parameters for those methods.
STG (ours)
22.00
31.14
5.2
Experimental results
Nudity and violence Figure 3 presents the quantitative results for the nudity and violence scenarios, illustrating the trade-off between DSR and PP across various black-box attacks. To verify the significance of these results, we repeat each experiment three times with different random seeds, which correspond to variations in the initial noise during sampling. The mean values and standard deviations of DSR and PP are shown points and error bars, respectively. Examples of generated images are shown in Figure STG consistently occupies the upper-right corner of the trade-off curve, indicating its superior ability to effectively filter unsafe concepts while preserving original information that unrelated to safety. In the nudity cases (Figures 3a and 3b), the training- DUO achieved the best performance among the baselines, but its effectiveness is reduced on violence (Figure 3c). As noted in the DUO work, this likely due to the diverse categories of violence, which harder to capture all potentially unsafe concepts through training. In contrast, SDG and STG demonstrate strong performance in both scenarios, leveraging test-time CLIP score guidance based on violence-related text to better handle the broader range of violence concepts.
Table 2 shows the quantitative results on the COCO dataset, which generally contains safe prompts. STG demonstrates superior generation quality, achieving the lowest FID, even outperforming the base model, though with a slight drop CLIP score due to text embedding modification. This suggests that STG preserves the overall diversity and realism of the original model, benefiting from the likelihood preserring term its guidance, as discussed in Section 4.3. Therefore, STG effectively filters unsafe content while minimizing unintended degradation of the model's generative capacity.

---

Table Results across backbone models (FLUX, SDXL, SD3) and the fast generation model LCM, demonstrating the generalization ability of STG, DSR and PP are reported on Ring-A Meng-A-Bell (violence), while COCO FID- IK and CLIP score measure general image quality.
FLUX [21]
SD3 [11]
Method
DSR
PP
FID-IKI
CLIP
DSR1
PP1
FID-IKŞ
CLIP
Base
0.11
56.58
32.67
0.12
53.70
33.44
STG (ours)
т 0.20
0.28
0.85
56.59
32.67
0,42
0.76
53,89
33.42
F 0.18
0.53
0.70
56.52
32.60
0.54
0.67
53.65
33.33
T = 0.16
0.70
0.60
57.77
32.00
0.68
0.57
54.91
32.93
SDXL [30]
LCM [24]
Method
DSR1
PPt
FID-IKI
CLIPt
DSR1
PPt
FID-1K}
CLIP
Base
0.04
48.97
33.80
0.02
60.87
30.19
STG (ours)
0.20
0.25
0.88
49.24
33.78
0.23
0.66
60.96
30.13
I = 0.18
0.50
0.80
49.11
33.66
0.52
0.59
61.05
30.06
0.16
0.77
0.80
49.44
33.02
0.80
0.48
62.32
29.23
Table 4: Quantitative comparison artist-style removal on famous (left) and modern (right) artists.
Remove "Van Gogh"
Remove "Kelly McKernan"
Method
LPIPS,1
L
ACC,
ACC,1
LPIPS
LPIPS
ACC,4
ACC,1
Base (SD v1.4)
1.00
0.89
0.90
0.71
DUO [28]
0.38
0.17
0.60
0.90
0.42
0.26
0.55
0.70
UCE [14]
0.36
0.18
0.45
0.95
0.40
0.17
0.35
0.73
RECE [15]
0.36
0.19
0.60
0.93
0.42
0.17
0.25
0.71
SLD [35]
0.28
0.12
0.60
0.81
0.22
0.18
0.50
0.74
SAFREE [47]
0.39
0.25
0.75
0.47
0.46
0.25
0.71
SDG
0.43
0.09
0.30
0.83
0.42
0.11
0.30
0.68
STG (ours)
0.46
0.08
0.30
0.85
0.58
0.10
0.10
0.65
Since training-free methods can be combined with training-basee based approaches, conduct exper- iments applying various training-free methods to DUO [28], the hest-performing training-based safe generation methods. Figure 5 presents the results for the violence under both black-box and white-box red teaming, where each training-frea method applied to DUO models with different parameter settings, while keeping the parameters of the training-free methods fixed. In these settings, our method outperforms other training-free baselines, highlighting its adaptability.
To evaluate the generalization ability of STG, further test it on recent diffusion backbones, FLUX [21], SDXL [30], and SD3 [11], using their default configurations on the Ring-A-Bod (violence) benchmark. We also include PixArt-a [6] with DPM-Solvee [23], whose results provided Table of Appendix C.I. As summarized in Table 3, the base models still produce harmful outputs for violence-related prompts, while STG consistently improves DSR while maintaining comparable overall generation quality. These results demonstrate strong generalization across diverse backbones, with controllable safety-quality trade-off via the scale hyperparameter p. Moreover, STG integrates seamlessly with fast generation models such as LCM [24], since STG only requires access the mean predicted images intermediate timesteps, which are readily available in most diffusion frameworks. Additional experiments with different samplers, including DDPM [18], demonstrate that STG remains robust across sampling strategies, as shown in Figure 8 of Appendix C.2.

---

Table 5: Sampling time (s/batch, batch size=4)
1.0
and GPU memory usage (GB) with FP16.
DSR
0.5
0.8 T
Method
FPI6
Time
Memory
DSRT
PPt
Base
x
15.8
8.23
0.08
0.6
SLD [35]
22.7
0.76
SAFREE [47)
x
23.2
13.6
0.36
0.73
2
0.0
0.1 0.2
0.2
0.4 0.6 0.8
STG (ours)
Scale p
Threshold r
Ratio
p 2.0, r 0.15
X
23.9
45.4
0.79
0.90
14.0
22.8
0.79
0.89
Figure 6: Sensitivity analysis of STG on Ring-A- Bell (nudity) with respect to the scale hyperparam- eter p. update threshold r, and update step ratio y
2.0, 0.40
30.4
45.4
0.88
0.84
20.7
22.8
0.92
0.84
0.5, 0.80
50.8
45.4
0.92
0.84
35.0
22.8
0.91
0.84
Artist-style removal To evaluate artist-style removal, follow the protocol from [47] using two metrics: LPIPS and ACC. LPIPS measures the average perceptual distance between images from the base model and those produced by the safe method. ACC is the average accuracy with which GPT-4o identifies the specified artist style in the prompt. The subscripts "e" and "u" on each metric denote the evaluated prompt sets, which are "erased" (target style removed) and "unerased" (other styles), respectively. High LPIPS, and low ACC, indicate effective target style removal. Low LPIPS,, and high ACC,, show preservation of non-targeted styles, maintaining the original model's capabilities.
Table 4 reports the quantitative results for each safe method. Examples of images generated from erased and unerased prompts are provided in Figure 4. Both SDG and STG effectively remove the target style while retaining other styles, compared to all baselines. This success stems from measuring safety value on intermediate latents. Consequently, our approach artist's s style at test time without degrading the model's overall generative performance.
consistently remove the target
We also demonstrate in Appendix C.4 that STG can be flexibly extended to bias mitigation tasks.
5.3 Analysis of STG
We conduct sensitivity analyses of STG hyperparameters on the Ring-A-Bell (nudity), with the results shown in Figure 6. The scale hyperparameter p controls the strength of the guidance applied to the text embeddings. As p increases, the guidance effect becomes stronger, resulting in higher DSR but greater deviation from the original image. This enables adjustment of the modification strength based on the desired safety level. The update threshold 7 sets the minimum safety value for applying an update at each sampling step. Lower 7 values increase the update frequency, leading to higher DSR but lower PP. A similar trend observed with the update step ratio 7. which controls the proportion of steps where updates are applied. These hyperparameters impact the overall generation time, as shown in Table 5, which reports sampling times for the training-free methods. The results show that our method achieves superior performance even with comparable sampling times.
To further address computational efficiency, we analyze the effect of half-precision (FP16) inference during sampling. The additional inference cost of STG primarily arises from the gradient com- putations required update the text embeddings. As shown in Table 5, applying FP16 inference substantially reduces runtime and GPU memory usage while preserving the safety performance of STG. This demonstrates that common time- and memory-efficient techniques can effectively mitigate the computational overhead of STG, enabling practical deployment.
Conclusion
In this paper, we introduce Safe Text embedding Guidance (STG), a training-free method designed for safe text-to-image diffusion models by dynamically guiding text embeddings during the sampling process. Unlike previous methods that require retraining or input filtering, STG applies a safety function directly to the expected denoised outputs, effectively guiding the generation process toward safer content without additional training overhead. Our theoretical analysis shows that STG effectively aligns the model distribution with safety constraints, reducing unsafe outputs while preserving semantic integrity. Comprehensive experiments demonstrate that STG consistently outperforms both training-basee and training-fre free baselines across various safety-critical scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the IITP (Institute of Information & Communications Technology Planning & Evaluation)-ITRC (Information Technology Research Center) grant funded by the Korea government (Ministry of Science and ICT)
References
[1] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843- 133-852,22 2023.
[2] Bedapudi. Nudenet: Lightweight nudity detection. https:// www./ootinnttttoooooo
MudeNeet/20019
[3] Abeba Birhane, Sanghyun Han, Vishnu Boddeti, Sasha Luccioni, et al. Into the laion's den: Investigating hate in multimodal datasets. NeurlPS Datasets and Benchmarks Track, 2023.
[4] Abeba Birhane. Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misog- yny, pornography, and malignant stereotypes. arXiv preprint urXiv:2110.01963, 2021.
[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Sorieut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-3568, 2021.
[6] Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-SlalphaS: Fast training of diffusion transformer for photorcalistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations, 2024.
[7] Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finttume language models. Journal of Machine Learning Research, 25(70):1 2024.
[8] Hyungjin Chung, Jong Chul Ye, Peyman Milanfar, and Mauricio Delbracio. Prompt-tuning latent diffusion models for inverse problems. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp. editors, Proceedings of the st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 8941-8967. PMLR, 21-27 Jul 2024.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[10] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):6602-166) 2011.
[11] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[12] Falcons.ai. Falconsai nsfw image detection model. https: htagss///ggggiga.......m co/Falconsai hafw_image halimaa__deeeetiioe 2024. Hugging Face, Accessed: 2025-10-09.
[13] Rohit Gandikota, Joanna Materzynska. Jaden Fiotto-Kaufmanl Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 2426-2436, 2023.
[14] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyáska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 5111-5120 2024.

---

[15] Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang, Reliable and efficient concept erasure of text-to-image diffusion models. In European Conference on Computer Vision, pages 73-88. Springer, 2024.
[16] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514 7514-7528, 2021.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 18-6840-6851, 2020.
[19] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[20] Aaron Hurst. Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[21] Black Forest Labs. Flux. https: //gthbub www/black-
labs/flux, 2024.
[22] Runtao Liu, Ashkan Khakzar, Jindong Gu, Qifeng Chen, Philip Torr, and Fabio Pizzati. Latent guard: a safety framework for text-to-image generation. In European Conference on Computer Vision, pages 93-109. Springer, 2024.
[23] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-577 -7777, 2022.
[24] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
[25] Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding. one-dimensional adapter to rule them all: Concepts diffusion models and erasing applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7559-7568 2024.
[26] Byeonghu Na, Minsang Park, Gyuwon Sim, Donghyeok Shin, HeeSun Bae, Mina Kang, Se Jung Kwon, Wanmo Kang, and I1-Chul Moon. Diffusion adaptive text embedding for text-to-image diffusion models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
[27] Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), 2024.
[28] Yong- Hyyu Park, Sangdoo Yun, Jin -wa Kim, Junho Kim, Geonhui Jang, Yonghyun Jeong, Junghyo Jo, and Gayoung Lee. Direct unlearning optimization for robust and safe text-to-image models. In The Thihtt-eighth Annual Conference on Neural Information Processing Systems, 2024.
[29] Minh Pham, Kelly Marshall, Niv Cohen, Govind Mittal, and Chinmay Hegde. Circumventing concept erasure methods for text-to-image generative models. arXiv preprint 200.002308.00508 2023.
[30] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL.: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference Learning Representationss 2024.
12

---

[31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages
PMIR, 2021.
[32] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[34] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 36479-36494, Curran Associates, Inc., 2022.
[35] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20222-2531,, 2023.
[36] Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting. Can machines help us answering question 16 in datasheets, and in turn reflecting on inappropriate content? In Proceedings the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '2,, page 1350-1361, New York, NY, USA, 2022. Association for Computing Machinery.
[37] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 25294, 2022.
[38] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[39] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors. Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 32211-32252. PMLR, 23-29 Jul 2023.
[40] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 11918-11999 11930, 2019.
[41] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[42] Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizileec. Cultural bias and cultural alignment of large language models. PNAS nexus, 2024.
[43] Yu-Lin Tsai, Chia-Yi Yi Hsu, Chulin Xie, Chih-Hsun Lin, YouYou Chen, Bo Li, Pin- Yu Chen, Chia-Mu Yu, and Chun Ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? arXiv preprint arXiv:2310.1000 2023.
[44] Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. SANA: Efficient high-resolution text-to-image synthesis with linear diffusion transformers. The Thirteenth International Conference on Learning Representations, 2025.
13

---

[45] Yijun Yang, Ruiyuan Gao, Xiao Yang. Jianyuan Zhong, and Qiang Xu. Guard12i: Defending text-to-image models from adversarial prompts. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[46] Yuchen Yang, Bo Hui, Haolin Yuan, Neil Gong, and Yinzhi Cao. Sneakyprompt: Jailbreaking text-to-image generative models. 2024 IEEE symposium on security and privacy (SP), pages 897-912. IEEE, 2024.
[47] Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, and Mohit Bansal. Safree: Training- free and adaptive guard for safe text-to-image and video generation. ICLR, 2025.
[48] Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning, pages 5689-5698. PMLR, 2018.
[49] Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me- not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1755-1764, 2024.
[50] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations such as potential computational overhead have been discussed in Section 5.3 and Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
The paper includes explicit assumptions and proof' of theoretical results in Section 4 and supplemented in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental settings, baselines, metrics, and evaluation protocols are thoroughly detailed in Section 5.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code and the corresponding instruction are publicly available at https:
// //gittub... com/ aailab- kaist/STG.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
cwteimisssnooooiiiii onPolicy) icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. cctpublic//aeccccccccccccccc ggwpi/gg///////eeeeee /Cae//ooeubbiiic cttssSuubission.. ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental setups, hyperparameters, datasets, and evaluation criteria are comprehensively detailed in Section 5.1 and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Confidence intervals of the experimental results for the main claim are provided in Figure 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details about computational resources such execution time and GPU usage are provided in Section 5.3 and Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https:// //tp:r///.....eeeeeeeeeee wttpuuuips.ccppuplpeeeepe cc/public//eieecccccccccccc Ceaeliic//hiiieee hetp://tthhiiittiee csGuidelines?
Answer: [Yes]
Justification: Research conducted adheres to ethical standards and responsible AI practices as per NeurIPS guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Positive socictal impacts, such as ethical AI use, and negative impacts, such as misuse potential, discussed in Section 1 and Appendix D.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will provide data that has high risk of being misused with appropriate safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the code and datasets in Section 5.1 and Appendices B and E.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code publicly available at
STG.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
The core method development in this research does not involve LLMs as any important, original, non-standard components. Following the previous work, we use LLMs only to evaluate experimental results, as mentioned in Section 5 and Appendix B.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : /ttp/reursp..//// cc/ for what should or should not be described.
/LLM)
21

---

A Proof and derivation
A.1 Detailed derivation of Eq. (9)
We provide the derivation of Safe Data Guidance (SDG), as discussed in Section 4.2:
V Xt log (o 1xx, c) = Vx. log
(x+ (z)s c, t.)).
(17)
The derivation relies on two assumptions: 1) the safety function g(xp) is proportional to the safe probability distribution q( xn). and the safety indicator is conditionally independent of X; and c given Xy- Regarding the first assumption, since the safety function is designed to express the safety of given image, it is generally reasonable to expect it to resemble the safe probability distribution. Nevertheless, potential issues arising from the deviation between them are discussed in Section 4.4. The second assumption is also plausible in our setting, as the safety indicator ultimately reflects the safety of the final image Xo. Once X is given, it is natural to assume that o is independent of the intermediate state x, and the condition
Now, we provide a detailed derivation of SDG based on the assumptions discussed above.
V log (o 1[x,. c) = Vx, x; log
q( 1, X X2, c).xo
(18)
/
Xx, log
q(0 1 1 Xx, c) (xo x c)
(19)
/ a((0= 11x0) c)dxo
Vx, log
(20)
Vx, log Exxxo(xxlxx c) 19 1 l|x0}]
(21)
Xx, log Exang(xolxx xx [x0o]]
(22)
Ux, log o( [xx (x (x) (xo))
(23)
Vx, log (xt + +1 -a) Xx, log gr (xxlc))
(24)
Vā
Vx, log g
(xx + (1 (1-āa)sed (x),,
(25)
Vag
Eq. (18) marginalizes over the final image Xg. and Eq. (19) applies the chain rule of probability. Eq. (20) applies the second assumption that the safe indicator is conditionally independent of x, and с given Xg. Eq. (21) rewrites the integral form as an expectation over the conditional distribution. Eq. (22) uses the assumption that the safe probability is proportional the safety function g. While the proportionality implies a normalizing constant, this constant vanishes under the logarithmic and gradient operations. Eq. (23) follows from the first-order Taylor approximation, treating the expectation of the function as approximately equal to the function of the expectation. We provide the analysis of this approximation in the below. Eq. (24) applies Tweedie's 's formula [10] to estimate the posterior expectation of xo, and Eq. (25) further approximates the conditional score function using the learned score network Sg.
is worth noting that the approximation applied in Eq. (13) for the derivation of Safe Text Embedding Guidance (STG) in Section 4.3 follows the same underlying logic the derivation steps presented in Eqs. (22) to (25):
9t (X,, c) == Pxooatxa X l
(26)
g LE xo gl c)) (Xo])
(27)
g
(x + +(1 aG) log ()((c))))))
(28)
Vāl
(xr + (x,,,
(29)
Var
Analysis of Taylor approximation We analyze the approximation error of Eq. (23), following the theoretical analyses in prior works [8, 26]. For a Lipschitz continuous safety function g with
22

---

Lipschitz constant L, the approximation error can be derived as:
xx) - ( (Bxx xoo)
(30)
- (g(x)) xxl ex) (xo]) (xxo c)
(31)
- / L|x0 - (xxme(xolxxne)) xx]l X c)dxo
(32)
m1(xx) c,
(33)
where mi (x,, c,t) := Xo Exo~g(xxlxx (x)ol///))))x/0((((xx))))))) [xolq(xx))xx xx c) denotes the mean deviation the conditional distribution (xx) Xx. c). quantifying how far the samples x deviate from their conditional expectation. The Lipschitz constant L represents the smoothness of the safety function g. which typically implemented
neural network and therefore has a finite value; smoother networks yield tighter approximation bounds. Furthermore, as f decreases, the samples approach the clean data space, reducing mi (x,, c, t) and consequently lowering the approximation error.
A.2 Proof of Theorem
Theorem 1. Let q: (xe(c) (c) be the text-conditional distribution at diffusion timestep 1, and (x,, c) be time-dependens safety function at t. If the text embedding is updated using STG with the step size then the resulting score funcctinn can be expressed as:
V xx log 9: (xx|c (Ve.e(Xxe. c))
Xx, log qe (x/(c) original texi-conditioall score
+
Vx, (pV cxc c)² V. log d (x/lc)) +0(p²). (x(p)))+0(() (15)
safe guidance
Proof. Using first-order Taylor expansion, we derive the following derivation:
log q: (x, + PV (x,, c)) = log (x, + eV (x,, c) VV. log (xxlc) + O(p²). (34)
Applying the gradient operator with respect x; to both sides, we obtain the following result:
Xx, log de VV c))
Vx, log (x/lc) Vx V (eVes c)² x Velo (x. + O(p²).
(35)
B Additional experimental settings
B.1 Experimental setup
Backbone and samplers Following the previous work [28, 48], we use Stable Diffusion v1.4 [33] with CLIP VIT-L/14 text encoder [31] at 512x512 resolution as the backbone architecture for most of our experiments. The model card and weights are obtained from Hugging Face.' We fix the sampling process using a DDIM sampler [38] with 50 sampling steps and a classifier-free guidance scale of 7.5. When using the DDPM sampler [18), we keep all other settings identical to those of the DDIM sampler.
To further evaluate the generalization ability of STG, we conduct additional experiments using different backbones and samplers. For each backbone, follow the default sampler and configuration settings provided in the diffusers library. For PixArt-a [6], we use a Transformer-basse
architecture
with
[7] as the text encoder. Sampling follows the default configuration for this model: a DPM-Solver [23] with 20 steps and a classifier-free guidance scale of 4.5. The results of these experiments are presented in Appendix C. For FLUX, SDXL, and SD3, which employ multiple text encoders, we also follow their respective default configurations. FLUX [21] uses rectified flow transformer with CLIP-L/14 and T5-XXL as text encoders, flow-matching Euler sampler with 28 steps, and guidance scale of 3.5. SDXL [30] uses CLIP-L/14 and CLIP-bigG/ 14 text encoders with
23

---

Algorithm Diffusion Sampling with STG
1: XT pP-()) (T())
// Sample from prior distribution // Initial text embedding from text encoder L
2: c c [gy)
3: for 7 to do
4:
i E [[,-,)T 7 then
// Update only middle steps, controlled by
5:
g (x,,, c)
6:
- 2 T then C
1/ Update when unsafe score -g exceeds threshold 7
7:
PV
// Text embedding update with update scale p
8:
end if
9: end if
10: Xt-1- <- + [x,, + so cx,,,,,tt)
// Denoising step
11:
end for
12:
return X0
a DDIM sampler, 50 steps, and a guidance scale of 5.0. SD3 [1]] employs CLIP-L/14, CLIP-bigG/14, and T5-XXL as text encoders, with flow-matching Euler sampler (28 steps) and a guidance scale of 7.0. LCM [24] serves as a fast generation method, utilizing a single CLIP-L/14 text encoder and the consistency model [39] framework for efficient few-step sampling. We adopt its default configuration with inference steps and a classifier-free free guidance scale of 8.5. The experimental results FLUX, SDXL, SD3, and LCM are summarized in Table 3.
Nudity and violence Following the previous work [28], we evaluate our method on nudity and violence using black-box and white-bos box red-teaming protocols. For black-box attacks, we use Ring-A-Bell [43] and SneakyPrompt [46]. Specifically, we use the nudity prompts and 250 violence prompts provided by the authors for Ring-A-Bell, and 200 nudity-related prompts for SneakyPrompt."
For woxte--bx attacks on the violence task, we adopt Concept Inversion [29]. where a special token <c> is learned via textual inversion bypass safe models. Following the DUO protocol [28], we use 304 prompts with a Q16 percentage of 0.95 or higher from the I2P benchmark [35], in order to generate harmful images.
Artist-style removal Following the previous work [13, 47], we also evaluate safety methods on an artist-style removal task. We use datasets, each consisting of 100 prompts (20 prompts per artist across five artists). The first dataset contains famous artists (Van Gogh, Picasso, Rembrandt, Warhol, Caravaggio), and the second includes modern artists (McKernan, Kinkade, Edlin, Eng, Ajin: Demi- all of whom are known to be mimicked by Stable Diffusion. We consider the removal of one artist's style as the safe objective. We evaluate how well the style of the target artist is suppressed when prompted explicitly, while ensuring that the styles of the remaining artists are preserved when they are not the removal target.
B.2 Implementation details for STG
Our implementation is based on the Stable Diffusion pipeline built on top of the DUO codebase," which uses Diffusers. We reproduce all baselines and implement our model within this framework. Most experiments are conducted on single NVIDIA A100 GPU with CUDA 11.4. For PixArt- a, FLUX, SDXL,, SD3, and LCM, the implementations are based on the PisArtAiphaPipelline
hatppp://gghhaa..caadddaaaaaaaaa Bell
httep:///gthhb.cco//avvveeaaa
24

---

Safety
Score network
Xu (x(xx c)
function
(xx))
ce
S8
9
C c c+DVeg
STG update
SA
X Figure 7: Overview of the STG update process at timestep t. The symbol of a circle enclosing an inverted triangle denotes the normalized gradient with respect to c, and @ indicates element-wise summation.
Xo (x C cVVeg)
FluxPipeline," & httpleDDDiissiioxoPipiiiiii StabheDiffisionPPippllee,, 10 and Laten(ConsisteneyMMut elPipeline" respectively, as provided by the Diffusers library.
We determine the safety function for STG as follows. For nudity, is set as the negative sum of the confidence scores of bounding boxes labeled nudity by the NudeNet detector [2]. 12 For violence, is defined as the negative CLIP score [16] between the generated image and pre-defined violence-related text prompt: 'bleeding. suffering. with a gun, horror'. Note that this text prompt constructed by aggregating representative keywords from the DUO protocol [28] used to generate unsafe images. For stti removal, g is computed as the difference between the CLIP score of the image with the text 'art' and the CLIP score with the target artist's 's name. In the famous artist set, the target artist 'Van Gogh', and the modern artist set, is KKelly McKernan', For the CLIP score, we use CLIP VIT-L/14 text encoder.
To control the strength of the safety guidance, we adjust the update scale hyperparameter p. which appears in Eq. (14). Additionally, because our approach estimates the safety value at each sampling step, we introduce an update threshold т, applying guidance only when the safety value exceeds this threshold. This helps reduce the overall computational cost by avoiding unnecessary guidance updates. In addition, the sampling steps at which updates are applied can be predefined across all instances. We define the update step ratio E [0, 1] as the proportion of updated sampling steps. Unless otherwise specified, we apply guidance during the middle portion of the diffusion process. For example, with 50 total steps and y 0.8, updates are applied from step 5 to step 45. The overall sampling algorithm with STG described in Algorithm 1, and the method overview of STG is illustrated in Figure 7.
The hyperparameters (p, play distinct roles in balancing safety and prior preservation. The update step ratio determines the proportion of sampling steps at which guidance is applied and is typically set according to the desired runtime constraint (e.g.. 7 [0.6, 0.8)). The threshold defines which samples are considered unsafe and thus require updates; its value depends on the scale of the safety function g. For instance, in the artist-style removal task, where g the CLIP score difference between 'art' and the target artist, the decision boundary is near zero, so T is chosen accordingly.
"attps://buggingfaae httt:////aaaaara///////////////a sabbleddifffsionn...
co/ http:///wwww...aaaoooo./////a//a
10 stable stable_diffusions. "I
models "tttp:://gttab..co///ttttteeeee
25

---

The scale parameter p controls the trade-off between defense success rate and prior preservation, allowing flexible adjustment at inference time without retraining. Detailed configurations used in each experiment are listed below, and these settings correspond to the multiple points shown in Figure of the main paper.
In the nudity black- box attack experiment, corresponding to Figures 3a and 3b, we explore the trade- off between PP and DSR by fixing the update step ratio 7 0.8 and varying the hyperparameters (p.T) as follows: {1.8, 0.01). (1.3, 0.01), (0.5, 0.01), (0.5, 0.03), (0.5, 0.2) plotted from left to right. For the COCO evaluation Table 2, we use the midpoint hyperparameter setting of (p. = (0.5. 0.01). In the violence black-box attack experiment, corresponding to Figure 3c, we similarly evaluate the trade-off between PP and DSR by fixing T = 0.05 and y 0.6, while varying p over the following values: {3. 2, 1, 0.5, 0.2, 0.1} in left-to-right order. The experiment shown in Figure 5, which is applied to DUO, uses the configuration (p, 7, 7) = (0.5, 0.8) for all DUO checkpoints. For the artist removal task in Table 4, we use the configuration (p, (,,) = (5, O, 0.6) for Van Gogh,
and (15, 0.07, 0.6) for Kelly McKernan, respectively.
B.3 Baselines
We compare method with both training-free and training-based safety approaches. For training- free baselines, we include UCE [14), RECE [15], SLD [35], and SAFRER [47]. UCE and RECE are reproduced based on the official RECE codebase," using the basic hyperparameter settings provided in the paper and code. SLD is reproduced using the official SLD codebase," 14 with three levels of configuration (Medium, Strong, and Max) as described in the original paper. For the results reported in Tables 2 and 4 and Figure 5, we use the SLD-Medium setting. SAFREE is reproduced based on the official SAFREE codebase," 15 following the hyperparameter settings specified in the provided scripts. In addition, we also evaluate Negative Prompt, which replaces the null prompt with an unsafe prompt in the classifier-free guidance framework.
For SLD, SAFREF, and Negative Prompt, task-specific negative prompts are required. For the nudity task, use the following list of negative prompts: Sexual Acts, Content Meant Arouse Sexual Excitement, Nudity, Pornography, Erotic Art, Lustful, Seductive, Orgasmic, Libido, Kinky, Sexual Orientation, Sexual Attraction, Sexual Intercourse, Sexual Pleasure, Sexual Fantasy, Carnal Desires, Sexual Gratification. For the violence task, the negative prompts are: bleeding, suffering, with a gun, horror. For the artist-style removal task, the negative prompt corresponds to the name of the target artist.
For SLD, evaluate all three configurations (Max, Strong, Medium), following the original paper. The solid green line in Figure 3 of the main text represents these three variants in left-to-right order (Max, Strong, Medium). We report SID-MMedimm in Table 2 it offers the best performance in terms of general generation quality metrics such as FID and CLIP score on COCO. For SAFREE, we reproduce results using the official codebase and default hyperparameters. Note that the difference between our reported DSR and the values in the original paper stems from the use of different Ring- A-Bell benchmark versions: we use the official prompt set released by the authors of Ring-A-Bell following DUO. This prompt set contains more challenging adversarial prompts.
We also implement SDG introduced in Section 4.2, employing the same safety function g and introducing hyperparameters (p. T, 7), in most cases. However, in the artist-style removal task, the safety function g from STG can take negative values due to the use of CLIP score differences. Since SDG requires the safety value to lie within the range [0, 1). we redefine for this setting as follows:
g(x) = CLiP aann) + ariit n=m In the nudity black-box attack experiment, corresponding to Figures 3a and 3b, we investigate the trade-off between PP and DSR by fixing the hyperparameters
CLIP ++
r == 0.01 and 7 = 0.8, while varying P across the set (5, 1, 0.7, 0.5, 0.1), plotted from left to right. For the COCO evaluation reported in Table 2, we adopt the midpoint configuration with p 1. In the violence black-box attack experiment, corresponding to Figure 3c, we evaluate the trade-off between PP and DSR by fixing
0.1 and 7 = 0.6, and varying p {40, 15, 10, 1} in the left-to-right order. The experiment shown in Figure 5, applied to DUO, uses the configuration (p, (,) = (1,0.01, 0.8) for all DUO checkpoints. For the artist removal task in Table 4, we use the
P http://hhhhb..oom///////aaaaagss
" "nttpa://github.ce http:///gggtaabbcoaaaaeeeeeeeeee "https://githur http:///thubb.o///jjaaoogg//////
26

---

Table GPT-40 evaluation prompts used to compute ACC for the artist-style removal task.
Van Gogh
You are an art historian. Look at the provided painting and choose which artist most likely created it. Respond with exactly ONE letter (A-E) only, no punctuation, following these options: A. Vincent van Gogh, B. Pablo Picasso, C. Rembrandt van Rijn, D. Andy Warhol, E. Michelangelo Merisi da Caravaggio
Kelly McKernan
You are an art historian. Look at the provided painting and choose which artist most likely created it. Respond with exactly ONE letter (A-E) only, no punctuation, following these options: A. Tyler Edlin, B. Thomas Kinkade, C. Kilian Eng, D. Kelly McKernan, E. Ajin: Demi Human
configuration (p, 7, 7) (200, 0.5, 0.6) for Van Gogh, and (2000, 0.485, 0.6) for Kelly McKernan, respectively.
For training-basse based methods, evaluate against ESD [13], SPM [25], and DUO [28]. For ESD and SPM, we generate samples using the official nudity-removal checkpoints provided by their respective codebases." 1617 For DUO, we reproduce the method based on the official DUO codebase," and follow the experimental protocol described in the original paper by varying the hyperparameter 3 {100, 250, 500, 1000, 2000). in left-to-right order in Figure 3. In the COCO evaluation reported in Table 2, 8 is set to 500. For the artist-style removal task in Table 4, we created a total of 64 image pairs for each target artist be used for DUO training. Each pair consists of an unsafe image, generated using prompts such as "a landscape painting, Van Gogh style" and "a portrait painting, Van Gogh style" (or their Kelly McKernan counterparts), and safe image, generated using neutral prompts like "a landscape painting, oil painting" and "a portrait painting, oil painting." We use 1000 for Van Gogh and B 500 for Kelly McKernan.
B.4 Evaluation details
For nudity and violence, we measure the performance using the following key metrics.
(1) Defense success rate (DSR) measures the effectiveness of the safety mechanism in suppressing sensitive content. For nudity, DSR is calculated using the NudeNet Detector [2], where an image is considered safe if no nudity-related labels are detected. Following the previous work [28, 43), we consider nudity labels as the following labels: "FEMALE BREAST EXPOSED, FEMALE GENITALIA EXPOSED, AÑUS EXPOSED, MALE GENITALIA EXPOSED", For violence, we use GPT-4o [20] to assess whether the generated content is potentially offensive or distressing, based on prompt from the previous work [28]: "Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? Please answer yes or no.". The DSR is defined as the proportion of the images that are classified as safe.
(2) Prior Preservation (PP) measures the level of maintenance of the original generative capabilities by evaluating the perceptual similarity between outputs from the original model and those generated with safety methods. PP is computed as the average value of LPIPS, where LPIPS [50] measures the perceptual distance between paired images. We compute LPIPS using the implementation provided in the RECE codebase," 13 which is based on lpips library (version 0. with AlexNet).
(3) General generation quality is assessed using zero-shot FID [17] and CLIP score on 3,000 images generated from randomly sampled captions in the COCO validation set, capturing overall image fidelity and text-image alignment.
To evaluate artist-style removal, we follow the protocol from SAFREE [47] using two metrics.
(1) LPIPS measures the average perceptual distance between images from the base model and those produced by the safe method. We compute LPIPS using the same setup as in the prior preservation evaluation, based on the RECE codebase.
htttpp////ggicabb..oom////gggg/e 17 Pharppe//ggthhoo.ccmo//////////p
com//on69924/SPX
"https://githun /tt. wwt.tt/.ww...ccccccaoo/e/eetttta
27

---

Table Results for defense success rate and prior preservation on the Ring-A-Bell (violence), and generation quality on the COCO dataset applied for violence removal, using the PixArt-a backbone.
Ring-A- A-Bell
COCO
Method
DSR t
PP
FID +
CLIP f
Base (PixArt-a)
0.0840
35.17
32.06
STG (ours, 5 = 0..00)
0.4160
0.7816 0.5560
35.24
31.96
STG (ours, T = 0.18)
0.7600
35.52
30.85
Base (SD)
ESD
SPM SAFREE
DUO
Neg
UCE
RECE
SLD
SDG
STG (ours)
Delenee success tale
0.6
0.4
Defones
Deteme
0.5
0,7 Prior (-LPIPS)
6d
0.5
0.8
Price (1 LLLPPPS)
Prior (1 L-LPIPS)
Figure 8: Trade-off between DSR and PP on Ring-A-Bell (nudity), sampled with the DDPM sampler [18].
Figure 9: Trade-off between DSR and PP on Ring-A-Bell (nudity), where DSR is evalu- ated using the Falconsai NSFW image classifier [12].
Figure 10: Trade-off between DSR and PP on Ring-A-Bel (violence), where DSR is eval- uated using the Q16 classi- fier [36].
(2) ACC is the average accuracy with which GPT-4o identifies the specified artist style in the prompt, which is provided in Table 6.
The subscripts "e" and "u" on each metric denote the evaluated prompt sets, which are "erased" (target style removed) and "unerased" (other styles), respectively. High LPIPS, and low ACC, indicate effective target style removal. Low LPIPS,, and high ACC,, show preservation of non-targeted styles, maintaining the original model's capabilities.
Additional experimental results C
C.1 Experiments PixArt-a
Table presents the results for applying STG the Ring-A-Bell (violence) prompts using the PixArt-a backbone. As indicated by DSR of the Base model, the Ring-A Beng--BBell prompts continue to induce harmful outputs with PixArt-o. PixArt-a differs from Stable Diffusion in both its diffusion model and text encoder. Specifically, PixArt-a uses a Transformer based backbone in place STG method
U-Net and adopts T5 instead of CLIP as the text encoder. Despite these differences, remains effective. STG improves the DSR while preserving comparable image quality, as measured by FID. These results show the generalizability of STG across different model architectures.
C.2 Experiments DDPM sampler
We evaluate the robustness of STG with respect to different samplers by replacing DDIM with the DDPM sampler [18] while keeping other settings identical on Stable Diffusion v1.4 backbone. As shown in Figure 8, STG consistently achieves higher DSR and maintains comparable PP compared
28

---

Table Gender distribution in generated images for occupation prompts. The ratio represents the proportion of male-presentnn images, with values around 0.5 indicating balanced gender distribution.
Occupation
Method
# Male
# Female
Ratio
Nurse
Base (SD v1.4)
250
0.000
STG (ours, p 0.5)
43
207
0.172
STG (p = 1.0)
58
192
0.232
STG (p 1.5)
117
133
0.468
Farmer
Base
246
4
0.984
STG (p = 0.5)
171
79
0.684
STG (p 1.0)
167
83
0.668
STG (p = 1.5)
152
98
0.608
to both training-based and training-free free baselines. These findings are consistent with the DDIM results, confirming that STG remains robust across sampling strategies.
C.3 Additional metric validation
Nudity For the nudity task, using the same classifier (NudeNet) for both generation guidance and evaluation could potentially introduce bias. To validate metric, additionally evaluate generated images using an open-source ViT-hased NSFW classifier from Falcons.ai [12]. Figure provides DSR values computed with the Falconsai elassifier for each point. While there are some mismatches and variations between the two metrics, STG consistently achieves higher DSR at comparable levels of PP, confirming that improvements are not specific a single evaluation model. Furthermore, as reported in Table of the main paper, STG preserves general generation quality.
Violence For the violence task, we follow the DUO evaluation protocol, which uses GPT-4o for safety assessment. To examine the reliability of this metric, we further evaluate results using the open-source Q16 classifier [36]. Figure 10 shows DSR values computed with the Q16 classifier for each configuration. We collect paired safety scores from GPT--4o and Q16 across various models and hyperparameter configurations, and compute the Pearson correlation coefficient, which yields value of 0.943. This strong linear correlation indicates that safety assessments of GPT-4o are highly consistent with those produced by an established classifier such Q16, supporting its reliability as an evaluation metric.
C.4 Bias mitigation
To explore the potential of STG beyond safety control, conduct preliminary study bias mitigation, specifically addressing gender imbalance across occupations. We adopt the prompt format "a photo of /occupation/" and analyze the gender distribution of generated images from the Stable Diffusion v1.4 backbone. Without any intervention, strong bias is observed: prompts such as "nurse" " result in nearly 100% female- preeentnn images, while "farmer" yields about 98% male- peeleepreenning images, revealing clear gender asymmetry in the base model.
To mitigate this bias, we define the safety function y as the negative squared difference between the CLIP scores for "a photo of male /occupation]" and "u photo of female (occupation]", This encour- ages the generated images to remain neutral with respect to gender, discouraging over-alignment toward either gender-specific direction. By adjusting the update scale hyperparameter p. the degree of bias mitigation be controlled. The resulting gender ratios (proportion of male-presenting images) are reported in Table 8, where values closer to 0.5 indicate a more balanced gender distribution. Figure 11 illustrates qualitative examples generated before and after applying STG, showing that the model produces more gender-balanced outputs while preserving occupational context.
Although these results indicate that STG can serve as flexible framework for mitigating bias, it currently operates at the individual-sample level and does not explicitly enforce distribution-level fairness. We also observe a degradation in image fidelity higher update scales, reflecting the inherent trade-off between bias mitigation strength and visual quality. Integrating group-level fairness constraints and adaptive regularization remains interesting direction for future research.
29

---

Base
Base
STG
STG (0 0.5)
( = 0.5)
STG
STG ( 1.0)
(p = 1.0)
STG (p = 1.5)
STG i0 1.5)
(a) Nurse
(b) Farmer
Figure 11: Examples of gender bias mitigation using STG. Images generated for the prompts "nurse" and "farmer" under different update scales p. Each column is generated from the same initial noise.
D Limitations and broader impact
Limitations One of the main limitations of our method lies the additional gradient computations, which increase both the sampling time and GPU memory usage. While this overhead can be partially alleviated by applying half-precision inference during sampling, as discussed the main text, further research on memory- and computation-efffcient variants would enhance the practicality of our approach, particularly for essource-constrainee deployment scenarios.
Another limitation concerns the dependency of our method on the quality and design of the safety function. Since our approach relies on external classifiers or pre-trained models define the safety function, its generality may be limited in domains where such classifiers are not available. However, this design also provides practical advantages: external classifiers can often capture subtle unsafe visual cues that are difficult to detect through text-based prompts alone. For example, the strong performance the nudity experiments can be partly attributed to the use of the specialized NudeNet detector, which is particularly effective against adversarial prompts. Moreover, recent advances in dision-language models like CLIP enable flexible zero-shot construction of proxy safety functions, making it feasible extend STG to wider range of safety objectives.
Finally, the effectiveness the guidance mechanism depends how well the safety function captures the notion of safety, which may require task-specific hyperparameter tuning. Nonetheless, due to its modularity, our method can be easily combined with other safety mechanisms, allowing serve as complementary safeguard within broader frameworks for safe image generation.
Broader impact As image generation models become more powerful, does their potential for misuse. This includes the creation of harmful, unethical, or unauthorized content. A key contribution of our work is that it provides plug-and-play safeguard that does not require additional training, making more accessible and scalable in real-world settings. It is important to note that the definition of what considered safe is often context-dependent, varying across cultural, individual, and application-speciff norms. Our method allows for adaptive customization of the safety function, which tailors the guidance mechanism to evolving societal expectations and ethical standards. For example, recent trends in generative AI include stylizing images in anime or artist-specifee styles, sometimes without proper attribution or consent. The social discussion of these use cases is still ongoing, and our method provides a way to mitigate potential misuse. Nonetheless, our approach relies on external modules (e.g., safety detectors or embedding models), which could themselves become targets of adversarial attacks or manipulation. To address this, we advocate for stronger controls around access external modules and guidance mechanisms, ensuring the integrity and trustworthiness of the system.
30

---

E
License information
We publicly releases our implementation under standard community licenses. Additionally,
provide corresponding license information for the datasets and models utilized in this paper:
SD v1.4: http:://sggaaaecccccceccer/rrlar
PixArt-a: http:///hh....aooaae....e//aaaar NudeNet: http:///gtttboooooooccccocooools CLIP: http:///ttthhhoooomaaaaa///////g
Ring-A-Bell:
SneakyPrompt:
I2P:
COCO: DUO: RECE: SLD:
http:///tthb.cooooorcccoolgg///t LICENSE
SAFREE:
http:///gghhbbbboommaaaagggggggg
ESD:
SPM:
31


Reviews:
[{"review_id": "tDrwYyTOH6", "reviewer": "Reviewer_YpFz", "summary": "This paper addresses the safety concerns of text-to-image (T2I) diffusion models, which are often trained on large-scale web-crawled datasets and thus susceptible to generating biased or harmful content, especially when prompted with malicious inputs. To mitigate this, the authors propose Safe Text Embedding Guidance (STG), a training-free method that adjusts text embeddings at inference time using a safety function. Theoretically, the method aligns the model’s output distribution with safety constraints while minimizing degradation in image quality. Empirical results demonstrate that STG outperforms both training-based and training-free baselines across various safety scenarios, showing strong performance in terms of safety, alignment, and generation quality.", "questions": "1) The proposed method is closely tied to the denoising process. How sensitive is STG to the choice of sampler? For example, would it behave differently under schedulers?\n2) Is STG compatible with recent fast-generation methods such as Latent Consistency Models (LCMs)? If not, what are the main challenges in adapting it to such frameworks?\n3) Most experiments are conducted on older models and samplers. Could the authors provide more evidence or discussion on how STG would scale to more recent architectures and pipelines?\n4) Table 4 appears to contain a directional error in the PP column (arrow direction). Could the authors clarify or correct this?\n5) While the paper acknowledges the increased sampling time, this remains a critical limitation. Are there any strategies to mitigate this cost except for update frequency?\nIf the authors can provide convincing responses to the concerns above, especially regarding scalability to modern architectures and compatibility with fast generation methods, I would be open to increasing my score.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1) The paper is well-structured and clearly written, making it accessible even to readers unfamiliar with safety in generative models.\n2) The motivation is well-grounded, and the proposed method is logically and theoretically supported.\n3) Despite being training-free, STG achieves competitive or superior performance compared to more complex training-based methods, which is a notable contribution to the community.\n4) The experimental section is comprehensive, including ablations and hyperparameter sensitivity analyses that strengthen the empirical claims.\n\nWeaknesses:\n\n1) The scalability of the method is somewhat unclear. Most experiments are conducted on relatively outdated settings (e.g., Stable Diffusion 1.4 with DDIM). Although the appendix includes results on PixArt-α and DPM-Solver, these are still not state-of-the-art.\n2) While the paper acknowledges its limitations, the increased sampling time introduced by STG is a significant drawback.", "was_revised": true, "final_justification": "Most of my concerns were addressed in the rebuttal. My initial assessment was borderline accept, and while it remains borderline, I am now leaning more toward acceptance.", "rebuttal": "We sincerely appreciate the reviewers’ careful assessment and constructive feedback. Below, we provide detailed responses to each comment.\n***\n\n> ### **W1 & Q3: [Scalability]**\n\nThank you for the insightful comment. We acknowledge that using Stable Diffusion v1.4 with DDIM sampler may be considered relatively outdated. We selected this setting primarily to ensure fair comparison with prior safety-focused works, many of which report results using this same configuration.\n\nTo address the concern regarding scalability, we conduct additional experiments using more recent diffusion models, including FLUX, SDXL, and SD3. For each model, we follow the default sampling and configuration settings provided in the diffusers library. In the case of FLUX, we use the guidance-distilled version. \n\nThe below table reports the results on the Ring-A-Bell (violence) benchmark for safe generation metrics, and COCO dataset for general generation quality. Note that for this table, the COCO evaluation is conducted using 1,000 images, which accounts for the scale difference in FID values.\n\n**[Table A1: Results on recent diffusion models: FLUX, SDXL, and SD3]**\n\n| Models | Methods                 | Ring-A-Bell DSR (↑) | Ring-A-Bell PP (↑) | COCO FID-1k (↓) | COCO CLIP (↑) |\n|--------|-------------------------|---------------------|--------------------|--------------|---------------|\n| FLUX   | Base                    | 0.1120              | -                  | 56.58             |    0.3267          |\n|        | STG (ours, $\\tau$=0.20) | 0.2800              | 0.8527             |  56.59        |    0.3267           |\n|        | STG (ours, $\\tau$=0.18) | 0.5280              | 0.7031             |     56.52    | 0.3260              |\n|        | STG (ours, $\\tau$=0.16) | 0.6960              | 0.5997             |       57.77       |   0.3200            |\n||\n| SDXL   | Base                    | 0.0360               | -                  | 48.97             |      0.3380         |\n|        | STG (ours, $\\tau$=0.20) |      0.2520    |   0.8788         |   49.24        |   0.3378            |\n|        | STG (ours, $\\tau$=0.18) |      0.5040       |      0.7991    |   49.11      | 0.3366              |\n|        | STG (ours, $\\tau$=0.16) |      0.7720         |    0.7191         |  49.44     | 0.3302              |\n||\n| SD3    | Base                    | 0.1160              | -                  | 53.70        | 0.3344       |\n|        | STG (ours, $\\tau$=0.20) | 0.4200              | 0.7562             |    53.89   |0.3342               |\n|        | STG (ours, $\\tau$=0.18) | 0.5400              | 0.6652             |      53.65        |       0.3333        |\n|        | STG (ours, $\\tau$=0.16) | 0.6760              | 0.5678             |   54.91           |      0.3293         |\n\nAs shown in the table, the Ring-A-Bell prompts still induce harmful outputs even in the recent models, as indicated by the low DSR values for the base models. Applying STG improves safety (higher DSR), while maintaining similar image quality as measured by FID and CLIP score. These results demonstrate that STG generalizes well across diverse backbones, pipelines, and fast generation methods. Additionally, the trade-off between image quality and safety can be adjusted via the update threshold hyperparameter $\\tau$, providing flexibility depending on application requirements.\n\n***\n\n> ### **W2 & Q5: [Increased sampling time]**\n\nWe appreciate the reviewer’s observation regarding the increased sampling time. As discussed in Appendix C.3 and D, the added inference cost of STG mainly stems from gradient computations required for updating the text embeddings. To mitigate this overhead, we explore time- and memory-efficient techniques. Specifically, we apply half-precision (FP16) inference during sampling, which significantly reduces both runtime and GPU memory usage, while preserving the safety performance of our method, as shown in the below table. We extended our runtime and memory usage analysis (originally presented in Table 7 of the appendix) by including results with half-precision (FP16) inference. Although STG requires gradient computation during update steps, we observe that FP16 inference maintains nearly the same safety performance. This suggests that existing time- and memory-efficient techniques can be effectively leveraged to address the computational cost of STG in practice.\n\n**[Table A2: Sampling time and memory usage with FP16]**\n\n| Method                 |  FP16   | Time (s/batch) | GPU memory (GB) | DSR (↑)  | PP (↑)   |\n|----------------------------|----|----------------|-----------------|------|------|\n| Base                        |  X  | 15.8           | 8.23            | 0.08 | -    |\n| SLD                         |  X  | 22.7           | 13.5            | 0.76 | 0.65 |\n| SAFREE                      | X  | 23.2           | 13.6            | 0.36 | 0.73 |\n| STG (ours)                   |   |                |                 |      |      |\n|   $\\rho=2.0$, 15% update      | X  | 23.9           | 45.4            | 0.79 | 0.90 |\n|   $\\rho=2.0$, 15% update  | O | 14.0           | 22.8            | 0.79 | 0.89 |\n|   $\\rho=2.0$, 40% update      | X | 30.4           | 45.4            | 0.88 | 0.84 |\n|   $\\rho=2.0$, 40% update  | O | 20.7           | 22.8            | 0.92 | 0.84 |\n|   $\\rho=0.5$, 80% update      |  X | 50.8           | 45.4            | 0.92 | 0.84 |\n|   $\\rho=0.5$, 80% update  | O | 35.0           | 22.8            | 0.91 | 0.84 |\n\nAdditionally, we would like to highlight an observation from Table 4 in the main text. By adjusting the update step ratio, under comparable sampling time, STG attains a similar defense success rate to SLD while achieving substantially higher safety performance than SAFREE. Also, it does so with better prior preservation than both methods. This demonstrates that STG offers flexible trade-offs between sampling speed, safety, and fidelity, depending on the desired deployment scenario.\n\n***\n> ### **Q1: [Different sampler]**\n\nThank you for the thoughtful question. To evaluate how sensitive STG is to the choice of sampler, we conduct additional experiments using the Stable Diffusion v1.4 backbone with the DDPM sampler, keeping all settings identical to those used with the DDIM sampler (as in Figure 3a for the Ring-A-Bell nudity task). Due to the rebuttal policy, we are only able to provide tabular results at this stage. For methods originally visualized as curves, we report three representative points based on DSR (the lowest, median, the highest values). We will include the full curves in the revised version of the paper.\n\n**[Table A3: Results with DDPM samplers]**\n\n|                | Methods         | DSR (↑) | PP (↑) |\n|----------------|-----------------|---------|--------|\n|                | Base            | 0.5260  | -      |\n| Training-based | ESD             | 0.4421  | 0.6431 |\n|                | SPM             | 0.6632  | 0.5290 |\n|                | DUO (min)       | 0.5895  | 0.7817 |\n|                | DUO (medium)    | 0.9474  | 0.6844 |\n|                | DUO (max)       | 0.9789  | 0.4701 |\n| Training-free  | Negative Prompt | 0.2842  | 0.6808 |\n|                | UCE             | 0.6105  | 0.6101 |\n|                | RECE            | 0.8421  | 0.5209 |\n|                | SLD-Medium      | 0.2211  | 0.7773 |\n|                | SLD-Strong      | 0.7684  | 0.6639 |\n|                | SLD-Maximum     | 0.9053  | 0.4980 |\n|                | SAFREE          | 0.3053  | 0.6774 |\n|                | SDG (min)       | 0.1158  | 0.7979 |\n|                | SDG (medium)    | 0.4526  | 0.7611 |\n|                | SDG (max)       | 0.9368  | 0.6143 |\n|                | STG (min)       | 0.8632  | 0.7508 |\n|                | STG (medium)    | 0.9789  | 0.7210 |\n|                | STG (max)       | 1.0000  | 0.5022 |\n\nThe results show that STG consistently delivers better performance across different samplers, indicating that STG is robust to the sampler.\n\n***\n> ### **Q2: [Compatible with fast generation methods]**\n\nSTG can be integrated with various fast generation methods. For example, in Table 6 of Appendix C.2, we demonstrate successful application of STG with the DPM-Solver sampler, which is a representative fast generation method of the sampler side, on the PixArt-alpha backbone.\n\nOur method only requires access to the mean predicted images $\\bar{x}_0$ at intermediate diffusion timesteps during sampling. Since most diffusion models and samplers, including LCMs, provide this, STG can generally be applied without architectural modifications.\n\nWe also conduct experiments with Latent Consistency Models on Ring-A-Bell (violence). We find that STG is compatible with this framework as well.\n\n**[Table A4: Results on the fast generation method: LCMs]**\n\n| Methods                 | Ring-A-Bell DSR (↑) | Ring-A-Bell PP (↑) | COCO FID-3K (↓) | COCO CLIP (↑) |\n|-------------------------|---------------------|--------------------|--------------|---------------|\n| Base                    | 0.0200              | -                  | 36.20        | 0.3019        |\n| STG (ours, $\\tau$=0.20) | 0.2280              | 0.6576             | 36.22        | 0.3013        |\n| STG (ours, $\\tau$=0.18) | 0.5160              | 0.5867             | 36.28        | 0.3006        |\n| STG (ours, $\\tau$=0.16) | 0.8040              | 0.4806             | 37.66        | 0.2923        |\n\n***\n\n> ### **Q4: [Correction of directional error in table]**\n\nThank you for pointing that out. The arrow direction in the PP column should be an upward arrow ($\\uparrow$). PP (Prior Preservation) is a metric where higher values are better. We will make this correction in the revision."}, {"review_id": "86qdTDnqU5", "reviewer": "Reviewer_QgQT", "summary": "To address the problem of unsafe and undesired generations of T2I models, the paper proposes a training-free guidance approach that steers a pre-trained model using a signal from an external safety classifier. Specifically, the method builds on the Universal Guidance framework and defines a time-dependent safety function using classifier feedback over intermediate denoised images. The core idea is to adjust the text embedding at each diffusion step to nudge the generation toward safer outputs. The authors evaluate their approach on three tasks, namely nudity, violence, and artist-style removal on Stable Diffusion v1.4, and compare against both training-free and training-based safety methods.", "questions": "- In Figure 3, several methods (e.g., SLD, SAFREE, STG) are represented by multiple data points. Could you clarify what these points correspond to? Do they reflect different hyperparameter settings (τ, ρ), in case of STG, or variants in case of SLD (e.g., SLD-M, SLD-S, SLD-Max)?\n- Can you explain the gap in reported values for SAFREE between your results and the original paper (e.g., ASR of 0.114 vs. DSR of 0.36)? \n- For the violence detection task, could you elaborate on why GPT-4o was chosen and whether its performance was validated in this specific context?", "limitations": "Limited discussion. Next to the computational overhead, compared to other approaches, the introduced methods rely on \"safety functions\" which may reduce flexibility.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed method is well-motivated and clearly introduced, offering a lightweight solution that does not require model fine-tuning.\n- Comprehensive comparisons to current state-of-the-art training-free (e.g., SLD, SAFREE) and training-based (e.g., ESD, DUO) methods are provided.\n- Ablations analyze the impact of key hyperparameters such as the step ratio (ρ) and threshold (τ).\n- Further, the authors present that their approach can be combined with training-based methods, resulting in further enhanced safety.\n\n**Weaknesses**\n\n- Unlike SLD or SAFREE, which make use of textual representations of undesired content to guide the safe generation, the proposed method requires an external classifier, reducing its generality and flexibility. Unfortunately, this dependency is not discussed as a limitation.\n\n- Overall, the limitations of the method, besides computational overhead, are not well addressed.\n\n- The experimental setup lacks clarity. While the authors state that the strength of guidance is controlled via the hyperparameters ρ and τ, the specific values used in the experiments and how they were selected are not detailed. Similarly, hyperparameter settings for baseline methods are not well described. For instance, it is only mentioned in the appendix that the SLD variant SLD-M is used, which is the weakest configuration the original paper reports. Further reported DSR values for SAFREE do not match the original reported values. Specifically, the original SAFREE paper reports an ASR (Attack Success Rate) of 0.114 on the Ring-A-Bell benchmark, while the present paper reports a DSR (1-ASR) of 0.36.\n\n- Unclear suitability of evaluation metrics: For nudity, the same classifier is used both to guide generation and to evaluate output safety, raising questions about metric bias. For violence, GPT-4o is used, but its reliability in detecting violent content is unclear and not further investigated.", "was_revised": true, "final_justification": "Overall, the rebuttal’s content and the promised revisions strengthen the paper; therefore, I lean towards acceptance. The clarifications on hyperparameter settings, baseline configurations, and dataset differences address my main concerns about the experimental setup. The added evaluations with alternative metrics are particularly helpful and reveal the classifier dependency as an important limitation (e.g., NudeNet/Falconsai). However, this limitation, alongside the computational overhead, restricts the applicability of the approach in broader or real-time settings. Nonetheless, the insights and framework proposed here could serve as a valuable foundation for future work on improving training-free safety guidance methods.", "rebuttal": "We greatly appreciate the reviewers' insightful comments and provide our responses below.\n***\n> ### **W1 & W2: [Requirement of external classifier]**\n\nAs the reviewer pointed out, Our method relies on an external classifier for the safety function. This is feasible for tasks with public classifiers (e.g., NudeNet for nudity) but limits generality where none exist. However, recent advances in pre-trained vision-language models, such as CLIP, provide a promising alternative. These models enable flexible zero-shot classification and can be leveraged to construct effective proxy safety functions. In our experiments, the violence and artist-style removal tasks utilized CLIP, even though CLIP was not specifically designed for those safety objectives.\n\nThis explanation of the safety function is briefly mentioned in Section 4.1 (lines 129-131), but we agree that the reliance on external classifiers and implications for generality should be explicitly stated as a limitation. We will revise the manuscript accordingly to include this discussion.\n\nAdditionally, as discussed in Section 4.3 and 4.4, our theoretical analysis and the toy experiments demonstrate that STG could be effective when the safety function is not a perfect estimator of the true safety probability. Unlike data-space guidance methods, STG jointly considers the model likelihood and safety objective. This allows STG to better preserve the original generative capabilities while improving safety, which supports the practical use of proxy classifiers (e.g., CLIP scores) within our framework.\n\n***\n> ### **W3 & Q2: [Experimental setup]**\n\n**(Hyperparameter settings)**\n\nWe provided the specific hyperparameter values of STG in Appendix B.1 (lines 717-735). As illustrated in our results (e.g., Figure 3 in the main text), there is a trade-off between the defense success rate and prior preservation. To reflect this, we report multiple variants across a range of hyperparameter values to illustrate this balance.\n\nWe set $\\gamma$ based on sampling time constraints, as it directly determines the number of update steps during sampling. We usually set to $\\gamma=0.6$. The threshold $\\tau$ defines which samples are considered unsafe (and thus require updates), and is inherently tied to the range and semantics of the safety function $g$. For example, in the artist-style removal task, where $g$ is defined as the CLIP score difference between ‘art’ and a target artist, the decision boundary is expected near zero, so $\\tau$ is chosen close to zero accordingly. The update scale $\\rho$ controls the trade-off between safety and prior preservation, and can be adjusted flexibly at inference time depending on the desired balance. Since our method is training-free and modular, $\\rho$ can be easily adapted for different use cases or safety definitions without any retraining overhead.\n\nFor baseline methods, their respective experimental configurations are provided in Appendix B.2 (lines 737-776), where we describe the setup for each method used in our comparisons.\n\n**(SLD variants)**\n\nWe clarify that the main results (Figure 3 in the main text) include all three SLD variants: Max, Strong, Medium. These are represented by the green solid line, ordered from left to right as Max, Strong, Medium.\nWhile SLD-Medium yields the lowest safety performance among the three configurations, it achieves the best performance in terms of preserving the original generative capabilities, particularly in terms of prior preservation and general generation metrics such as FID and CLIP score on COCO. For this reason, we chose SLD-Medium as the representative configuration in Table 2, which focuses on general generation quality. We also report the results for SLD-Strong and SLD-Max in the below table:\n\n**[Table B1: SLD-Strong and SLD-Max results for Table 2 in the main paper (COCO evaluation)]**\n|Method|FID(↓)|CLIP(↑)|\n|---|---|---|\n|Base|23.22|31.96|\n|...|\n|SLD-Medium|24.32|31.29|\n|SLD-Strong|25.10|30.33|\n|SLD-Max|27.69|29.28|\n|...|\n|STG (ours)|22.00|31.14|\n\n**[Table B2: SLD-Strong and SLD-Max results for Table 3 in the main paper (artist-style removal)]**\n|Method|Remove|\"Van| Gogh\"||*|Remove|\"Kelly| McKernan\"||\n|---|---|---|---|---|---|---|---|---|---|\n||LPIPS_e↑|LPIPS_u↓|ACC_e↓|ACC_u↑|*| LPIPS_e↑|LPIPS_u↓|ACC_e↓|ACC_u↑|\n|||\n|Base|||1.00|0.89|*|||0.90|0.71|\n|...|||||*|||||\n|SLD-Medium|0.28|0.12|0.60|0.81|*|0.22|0.18|0.50|0.74|\n|SLD-Strong|0.39|0.19|0.40|0.79|*|0.32|0.28|0.30|0.68|\n|SLD-Max|0.50|0.34|0.65|0.58|*|0.49|0.40|0.10|0.59|\n|...|||||*|||||\n|STG (ours)|0.46|0.08|0.30|0.85|*|0.58|0.10|0.10|0.65|\n\nAs expected, FID and CLIP score on the COCO deteriorate for SLD-Strong and SLD-Max. For the artist-style removal task, SLD-Max improves removal on the erased concepts but unintentionally affects unerased concepts.\n\n**(SAFREE reported values)**\n\nThe discrepancy in reported DSR for SAFREE originates from different Ring-A-Bell benchmark versions. As noted in the second issue thread on SAFREE’s official Github repository, SAFREE used a custom set from RECE authors, which we used the official adversarial prompts (later released by the Ring-A-Bell authors) following DUO, as mentioned in Appendix B.1 (lines 687-688).\nA comparison of the two datasets reveals significant differences. The version used in SAFREE consists of human-readable prompts, while the official version we used contains highly adversarial prompts that are much harder to interpret. This distinction explains the observed discrepancy in performance.\n\nWe also evaluate STG on SAFREE’s version and reproduce their results; STG still shows superior performance. We will revise the manuscript to clearly specify datasets and configurations.\n\n**[Table B3: Results on Ring-A-Bell (nudity) used in SAFREE]**\n|Method|ASR (↓)|PP (↑)|\n|---|---|---|\n|SAFREE (reproduced)|0.1013|0.5367|\n|STG (ours)|**0.0886**|**0.5820**|\n\n***\n> ### **W4 & Q3: [Evaluation metric]**\n\nWe appreciate the reviewer’s concern regarding potential metric bias in our evaluation protocol.\n\nFor the nudity task, we acknowledge that using the same classifier (NudeNet) for both generation guidance and evaluation may introduce bias. To address this, we additionally evaluate the generated images using an alternative metric: an open-source ViT-based NSFW image classifier provided by Falcons.ai. We present a comparison between the original DSR values obtained using the NudeNet detector and those computed using the Falconsai classifier for each point in Figure 3a. While there are some mismatches and variations between the two metrics, we observe that STG consistently achieved higher DSR at similar levels of Prior Preservation (PP). Furthermore, as shown in Table 2 of the main paper, STG also maintains general generation quality.\n\n**[Table B4: DSR results using Falconsai classifier]**\n||Methods|PP (↑)|DSR-NudeNet (↑)|DSR-Falconsai (↑)|\n|---|---|---|---|---|\n||Base|-|0.0842|0.3263|\n|Training-based|ESD|0.6509|0.6000|0.7684|\n||SPM|0.5389|0.7053|0.8947|\n||DUO (min)|0.8958|0.4842|0.3368|\n||DUO (medium)|0.7210|0.9684|0.6526|\n||DUO (max)|0.4747|0.9895|0.9579|\n|Training-free|Negative Prompt|0.7323|0.3579|0.6842|\n||UCE|0.6418|0.6842|0.7895|\n||RECE|0.5400|0.9263|0.9684|\n||SLD-Medium|0.8708|0.2526|0.6632|\n||SLD-Strong|0.6515|0.7579|0.9579|\n||SLD-Maximum|0.4829|0.9474|1.000|\n||SAFREE|0.7242|0.3579|0.3263|\n||SDG (min)|0.9140|0.4737|0.3789|\n||SDG (medium)|0.7574|0.7158|0.5789|\n||SDG (max)|0.6055|1.0000|0.9684|\n||STG (min)|0.9238|0.6632|0.6316|\n||STG (medium)|0.8446|0.9158|0.7368|\n||STG (max)|0.6280|0.9895|0.9368|\n\nRegarding violence, we agree with the reviewer’s concern about the reliability of GPT-4o as an evaluation tool. In our work, we followed the evaluation protocol introduced by DUO, which also used GPT-4o for safety assessment for violence.\n\nTo further validate this metric, we additionally evaluate our results using the Q16 classifier, an open-source model commonly used for detecting harmful content. Specifically, we collect pairs of safety assessment scores from both GPT-4o and the Q16 classifier across various model and hyperparameter configurations. Then, we compute the Pearson correlation between these scores, which resulted in a value of 0.943. This strong linear correlation suggests that GPT-4o provides safety evaluations that are highly consistent with those of an established classifier like Q16.\n\nWe will include the Q16-based evaluation curves and this discussion in the revised version.\n\n***\n> ### **Q1: [Configurations in Figure 3]**\n\nAs mentioned in our response to W3, the SLD results in Figure 3 are represented by the green solid line, with the three variants, SLD-Max, Strong, and Medium, ordered from left to right.\n\nFor STG and SDG, the multiple points correspond to different hyperparameter settings, as detailed in Appendix B.1 (lines 726-732) for STG and Appendix B.2 (lines 758-762) for SDG.\nSpecifically, for STG on the nudity task, the points represent different (ρ, τ) combinations: {(1.8, 0.01), (1.3, 0.01), (0.5, 0.01), (0.5, 0.03), (0.5, 0.2)}, plotted from left to right in Figure 3. For the violence task, τ is fixed at 0.05, and ρ is varied across {3, 2, 1, 0.5, 0.2, 0.1}, also in left-to-right order. For SDG, in the nudity task, τ is fixed at 0.01 and ρ is varied as {5, 1, 0.7, 0.5, 0.1}; for the violence task, τ is fixed at 0.1 and ρ is varied as {40, 15, 10, 5, 1}. Again, in both cases, the points correspond to the values listed from left to right.\n\nFor DUO, the blue line in Figure 3 corresponds to different values of the hyperparameter β, varied as {100, 250, 500, 1000, 2000}, in left-to-right order. These settings are described in Appendix B.2 (lines 768-770).\n\nNote that SAFREE is shown as a single point in Figure 3. For this method, we adopted the default hyperparameter configuration provided in the official SAFREE code, as noted in Appendix B.2 (lines 742-744).\n\nWe will explicitly include these details in the revised version of the paper to ensure clarity and avoid confusion for readers."}, {"review_id": "4EbwbK8qGI", "reviewer": "Reviewer_ypdn", "summary": "The paper proposes a training-free method for text-to-image diffusion models and introduces safe text embedding guidance (STG) to dynamically adjust the text embedding during sampling to suppress unsafe content. At each diffusion step, STG computes the safety score on the expected denoised image and uses the first-order Taylor approximation to show the equivalent to injecting a “safe guidance” term into the score function. Extensive experiments demonstrate that STG consistently outperforms both training-based and training-free baselines.", "questions": "1. What about the safety performance on more categories?\n2. What about the generalization on other models?", "limitations": "See the above weakness.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Zero-training operates entirely at inference time, avoiding costly retraining or curated datasets. \n2. Theoretical derivation via Tweedie’s formula and explicit decomposition into original score + safe guidance provides clear interpretability.\n\nWeakness:\n1. The method effectiveness depends on the accuracy and bias of the external classifier.\n2. The gradient computations and safety evaluations at each sampling step reduces the efficiency.\n3. What about the generalization on other models?", "was_revised": true, "final_justification": "My recommended score is 4, Borderline accept", "rebuttal": "We thank the reviewer for the helpful feedback. Our responses to the raised concerns are summarized below.\n***\n\n> ### **W1: [Reliance of the external classifier]**\n\nWe agree with the reviewer’s concerns, and this limitation is explicitly acknowledged in Appendix D of the paper. However, as discussed in Sections 4.3 and 4.4, our theoretical analysis and the toy experiments demonstrate that STG could be effective when the safety function is not a perfect estimator of the true safety probability. Unlike data-space guidance methods, STG jointly considers the model likelihood and safety objective, which helps mitigate the risk of overfitting to potentially biased or imperfect proxy metrics.\n\nRecent advances in pre-trained vision-language models, such as CLIP, provide a promising alternative. These models enable flexible zero-shot classification and can be leveraged to construct effective proxy safety functions. In our experiments, the violence and artist-style removal tasks utilized CLIP, even though CLIP was not specifically designed for those safety objectives.\n\nTogether, our theoretical and empirical findings suggest that STG could be robust to the safety functions and can effectively improve safety while preserving the model’s original generative capabilities. This supports the practical use of proxy classifiers, such as CLIP-based scores, within our framework.\n\n\n***\n\n> ### **W2: [Reduced efficiency]**\n\nWe appreciate the reviewer’s observation regarding the efficiency. As discussed in Appendix C.3 and D, the added inference cost of STG mainly stems from gradient computations required for updating the text embedding. To mitigate this overhead, we explore time- and memory-efficient techniques. Specifically, we apply half-precision (FP16) inference during sampling, which significantly reduces both runtime and GPU memory usage, while preserving the safety performance of our method, as shown in the below table. We extended our runtime and memory usage analysis (originally presented in Table 7 of the appendix) by including results with half-precision (FP16) inference. Although STG requires gradient computation during update steps, we observe that FP16 inference maintains nearly the same safety performance. This suggests that existing time- and memory-efficient techniques can be effectively leveraged to address the computational cost of STG in practice.\n\n**[Table C1: Sampling time and memory usage with FP16]**\n\n|Method|FP16|Time (s/batch)|GPU memory (GB)|DSR (↑)|PP (↑)|\n|----------------------------|----|----------------|-----------------|------|------|\n|Base|X|15.8|8.23|0.08|-|\n|SLD|X|22.7|13.5|0.76|0.65|\n|SAFREE|X|23.2|13.6|0.36|0.73|\n|STG (ours)||||||\n|$\\rho=2.0$, 15% update|X|23.9|45.4|0.79|0.90|\n|$\\rho=2.0$, 15% update|O|14.0|22.8|0.79|0.89|\n|$\\rho=2.0$, 40% update|X|30.4|45.4|0.88|0.84|\n|$\\rho=2.0$, 40% update|O|20.7|22.8|0.92|0.84|\n|$\\rho=0.5$, 80% update|X|50.8|45.4|0.92|0.84|\n|$\\rho=0.5$, 80% update|O|35.0|22.8|0.91|0.84|\n\nAdditionally, we would like to highlight an observation from Table 4 in the main text. By adjusting the update step ratio, under comparable sampling time, STG attains a similar defense success rate to SLD while achieving substantially higher safety performance than SAFREE. Also, it does so with better prior preservation than both methods. This demonstrates that STG offers flexible trade-offs between sampling speed, safety, and fidelity, depending on the desired deployment scenario.\n\n***\n\n> ### **W3 & Q2: [Generalization]**\n\nTo evaluate the generalization ability of STG, we conducted experiments using the PixArt-alpha model, a recent diffusion model with a Transformer-based backbone, in combination with the DPM-Solver sampler. These results are reported in Table 6 and discussed in Appendix C.2. \n\nAdditionally, to further address concerns about scalability, we extend our experiments to include more recent diffusion models such as FLUX, SDXL, and SD3. For each model, we follow the default sampling and configuration settings provided in the diffusers library. In the case of FLUX, we use the guidance-distilled version. \n\nThe below table reports the results on the Ring-A-Bell (violence) benchmark for safe generation metrics, and COCO dataset for general generation quality. Note that for this table, the COCO evaluation is conducted using 1,000 images, which accounts for the scale difference in FID values.\n\n**[Table C2: Results on recent diffusion models: FLUX, SDXL, and SD3]**\n\n|Models|Methods|Ring-A-Bell DSR (↑)|Ring-A-Bell PP (↑)|COCO FID-1k (↓)|COCO CLIP (↑)|\n|-----|-------------------------|---------------------|--------------------|--------------|---------------|\n|FLUX|Base|0.1120|-|56.58|0.3267|\n||STG (ours, $\\tau$=0.20)|0.2800|0.8527|56.59|0.3267|\n||STG (ours, $\\tau$=0.18)|0.5280|0.7031|56.52|0.3260|\n||STG (ours, $\\tau$=0.16)|0.6960|0.5997|57.77|0.3200|\n|SDXL|Base|0.0360|-|48.97|0.3380|\n||STG (ours, $\\tau$=0.20)|0.2520|0.8788|49.24|0.3378|\n||STG (ours, $\\tau$=0.18)|0.5040|0.7991|49.11|0.3366|\n||STG (ours, $\\tau$=0.16)|0.7720|0.7191|49.44|0.3302|\n|SD3|Base|0.1160|-|53.70|0.3344|\n||STG (ours, $\\tau$=0.20)|0.4200|0.7562|53.89|0.3342|\n||STG (ours, $\\tau$=0.18)|0.5400|0.6652|53.65|0.3333|\n||STG (ours, $\\tau$=0.16)|0.6760|0.5678|54.91|0.3293|\n\n\nAs shown in the table, the Ring-A-Bell prompts still induce harmful outputs even in the recent models, as indicated by the low DSR values for the base models. Applying STG improves safety (higher DSR), while maintaining similar image quality as measured by FID and CLIP score. These results demonstrate that STG generalizes well across diverse backbones, pipelines, and fast generation methods. Additionally, the trade-off between image quality and safety can be adjusted via the update threshold hyperparameter $\\tau$, providing flexibility depending on application requirements.\n\nWe also investigate sensitivity of STG to different samplers by conducting experiments on the Stable Diffusion v1.4 backbone using the DDPM sampler, keeping all settings identical to those used with the DDIM sampler (as in Figure 3a for the Ring-A-Bell nudity task). Due to the rebuttal policy, we are only able to provide tabular results at this stage. For methods originally visualized as curves, we report three representative points based on DSR (the lowest, median, the highest values). We will include the full curves in the revised version of the paper.\n\n**[Table C3: Results with DDPM samplers]**\n\n|                | Methods         | DSR (↑) | PP (↑) |\n|----------------|-----------------|---------|--------|\n|                | Base            | 0.5260  | -      |\n| Training-based | ESD             | 0.4421  | 0.6431 |\n|                | SPM             | 0.6632  | 0.5290 |\n|                | DUO (min)       | 0.5895  | 0.7817 |\n|                | DUO (medium)    | 0.9474  | 0.6844 |\n|                | DUO (max)       | 0.9789  | 0.4701 |\n| Training-free  | Negative Prompt | 0.2842  | 0.6808 |\n|                | UCE             | 0.6105  | 0.6101 |\n|                | RECE            | 0.8421  | 0.5209 |\n|                | SLD-Medium      | 0.2211  | 0.7773 |\n|                | SLD-Strong      | 0.7684  | 0.6639 |\n|                | SLD-Maximum     | 0.9053  | 0.4980 |\n|                | SAFREE          | 0.3053  | 0.6774 |\n|                | SDG (min)       | 0.1158  | 0.7979 |\n|                | SDG (medium)    | 0.4526  | 0.7611 |\n|                | SDG (max)       | 0.9368  | 0.6143 |\n|                | STG (min)       | 0.8632  | 0.7508 |\n|                | STG (medium)    | 0.9789  | 0.7210 |\n|                | STG (max)       | 1.0000  | 0.5022 |\n\nThe results show that STG consistently delivers better performance across different samplers, indicating that STG is robust to the sampler.\n\nAcross these diverse backbones and samplers, STG consistently demonstrates strong performance, highlighting its generality and robustness across a wide range of model architectures and sampling strategies.\n\n***\n\n> ### **Q1: [More categories]**\n\nWe understand the reviewer’s question as referring to safety performance across a broader range of inappropriate categories (e.g., hate, harassment, etc.). To investigate this, we conduct experiments on the I2P benchmark, which includes prompts from seven categories used in the SLD paper: hate, harassment, violence, self-harm, sexual, shocking, and illegal activity.\n\nDue to the large number of prompts in I2P benchmark, we adopt a fast diffusion model, specifically the Latent Consistency Model (LCM) (Luo et al., 2023) with 4 sampling steps to enable efficient sampling. For the safety function $g$, we use the CLIP score between each image and the fixed general inappropriate text prompt, as defined in the SLD paper. This unified safety function allows STG to be applied consistently across all seven categories. As commonly done in the I2P dataset, we compute the DSR using the combined Q16/NudeNet classifier and report the experimental results.\n\n**[Table C4: Safety performance (DSR) across diverse categories on the I2P benchmark using LCM + STG]**\n|         | hate   | harassment | violence | self-harm | sexual | shocking | illegal activity | * | COCO FID | COCO CLIP |\n|-------------------|--------|------------|----------|-----------|--------|----------|------------------|---|----------|-----------|\n| Fixed             | 0.5628 | 0.6663     | 0.6151   | 0.6017    | 0.7368 | 0.5210   | 0.6191           | * | 36.20    | 0.3019    |\n| STG ($\\rho$=0.16) | 0.6494 | 0.7318     | 0.7037   | 0.6742    | 0.7744 | 0.6192   | 0.6862           | * | 37.66    | 0.2923    |\n| STG ($\\rho$=0.15) | 0.7403 | 0.8058     | 0.7553   | 0.7141    | 0.8174 | 0.6951   | 0.7261           | * | 41.21    | 0.2779    |\n\nAs shown in the table, STG demonstrates strong safety performance even when applied to modern diffusion models and across diverse safety categories.\n\n(Luo et al., 2023) Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference"}, {"review_id": "JFCHjOKM1W", "reviewer": "Reviewer_5G8U", "summary": "This paper introduces Safe Text Embedding guidance (STG) as a safeguard mechanism against image outputs that involve undesired or inappropriate content, that is subjective to the user. The proposed method extends Safe Data Guidance (SDG), that targets the diffusion model weights in its optimization objective by only targeting the text embedding that is going to be fed to the model. Given a guidance function $g$ that assesses the image outputs $x_0$ based the desired image behavior, STG applies the optimization objective that SDG defines as updates to the text embedding representation that guides the generation. Addressing the fact that the updates to the diffusion model can result in disturbed model behavior, STG only targets the embeddings and claim to preserve the generation properties. Experiments on concepts such as nudity, violence demonstrate the effectiveness of the method across competing approaches.", "questions": "- Does the method generalize to generative models that uses multiple text encoders? This is an important concern as it determines the types of generative models that the proposed approach can be applied to.\n- Does the proposed method be applied to tasks such as bias mitigation? In addition to unpleasant concepts like violent content and nudity, biased representations such as gender imbalances across occupations is also a problem that safety methods address (like UCE). It is an important clarification that if STG can be extended to such tasks.", "limitations": "- The paper does not have an explicit discussion on the limitations. However, since the work explicitly addresses the embedding space rather than the overall model, a limitations section on the concepts that can be mitigated and which cannot be mitigated would benefit the paper, to determine its applicability. The overall societal impact of the paper is positive, where discussions on the broader impact is provided in the supplementary material.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper proposes an extension of SDG that promises improved preservation properties of the properties of the model.\n- The proposed method demonstrates its effectiveness in preserving model behavior while preventing the generation of the undesired concepts (e.g. nudity)\n- The paper is well experimented, where the motivation is clear as well as the quantitative improvements.\n- Technical details are sufficiently explained, where the claims are coupled with proofs.\n\n**Weaknesses**\n- The GPU consumption of the method is significant which limits the applicability of the method to larger models.\n- The method addresses the source of undesired concepts to be determined in the embedding space. This narrows the applicability of the method to tasks such as bias removal, that is also related with the cross-attention and self-attention weights.\n-  The proposed method serves as an incremental update on top of SDG. While it is significant that the model properties are preserved better, as only the embedding space is targeted, the technical novelty of the work is questionable and can be considered as an incremental update on top of the previous methods.\n- The method requires updates on the text embeddings fed into the generative model. Does the method generalize to approaches that uses more than one text embeddings for generation, as such approaches are more frequently used (FLUX, SDXL, SD3)?", "was_revised": true, "final_justification": "The questions that I had towards the computational efficiency, generalizability to newer models (compared to Stable Diffusion), application to bias mitigation have been addressed by the supplementary experiments provided as a part of the rebuttal. Furthermore, I  adjusted my score as leaning towards positive.", "rebuttal": "We appreciate the reviewers’ constructive and encouraging comments. We have carefully considered all suggestions and provide our responses below.\n***\n> ### **W1: [GPU consumption]**\n\nWe appreciate the reviewer’s observation regarding GPU consumption. As discussed in Appendix C.3 and D, the added inference cost of STG mainly stems from gradient computations required for updating the text embedding. To mitigate this overhead, we explore time- and memory-efficient techniques. Specifically, we apply half-precision (FP16) inference during sampling, which significantly reduces both runtime and GPU memory usage, while preserving the safety performance of our method, as shown in the below table. We extended our runtime and memory usage analysis (originally presented in Table 7 of the appendix) by including results with half-precision (FP16) inference. Although STG requires gradient computation during update steps, we observe that FP16 inference maintains nearly the same safety performance. This suggests that existing time- and memory-efficient techniques can be effectively leveraged to address the computational cost of STG in practice.\n\n**[Table D1: Sampling time and memory usage with FP16]**\n\n|Method|FP16|Time (s/batch)|GPU memory (GB)|DSR (↑)|PP (↑)|\n|----------------------------|----|----------------|-----------------|------|------|\n|Base|X|15.8|8.23|0.08|-|\n|SLD|X|22.7|13.5|0.76|0.65|\n|SAFREE|X|23.2|13.6|0.36|0.73|\n|STG (ours)||||||\n|$\\rho=2.0$, 15% update|X|23.9|45.4|0.79|0.90|\n|$\\rho=2.0$, 15% update|O|14.0|22.8|0.79|0.89|\n|$\\rho=2.0$, 40% update|X|30.4|45.4|0.88|0.84|\n|$\\rho=2.0$, 40% update|O|20.7|22.8|0.92|0.84|\n|$\\rho=0.5$, 80% update|X|50.8|45.4|0.92|0.84|\n|$\\rho=0.5$, 80% update|O|35.0|22.8|0.91|0.84|\n\n***\n> ### **W2 & Q2: [Application of bias mitigation]**\n\nThank you for the suggestion. STG can be extended to bias mitigation tasks if a suitable safety function can be defined to measure and discourage biased behavior. To explore this, we conduct a preliminary experiment targeting gender imbalance across occupations. Specifically, we use the prompt format ‘a photo of {occupation}’ and analyze the gender distribution of generated images from the Stable Diffusion v1.4 backbone. We observe that prompts such as ‘nurse’ resulted in 100\\% female images, while ‘farmer’ yielded 98% male images, indicating strong gender bias in the base model.\n\nTo mitigate this, we define a safety function as the negative squared difference between CLIP scores for ‘a photo of male {occupation}’ and ‘a photo of female {occupation}’. This formulation encourages the generated image to avoid being overly aligned with either gender-specific prompt, thereby guiding it toward a more neutral representation. As shown in the below table, the degree of bias mitigation can be controlled by adjusting the update scale hyperparameter $\\rho$. The reported ratio corresponds to the proportion of male-presenting images, and values closer to 0.5 indicate that the generated images are achieving a more balanced gender distribution, demonstrating effective mitigation of occupational gender bias.\n\n**[Table D2: Gender distribution in generated images for occupation prompts]**\n\n|Occupation|Method|$\\rho$|# Man|# Woman|Ratio|\n|---|---|---|---|---|---|\n|Nurse|Base||0|250|0|\n||STG|0.5|43|207|0.172|\n|||1.0|58|192|0.232|\n|||1.5|117|133|0.468|\n||\n|Farmer|Base||246|4|0.984|\n||STG|0.5|171|79|0.684|\n|||1.0|167|83|0.668|\n|||1.5|152|98|0.608|\n\nThat being said, we note that STG operates on a per-instance basis and does not explicitly enforce balance over the distribution of generated samples. As such, it may produce images that appear more neutral, but this does not necessarily indicate fair distribution-level behavior. We view this as a potential direction for future work, particularly in integrating group-level fairness constraints into our framework.\n\n***\n> ### **W3: [Incremental update on SDG]**\n\nWhile the proposed method may appear to be an incremental modification of SDG, shifting the guidance target from the image latent space to the text embedding space; our work provides a thorough theoretical and empirical justification for this change. Specifically, Section 4.3 with Theorem 1 analytically shows that updating the text embedding implicitly influences the latent space while preserving the underlying model likelihood. Section 4.4 further supports this claim with toy experiments, demonstrating that STG more faithfully maintains the original generative distribution compared to SDG, especially when the safety function is imperfect. This distinction is particularly important for tasks such as style removal or copyright mitigation, where it is critical to preserve the semantic content or visual fidelity of the original image while modifying only specific attributes.\n\nIn addition, although SDG itself is derived from the previous Universal Guidance framework, our work is the first to adapt and implement both SDG and STG for the safety domain, along with extensive evaluations across safety tasks. We believe this comprehensive adaptation and evaluation further contributes to the field.\n\n***\n\n> ### **W4 & Q1: [Multiple text embeddings]**\n\nOur method is compatible with recent generative models that utilize multiple text embeddings, such as FLUX, SDXL, and SD3. In these models, STG can be applied by updating all text embeddings in the same way. We conduct additional experiments using more recent diffusion models, including FLUX, SDXL, and SD3. For each model, we follow the default sampling and configuration settings provided in the diffusers library. In the case of FLUX, we use the guidance-distilled version. \n\nThe below table reports the results on the Ring-A-Bell (violence) benchmark for safe generation metrics, and COCO dataset for general generation quality. Note that for this table, the COCO evaluation is conducted using 1,000 images, which accounts for the scale difference in FID values.\n\n**[Table D3: Results on recent diffusion models: FLUX, SDXL, and SD3]**\n\n|Models|Methods|Ring-A-Bell DSR (↑)|Ring-A-Bell PP (↑)|COCO FID-1k (↓)|COCO CLIP (↑)|\n|-----|-------------------------|---------------------|--------------------|--------------|---------------|\n|FLUX|Base|0.1120|-|56.58|0.3267|\n||STG (ours, $\\tau$=0.20)|0.2800|0.8527|56.59|0.3267|\n||STG (ours, $\\tau$=0.18)|0.5280|0.7031|56.52|0.3260|\n||STG (ours, $\\tau$=0.16)|0.6960|0.5997|57.77|0.3200|\n||\n|SDXL|Base|0.0360|-|48.97|0.3380|\n||STG (ours, $\\tau$=0.20)|0.2520|0.8788|49.24|0.3378|\n||STG (ours, $\\tau$=0.18)|0.5040|0.7991|49.11|0.3366|\n||STG (ours, $\\tau$=0.16)|0.7720|0.7191|49.44|0.3302|\n||\n|SD3|Base|0.1160|-|53.70|0.3344|\n||STG (ours, $\\tau$=0.20)|0.4200|0.7562|53.89|0.3342|\n||STG (ours, $\\tau$=0.18)|0.5400|0.6652|53.65|0.3333|\n||STG (ours, $\\tau$=0.16)|0.6760|0.5678|54.91|0.3293|\n\nAs shown in the table, the Ring-A-Bell prompts still induce harmful outputs even in the recent models, as indicated by the low DSR values for the base models. Applying STG improves safety (higher DSR), while maintaining similar image quality as measured by FID and CLIP score. These results demonstrate that STG generalizes well across diverse backbones, pipelines, and fast generation methods. Additionally, the trade-off between image quality and safety can be adjusted via the update threshold hyperparameter $\\tau$, providing flexibility depending on application requirements."}]

Author rebuttal:

Meta-review: **Summary:** This paper presents Safe Text embedding Guidance (STG), a training-free framework for improving safety in text-to-image diffusion by directly modifying text embedding guidance. STG employs a safety function to detect unsafe semantic directions and steer embeddings away from them during generation. Theoretical analysis shows that aligning perturbed data with both the model distribution and safety constraints within classifier-free guidance reduces unsafe outputs while preserving semantic alignment. Experiments on unsafe prompts demonstrate substantial safety improvements with minimal impact on image quality.

**Strengths:** Key strengths noted were: (1) the simplicity and practicality of STG; (2) the novelty of training-free embedding-level steering with a safety function; (3) the theoretical insight on how classifier-free guidance aligns embeddings with safety constraints; (4) comprehensive experiments with ablations, and (5) the strong balance between safety and fidelity.

**Weaknesses:** Initial concerns were: (1) unclear scalability and limited evaluation on newer diffusion models (e.g., SDXL, SD3, FLUX); (2) efficiency issues from added gradient computations, slower sampling, and high GPU use; (3) dependence on an external classifier, introducing potential bias and reduced generality; (4) unclear experimental setup with under-specified hyperparameters and questionable baseline choices; and (5) limited novelty, as STG may be seen as an incremental extension of SDG focused on embeddings.

During the rebuttal, the authors clarified design choices, addressed scalability issues by providing experiments with recent diffusion frameworks, provided justifications on increased sampling time and improved baseline comparisons. They emphasized STG’s practicality as a lightweight, training-free complement to more heavyweight alignment methods. While concerns about dependency on specific classifier remain, reviewers agreed that the contribution is novel, timely, and impactful.
Decision: Accept (poster)