SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation

## Abstract

Abstract Recent advances in diffusion models have significantly enhanced their ability to generate high-quality images and videos, but they have also increased the risk of producing unsafe content.
Existing unlearning/editing-based methods for safe generation remove harmful concepts from models but face several challenges: (1) They cannot instantly remove harmful or undesirable concepts (e.g., artist styles) without additional training.
(2) Their safe generation capabilities depend on collected training data.
(3) They alter model weights, risking degradation in quality for content unrelated to the targeted toxic concepts.
To address these challenges, we propose SAFREE, a novel, training-free approach for safe text-to-image and video generation, that does not alter the model’s weights.
Specifically, we detect a subspace corresponding to a set of toxic concepts in the text embedding space and steer prompt token embeddings away from this subspace, thereby filtering out harmful content while preserving intended semantics.
To balance the trade-off between filtering toxicity and preserving safe concepts, SAFREE incorporates a novel self-validating filtering mechanism that dynamically adjusts the denoising steps when applying the filtered embeddings.
Additionally, we incorporate adaptive re-attention mechanisms within the diffusion latent space to selectively diminish the influence of features related to toxic concepts at the pixel level.
By integrating filtering across both textual embedding and visual latent spaces, SAFREE ensures coherent safety checking, preserving the fidelity, quality, and safety of the generated outputs.
Empirically, SAFREE achieves state-of-the-art performance in suppressing unsafe content in T2I generation (reducing it by 22% across 5 datasets) compared to other training-free methods and effectively filters targeted concepts, e.g., specific artist styles, while maintaining high-quality output.
It also shows competitive results against training-based methods.
We further extend SAFREE to various T2I backbones and T2V tasks, showcasing its flexibility and generalization.
As generative AI rapidly evolves, SAFREE provides a robust and adaptable safeguard for ensuring safe visual generation.

## 1 Introduction

Recent advancements in Generative AI have significantly impacted various modalities, including text (Brown, ), code (Chen et al., ), audio (Kreuk et al., ; Copet et al., ), image (Podell et al., ; Ho et al., ), and video generation (Ho et al., ; Kondratyuk et al., ; Yoon et al., ; openai, ).
Generation tools such as DALL·E 3, Midjourney, Sora, and KLING have seen significant growth, enabling a wide range of applications in digital art, AR/VR, and educational content creation.
However, these tools/models also carry the risk of generating content with unsafe concepts such as bias, discrimination, sex, or violence.
Moreover, the definition of “unsafe content” varies according to societal perceptions. For example, individuals with Post-Traumatic Stress Disorder (PTSD) might find specific images (e.g., skyscrapers, deep-sea scenes) distressing. This underscores the need for an adaptable, flexible solution to enhance the safety of generative AI while considering individual sensitivities.

To tackle these challenges, recent research has incorporated safety mechanisms in diffusion models.
Unlearning methods (Zhang et al., ; Huang et al., ; Park et al., ; Wu et al., ) fine-tune models to remove harmful concepts, but they lack adaptability and are less practical due to the significant training resources they require. Model editing methods (Orgad et al., ; Gandikota et al., ; Xiong et al., ) modify model weights to enhance safety, but they often degrade output quality and make it challenging to maintain consistent model behavior.
A promising alternative is training-free, filtering-based methods that exclude unsafe concepts from input prompts without altering the model’s original capabilities.
However, prior training-free, filtering-based methods encounter two significant challenges: (1) they may not effectively guard against implicit or indirect triggers of unsafe content, as highlighted in earlier studies (Deng & Chen, ), and (2) our findings indicate that prompts subjected to hard filtering can result in distribution shifts, leading to quality degradation even without modifying the model weights. Thus, there is an urgent need for an efficient and adaptable mechanism to ensure safe visual generation across diverse contexts.

This paper presents SAFREE, a training-free, adaptive plug-and-play mechanism for any diffusion-based generative model to ensure safe generation without altering well-trained model weights.
SAFREE employs unsafe concept filtering in both textual prompt embedding and visual latent space, thereby enhancing the fidelity, quality, and efficiency for safe visual content generation. Specifically, SAFREE first identifies the unsafe concept subspace, i.e., the subspace within the input text embedding space that corresponds to undesirable concepts, by concatenating the column vectors of unsafe keywords.
Then, to measure the proximity of each input prompt token to the unsafe/toxic subspace, we mask each token in the prompt and calculate the projected distance of the masked prompt embedding to the subspace. Given the proximity, SAFREE aims to filter out tokens that drive the prompt embedding closer to the unsafe subspace. Rather than directly removing or replacing unsafe tokens—which can compromise the coherence of the input prompt and degrade generation quality—SAFREE efficiently projects the identified unsafe tokens into a space orthogonal to the unsafe concept subspace while maintaining them on the original input embedding space.
Such orthogonal projection design aims to preserve the overall integrity and the safe content within the original prompt while filtering out harmful content in the embedding space.
To balance the trade-off between filtering out toxicity and preserving safe concepts (examples shown in Fig. 1), SAFREE incorporates a novel self-validating filtering scheme, which dynamically adjusts the number of denoising steps for applying filtered embeddings, enhancing the suppression of undesirable prompts when needed.
Additionally, since we observe that unsafe content usually emerges at the regional pixel level, SAFREE extends filtering to the pixel space using a novel adaptive latent re-attention mechanism within the diffusion latent space.
It selectively reduces the influence of features tied to the detected unsafe prompt, ensuring desirable outputs without drawing attention to those content.
In the end, SAFREE filters out unsafe content simultaneously in both the embedding and pixel spaces.
This approach ensures flexible and adaptive safe T2I/T2V generation that efficiently handles a broad range of unsafe concepts without extra training or modifications to model weights,
while preserving the quality of safe outputs.

Empirically, SAFREE achieves the state-of-the-art performance on five popular safe T2I benchmarks (I2P (Schramowski et al., ), P4D (Chin et al., ), Ring-A-Bell (Tsai et al., ), MMA-Diffusion (Yang et al., ), and UnlearnDiff (Zhang et al., )) compared to other training-free methods, while also demonstrating competitive results against top training-based methods along with better efficiency, lower resource requirements and better inference-time adaptability across diverse unsafe concepts.
We further apply our SAFREE to various T2I diffusion backbones (e.g., SDXL (Podell et al., ), SD-v3 (stabilityai, )) and T2V models (ZeroScopeT2V (zeroscope, ), CogVideoX (Yang et al., )), showcasing SAFREE’s strong generalization and flexibility by effectively managing unsafe concept outputs across different models and tasks.
As generative AI evolves rapidly, our training-free safeguard, SAFREE, sets a new strong baseline for generative AI safety, ensuring safe and ethical generative practices across diverse applications, from image to video synthesis, meeting the growing needs of the AI community.

Our contributions are summarized as:

- •
We propose SAFREE, a strong, adaptive, and training-free safeguard for text-to-image and text-to-video generation that ensures safety by jointly filtering out unsafe concepts in both textual embedding and visual latent spaces through conceptual proximity analysis.
- •
SAFREE achieves state-of-the-art performance among training-free methods for concept removal in visual generation while maintaining high-quality outputs for desirable concepts, and it exhibits competitive results compared to training-based methods.
- •
SAFREE effectively operates across various visual diffusion model architectures and applications, demonstrating strong generalization and flexibility,

## 2 Related Work

### 2.1 T2I attacks

Recent works address vulnerabilities in generative models, including LLMs (Patil et al., ; Liu et al., ), VLMs (Zhao et al., ), and T2I models (Yang et al., ; Wang et al., ; Li et al., ). Cross-modality jailbreaks, like Shayegani et al. (), pair adversarial images with prompts to disrupt VLMs without accessing the language model.
Tools like Ring-A-Bell (Tsai et al., ) and automated frameworks by Kim et al. () and Li et al. () focus on model-agnostic red-teaming and adversarial prompt generation, revealing safety flaws. Methods by Ma et al. (), Yang et al. (), and Mehrabi et al. () exploit text embeddings and multimodal inputs to bypass safeguards, using strategies like adversarial prompt optimization and in-context learning (Chin et al., ; Liu et al., ). These works highlight vulnerabilities in T2I models.

### 2.2 Safe T2I generation

Training-based: Training-based approaches (Lyu et al., ; Pham et al., ; Zhang et al., ) ensure safe T2I generation by removing unsafe elements, as in Li et al. () and Gandikota et al. (), or using negative guidance. Adversarial training frameworks like Kim et al. () neutralize harmful text embeddings, while works like Das et al. () and Park et al. () filter harmful representations through concept removal and preference optimization. Fine-tuning methods such as Lu et al. ()’s cross-attention refinement and Heng & Soh ()’s continual learning remove inappropriate content. Latent space manipulation, explored by Liu et al. () and Li et al. (), enhances safety using self-supervised learning. While effective, they require extensive fine-tuning, degrade image quality, and lack inference-time adaptation. Our SAFREE is training-free, dynamically adapts to concepts, and controls filtering strength w/o modifying weights, offering efficient safety across T2I and T2V models.

Training-free: Training-free methods for safe T2I generation adjust model behavior without retraining. These include (1) Closed-form weight editing, like Gandikota et al. ()’s model projection editing and Gong et al. ()’s target embedding methods, which remove harmful content while preserving generative capacity, and Orgad et al. ()’s minimal parameter updates to diffusion models. (2) Non-weight editing, such as Schramowski et al. ()’s Safe Latent Diffusion using classifier-free guidance and Cai et al. ()’s prompt refinement framework. However, these methods lack robustness and test-time adaptation. Our method is training-free, dynamically adjusts filtering based on prompts, and extends to other architectures and video tasks without weight edits, offering improved scalability and efficiency.

## 3 SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image and Video Generation

Recent approaches (Gandikota et al., ; Gong et al., ; Lu et al., ; Chavhan et al., ; Yang et al., ) have demonstrated the effectiveness of weight modification through unlearning or model editing to prevent the generation of harmful (e.g., pornography, self-harm, violence), biased (e.g., racial or social stereotypes, ageism), or otherwise undesirable (e.g., public, copyrighted) visual content in text-to-image generation models.
However, these methods have limited flexibility because they 1) require storing individual model weights for each concept to be removed, 2) inherently reduce the backbone model’s generative capabilities through unlearning, and 3) necessitate distinct solutions for safe generation across different models (i.e., modified model weights).
To address this, we propose SAFREE, a training-free, yet adaptive remedy for safe T2I and T2V generation (See Fig. 2).
We first determine the trigger tokens that can potentially induce toxicity based on the proximity between masked input prompt embeddings and toxic concept subspace (Sec. 3.1).
Next, we propose projecting the detected undesirable tokens’ embeddings into a space orthogonal to the toxic concept subspace, while ensuring they remain within the input embedding space (Sec. 3.2).
Additionally, we introduce an adaptive re-attention strategy, called self-validating filtering, in the latent space during the de-noising process, facilitating a robust joint filtering mechanism across both text and visual embedding spaces (Sec. 3.3).
Finally, we extend our approach to high-resolution models like SDXL (Podell et al., ), DiT-based image diffusion (SD-v3) (Peebles & Xie, ), and representative text-to-video generative models such as ZeroScopeT2V (zeroscope, ) and CogVideoX-5B (Yang et al., ) (Sec. 3.4).

Figure: Figure 2: Illustration of SAFREE framework. Based on proximity analysis between the masked token embeddings and the toxic subspace $\mathcal{C}$, we detect unsafe tokens and project them into orthogonal to the toxic concept (in red), but still be in the input space $\mathcal{I}$ (in green). SAFREE adaptively controls the filtering strength in an input-dependent manner, which also regulates a latent-level re-attention mechanism. Note that our approach can be broadly applied to various image and video diffusion backbones.
Refer to caption: x2.png

### 3.1 Adaptive Token Selection based on Toxic Concept Subspace Proximity

Random noise $\epsilon_{0}\sim\mathcal{N}(0,\bm{I})$ sampled from a Gaussian distribution can lead to the generation of unsafe or undesirable images in diffusion models, primarily due to inappropriate semantics embedded in the text, which conditions the iterative denoising process (Rombach et al., ; Ho & Salimans, ). To mitigate this risk, recent studies (Miyake et al., ; Schramowski et al., ; Ban et al., ) have demonstrated the effectiveness of using negative prompts. In this approach, the model aims to predict the refined noise from $\epsilon_{0}$ over several autoregressive denoising steps, synthesizing an image conditioned on the input (i.e., the input text prompt).
The denoising process of diffusion models, parameterized by $\bm{\theta}$, at timestep $t$ follows the classifier-free guidance approach:

$$ $\epsilon_{t}=\left(1+\omega\right)\epsilon_{\bm{\theta}}\left(\bm{z}_{t},\bm{p }\right)-\omega\epsilon_{\bm{\theta}}\left(\bm{z}_{t},\emptyset\right),$ (1) $$

where $\omega$ is a hyperparameter controlling the guidance scale.
$\bm{p}$ and $\emptyset$ denote the embedding of the input prompt and null text, respectively.
The negative prompt is applied by replacing $\emptyset$ with the embedding of the negative prompt. We note that even if the input prompts are adversarial and unreadable to humans, such as those generated by adversarial attack methods through prompt optimization (Tsai et al., ; Yang et al., ; Chin et al., ), they are still encoded within the same text embedding space, like CLIP (Radford et al., ), highlighting the importance and necessity of the feature embedding level unsafe concepts identification for safeguard.
To this end, we propose to detect token embeddings that trigger inappropriate image generation and transform them to be distant from the toxic concept subspace: $\mathcal{C}=\left[\bm{c}_{0};\bm{c}_{1};...;\bm{c}_{K-1}\right]\in\mathbb{R}^{
D\times K}$, which represents the embedding matrix that denotes the toxic subspace, where each column vector $\bm{c}_{k}$ corresponds to the embedding of the relevant text associated with the user-defined toxic concept, such as Sexual Acts or Pornography for Nudity concept.

To assess the relevance of tokens in the input prompt to the toxic concept subspace, we design a pooled input embedding $\overline{\bm{p}}_{\backslash i}\in\mathbb{R}^{D}$ that averages the token embeddings in $\bm{p}$ while masking out the $i$-th token.
Suppose $\bm{z}\in\mathbb{R}^{K}$ be the vector of coefficients (i.e., the projection coordinates) of $\overline{\bm{p}}_{\backslash i}$ onto $\mathcal{C}$, it satisfies,

$$ $\begin{split}{\mathcal{C}}^{\top}\left(\overline{\bm{p}}_{\backslash i}- \mathcal{C}\bm{z}\right)=0,\quad\quad\quad\quad\bm{z}=\left({\mathcal{C}}^{ \top}\mathcal{C}\right)^{-1}{\mathcal{C}}^{\top}\overline{\bm{p}}_{\backslash i }.\end{split}$ (2) $$

We estimate the conceptual proximity of a token in the input prompt with $\mathcal{C}$ by computing the distance between the pooled text embedding obtained after masking out (i.e., removing) the corresponding token and $\mathcal{C}$.
The residual vector $\bm{d}_{\backslash i}$, which is a component of $\overline{\bm{p}}_{\backslash i}$, is orthogonal to the subspace $\mathcal{C}$ is then formulated as follows:

$$ $\begin{split}\bm{d}_{\backslash i}=\overline{\bm{p}}_{\backslash i}-\mathcal{C }\bm{z}&=\left(\bm{I}-\mathcal{C}\left({\mathcal{C}}^{\top}\mathcal{C}\right)^ {-1}{\mathcal{C}}^{\top}\right)\overline{\bm{p}}_{\backslash i}\\ &=\left(\bm{I}-\bm{P}_{\mathcal{C}}\right)\overline{\bm{p}}_{\backslash i},~{} ~{}~{}~{}\text{where}~{}~{}\bm{P}_{\mathcal{C}}=\mathcal{C}\left({\mathcal{C}} ^{\top}\mathcal{C}\right)^{-1}{\mathcal{C}}^{\top}\end{split}$ (3) $$

and $\bm{I}\in\mathbb{R}^{D\times D}$ denotes the identity matrix (See Fig. 2 middle left).

Figure: Figure 3: Left: Correlation between the toxicity score (predicted by Nudenet detector) and distance to the subspace of nudity concept. Right: Gaussian distributions of the distance between the nudity subspace and text embeddings of Ring-a-bell or COCO 30k prompts.
Refer to caption: x3.png

A longer residual vector distance indicates that the removed token in the prompt is more strongly associated with the concept we aim to eliminate. In the end, we derive a masked vector $\bm{m}\in\mathbb{R}^{N}$ (where $N$ denotes the token length) to identify tokens related to the target concept, allowing us to subtly project them within the input token subspace while keeping them distant from the toxic concept subspace.
We obtain a set of distances of masked token embeddings $\overline{\bm{p}}_{\backslash i},i\in[0,N-1]$ to the concept subspace, $D(\bm{p}|\mathcal{C})$, and select tokens for masking by evaluating the disparity between each token’s distance and the average distance of the set, excluding the token itself:

$$ $\begin{split}D(\bm{p}|\mathcal{C})&=\left[\|\bm{d}_{\backslash 0}\|_{2},\|\bm{ d}_{\backslash 1}\|_{2},...,\|\bm{d}_{\backslash N-1}\|_{2}\right],\\ m_{i}&=\begin{cases}1&\text{if}~{}~{}\|\bm{d}_{\backslash i}\|_{2}>\left(1+ \alpha\right)\cdot\text{mean}\left(D(\bm{p}|\mathcal{C}).\text{delete}\left(i \right)\right),\\ 0&\text{otherwise},\end{cases}\end{split}$ (4) $$

where $\alpha$ is a non-negative hyperparameter that controls the sensitivity of detecting concept-relevant tokens. $X.\text{delete}(i)$ denotes an operation that produces a list $X$ removing the $i$-th item. We set $\alpha=0.01$ for all experiments in this paper, demonstrating the robustness of our approach to $\alpha$ across T2I generation tasks with varying concepts. We project the detected token embedding (i.e., $m_{i}=1$) to safer embedding space (See Sec. 3.2).

To assess the soundness of our approach across various T2I generation tasks, we present visualizations of predicted toxicity scores obtained from the Nudenet detector (notAI tech, ). These scores are based on adversarial prompts from the Ring-A-Bell (Tsai et al., ) and normal (non-toxic) prompts from COCO-30k (Lin et al., ) datasets and are plotted against the distance between token embeddings and the subspace of nudity concept. Fig. 3 Left plots toxicity scores against the distance from the toxic subspace, measured using the Nudenet detector. Non-toxic COCO prompts show lower toxicity and are farther from the toxic subspace, while Ring-a-Bell prompts have higher toxicity and tend to be closer. This demonstrates that prompts with higher toxicity tend to be nearer to the toxic concept subspace, validating our method for identifying potentially undesirable prompts based on their distance from the target subspace. This is further supported by the significant disparity between the Gaussian distribution of the distances of COCO and Ring-a-Bell prompt embeddings to the toxic subspace, as visualized in Fig. 3 Right.

### 3.2 Safe Generation via Concept Orthogonal Token Projection

We aim to project toxic concept tokens into a safer space to encourage the model to generate appropriate images. However, directly removing or replacing these tokens with irrelevant ones, such as random tokens or replacing the token embeddings with null embeddings, disrupts the coherence between words and sentences, compromising the quality of the generated image to the safe input prompt, particularly when the prompt is unrelated to the toxic concepts. To address this, we propose projecting the detected token embeddings into a space orthogonal to the toxic concept subspace while keeping them within the input space to ensure that the integrity of the original prompt is preserved as much as possible.
We begin by formalizing the input space $\mathcal{I}$ using pooled embeddings from masked prompts as described in Sec. 3.1, such that
$\mathcal{I}=\left[\overline{\bm{p}}_{\backslash 0};\overline{\bm{p}}_{
\backslash 1};...;\overline{\bm{p}}_{\backslash N-1}\right]\in\mathbb{R}^{D
\times N}$. Given the projection matrix into input space $\mathcal{I}$ formulated by $\bm{P}_{\mathcal{I}}=\mathcal{I}\left({\mathcal{I}}^{\top}\mathcal{I}\right)^{
-1}{\mathcal{I}}^{\top}$ (derived by Eq. 3), we perform selective detoxification of input token embeddings based on the obtained token masks that project assigned tokens into $\bm{P}_{\mathcal{I}}$ and to be orthogonal to $\bm{P}_{\mathcal{C}}$:

$$ $\begin{split}\bm{p}_{proj}&=\bm{P}_{\mathcal{I}}\left(\bm{I}-\bm{P}_{\mathcal{ C}}\right)\bm{p},\\ \bm{p}_{safe}&=\bm{m}\odot\bm{p}_{proj}+(\bm{1}-\bm{m})\odot\bm{p},\end{split}$ (5) $$

where $\odot$ indicates an element-wise multiplication operator.
While our approach so far adaptively controls the number of token embeddings to be updated, it sometimes lacks flexibility in preserving the original generation capabilities for content outside the target concept. Recent observations (Kim et al., ; Ban et al., ) suggest that different denoising timesteps in T2I models contribute unevenly to generating toxic or undesirable content. Based on this insight, we propose a self-validating filtering mechanism during the denoising steps of the diffusion model that automatically adjusts the number of denoising timesteps conditioned on the obtained embedding (middle in Fig. 2). This mechanism amplifies the model’s filtering capability when the input prompt is deemed undesirable, while approximating the original backbone model’s generation for safe content. In the end, our updated input text embedding $\bm{p}_{safree}$ at a different denoising step $t$ is determined as follows:

$$ $\begin{split}t^{\prime}&=\gamma\cdot\text{sigmoid}(1-\text{cos}(\bm{p},\bm{p}_ {proj})),\\ \bm{p}_{safree}&=\begin{cases}\bm{p}_{safe}&\text{if}~{}~{}t\leq\text{round}(t ^{\prime}),\\ \bm{p}&\text{otherwise},\end{cases}\end{split}$ (6) $$

where $\gamma$ is hyperparameter ($\gamma=10$ throughout the paper) and cos represents cosine similarity. $t^{\prime}$ denotes the self-validating threshold to determine the number of denoising steps applying to the proposed safeguard approach. Specifically, we adopt the cosine distance between the original input embedding $\bm{p}$ and the projected embedding $\bm{p}_{proj}$ to compute $t^{\prime}$. A higher similarity indicates that the input prompt has been effectively disentangled from the toxic target concept to be removed.

### 3.3 Adaptive Latent Re-attention in Fourier Domain

Recent literature (Mao et al., ; Qi et al., ; Ban et al., ; Sun et al., ) has demonstrated that the initial noise sampled from a Gaussian distribution significantly impacts the fidelity of T2I generation in diffusion models. To further guide these models in creating content while suppressing the appearance of inappropriate or target concept semantics, we propose a novel visual latent filtering strategy during the denoising process. Si et al. () note that current T2I models frequently experience oversmoothing of textures during the denoising process, resulting in distortions in the generated images.
Building on this insight, we suggest an adaptive re-weighting strategy using spectral transformation in the Fourier domain. At each timestep, we initially perform a Fourier transform on the latent features, conditioned on the initial prompt $\bm{p}$ (which may incorporate unsafe guidance) and our filtered prompt embedding $\bm{p}_{safree}$.
The low-frequency components typically capture the global structure and attributes of an image, encompassing its overall context, style, and color.
In this context, we reduce the influence of low-frequency features, which are accentuated by our filtered prompt embedding, while preserving the visual regions that are more closely aligned with the original prompt to avoid excessive oversmoothing. Let $h(\cdot)$ be a latent feature, to achieve this, we attenuate the low-frequency features in $h(\bm{p}_{safree})$ by a scalar $s$ when their values are lower in magnitude than those from $\bm{p}$:

$$ $\begin{split}\mathcal{F}({\bm{p}})&=\bm{b}\odot\text{FFT}(h(\bm{p})),~{}~{}~{} \mathcal{F}({\bm{p}_{safree}})=\bm{b}\odot\text{FFT}(h(\bm{p}_{safree})),\end{split}$ (7) $$

$$ $\begin{split}\mathcal{F}^{\prime}_{i}&=\begin{cases}s\cdot\mathcal{F}({\bm{p}_ {safree}})_{i}&\text{if}~{}~{}\mathcal{F}({\bm{p}_{safree}})_{i}>\mathcal{F}({ \bm{p}})_{i},\\ \mathcal{F}({\bm{p}_{safree}})_{i}&\text{otherwise}.\end{cases}\end{split}$ (8) $$

where $s<1$, $\bm{b}$ represents the binary masks corresponding to the low-frequency components (i.e., the middle in the width and height dimension), and FFT denotes a Fast Fourier Transform operation. We first obtain low frequency features from $h(\bm{p})$ and $h(\bm{p}_{safree})$ in Eq. 7. This reduces the oversmoothing effect in safe visual components, encouraging the generation of safe outputs without emphasizing inappropriate content. This process is enabled by obtaining the refined features $h^{\prime}$ via inverse FFT, $h^{\prime}=\text{IFFT}(\mathcal{F}^{\prime})$, as described in Eq. 8. Note that this equation doesn’t affect the original feature if $t>\text{round}(t^{\prime})$ since $\mathcal{F}({\bm{p}_{safree}})_{i}==\mathcal{F}({\bm{p}})_{i}$, allowing automatic control of filtering capability through self-validated filtering.

### 3.4 SAFREE for Advanced T2I Models and Text-to-Video Generation

Unlike existing unlearning-based methods limited to specific models or tasks, SAFREE is architecture agnostic and can be integrated across diverse backbone models without model modifications, offering superior versatility in safe generation.
This flexibility is enabled by concept-orthogonal, selective token projection and self-validating adaptive filtering, allowing SAFREE to work across a wide range of generative models and tasks.
It operates seamlessly with models beyond SD v-1.4, like, SDXL (Podell et al., ) and SD-v3 (stabilityai, ) in a zero-shot, training-free manner, and extends its applicability to text-to-video (T2V) generation models like ZeroScopeT2V (zeroscope, ) and CogVideoX (Yang et al., ), making it highly flexible as a series of plug-and-play modules.
We present both qualitative and quantitative results showing its effectiveness across various model backbones (UNet (Ronneberger et al., ) and DiT (Peebles & Xie, )) and tasks (T2I and T2V generation) in Sec. 4.6.

## 4 Experimental Results

### 4.1 Experimental Setup

We use StableDiffusion-v1.4 (SD-v1.4) (Rombach et al., ) as the primary T2I backbone, following recent work (Gandikota et al., ; ; Gong et al., ). All methods are tested on adversarial prompts from red-teaming methods: I2P (Schramowski et al., ), P4D (Chin et al., ), Ring-a-Bell (Tsai et al., ), MMA-Diffusion (Yang et al., ), and UnlearnDiff (Zhang et al., ). Following Gandikota et al. (), we also evaluate models on artist-style removal tasks, using two datasets: one with five famous artists (Van Gogh, Picasso, Rembrandt, Warhol, Caravaggio) and the other with five modern artists (McKernan, Kinkade, Edlin, Eng, Ajin: Demi-Human), whose styles can be mimicked by SD. We extend SAFREE to text-to-video generation, applying it to ZeroScopeT2V (zeroscope, ) and CogVideoX (Yang et al., ) with different model backbones (UNet (Ronneberger et al., ) and Diffusion Transformer (Peebles & Xie, )). For quantitative evaluation, we use SafeSora (Dai et al., ) with 600 toxic prompts across 12 concepts, constructing a benchmark of 296 examples across 5 categories.

**Table 1: Attack Success Rate (ASR) and generation quality comparison with training-free and training-based safe T2I generation methods. The best results are bolded. We gray out training-based methods for a fair comparison. SD-v1.4 is the backbone model for all methods. We measure the FID scores of safe T2I models by comparing their generated outputs with the ones from SD-v1.4.**
| Method | T | S | L | Adversarial Prompt | CoCo |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| P4D$\downarrow$ | MMA-Diffusion$\downarrow$ | FID$\downarrow$ | CLIP$\uparrow$ |  |  |  |  |
| SD-v1.4 | - | - | - | 0.987 | 0.957 | - | 31.3 |
| SAFREE<br>Ours | N | - | - | 0.430 | 0.512 | 37.6 | 30.9 |
| P | - | - | 0.417 | 0.597 | 36.5 | 31.1 |  |
| P | $\checkmark$ | - | 0.461 | 0.598 | 36.1 | 31.1 |  |
| P | - | $\checkmark$ | 0.410 | 0.588 | 42.2 | 30.7 |  |
| P | $\checkmark$ | $\checkmark$ | 0.384 | 0.585 | 36.4 | 31.1 |  |

### 4.2 Baselines and Evaluation Metrics

Baselines. We compare our method with training-free approaches: SLD (Schramowski et al., ) and UCE (Gandikota et al., ), as well as training-based methods including ESD (Gandikota et al., ), SA (Heng & Soh, ), CA (Kumari et al., ), MACE (Lu et al., ), SDID (Li et al., ), and RECE (Gong et al., ). Additional details are in the Appendix.

Evaluation Metrics. We assess safeguard capability via Attack Success Rate (ASR) on adversarial nudity prompts (Gong et al., ). For generation quality, we use FID (Heusel et al., ), CLIP score, and TIFA (Hu et al., ) on COCO-30k (Lin et al., ), evaluating 1k samples. For artist-style removal, LPIPS (Zhang et al., ) measures perceptual difference. We frame style removal as a multi-choice question-answering task, using GPT-4o (gpt 4o, ) to identify the artist from generated images. Safe T2V Metrics follow ChatGPT-based evaluation from T2VSafetybench (Miao et al., ). We provide 16 sampled video frames, following the prompt design outlined in T2VSafetybench, to GPT-4o (gpt 4o, ) for binary safety assessment.

### 4.3 Evaluating the Effectiveness of SAFREE

SAFREE achieves training-free SoTA performance without altering model weights. We compare different methods for safe T2I generation, extensively and comprehensively evaluating each model’s vulnerability to adversarial attacks (i.e., attack success rate (ASR)) and their performance across multiple attack scenarios.
As shown in Sec. 4.1 and Fig. 4, the proposed safeguard SAFREE consistently achieves significantly lower ASR than all training-free baselines, across all attack types. Notably, it demonstrates 47%, 13%, and 34% lower ASR compared to the best-performing counterparts— I2P, MMA-diffusion, and UnlearnDiff, respectively—highlighting its strong resilience against adversarial attacks.

Figure: Figure 4: Generated examples of SAFREE and safe T2I baselines. Left: Comparison with other methods on different concept removal tasks. Right: SAFREE incorporates with different T2I backbone models (SDXL and SD-v3) and T2V models (ZeroScopeT2V and CogVideoX). More visualizations are in the Appendix.
Refer to caption: x5.png

SAFREE show competitive results against training-based methods. In addition, we compare our approach with training-based methods. Surprisingly, our approach achieves competitive performance against these techniques. While SA and MACE exhibit strong safeguarding capabilities, they significantly degrade the overall quality of image generation due to excessive modifications of SD weights, often making them impractical for real-world applications as they frequently cause severe distortions. Notably, SAFREE delivers comparable safeguarding performance while generating high-quality images on the COCO-30k dataset, all within a training-free framework.

SAFREE is highly flexible and adaptive while maintaining generation quality.
Our SAFREE does not require additional training or model weight modifications (more detailed comparison in later Tab. 4), providing key advantages over other methods (e.g., ESD, SA, and CA) which depend on unlearning or stochastic optimization, thereby increasing complexity.
SAFREE allows for dynamic control of the number of filtered denoising steps based on inputs in an adaptable manner without extensive retraining or model modifications.
Furthermore, it is worth noting that, compared with other methods, SAFREE is able to preserve the safe content within the original prompt through targeted filtering in both the token and pixel spaces, ensuring that the projected embeddings remain within the input space, which makes SAFREE a highly efficient and reliable solution for real-world applications.
As shown in Fig. 4 left-top, when requiring removing the artist concept (“Van Gogh”), our SAFREE deletes this targeted art style (first row) while preserving other artists style (second row) by producing semantically similar outputs to the original SD. The examples of removing the “nudity” concept as shown in Fig. 4 left-bottom draw a similar conclusion. It demonstrates our SAFREE is a highly adaptive safe generation solution that is able to maintain untargeted safe concepts well.

Ablations. We validate the effectiveness of three components of SAFREE in Tab. 4 using SD-v1.4. First, we examine the impact of the adaptive toxic token selection (T, Sec. 3.1). We replace the selected token embeddings with either the null token embedding (N) or our proposed projected embeddings (P, Sec. 3.2). Both variants significantly reduce ASR from adversarial prompts, demonstrating the effectiveness of our toxic token selection. However, we observe that using N results in degraded image quality, as inserting null tokens disrupts the prompt structure and shifts the input embeddings outside their original space. In contrast, our orthogonal projection (P) achieves competitive safeguard performance with better COCO evaluation results. Incorporating self-validating filtering (S, Eq. 6) further enhances image quality by amplifying the filtering when input tokens are relevant to toxic concepts, although it can slightly reduce filtering capability. By integrating these components with latent-level re-attention (L, Sec. 3.3), our final model strikes a strong balance between effective, safe filtering and preserving the quality of images generated from prompts unrelated to toxic concepts.

**Table 3: Ablation Study on the proposed SAFREE. T: Token Projection. S: Self-validating filtering. L: Latent Re-attention. N: Replacing with Null embedding. P: Orthogonal Token Projection.**
| Method | T | S | L | Adversarial Prompt | CoCo |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| P4D$\downarrow$ | MMA-Diffusion$\downarrow$ | FID$\downarrow$ | CLIP$\uparrow$ |  |  |  |  |
| SD-v1.4 | - | - | - | 0.987 | 0.957 | - | 31.3 |
| SAFREE<br>Ours | N | - | - | 0.430 | 0.512 | 37.6 | 30.9 |
| P | - | - | 0.417 | 0.597 | 36.5 | 31.1 |  |
| P | $\checkmark$ | - | 0.461 | 0.598 | 36.1 | 31.1 |  |
| P | - | $\checkmark$ | 0.410 | 0.588 | 42.2 | 30.7 |  |
| P | $\checkmark$ | $\checkmark$ | 0.384 | 0.585 | 36.4 | 31.1 |  |

**Table 5: Ours with SDXL and SD-v3. We include more visualization in Appendix.**
| Methods | P4D $\downarrow$ | Ring-a-Bell $\downarrow$ | MMA-Diffusion $\downarrow$ | UnlearnDiffAtk $\downarrow$ | CLIP $\uparrow$ | TIFA $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| SDXL (Podell et al., ) | 0.709 | 0.532 | 0.501 | 0.345 | 27.82 | 0.705 |
| SDXL + SAFREE | 0.285 | 0.241 | 0.169 | 0.246 | 27.84 | 0.697 |
| SD-v3 (stabilityai, ) | 0.715 | 0.646 | 0.528 | 0.598 | 31.80 | 0.884 |
| SD-v3 + SAFREE | 0.271 | 0.430 | 0.165 | 0.302 | 31.55 | 0.872 |

### 4.4 Evaluating SAFREE on Artist Concept Removal Tasks

As shown in Sec. 4.1, the proposed SAFREE achieves higher LPIPSe and LPIPSu scores compared to the baselines. The latter is likely due to our approach performing denoising processes guided by a coherent yet projected conditional embedding within the input space. As illustrated in Fig. 5, SAFREE enables generation models to retain the artistic styles of other artists very clearly even with larger feature distance (i.e., high LPIPSu).
To validate whether the generated art styles are accurately removed or preserved, we frame these tasks as a multiple-choice QA problem, moving beyond feature-level distance assessments. Here, Acce and Accu represent the average accuracy of erased and unerased artist styles predicted by GPT-4o based on corresponding text prompts. As shown in  Sec. 4.1, SAFREE demonstrates impressive ability in removing targeted artist concepts, while baseline methods struggle to erase essential representations of target artists.

### 4.5 Efficiency of SAFREE

This section compares the efficiency of various methods, including the training-based ESD/CA, which update models through online optimization and loss, and the training-free UCE/RECE, which modify model attention weights using closed-form edits. Our method (SAFREE), similar to SLD, is training-free and filtering-based, without altering diffusion model weights. As shown in Tab. 4, while UCE/RECE offer fast model editing, they still require additional time for model updates. In contrast, SAFREE requires no model editing or modification, providing flexibility for model development across different conditions while maintaining competitive generation speeds.
Based on Sec. 4.1 and Tab. 4, SAFREE delivers the best overall performance in concept safeguarding, generation quality, and flexibility.

### 4.6 Generalization and Extensibility of SAFREE

To further validate the robustness and generalization of SAFREE, we apply our method to various Text-to-Image (T2I) backbone models and Text-to-Video (T2V) applications.
We extend SAFREE from SD-v1.4 to more advanced models, including SDXL, a scaled UNet-based model, and SD-V3, a Diffusion Transformer (Peebles & Xie, ) model.
Our SAFREE demonstrates strong, training-free filtering of unsafe concepts, seamlessly integrating with these backbones. As shown in Tab. 6, SAFREE reduces unsafe outputs by 48% and 47% across benchmarks/datasets for SD-XL and SD-V3, respectively.
We also extend SAFREE to T2V generation, testing it on ZeroScopeT2V (zeroscope, ) (UNet based) and CogVideoX-5B (Yang et al., ) (Diffusion Transformer based) using the SafeSora (Dai et al., ) benchmark.
As listed in Tab. 6, SAFREE significantly reduces a range of unsafe concepts across both models.
It highlights SAFREE’s strong generalization across architectures and applications, offering an efficient safeguard for generative AI. This is also evident in Fig. 4 right, demonstrating that SAFREE with recent powerful T2I/T2V generation models can produce safe yet faithful (e.g., preserve the concept of ‘woman’ in CogVideoX + SAFREE) and quality visual outputs. More visualizations for T2I and T2V models are included in the Appendix.

## 5 Conclusion

Recent advances in image and video generation models have heightened the risk of producing toxic or unsafe content. Existing methods that rely on model unlearning or editing update pre-trained model weights, limiting their flexibility and versatility. To address this, we propose SAFREE, a novel training-free approach to safe text-to-image and video generation. Our method first identifies the embedding subspace of the target concept within the overall text embedding space and assesses the proximity of input text tokens to this toxic subspace by measuring the projection distance after masking specific tokens. Based on this proximity, we selectively remove critical tokens that direct the prompt embedding toward the toxic subspace. SAFREE effectively prevents the generation of unsafe content while preserving the quality of benign textual requests. We believe our method will serve as a strong training-free baseline in safe text-to-image and video generation, facilitating further research into safer and more responsible generative models.

## Ethics Statement

In recent text-to-image (T2I) and text-to-video (T2V) models, there are significant ethical concerns related to the generation of unsafe or toxic content. These threats include the creation of explicit, violent, or otherwise harmful visual content through adversarial prompts or misuse by users. Safe image and video generation models, including our proposed SAFREE, play a crucial role in mitigating these risks by incorporating unlearning techniques, which help the models forget harmful associations, and filtering mechanisms, which detect and block inappropriate content. Ensuring the ethical usage of these models is essential for promoting a safer and more responsible deployment in creative, social, and educational contexts.

## Acknowledgement

We thank Jaemin Cho, Zhongjie Mi, and Chumeng Liang for the useful discussion and feedback. This work was supported by DARPA ECOLE Program No. HR00112390060, NSF-AI Engage Institute DRL-2112635, DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031, ARO Award W911NF2110220, and ONR Grant N00014-23-1-2356. The views contained in this article are those of the authors and not of the funding agency.

## Appendix

In this Appendix, we present the following:

- •
Experiment Setups including our method implementation details (Sec. A.1), baseline implementation details, and evaluation metrics details (Sec. A.1).
- •
Extra visualization of our methods and other baselines, T2I generation with other backbone models (SDXL and SD-v3), video generation results with T2V backbones (ZeroScopeT2V and CogVideoX) (Sec. B).
- •
Limitations and Broader Impact of our proposed SAFREE (Sec. C).
- •
License information (Sec. D).

## Appendix A Experimental Details

### A.1 Experimental Setup

We employ StableDiffusion-v1.4 (SD-v1.4) (Rombach et al., ) as the main text-to-image generation backbone, following the recent literature (Gandikota et al., ; ; Gong et al., ).
We evaluate our approach and baselines on inappropriate or adversarial prompts from multiple red-teaming techniques:

I2P (Schramowski et al., ): contains $4703$ unsafe prompts related to seven inappropriate/toxic concepts, including hate, harassment, violence, self-harm, sexual, shocking, and illegal activity. We evaluate our model and baselines by removing only the nudity concept following Gong et al. (), as the other concepts are ambiguous to determine their degree.

P4D (Chin et al., ): we use the P4D-N-16 dataset which contains $151$ unsafe prompts collected by a red-teaming tool, Prompting4Debugging (P4D). This is obtained by the prompt engineering to generate inappropriate concepts/objects from (safe) SD variants, including SD with negative prompts, ESD (Gandikota et al., ), and SLD (Schramowski et al., ).

Ring-a-bell (Tsai et al., ): we use $79$ problematic prompts collected by a model agnostic red-teaming framework, Ring-a-bell, which is built upon a text encoder (e.g., text encoder in
CLIP model) and is executed offline without dependency on T2I models.

MMA-Diffusion (Yang et al., ): is a red-teaming framework that is designed to generate inappropriate attack prompts by considering both textual and visual inputs to bypass defensive approaches for T2I models. We use $1$K strong adversarial prompts obtained by this framework.

UnlearnDiffAtk (Zhang et al., ): we also use $142$ adversarial prompts generated by an adversarial prompt attack framework, named UnLearnDiffAtk for unlearned T2I diffusion models (i.e., diffusion models with additional unlearning on unsafe concepts/objects).

In addition to evaluating safe T2I generation, we further assess the models’ reliability in artist-style removal tasks. Following Gandikota et al. (), we employ two datasets: The first includes five famous artists: Van Gogh, Pablo Picasso, Rembrandt, Andy Warhol, and Caravaggio, while the second contains five modern artists: Kellly McKernan, Thomas Kinkade, Tyler Edlin, Kilian Eng, and Ajin: Demi-Human, whose styles have been confirmed to be imitable by SD. Each dataset contains $20$ prompts per artist style; therefore, it has $100$ prompts in total.

We further extend our SAFREE to text-to-video generation. We apply our method to two video generation models, ZeroScopeT2V (zeroscope, ) and CogVideoX (Yang et al., ) with different model backbones (UNet and Diffusion Transformer (Peebles & Xie, )). To quantitatively evaluate the unsafe concept filtering ability on T2V, we choose SafeSora (Dai et al., ), which contains 600 toxic textual prompts across 12 toxic concepts as our testbed. We further select 5 representative categories within 12 concepts, and thus construct a safe video generation benchmark with 296 examples.

### A.2 Baselines and Evaluation Metrics

Baselines. We primarily compare our method with recently proposed training-free approaches allowing instant weight editing or filtering: variants of SLD (Schramowski et al., ) and UCE (Gandikota et al., ) and RECE (Gong et al., ). UCE and RECE are training-free weight modification methods with a closed-form solution. We note that the editing time of RECE largely varies according to the concept to remove (e.g., several seconds to minutes).
In addition, we compare SAFREE with training-based baselines to highlight the advantages of our approach encompassing decent safeguard capability through a training-free framework:
ESD (Gandikota et al., ), SA (Heng & Soh, ),
CA (Kumari et al., ), MACE (Lu et al., ), SDID (Li et al., ).

Hyperparameters. Following Si et al. (), we use two different values for $s$: $s_{1}=0.9$ and $s_{2}=0.2$ for the first and second stage blocks of the decoder, respectively. $\alpha=0.01$, $\gamma=10$ throughout the paper.

Evaluation Metrics. We measure the Attack Success Rate (ASR) on adversarial prompts in terms of nudity following Gong et al. () to evaluate the safeguard capability of methods. To evaluate the original generation quality of safe generation or unlearning methods, we measure the FID (Heusel et al., ), CLIP score, and a fine-grained faithfulness evaluation metric TIFA score (Hu et al., ) on COCO-30k (Lin et al., ) dataset. Among these, we randomly select 1k samples for evaluating FID and TIFA. In artist concept removal tasks, we use LPIPS (Zhang et al., ) to calculate the perceptual difference between SD-v1.4 output and filtered images following Gong et al. ().
To more accurately evaluate whether the model removes characteristic (artist) ”styles” in its output while preserving neighbor and interconnected concepts, we frame the task as a Multiple Choice Question Answering (MCQA) problem. Given the generated images, we ask GPT-4o (gpt 4o, ) to identify the best matching artist name from five candidates.
For safe Text-to-Video evaluation metrics, we follow the automatic evaluation via ChatGPT proposed by T2VSafetybench (Miao et al., ).
We input sampled 16 video frames along with the same prompt design presented in T2VSafetybench to GPT-4o (gpt 4o, ) for binary safety checking.

## Appendix B More Qualitative Visualization

We provide more visualization in this Appendix.
We provide visualization of artist concept removal in Figs. 5, 6, 7 and 8, where we remove ’Van Gogh’ in the model. Across  Figs. 5, 6 and 7, we observe that our SAFREE can effectively remove ’Van Gogh’ without updating any model weights while other methods, even for training-based method, still struggle for removing this concept. Meanwhile, SAFREE preserves the quality of the desirable concepts in the given prompts.
SAFREE can generate the same subjects/scenes as the base model did but remove the targeted style concepts.
Furthermore, as shown in Fig. 8, we test both SAFREE and other baseline methods with text prompts containing other artist concepts. All models removed the ’Van Gogh’ concept in their own way. SAFREE successfully preserved other artist styles by maintaining a high similarity to the original SD-1.4 outputs. Meanwhile, other methods like CA/SLD failed to hold the desirable concept. We further show more results by removing the ’nudity’ concept in Figs. 9, 10 and 11, and draw a similar conclusion.

We further change our diffusion model backbones to more advanced SDXL (Podell et al., ) and SD-v3 (stabilityai, ), as well as Text-to-Video generation backbone models, ZeroScopeT2V (zeroscope, ) and CogVideoX (Yang et al., ).
As shown in Fig. 12, our method shows robustness across Text-to-Image model backbones, and can effectively filter user-defined unsafe concepts but still preserve the safe concepts in the given toxic prompts.
As illusrated in Figs. 13, 14, 15, 17, 18 and 19, our SAFREE shows good generalization ability to Text-to-Video settings. It helps to guard against diverse unsafe/toxic concepts (e.g., animal abuse, porn, violence, terrorism) while preserving quality to the remaining desirable content (e.g., building/human/animals).

## Appendix C Limitation & Broader Impact

While our SAFREE demonstrates remarkable effectiveness in concept safeguarding and generalization abilities across backbone models and tasks. We notice that it is still not a perfect method to ensure safe generation in any case. Specifically, our filtering-based SAFREE method exhibits limitations when toxic prompts become much more implicit and in a chain-of-thought style or under an Image-to-Video settting (Yu et al., ). such kind of toxic prompts can still jailbreak our SAFREE and yield unsafe/inappropriate content generation.
However, we also note that perfect safeguarding in generative models is a challenging open problem that needs more future studies.

Photorealistic Text-to-Image/Video Generation inherits biases from their training data, leading to several broader impacts, including societal stereotypes, biased interpretation of actions, and privacy concerns.
To mitigate these broader impacts, it is essential to carefully develop and implement generative and video description models, such as considering diversifying training datasets, implementing fairness and bias evaluation metrics, and engaging communities to understand and address their concerns.

## Appendix D License Information

We will make our code publicly accessible.
We use standard licenses from the community and provide the following links to the licenses for the datasets and models that we used in this paper.
For further information, please refer to the specific link.

StableDiffusion 1.4: [https://huggingface.co/spaces/CompVis/stable-diffusion-license](https://huggingface.co/spaces/CompVis/stable-diffusion-license)

SDXL: [https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/LICENSE.md)

SD-v3: [https://huggingface.co/stabilityai/stable-diffusion-3-medium/blob/main/LICENSE.md](https://huggingface.co/stabilityai/stable-diffusion-3-medium/blob/main/LICENSE.md)

ZeroScopeT2V: [https://spdx.org/licenses/CC-BY-NC-4.0](https://spdx.org/licenses/CC-BY-NC-4.0)

CogVideoX: [https://github.com/THUDM/CogVideo/blob/main/LICENSE](https://github.com/THUDM/CogVideo/blob/main/LICENSE)

I2P: [https://github.com/ml-research/safe-latent-diffusion?tab=MIT-1-ov-file#readme](https://github.com/ml-research/safe-latent-diffusion?tab=MIT-1-ov-file#readme)

P4D:[https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md)

Ring-A-Bell: [https://github.com/chiayi-hsu/Ring-A-Bell?tab=MIT-1-ov-file#readme](https://github.com/chiayi-hsu/Ring-A-Bell?tab=MIT-1-ov-file#readme)

MMA-Diffusion [https://github.com/cure-lab/MMA-Diffusion/blob/main/LICENSE](https://github.com/cure-lab/MMA-Diffusion/blob/main/LICENSE)

UnlearnDiffAtk:[https://github.com/OPTML-Group/Diffusion-MU-Attack?tab=MIT-1-ov-file#readme](https://github.com/OPTML-Group/Diffusion-MU-Attack?tab=MIT-1-ov-file#readme)

COCO:[https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/cc-by-4.0.md)

SafeSora: [https://spdx.org/licenses/CC-BY-NC-4.0](https://spdx.org/licenses/CC-BY-NC-4.0)

Figure: Figure 5: Visualization of concept removal for famous artist styles. Each row from top to bottom represents generated artworks of Van Gogh, Pablo Picasso, Rembrandt, Andy Warhol, and Caravaggio with corresponding text prompts, where we remove only Van Gogh’s art style (i.e., the first row).
Refer to caption: x6.png

Figure: Figure 6: More Text-to-Image generated examples. We filter the Van Gogh style/concept in the diffusion model.
Refer to caption: x10.png

Figure: Figure 7: More Text-to-Image generated examples. We filter the Van Gogh style/concept in the diffusion model.
Refer to caption: x11.png

Figure: Figure 8: More Text-to-Image generated examples. We filter the Van Gogh style/concept in the diffusion model.
Refer to caption: x12.png

Figure: Figure 9: More T2I generated examples. We filter the unsafe nudity concept in the diffusion model. We manually masked unsafe generated results for display purposes.
Refer to caption: x13.png

Figure: Figure 10: More T2I generated examples. We filter the unsafe nudity concept in the diffusion model. We manually masked unsafe generated results for display purposes.
Refer to caption: x14.png

Figure: Figure 11: More T2I generated examples. We filter the unsafe nudity concept in the diffusion model. We manually masked unsafe generated results for display purposes.
Refer to caption: x15.png

Figure: Figure 12: More T2I generated examples with different diffusion model backbone (SDXL and SD-v3). Our SAFREEcan guard the ’nudity’ concept in any given diffusion models and still preserve safe concepts in the toxic prompts. We manually masked unsafe generated results for display purposes.
Refer to caption: x16.png

Figure: Figure 13: More Text-to-Video generated examples with CogVideoX. We manually blurred unsafe video and masked out sensitive text prompts for display purposes.
Refer to caption: x17.png

Figure: Figure 14: More Text-to-Video generated examples with CogVideoX. We manually blurred unsafe video and masked out sensitive text prompts for display purposes.
Refer to caption: x18.png

Figure: Figure 15: More Text-to-Video generated examples with CogVideoX. We manually blurred unsafe video and masked out sensitive text prompts for display purposes.
Refer to caption: x19.png

Figure: Figure 16: More Text-to-Video generated examples with CogVideoX. We manually blurred unsafe video and masked out sensitive text prompts for display purposes.
Refer to caption: x20.png

Figure: Figure 17: More Text-to-Video generated examples with ZeroScopeT2V. We manually blurred unsafe video and masked out sensitive text prompts for display purposes.
Refer to caption: x21.png

Figure: Figure 18: More Text-to-Video generated examples with ZeroScopeT2V. We manually blurred unsafe video and masked out sensitive text prompts for display purposes.
Refer to caption: x22.png

Figure: Figure 19: More Text-to-Video generated examples with ZeroScopeT2V. We manually blurred unsafe video and masked out sensitive text prompts for display purposes.
Refer to caption: x23.png

Figure: Figure 20: More Text-to-Video generated examples with ZeroScopeT2V. We manually blurred unsafe video and masked out sensitive text prompts for display purposes.
Refer to caption: x24.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - How is inference time compared to vanilla diffusion without filtering?
- How many diffusion steps are needed for filtering to work? Does the method support fast samplers that run <10 sampling steps?
soundness: 2
strengths: - The method solves a meaningful task. It addresses toxic concept removal from generative models, a key step towards their safe and responsible deployment.
- The method is training-free, allowing easy integration into pre-trained models. Further, it generalizes well on different diffusion model architectures and can be extended to support text-to-video models.
- The experiment results look promising. It outperforms other training-free methods on adversarial prompt filtering and artistic style removal. It also demonstrates competitive performance in comparison to training-based methods while being more efficient and flexible.
summary: This paper presents a method to remove user-defined toxic concepts from pre-trained text-to-image diffusion models by manipulating the text embeddings and intermediate latents. The intuition is to steer the text embeddings away from a subspace spanned by toxic tokens, while keeping them close to the learned manifold to avoid image quality degradation. The method does not modify pre-trained model weights, can be applied on various diffusion model architectures, and can be extended to support text-to-video models. The experiment results demonstrate that the model outperforms other training-free methods on several benchmarks while being more efficient and flexible.
weaknesses: - The paper is very difficult to read, mainly because there are lengthy, distracting discussions all over the place. While the discussions help explain how ideas originated, they disrupt the flow of presentation and cover up the key ideas of the work. The authors are encouraged to sharpen the presentation of the paper by clearly separating background, their key innovations, and insights.

- Many key steps of the algorithm appear quite arbitrary to me. For example, I did not get the intuition behind the definitions of mask m and input space I, and thus got confused by what Eq. (5) means. The reasoning behind Eq. (6) (self-validating filtering) is also not clear. I am not sure I fully understand Eq. (7) either. In particular, why latents in the frequency domain has anything to do with toxic concepts?

- I am not an expert on concept removal from diffusion models, so I will not comment on the selection of baselines and benchmarks. In Table 2, what do LPIPS_e and LPIPS_u mean? Similarly, what do Acc_e and Acc_u mean? Why does the proposed method yield the worst LPIPS_u scores?

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses above
soundness: 3
strengths: - The method is training free and has no modification of model weights, which makes it efficient and perserve the generative ability.
- The method filters unsafe content in both textual embeddings and visual latent spaces, enhancing the robustness of the safeguard.
- The method shows strong performance on multiple benchmarks, reducing the attack success rate of adversarial prompts and maintaining high generation quality.
summary: The paper introduces SAFREE, a training-free and adaptive method designed to ensure safe content generation in text-to-image (T2I) and text-to-video (T2V) diffusion models without modifying the model’s weights. SAFREE works by identifying tokens in the input prompt that are related to undesirable concepts and projecting them into a space orthogonal to these concepts within the text embedding space. This approach maintains the coherence and intended meaning of the original prompt while filtering out unsafe content. Additionally, SAFREE adjusts the denoising steps in the diffusion process based on the input, enhancing the suppression of unwanted content when necessary. It also employs an adaptive re-attention mechanism in the visual latent space to reduce the influence of features tied to unsafe concepts at the pixel level. The method demonstrates strong performance across multiple benchmarks, effectively reducing the generation of unsafe content while preserving the quality and fidelity of the desired output.
weaknesses: - SAFREE may struggle with implicit or indirect triggers of unsafe content, especially when toxic prompts are subtle or crafted in a chain-of-thought style.
-  The effectiveness of the method relies on the accurate identification of the toxic concept subspace, which may not capture all forms of undesirable content.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - In Equation 4, you calculate the mean of distances for masked tokens from subspace C to assess the impact of individual tokens on the distance. Why did you choose not to directly measure the distance of $p$ (without masking any tokens) from $C$? Specifically, I'm referring to using a metric like $\|\| d_{\backslash i} \|\|_2 > (1+\alpha) \times d_p$, where $d_p$ is the distance of the entire prompt $p$ (averaged across token embeddings) from the subspace $C$. 
- Regarding Equation 6, I'm curious about the choice to use the sigmoid function. Given that your input ($1-Sim(p, p')$) ranges from 0 to 2, the sigmoid function's output in this range is nearly linear. What motivated this choice? Additionally, with the values of $t'$ ranging from 5 to 9, this translates to $t$ values between 50 and 90. Could you specify how many time steps you have used in the backward process of the diffusion model?
- In Section 3.4, you discuss the flexibility enabled by *"... concept-orthogonal, selective token projection and self-validating adaptive filtering ..."* allowing SAFREE to be effective across various models and tasks. However, is the adaptive latent re-attention mechanism applied to models like Stable Diffusion 3, or is it utilized differently?
soundness: 3
strengths: - Addressing the critical challenge of safe content generation in AI, this paper tackles an increasingly important issue within the field.
- The method introduced in this paper for filtering toxic concepts operates without the need for retraining or modifying the underlying model weights, which is advantageous. 
- The method has been evaluated across an array of model architectures, including newer models like Stable Diffusion 3. Additionally, it has undergone benchmarking across various performance metrics and benchmarks. 
- The mathematical notations are clear and properly used, enhancing the understanding of the method.
summary: The paper presents SAFREE, an approach designed to enhance the safety of generative AI models in text-to-image and text-to-video applications. This training-free method effectively preserves the original model weights, offering a non-invasive solution to mitigate the risk of generating unsafe content. SAFREE improves safety by identifying and modifying the embeddings of toxic concepts within both textual and visual domains, ensuring the output is devoid of harmful elements. The framework utilizes adaptive token selection based on proximity to a toxic concept subspace, concept orthogonal token projection which repositions toxic tokens away from harmful subspace, and adaptive latent re-attention that diminishes the impact of low-frequency features often emphasized by the filtered prompt embedding.
weaknesses: - The paper could benefit from a more thorough examination of scenarios where SAFREE may underperform, especially in instances where all toxic tokens are absent from the predefined space tokens. This would provide a clearer understanding of its limitations and potential areas for improvement.
- While Figures 3 and 4 demonstrate a correlation between toxic text and distance from subspace C, this might simply reflect that the words from the prompts are already included in subspace C, resembling a form of "dictionary lookup". For a more comprehensive analysis, it would be beneficial to provide metrics showing the extent to which words in the prompts overlap with those in subspace C.
- I have some reasonable guesses as to why the LPIPS_u metric is higher in Table 2, despite a lower score being preferable. However, this isn't addressed in the paper. Providing an explanation for this result would greatly aid readers in understanding the evaluation.
- Given that your method perturbs the text embedding fed into the model, discussing related works that utilize text embedding modifications for enhancing performance in various tasks would enrich the context. For instance, referencing works that apply text embedding modifications for continuous attribute control [1] or improving compositionally [2] would draw valuable parallels and situate your research within the broader field.
- Given that your method perturbs the text embeddings fed into the model, it would be insightful to discuss related works that have similarly applied text embedding modifications to enhance performance across various tasks. For instance, referring to studies like [1], which uses text embedding modifications for continuous attribute control, and [2], which applies them to improve compositionality, would provide valuable context and enhance the related work or background section of your paper.



[1] Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions. Baumann et al, 2024.

[2] Understanding and Mitigating Compositional Issues in Text-to-Image Generative Models. Zarei et al, 2024.

Meta-review: This paper presents SAFREE, a training free method to improve the safety of T2I or T2V models.  The key idea of SAFREE is to modify the embeddings in the text input space to be far away (i.e., orthogonal to) from the subspace of toxic tokens. Other ideas include adjusting the denoising steps based on whether similar the updated embedding to the original embedding (so that prompts with more harmful content will have larger steps),  an adaptive re-attention mechanism in Fourier domain.

The major strengths: 1) The proposed SAFREE to improve safety is training free and is flexible and easy to integrate into pre-trained models. 2) The proposed method is overall solid and novel to some extent 3) The proposed method shows strong performance on multiple benchmarks, outperforming other training-free methods, and showing competitive performance compared to training based methods.

The major weaknesses: 1) The proposed method relies on accurate identification of the toxic concept subspace 2) As reviwer 6QHh pointed out, some related works are not discussed/compared 3) Some technical parts are not clear enough.

Overall, a training free approach to improve safety is an important direction, and the proposed methods are interesting with promising results. So I would recommend "Accept (poster)".

Decision: Accept (Poster)