SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generation

Authors: Jaehong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, Mohit Bansal
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Safe generation, Text-to-image generation, Text-to-video generation

Abstract:
Recent advances in diffusion models have significantly enhanced their ability to generate high-quality images and videos, but they have also increased the risk of producing unsafe content. Existing unlearning/editing-based methods for safe generation remove harmful concepts from models but face several challenges: (1) They cannot instantly remove harmful or undesirable concepts (e.g., artist styles) without additional training. (2) Their safe generation capabilities depend on collected training data. (3) They alter model weights, risking degradation in quality for content unrelated to the targeted toxic concepts. To address these challenges, we propose SAFREE, a novel, training-free approach for safe text-to-image and video generation, that does not alter the model's weights. Specifically, we detect a subspace corresponding to a set of toxic concepts in the text embedding space and steer prompt token embeddings away from this subspace, thereby filtering out harmful content while preserving intended semantics. To balance the trade-off between filtering toxicity and preserving safe concepts, SAFREE incorporates a novel self-validating filtering mechanism that dynamically adjusts the denoising steps when applying the filtered embeddings. Additionally, we incorporate adaptive re-attention mechanisms within the diffusion latent space to selectively diminish the influence of features related to toxic concepts at the pixel level. By integrating filtering across both textual embedding and visual latent spaces, SAFREE ensures coherent safety checking, preserving the fidelity, quality, and safety of the generated outputs. Empirically, SAFREE achieves state-of-the-art performance in suppressing unsafe content in T2I generation (reducing it by 22% across 5 datasets) compared to other training-free methods and effectively filters targeted concepts, e.g., specific artist styles, while maintaining high-quality output. It also shows competitive results against training-based methods. We further extend SAFREE to various T2I backbones and T2V tasks, showcasing its flexibility and generalization. As generative AI rapidly evolves, SAFREE provides a robust and adaptable safeguard for ensuring safe visual generation.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730671616926

thread_type:
official_review

<thread_object>
creator: reviewer_6qhh

<content>
summary:
The paper presents SAFREE, an approach designed to enhance the safety of generative AI models in text-to-image and text-to-video applications. This training-free method effectively preserves the original model weights, offering a non-invasive solution to mitigate the risk of generating unsafe content. SAFREE improves safety by identifying and modifying the embeddings of toxic concepts within both textual and visual domains, ensuring the output is devoid of harmful elements. The framework utilizes adaptive token selection based on proximity to a toxic concept subspace, concept orthogonal token projection which repositions toxic tokens away from harmful subspace, and adaptive latent re-attention that diminishes the impact of low-frequency features often emphasized by the filtered prompt embedding.

strengths:
- Addressing the critical challenge of safe content generation in AI, this paper tackles an increasingly important issue within the field.
- The method introduced in this paper for filtering toxic concepts operates without the need for retraining or modifying the underlying model weights, which is advantageous. 
- The method has been evaluated across an array of model architectures, including newer models like Stable Diffusion 3. Additionally, it has undergone benchmarking across various performance metrics and benchmarks. 
- The mathematical notations are clear and properly used, enhancing the understanding of the method.

weaknesses:
- The paper could benefit from a more thorough examination of scenarios where SAFREE may underperform, especially in instances where all toxic tokens are absent from the predefined space tokens. This would provide a clearer understanding of its limitations and potential areas for improvement.
- While Figures 3 and 4 demonstrate a correlation between toxic text and distance from subspace C, this might simply reflect that the words from the prompts are already included in subspace C, resembling a form of "dictionary lookup". For a more comprehensive analysis, it would be beneficial to provide metrics showing the extent to which words in the prompts overlap with those in subspace C.
- I have some reasonable guesses as to why the LPIPS_u metric is higher in Table 2, despite a lower score being preferable. However, this isn't addressed in the paper. Providing an explanation for this result would greatly aid readers in understanding the evaluation.
- Given that your method perturbs the text embedding fed into the model, discussing related works that utilize text embedding modifications for enhancing performance in various tasks would enrich the context. For instance, referencing works that apply text embedding modifications for continuous attribute control [1] or improving compositionally [2] would draw valuable parallels and situate your research within the broader field.
- Given that your method perturbs the text embeddings fed into the model, it would be insightful to discuss related works that have similarly applied text embedding modifications to enhance performance across various tasks. For instance, referring to studies like [1], which uses text embedding modifications for continuous attribute control, and [2], which applies them to improve compositionality, would provide valuable context and enhance the related work or background section of your paper.



[1] Continuous, Subject-Specific Attribute Control in T2I Models by Identifying Semantic Directions. Baumann et al, 2024.

[2] Understanding and Mitigating Compositional Issues in Text-to-Image Generative Models. Zarei et al, 2024.

questions:
- In Equation 4, you calculate the mean of distances for masked tokens from subspace C to assess the impact of individual tokens on the distance. Why did you choose not to directly measure the distance of $p$ (without masking any tokens) from $C$? Specifically, I'm referring to using a metric like $\|\| d_{\backslash i} \|\|_2 > (1+\alpha) \times d_p$, where $d_p$ is the distance of the entire prompt $p$ (averaged across token embeddings) from the subspace $C$. 
- Regarding Equation 6, I'm curious about the choice to use the sigmoid function. Given that your input ($1-Sim(p, p')$) ranges from 0 to 2, the sigmoid function's output in this range is nearly linear. What motivated this choice? Additionally, with the values of $t'$ ranging from 5 to 9, this translates to $t$ values between 50 and 90. Could you specify how many time steps you have used in the backward process of the diffusion model?
- In Section 3.4, you discuss the flexibility enabled by *"... concept-orthogonal, selective token projection and self-validating adaptive filtering ..."* allowing SAFREE to be effective across various models and tasks. However, is the adaptive latent re-attention mechanism applied to models like Stable Diffusion 3, or is it utilized differently?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730753909268

thread_type:
official_review

<thread_object>
creator: reviewer_xsbb

<content>
summary:
The paper introduces SAFREE, a training-free and adaptive method designed to ensure safe content generation in text-to-image (T2I) and text-to-video (T2V) diffusion models without modifying the model’s weights. SAFREE works by identifying tokens in the input prompt that are related to undesirable concepts and projecting them into a space orthogonal to these concepts within the text embedding space. This approach maintains the coherence and intended meaning of the original prompt while filtering out unsafe content. Additionally, SAFREE adjusts the denoising steps in the diffusion process based on the input, enhancing the suppression of unwanted content when necessary. It also employs an adaptive re-attention mechanism in the visual latent space to reduce the influence of features tied to unsafe concepts at the pixel level. The method demonstrates strong performance across multiple benchmarks, effectively reducing the generation of unsafe content while preserving the quality and fidelity of the desired output.

strengths:
- The method is training free and has no modification of model weights, which makes it efficient and perserve the generative ability.
- The method filters unsafe content in both textual embeddings and visual latent spaces, enhancing the robustness of the safeguard.
- The method shows strong performance on multiple benchmarks, reducing the attack success rate of adversarial prompts and maintaining high generation quality.

weaknesses:
- SAFREE may struggle with implicit or indirect triggers of unsafe content, especially when toxic prompts are subtle or crafted in a chain-of-thought style.
-  The effectiveness of the method relies on the accurate identification of the toxic concept subspace, which may not capture all forms of undesirable content.

questions:
See weaknesses above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730881892666

thread_type:
official_review

<thread_object>
creator: reviewer_vdpx

<content>
summary:
This paper presents a method to remove user-defined toxic concepts from pre-trained text-to-image diffusion models by manipulating the text embeddings and intermediate latents. The intuition is to steer the text embeddings away from a subspace spanned by toxic tokens, while keeping them close to the learned manifold to avoid image quality degradation. The method does not modify pre-trained model weights, can be applied on various diffusion model architectures, and can be extended to support text-to-video models. The experiment results demonstrate that the model outperforms other training-free methods on several benchmarks while being more efficient and flexible.

strengths:
- The method solves a meaningful task. It addresses toxic concept removal from generative models, a key step towards their safe and responsible deployment.
- The method is training-free, allowing easy integration into pre-trained models. Further, it generalizes well on different diffusion model architectures and can be extended to support text-to-video models.
- The experiment results look promising. It outperforms other training-free methods on adversarial prompt filtering and artistic style removal. It also demonstrates competitive performance in comparison to training-based methods while being more efficient and flexible.

weaknesses:
- The paper is very difficult to read, mainly because there are lengthy, distracting discussions all over the place. While the discussions help explain how ideas originated, they disrupt the flow of presentation and cover up the key ideas of the work. The authors are encouraged to sharpen the presentation of the paper by clearly separating background, their key innovations, and insights.

- Many key steps of the algorithm appear quite arbitrary to me. For example, I did not get the intuition behind the definitions of mask m and input space I, and thus got confused by what Eq. (5) means. The reasoning behind Eq. (6) (self-validating filtering) is also not clear. I am not sure I fully understand Eq. (7) either. In particular, why latents in the frequency domain has anything to do with toxic concepts?

- I am not an expert on concept removal from diffusion models, so I will not comment on the selection of baselines and benchmarks. In Table 2, what do LPIPS_e and LPIPS_u mean? Similarly, what do Acc_e and Acc_u mean? Why does the proposed method yield the worst LPIPS_u scores?

questions:
- How is inference time compared to vanilla diffusion without filtering?
- How many diffusion steps are needed for filtering to work? Does the method support fast samplers that run <10 sampling steps?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732234538226

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment by Authors

comment:
We thank the reviewers for their time and valuable comments.

We appreciate the reviewers recognized:

- **SAFREE addresses the critical challenge of safe content generation in AI**, this paper tackles an increasingly important issue within the field. ```[VDpx, 6QHh]```
- The method is **training-free**, allowing easy integration into pre-trained models ```[VDpx,XsBb,6QHh]``` 
- **It generalizes well on different diffusion model architectures** and can be extended to support **text-to-video** models. ```[VDpx,6QHh]```
- **Enhancing the robustness of the safeguard** with joint filtering in both textual embeddings and visual latent spaces. ```[XsBb]```
- **Strong performance** on multiple benchmarks with various metrics ```[VDpx,XsBb,6QHh]```
- **The mathematical notations are clear and properly used**, enhancing the understanding of the method. ```[6QHh]```

---

During the rebuttal period, we made every effort to address all the reviewers' concerns faithfully.   
We hope that our responses have addressed your comments.   


We thank you again for reviewing our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732731835074

thread_type:
public_comment

<thread_object>
creator: ~anubhav_jain1

<content>
title:
Trouble reproducing results from the paper, some help/ clarifications would be appreciated

comment:
Thanks for your interesting work. We are working on the same problem and have been closely following your code base to reproduce some of the reported results. However, we have stumbled upon the following issues, some clarifications on this would be greatly appreciated. (for maintaining authors anonymity I have forked their GitHub repository and anonymized it, all links are to the anonymized version. We have forked the version as seen on Nov 27th 2024). 

1. I've tried running a few experiments with SLD-Max, SLD-Strong, RECE and UCE but these results do not match with the ones reported in your paper. But I was able to reproduce ones from your method (to some extent). 
For the P4D dataset I am getting the following results for your method - 40.84 (reported 38.4) but for other such as SLD-Max I am getting 35.76 (reported 74.2), for SLD-Strong I am getting 49.01 (reported 86.1), for UCE I am getting 29.31 (reported 66.7), for RECE I am getting 21.77 (reported 38.1). There are at least 4 different NudeNet evaluation scripts in your public codebase, which one have you used for getting these results and at what threshold? Which NudeNet version are you using - your requirements.txt file does not contain the NudeNet version? 

I believe this discrepancy could partially also be coming because you are using a threshold of 0.6 while evaluating your approach as seen from line 17 in your scripts/run_nudity.py script (https://anonymous.4open.science/r/SAFREE_copy_date_nov27_2024-281C/scripts/run_nudity.sh) and in your generate_safree.py script Line 407 (https://anonymous.4open.science/r/SAFREE_copy_date_nov27_2024-281C/generate_safree.py). You only set the threshold of 0.45 for the MMA-Diffusion benchmark. While based on nudenet/run_classify.py script Line 62 (https://anonymous.4open.science/r/SAFREE_copy_date_nov27_2024-281C/nudenet/run_classify.py) the baselines are evaluated with a hardcoded threshold 0.45. Could you please cross-check the evaluations and let me know if I am missing something? 

If I run experiments using your script 'run_classify.py' (https://anonymous.4open.science/r/SAFREE_copy_date_nov27_2024-281C/nudenet/run_classify.py) for the P4D benchmark your method yields ASR of 47.61% (reported 38.4) using the classifier model from this commented out line 25 from the script - https://anonymous.4open.science/r/SAFREE_copy_date_nov27_2024-281C/nudenet/classifier.py. 

2. On similar lines, we have had some trouble with the Ring-A-Bell benchmark. The dataset available on your github repository (https://anonymous.4open.science/r/SAFREE_copy_date_nov27_2024-281C/datasets/nudity-ring-a-bell.csv) is not the Ring-A-Bell dataset. Could you please clarify where you got this from? The one on your GitHub repository contains simple English prompts as compared to the actual Ring-A-Bell benchmark posted by the original authors on huggingface (https://huggingface.co/datasets/Chia15/RingABell-Nudity) that contains adversarial prompts. When we ran the experiments ourselves on the benchmark from the original authors and we got the following results for your method - 35.78 for subset K77, 47.36 for subset K38 and 55.78 for subset K16 as compared to the reported 11.4%. I have exactly followed the evaluation protocol from the original Ring-A-Bell authors with NudeNet detector v1 for getting these results to make them comparable with their reported results. Could you please also look into this and let me know if we are missing something? 

Thank you again for your interesting work. Some help and clarifications would be greatly appreciated.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents SAFREE, a training free method to improve the safety of T2I or T2V models.  The key idea of SAFREE is to modify the embeddings in the text input space to be far away (i.e., orthogonal to) from the subspace of toxic tokens. Other ideas include adjusting the denoising steps based on whether similar the updated embedding to the original embedding (so that prompts with more harmful content will have larger steps),  an adaptive re-attention mechanism in Fourier domain.

The major strengths: 1) The proposed SAFREE to improve safety is training free and is flexible and easy to integrate into pre-trained models. 2) The proposed method is overall solid and novel to some extent 3) The proposed method shows strong performance on multiple benchmarks, outperforming other training-free methods, and showing competitive performance compared to training based methods.

The major weaknesses: 1) The proposed method relies on accurate identification of the toxic concept subspace 2) As reviwer 6QHh pointed out, some related works are not discussed/compared 3) Some technical parts are not clear enough.

Overall, a training free approach to improve safety is an important direction, and the proposed methods are interesting with promising results. So I would recommend "Accept (poster)".