Prompt-Independent Safe Decoding to Restrain Unsafe Image Generation for Text-to-Image Models against White-Box Adversary

Authors: Shengyuan Pang, Jiangyi Deng, Jialin Wu, Yanjiao Chen, Huanlong Zhong, Xinfeng Li, Jie Zhang, Wenyuan Xu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Text-to-image generation, AI security, Model compliance

Abstract:
Text-to-image (T2I) models, developed through extensive training, are capable of generating realistic images from textual inputs, profoundly influencing various facets of our lives. Nevertheless, they can be exploited by adversaries who input malicious prompts, leading to the creation of unsafe content and posing serious ethical concerns. Current defense mechanisms primarily rely on external moderation or model modification, but they are inherently fragile against white-box adversaries who have access to the model's weights and can adjust them accordingly. 

To address this issue, we propose \sys, a novel defense framework that governs both the diffusion and the decoder module of the text-to-image pipeline, enabling them to reject generating unsafe content and resist malicious fine-tuning attempts. Concretely, we first fine-tune the diffusion and the decoder module with the denial-of-service samples: 1) for the diffusion module, the inputs are unsafe image-caption pairs, the ground truth is zero predicted noise, and 2) for the decoder module, the inputs are unsafe generations from the diffusion, the ground truth is zero decoding. Then, we employ adversarial training to ensure this denial-of-service behavior for unsafe queries remains effective even after the adversary's fine-tuning with unsafe data. Specifically, we continuously simulate potential fine-tuning processes that the adversary might adopt and expose them to the model, enabling it to learn how to resist.

Extensive experiments validate that \sys effectively prevents the generation of unsafe content without compromising the model’s normal performance. Furthermore, our method demonstrates robust resistance to malicious fine-tuning by white-box adversaries, rendering it resource-intensive to corrupt our protected model, thus significantly deterring the misuse of our model for nefarious purposes.

Decision: reject
Reviewer scores: 3, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1729995497385

thread_type:
official_review

<thread_object>
creator: reviewer_rpmp

<content>
summary:
This paper presents a framework to protect text-to-image models from generating unsafe content. The framework focuses on three key goals: (1) blocking unsafe content by embedding a safety filter in the decoder, and adjusting the diffusion process to ensure unsafe outputs result in black images, (2) resisting malicious finetuning through adversarial training that simulates such funtuning, and (3) maintaining the quality of benign content by adding a term that preserves it during generation. The design clearly outlines each goal and demonstrates effectiveness through experiments.

strengths:
1. The paper addresses a timely issue—preventing unsafe content generation—and shows effectiveness both in mitigating unsafe outputs and preserving benign ones. It also introduces a defensive measure against adversaries attempting to fintune the model toward unsafe outputs by using an adversarial training scheme.
2. The system is well-designed, with clearly defined objectives, and each loss term is crafted to meet a specific goal, making the approach easy to follow.
3. I appreciate the novel ideas, such as fine-tuning the safety filter within the decoder to act as a conditional decoder and the effective strategy of transforming the min-max optimization into min-min in Equation 23. Both are technically sound and effective solutions.

weaknesses:
1. The non-fine-tunable learning scheme seems largely based on Deng et al. (2024), so I question its contribution as a major novel element. However, I acknowledge that the loss function has been modified for the text-to-image task, providing some level of novelty.

2. The evaluation metrics for defending against unsafe generation only use the CLIP score, but this may not sufficiently distinguish your method from others, given the relatively small range of differences (e.g., values of 17-28 in Figure 3 for seven methods against I2P and 16-23 in Figure 4 for Sneakyprompt). Additional metrics are suggested, like NRR from SafeGen. Also, calculating the similarity between your generated images and a black image (your intended output for unsafe prompts) might provide a more direct assessment.

3. Some strategies lack clear explanations of their benefits and would benefit from experimental support. For example, Section 4.2.2 on Feature Space Calibration and the use of VGG for smoothing could be better justified. Please see detailed questions in the specific sections.

Minor issues:
1. There are inconsistencies between some figures and the descriptions, e.g., L_bd/L_fc in Figure 2 vs L_cd//Lfsc in EQ 11.
2. Some notation in the equations is hard to follow; it would help to provide explanations for each symbol directly near the equation.

I may consider raising my score if my concerns and questions are adequately addressed in the rebuttal.

questions:
1. 4.2.2 (Feature Space Calibration) mentions 'However, there is a gap between the feature distributions of the encoder output and the diffusion output, leading to PATRONUS’s occasional failure in the practical T2I working scenario.' Does this gap occur if the text input is left blank during alignment? Could you clarify the benefit of this calibration—does it primarily enhance unsafe content filtering, benign content preservation, or resistance to malicious fine-tuning? Additionally, could you also explain similar questions on VGG smoothing?

2. The paper mentions that “Φ is our simulated fine-tuning strategy set.” Could you specify the strategies within this set? Also, what impacts might arise if the defender uses different strategies from those of attackers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730434978876

thread_type:
official_review

<thread_object>
creator: reviewer_bgyq

<content>
summary:
The paper proposes PATRONUS, a new defense framework that protects text-to-image models from white-box adversarial attacks. With a special emphasis on the threats resulting from model fine-tuning, PATRONUS represents an internal moderator and a mechanism of non fine-tunable learning rather than traditional defenses such as content moderation or alignment, which could be manipulated by attackers even at the parameter level. The internal moderator transforms unsafe input features into zero vectors, ensuring that benign features are decoded correctly while rejecting unsafe prompts. In this regard, the authors prove the efficacy of PATRONUS against different adversarial attacks with much stronger robustness compared to the state-of-the-art methods by retaining low CLIP scores on unsafe inputs while keeping performance on benign inputs.

strengths:
The paper's main strengths lie in its method of applying the concepts of non-fine-tunable learning and inseparable content moderation to white-box adversaries in T2I models. it demonstrates a clear understanding of model architecture, with the authors skillfully leveraging each component to optimize the defense mechanism effectively.

weaknesses:
One limitation to the method could be expensive computational resources. Using 4 A100-80GB GPUs might not be ideal for all the users who wish to use this defense method.

questions:
Would the performance of non-fine-tunable mechanism maintain its effectiveness if PATRONUS is implemented on smaller or resource-limited T2I models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730493145456

thread_type:
official_review

<thread_object>
creator: reviewer_k1hz

<content>
summary:
This paper tries to propose a method to align text-to-image model to prevent unsafe image generation. The idea is to fine-tune the decoder and diffusion model (U-Net). Key challenge for alignment is that if the model is open-source, an attacker may fine-tune the model to remove the alignment. This paper aims to solve this challenge via aligning the model in a specific way. The alignment itself is a fine-tuning process. Evaluation is performed on several datasets and some malicious fine-tuning.

strengths:
This paper aims to solve a challenging problem. For an open-source model, malicious fine-tuning to remove the safety alignment is a key challenge. 

The paper follows a good formulation.

Evaluation is conducted on multiple datasets, and multiple baseline alignment methods are compared.

weaknesses:
The most important weakness is that the reviewer is not convinced the proposed method solves the challenge. It is not clear why the proposed alignment method can survive malicious fine-tuning. It's not clear to why the proposed method is different from baselines in terms of resistance against malicious fine-tuning. 

Evaluation metrics are too simple. Mainly CLIP is used to assess both attack effectiveness and performance preservation. More metrics are expected, e.g., the number of nudity parts in a generated image. FID scores or LPIPS scores to measure performance preservation for benign prompts. 

Adaptive attacks are shown in Appendix. I would say these are the most important results for such a defense work. I suggest to focus on these adaptive attacks, and show detailed results in main body using more fine-grained metrics for both attack effectiveness and performance preservation. 

Eventually, a strong attacker can remove any safety alignment with enough fine-tuning data. An extreme case is that the strong attacker re-trains a model without safety alignment given enough tuning data and computation resources. So, a right question to ask is how much more resources (data and computation) an attacker needs in order to remove the safety alignment. The paper can benefit from showing such experimental results. 

It's also helpful to show images generated for unsafe prompts (adaptive attacks) and safe prompts, for different methods.

questions:
Please see the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650639883

thread_type:
official_review

<thread_object>
creator: reviewer_my5p

<content>
summary:
This paper introduces a novel defensive solution called Patronus, designed to prevent T2Is from generating unsafe content. Patronus integrates an output moderator directly within the T2I's decoder. It guides the image decoder to perform conditional decoding based on safety features, transforming unsafe input features into zero vectors.

strengths:
1. This paper addresses a crucial safety issue: defending T2I models against white-box adversaries. As T2I models become more popular and widely deployed, this topic is increasingly important.

2. The paper presents a novel and intriguing idea.

3. The content is well-structured and easy to follow.

weaknesses:
While the paper introduces 'Patronus' as a novel defensive mechanism, the evaluation section of this paper is too weak, making the effectiveness of the proposed method unclear, which is essential for achieving a higher-scoring paper. 

1. Lacking Evaluation Results on SOTA adversarial attack for T2I. The authors should assess the proposed defense against leading Text-to-Image adversarial attacks, such as MMA-Diffusion [c0]. This attack has been reported to outperform I2P, posing a significant threat to T2I models. Evaluating the defense's effectiveness by reporting the Attack Success Rate after implementing Patronus would demonstrate its generalizability and robustness.

[c0] https://arxiv.org/abs/2311.17516

2. The related work section overlooks a key contribution, [c1], which fine-tunes a text decoder to defend against adversarial prompts. In contrast to this approach, the present work focuses on fine-tuning the image decoder. A discussion of [c1] should be included in the related work, highlighting this crucial difference in methodology and its potential implications. This comparison will provide a more complete context for the current work and clarify its unique contribution.
 
[c1] https://arxiv.org/pdf/2403.01446

3. Lacking comparison with close-related baselines.  The authors assert that existing content moderators "can be easily removed by white-box adversaries" and have other drawbacks (lines 037-041). However, it's unclear how effectively the proposed "Patronus" addresses these issues. To demonstrate its superiority, it is recommended that the authors compare their method with existing content moderation solutions such as LlamaGuard [c2] and OpenAI-Moderation [c3]. This comparison would highlight the effectiveness of Patronus in overcoming these challenges.

[c2] https://arxiv.org/abs/2312.06674

[c3] https://arxiv.org/pdf/2208.03274

4. Limited Evaluation Metrics: Relying solely on "CLIPScore" offers an incomplete view of the proposed solution's performance. The error bar in Figure 3 nearly obscures the performance gain. A more comprehensive evaluation should include metrics such as Attack Success Rate (ASR), Area Under the Receiver Operating Characteristic Curve (AUROC), and False Positive Rate (FPR), as recommended in [c1], [c2], and [c3]. These metrics provide a clearer understanding of the solution's impact on both normal and unsafe use cases.

5. Absence of Adaptive Attack Evaluation: The evaluation doesn't consider the critical scenario of adaptive attacks, where the attacker possesses full knowledge of both the T2I model and the implemented safeguards. Evaluating the defense's resilience against adaptive attacks is crucial for understanding its real-world effectiveness. The authors can follow the adaptive attack design principles suggested in [c4] to conduct experiments.

[c4] Tramer, F., Carlini, N., Brendel, W., & Madry, A. (2020). On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, 33, 1633-1645.

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732283018841

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response

comment:
Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely thank all four reviewers for their constructive comments and insightful questions, which helped us refine our work. Reviewers have acknowledged the impact and novelty of our work and the comprehensive analysis.

**[Problem Importance]**:

+ **Reviewer My5p**: This paper addresses a crucial safety issue: defending T2I models against white-box adversaries. As T2I models become more popular and widely deployed, this topic is increasingly important.
+ **Reviewer k1Hz**: This paper aims to solve a challenging problem. For an open-source model, malicious fine-tuning to remove the safety alignment is a key challenge.
+ **Reviewer RpMp:** The paper addresses a timely issue—preventing unsafe content generation.

**[Novelty]:**

+ **Reviewer My5p:** The paper presents a novel and intriguing idea.
+ **Reviewer RpMp:** I appreciate the novel ideas, such as fine-tuning the safety filter within the decoder to act as a conditional decoder and the effective strategy of transforming the min-max optimization into min-min in Equation 23. Both are technically sound and effective solutions.

**[Comprehensive Analysis]:**

+ **Reviewer k1Hz:** Evaluation is conducted on multiple datasets, and multiple baseline alignment methods are compared.

During the response period, we carefully try our best to provide feedback and conduct supplementary experiments to all comments from reviewers. We concisely summarize our responses to general concerns here (For details and more questions, please refer to rebuttals below):

+ **[Evaluation]:** We conduct supplementary experiments regarding seven new evaluation metrics and one SOTA attack.
+ **[Working Scenario]:** We give detailed explanation of our working scenario, i.e., defending white-box adversary, by comparing our defense with the moderator-based baselines and other alignment-based baselines.
+ **[Method Illustration]:** We clarify some key components of our defense, e.g., Non-Fine-Tunability Enhancement, Feature Space Calibration, smoothed denial-of-service loss, simulated fine-tuning set $\phi$ by providing more intuition, detailed explanation, and relationships with existing related work.

Thank you all again for your comments, and we hope our response can address your concerns. If you have any other concerns, please let us know~

Best regards,
Author #14032
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Patronus, a defense framework for text-to-image (T2I) models designed to prevent unsafe content generation. The proposed solution embeds a safety filter directly within the decoder and employs a non-fine-tunable learning scheme, making it resistant to malicious fine-tuning. The authors validate their framework on several datasets, comparing it to baseline methods, and highlight its effectiveness in mitigating unsafe content generation while maintaining performance on benign prompts.

Overall, the paper studies a timely and significant topic of mitigating unsafe content generation in T2I models. It proposes a novel solution to the problem. Experiments include multiple datasets and adversarial attacks, demonstrating the effectiveness of the method. However, this paper has weaknesses in the lack of clarity in the experimental settings and use of the CLIP score to measure the defense capability. Those concerns have not been fully addressed after author responses.

Although the paper shows promise and addresses an important research gap, further improvement and more comprehensive evaluations are needed to strengthen its contributions and make it suitable for publication. Therefore, the AC would recommend rejection.