Dark Miner: Defend against unsafe generation for text-to-image diffusion models

Authors: Zheling Meng, Bo Peng, Xiaochuan Jin, Yue Jiang, Jing Dong, Wei Wang, Tieniu Tan
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Text-to-Image Diffusion Models, Unsafe Generation, Concept Erasure

Abstract:
Text-to-image diffusion models have been demonstrated with unsafe generation due to unfiltered large-scale training data, such as violent, sexual, and shocking images, necessitating the erasure of unsafe concepts. Most existing methods focus on modifying the generation probabilities conditioned on the texts containing unsafe descriptions. However, they fail to guarantee safe generation for unseen texts in the training phase, especially for the prompts from adversarial attacks. In this paper, we re-analyze the erasure task and point out that existing methods cannot guarantee the minimization of the total probabilities of unsafe generation. To tackle this problem, we propose Dark Miner. It entails a recurring three-stage process that comprises mining, verifying, and circumventing. It greedily mines embeddings with maximum generation probabilities of unsafe concepts and reduces unsafe generation more effectively. In the experiments, we evaluate its performance on two inappropriate concepts, two objects, and two styles. Compared with 6 previous state-of-the-art methods, our method achieves better erasure and defense results in most cases, especially under 4 state-of-the-art attacks, while preserving the model's native generation capability. Our code can be found in Supplementary Material and will be available on GitHub.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730551585342

thread_type:
official_review

<thread_object>
creator: reviewer_tydk

<content>
summary:
This paper focuses on the model editing task of the text-to-image diffusion model. To address the challenges in erasing all unsafe prompts, this paper proposes a method called Dark MINER. This method consists of three steps: 1) mining the potential embeddings related to the unsafe images, 2) assessing whether the potential embeddings effectively induce the model to generate unsafe images, 3) if the mined embedding is effective, this paper conducts the erasing process. In step 3, to protect the generation of safe concepts, this paper also incorporates the regularization in three kinds of concepts: a predefined anchor concept, a null concept, and a concept '-c' that is defined as 'unsafe embedding * -1'.

strengths:
- The paper presents a clear logical flow and is well-written.
- Experiments demonstrate the effectiveness of the proposed method.

weaknesses:
1. Please clarify the difference from below existed studies [1,2].
  
2. In Table.1, the attack success rate of UnlearnDiff in the Violence concept is still 79%. Please provide an explanation for this. Similar phenomenon appear in P4D with the Violence concept (ASR is 46%) and the Church concept (ASR is 49).
  
3. In Eq.8, this paper proposes three regularization terms to protect the generation of safe concepts, but lacks of related ablation experiments to assess the effect of these three terms.

[1] RACE: Robust Adversarial Concept Erasure for Secure Text-to-Image Diffusion Model, ECCV 2024

[2] Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models, ECCV2024

questions:
Please help to address weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730592616643

thread_type:
official_review

<thread_object>
creator: reviewer_suxv

<content>
summary:
This work proposes, Dark Miner, an approach designed to address unsafe content generation in text-to-image diffusion models. Unlike existing methods that mainly adjust generation probabilities for known unsafe textual inputs, Dark Miner emphasizes minimizing unsafe generation probabilities for unseen or adversarial prompts. This is achieved through a recurring three-stage process: mining embeddings with high probabilities for unsafe content, verifying them, and circumventing unsafe generation pathways. Some experimental results demonstrate that Dark Miner outperforms six state-of-the-art methods in erasing and defending against unsafe content.

strengths:
1. The research topic in this paper is relevant to the community.
2. Organization of the paper is relatively clear, even not perfect.
3. Experimental details are clearly stated.

weaknesses:
1. The author claims that the following as one of the contributions:
“Based on the total probabilities, we analyze the reason why existing methods cannot completely erase concepts for text-to-image diffusion models and are vulnerable to attacks.”
I did not find a (sub)section for this part, although some discussion can be found in some paragraphs.
The reason cannot be found in conclusion section either.

2. Missing related work: The proposed Dark Miner aims to emphasize unsafe generation for unseen or adversarial prompts. One of the previous work [1] also handle unseen or adversarial harmful prompts via building a robust detection space, which is missing in the related work.

[1] Liu, Runtao, et al. "Latent guard: a safety framework for text-to-image generation." ECCV, 2024.

3. Experiments: Previous work show their effectiveness on many harmful concepts. This work only conducts experiments on two. The generalisation of the proposed approach remain to be further verified. The previous approach also shows very different formance on different harmful concepts.

4. The proposed approach cannot handle prompts to generate biased images, e.g. gender bias? Bias is also a harmful concept in responsible text-to-image generation [2,3].

[2] Li, Hang, et al. "Self-discovering interpretable diffusion latent directions for responsible text-to-image generation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[3] Friedrich, Felix, et al. "Fair diffusion: Instructing text-to-image generation models on fairness." arXiv preprint arXiv:2302.10893 (2023).

Minor issues:
- The citation format is incorrect across the paper.
- Confusing annotation, e.g. 0c in Equation 8

questions:
1. The paper claims that existing methods fail to guarantee safe generation for unseen texts in the training phase. Does the proposed Dark Miner can provide such guarantee? If so please provide more details or discussion regarding this.

2. In the mining stage of DM2, can DM2 obtain the novel harmful prompts? E.g. this work shows that text-to-image models also suffer from Multimodal Pragmatic Jailbreak prompts [1]. Can the Jailbreak prompts also be learned by the proposed DM2? Some discussion about the limitation of the mining stage will be helpful to the readers.

[1] Liu, Tong, et al. "Multimodal Pragmatic Jailbreak on Text-to-image Models." arXiv preprint arXiv:2409.19149 (2024).

3. How the proposed approach performs if the system is black-box? Is it still feasible?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661526431

thread_type:
official_review

<thread_object>
creator: reviewer_orfd

<content>
summary:
This paper proposed Dark Miner to eliminate unsafe content generation of T2I models. It searches and verifies the embeddings which contain the unsafe concept and reduces unsafe generation by adjusting LoRA adapter of T2I models.
The paper mainly made following contributions:
1.	Point out the previous methods fail to avoid unsafe generation on out-of-distribution prompts and easily being tricked by attacking methods.
2.	Propose Dark Miner which includes three stage to reduce the optimal embeddings related to unsafe concept and maintain generation ability on benign prompts.
3.	Evaluate efficient of proposed compared with 6 SOTA methods and conduct 4 SOTA attacks.

strengths:
Authors point out the limitation of existing method and provide theoretical analysis.
Authors propose an iterative scheme to mine the text c that with maximum likelihood to generate unsafe content while previous method usually predefined such text c.   
Authors propose a method to avoid overly concept removal by verifying the embedding before circumventing. Authors apply CLIP model to extract delta features from reference image and generated image, then a new metric based on cosine similarity of delta features.
Authors make effect to test proposed method against 4 different attacks.

weaknesses:
In section 3.2.2, authors use prompt “a photo” to generate reference image and compare the distance with target image generated by mined embedding.  However, due to the randomness of reference image, the difference between reference image and target image is always high. In an extreme case, when there are benign images or non-explicit sexual related images in the image pool, the verifying and circumventing will be affected.
 
And the prompt of target image has no relation with prompt of reference image. Another potential drawback is that, all the concepts from “unsafe” prompt are shifted away to random direction without guidance. Therefore, the image generated by “unsafe” prompt will not maintain any semantic information from its prompt even there are safe content expressed by “unsafe” prompt. The generation will be random which might degrade the utility of model.

In evaluation metrics, author use the mean classification score to evaluate inappropriateness by NudeNet, however, author didn’t mention what classes and threshold used in their implementation. And it is better if authors could also show the number of images being classified as inappropriate image instead of classification score.

For the CLIP score, the proposed method has relative low performance compared with other SOTA methods.

questions:
What classes are used in NudeNet classifier and threshold?
SLD and ESD original paper test their FID score on COCO-30k which is from COCO 2024 validation dataset, why authors didn’t use the same one?
According to another relevant paper in the task of nudity elimination as following: Li, Xinfeng, et al. "SafeGen: Mitigating Unsafe Content Generation in Text-to-Image Models." arXiv preprint arXiv:2404.06666 (2024). In the Table 3, they also give the FID score on COCO 2017 validation dataset, achieving 20.36 and 20.31 on ESD and their proposed method SafeGen, which is better than the author’s implementation. We suggest author should revisit their FID calculation on ESD on COCO 2017.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722998101

thread_type:
official_review

<thread_object>
creator: reviewer_n56b

<content>
summary:
This paper introduces Dark Miner for mitigating unsafe content generation in text-to-image diffusion models.  Dark Miner involves a recurring three-stage process of mining, verifying, and circumventing. It's designed to iteratively identify and suppress unsafe content generation. Comprehensive experiments demonstrate the superior performance of Dark Miner.

strengths:
1. This paper studies a critical area. The capability of these text2image models enables unauthorized usage of these models to generate harmful or disturbing content.
2. Experiments exhibit good overall performance.

weaknesses:
1. The performance of Dark Miner is largely limited by the image pool related to c. That is to say, a lot of text that leads to concept c, which is not linked to the images in the pool, will not identified by the mining step.
2. Questionable performance by CLIP in Section 3.2.2. Can you discuss how well CLIP performs this task, as this is critical to your methods, I believe this is an important part of the ablation study. 
3. Also, CLIP is used in both the model to be erased (i.e. SD) and Section 3.2.2 to identify the concept. However, texts that circumvent the defense will lead to images that cannot be identified by the CLIP.

4. Methods are only evaluated on two SD models. Will this method generalize well beyond the SD families?
5. Lack of ablation on the three steps of Dark Miner. For instance,
    * how well does the method perform with and without the verifying step?
    * Ablation over the size of the image pool
    * Ablation over different parameters and configurations of optimizing for the embeddings.

questions:
Please see the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732076306069

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We sincerely appreciate the time and the reviews of our manuscript by all the reviewers.

In pursuit of safer T2I generation, we analyze the limitations and underlying issues of existing erasure methods in defending against attacks and propose a new framework to address these challenges. We thank the reviewers for their acknowledgment of our methods (suXv), experiments (oRfd, suXv), and writing (oRfd, TYdk). 

We have thoroughly addressed all raised concerns in our rebuttal, and encourage you to review the individual responses for further details. In addition, the manuscript has been revised according to the suggestions of the reviewers, with clear markings indicating the revisions.
</content>
</thread_object>

</thread>


Metareview: