SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peiran Dong $^{1}$  Song Guo $^{2*}$  Junxiao Wang $^{3*}$  Bingjie Wang $^{1}$
Jiewei Zhang<sup>1</sup> Ziming Liu<sup>1</sup>
$^{1}$ Hong Kong Polytechnic University  $^{2}$ Hong Kong University of Science and Technology
$^{3}$ King Abdullah University of Science and Technology & SDAIA-KAUST AI
{peiran.dong,bingjie.wang,jiewei.zhang,ziming.liu}@connect.polyu.hk
songguo@cse.ust.hk
junxiao.wang@kaust.edu.sa
Generative models produce unbounded outputs, necessitating the use of refusal techniques to confine their output space. Employing generative refusals is crucial in upholding the ethical and copyright integrity of synthesized content, particularly when working with widely adopted diffusion models. "Concept negation" presents a promising paradigm to achieve generative refusals, as it effectively defines and governs the model's output space based on concepts, utilizing natural language interfaces that are readily comprehensible to humans. However, despite the valuable contributions of prior research to the field of concept negation, it still suffers from significant limitations. The existing concept negation methods, which operate based on the composition of score or noise predictions from the diffusion process, are limited to independent concepts (e.g., "a blonde girl" without "glasses") and fail to consider the interconnected nature of concepts in reality (e.g., "Mickey mouse eats ice cream" without "Disney characters"). Keeping the limitations in mind, we propose a novel framework, called PROTORE, to improve the flexibility of concept negation via test-time negative concept identification along with purification in the feature space. PROTORE works by incorporating CLIP's language-contrastive knowledge to identify the prototype of negative concepts, extract the negative features from outputs using the prototype as a prompt, and further refine the attention maps by retrieving negative features. Our evaluation on multiple benchmarks shows that PROTORE outperforms state-of-the-art methods under various settings, in terms of the effectiveness of purification and the fidelity of generative images.
The family of diffusion models [1, 2] has achieved remarkable performance in image synthesis [3, 4, 5]. Recent advancements in text-conditional diffusion models [6, 7, 8, 9, 10] have further improved the ability to generate images with precise control over their content. In text-conditional diffusion models, text prompts $^{2}$  are used as input during the diffusion process to guide the creation of images that align with the desired content. Glide [6] and Semantic Diffusion Guidance (SDG) [7] have explored the use of pre-trained vision-language models like CLIP [11] to encode text conditions into latent features. Latent diffusion [8], including its large-scale implementation (Stable Diffusion), efficiently leverages and expands the design of latent vectors throughout the denoising process, where convolutional neural networks and cross-attention mechanisms merge multi-modal latent features.
*Corresponding authors: Song Guo, Junxiao Wang.
2Text prompts can take various forms, such as sentences, phrases, or single words, and serve as descriptions of desired image content. They provide a natural language interface to specify image attributes like style, color, and texture, enabling the generation of objects, scenes, and abstract concepts.
Text-conditional diffusion models such as DALLE-2 [9] and Imagen [10] have unlocked the potential of generative models for various business applications with exceptional visual fidelity.
Nonetheless, the use of large-scale, web-scraped datasets like LAION [10, 12, 13] has raised ethical concerns among researchers. These unedited datasets often contain inappropriate and unauthorized content [12], posing risks. Users can manipulate text prompts to generate violent, pornographic, or copyright-infringing images, which can damage the reputation of the business model provider and lead to serious societal issues [14, 15, 16]. To address concerns about negative concepts generated by diffusion models, there is a growing interest in developing selective refusal techniques to confine the model's output space. Recently, researchers have been working on four approaches to reduce the generation of harmful content: filtering the dataset [6, 17], adversarial perturbations [16, 18, 19, 20], machine unlearning [21, 22], and implementing refusals during inference [23, 24]. Dataset filtering and perturbation-based methods primarily focus on preventive measures, making them less suitable for pre-trained well-established models. Unlearning-based methods often require modifications to global model parameters, which can limit scalability and hinder plug-and-play deployment capabilities. In contrast, refusals during inference time involve modifying the output of pre-trained models, making them more efficient for testing and deployment scenarios. Concept negation [22, 25] plays a crucial role in implementing such refusals by allowing the model to define and control its output space using language-based and human-understandable concepts. The current methods for concept negation perform on the composition of score or noise predictions from the diffusion process. However, these methods have limitations as they are confined to independent concepts and do not account for the interdependency of concepts in real-world scenarios.
Our Contributions. To address the aforementioned challenges, we propose a novel framework called PROTORE (Prototypical Refinement). Our approach enhances the flexibility of concept negation by introducing test-time negative concept identification and feature space purification. The PROTORE framework leverages CLIP's language-contrastive knowledge and follows a "Prototype, Retrieve, and Refine" pipeline. Here is a breakdown of the three steps involved: 1) Prototype: We utilize CLIP to encode a collection of text prompts obtained from social media platforms that express similar negative concepts. These encoded features are then aggregated into a comprehensive prototype feature, capturing the semantics of the negative concepts. 2) Retrieve: The negative prototype feature serves as a prompt to retrieve the model's output features that are correlated with the negative concepts. 3) Refine: We employ the retrieved negative features to refine the discriminative attention maps, purifying the influence of negative concepts in the feature space. By integrating these steps, our PROTORE framework offers a novel approach to concept negation, improving the flexibility and effectiveness of mitigating negative concepts in generative diffusion models. Moreover, this approach promotes scalability and enables easy deployment. Through comprehensive evaluations on multiple benchmarks, we demonstrate that PROTORE surpasses existing methods in terms of purification effectiveness and the fidelity of generated images across various settings.
Recently, researchers have focused on addressing the generation of harmful content by exploring four approaches: dataset filtering [6, 17], adversarial perturbations [16, 18, 19, 20], machine unlearning [21, 22], and refusals during inference [23, 24].
Dataset Filtering. One straightforward approach to mitigate undesired image outputs in generative models is to filter out specific images from the training dataset. This can involve excluding certain image categories [17], such as those featuring people [6], or meticulously curating the data. However, dataset filtering has the drawback of being a costly solution to address issues discovered post-training, as retraining large models demands substantial resources.
Adversarial Perturbations. Another promising approach to protect images from being generated by large-scale generative models involves adding adversarial perturbations to raw images before uploading them on the internet or utilizing them in text-to-image AI systems like DALLE-2 and Stable Diffusion. While deep learning architectures are vulnerable to adversarial perturbations, previous research has primarily focused on their application in classification tasks [26, 27, 28]. Kos et al.[18] was the first to generate adversarial perturbations against deep generative models such as the variational autoencoder (VAE) and VAE-GAN. Liang et al.'s AdvDM [16] introduces adversarial perturbations into raw images by estimating the "attack vector" through Monte Carlo estimation.
Figure 1: A diagram illustrating the logical relationship between negative concepts and benign concepts in ICN (left) and DCN (right). The generative space, representing the regions where diffusion models can generate valid samples, is visualized: the beige area represents valid samples conditioned on benign concepts, the green area represents valid samples conditioned on negative concepts, and the red area represents the overlap between these two spaces. In contrast to prior research [25, 31], we aim to remove the red area from the beige region.
Glaze [19] and PhotoGuard [20] apply perturbations that cause the model to confuse the perturbed image with an unrelated image or an image with a different artistic style. Similar to dataset filtering approaches, these methods typically adopt preventive measures to mitigate risks.
Machine Unlearning. The approach of machine unlearning [29, 30] involves the removal of specific knowledge from pre-trained models, including diffusion models. In their work, Moon et al.[21] tackled the challenge of unlearning a particular feature, such as a hairstyle, from pre-trained generative models. They devised an implicit feedback mechanism to identify the latent representation associated with the target feature and facilitate unlearning in the generative model. Gandikota et al.[22] proposed a method for fine-tuning diffusion model weights to eliminate specific concepts while minimizing interference with other concepts. In contrast to these approaches that globally modify model weights to unlearn undesirable outputs, our approach focuses on purifying negative outputs during inference, offering scalability and plug-and-play deployment capabilities.
Refusals during Inference Time. Previous studies have investigated methods of implementing refusals at inference time, as these methods are efficient for testing and deployment purposes. The core idea behind these approaches involves modifying the output after training using classifiers [23] or incorporating guidance into the inference process [24]. In contrast, our approach to refusal at inference time distinguishes itself by focusing on "concept negation" with a grounding in natural language. This enables defenders to easily specify normal and negative concepts in a text-conditional space, thereby enhancing the method's availability.
3 Concept Negation (NOT)
Preliminaries. A recent related work to our approach is the Composable Diffusion Models (CDM) [25], which provides an understanding of the diffusion model from the Energy-Based Model (EBM)'s perspective [31] and demonstrates how the additivity property of the EBM can be applied to diffusion generation. In this context, the generation process and scoring function of the diffusion model are referred to as  $p_{\theta}^{i}(x_{t - 1}|x_{t})$  and  $\epsilon_{\theta}^{i}(x,t)$ , respectively. If we consider a single score function in a diffusion model as the learned gradient of the energy function in an EBM, the combination of diffusion models results in a score function denoted as  $\sum_{i}\epsilon_{\theta}^{i}(x,t)$ . Consequently, the generative process for combining multiple diffusion models can be expressed as follows:
$$
p _ {\mathrm {C D M}} \left(x _ {t - 1} \mid x _ {t}\right) = \mathcal {N} \left(x _ {t} + \sum_ {i} \epsilon_ {\theta} ^ {i} \left(x _ {t}, t\right), \sigma_ {t} ^ {2}\right). \tag {1}
$$
CDM aims to generate images based on a given set of concepts  $\{c_1, c_2, \ldots, c_n\}$ . To achieve this, each concept  $c_i$  is represented as an individual diffusion model, and their score or noise predictions from the diffusion process are combined to generate the desired image. Drawing inspiration from EBMs, CDM introduces two combinatorial operators, namely conjunction (AND) and negation (NOT), to facilitate the combination of diffusion models. In the context of concept negation (NOT), consider the example of a user prompt "a blonde girl without glasses". In this case, the condition "a
Figure 2: Method Overview. Top: The diffusion process entangles negative concepts, where cross-attentions align visual and textual embeddings, resulting in the concealment of negative concepts within attention maps (green box). Bottom: Our approach comprises two main steps: ① Generating negative prototype prompts using the CLIP text encoder. ② Refining the discriminative attention features by incorporating retrieved negative features, effectively purifying the influence of negative concepts in the feature space.
blonde girl" represents the benign concept  $c_{i}$  that should be present in the generated images, while the text "glasses" represents the negative concept  $c_{j}$  whose score or noise predictions need to be subtracted from the diffusion process. CDM's approach can combine pre-trained diffusion models within inference time without any additional training.
Motivation. Although CDM shows promise, it makes a significant assumption that the negative concept  $c_{j}$  is independent of the benign concept  $c_{i}$ , which limits its flexibility in concept negation. For instance, consider a scenario where an image of "a Mickey Mouse eating ice cream" needs to be generated with "Mickey Mouse" or the broader condition "Disney character" as the negative concept. CDM would face challenges in handling such a situation. In our study, we propose a new taxonomy of concept negation, classifying CDM-like approaches as "independent concept negation" (ICN), while our work falls under the category of "dependent concept negation" (DCN). Figure 1 provides an intuitive illustration of the relationship between the ICN and DCN.
Problem Definition. Given a user prompt  $c$  and a certain negative concept  $\tilde{c}$ , we aim to generate high-quality images  $x$  describing  $c$  with the absence of  $\tilde{c}$ . If we view concept negation in diffusion models as a probabilistic instantiation of logical operators applied to concepts. Formally, ICN factorizes the conditional generation as the following composed probability distribution:
$$
\operatorname {I C N}: p (\boldsymbol {x} \mid \boldsymbol {c}, \text {n o t} \tilde {\boldsymbol {c}}) \propto p (\boldsymbol {x}, \boldsymbol {c}, \text {n o t} \tilde {\boldsymbol {c}}) \propto p (\boldsymbol {x}) \frac {p (\boldsymbol {c} \mid \boldsymbol {x})}{p (\tilde {\boldsymbol {c}} \mid \boldsymbol {x})}. \tag {2}
$$
However, the formula (2) holds only when  $c$  and  $\tilde{c}$  are independent. Considering a realistic scenario, DCN formulates a more general conditional generation:
$$
\text {D C N}: p (\boldsymbol {x} | \boldsymbol {c}, \text {n o t} \tilde {\boldsymbol {c}}) \propto p (\boldsymbol {x}, \boldsymbol {c}, \text {n o t} \tilde {\boldsymbol {c}}) \propto p (\boldsymbol {x}) \frac {p (\boldsymbol {c} | \boldsymbol {x})}{p (\tilde {\boldsymbol {c}} | \boldsymbol {c} , \boldsymbol {x})}. \tag {3}
$$
The natural ability of EBM [31, 32] and diffusion counterparts [25] to perform set-like composition through arithmetic on score or noise predictions would no longer hold when it comes to DCN. This implies that "A and not B" cannot be simply treated as the difference between log probability densities for A and B.
Let  $\mathcal{I}$  be an image generated using a diffusion model  $\psi(\cdot)$  based on a user prompt  $c$ . The prompt may include up to  $K$  pre-defined negative concepts  $\tilde{c}_k, k = 1,2,\dots,K$ , and our objective is to intervene in the image generation process to remove these concepts from  $\mathcal{I}$ . For example, if the prompt contains
"Mickey Mouse is eating ice cream", we may need to exclude the copyrighted content of "Mickey Mouse" to avoid legal issues while preserving other elements like "ice cream". This is crucial to prevent potential legal complications arising from using copyrighted or sensitive content. However, unlike previous approaches such as ICN [31, 25] and inpainting [8], we cannot rely on user-defined prompts or masks to determine where to remove or reconstruct. One intuitive approach is to create a list of prohibited words and remove them from the user's prompt whenever they appear. However, this approach has limitations, as synonyms may convey the same concept, and the list may not cover all possible negative expressions. Additionally, it may impede the normal use of the model. To overcome these challenges, we propose a plug-and-play concept negation method that enables text-conditional refusals for diffusion models during inference. The proposed method is illustrated in Figure 2.
Prototype Prompt Generation. To encode text prompts, we utilize the CLIP (Contrastive Language-Image Pre-training) text encoder  $\varphi(\cdot)$  [11], which is a state-of-the-art model of vision-language representation learning. CLIP is pre-trained on a vast corpus of text and images using a contrastive loss function, which encourages the model to map semantically similar text and images to nearby points in a shared embedding space. However, explicitly enumerating all possible prompt words of negative concepts  $\tilde{c}_k$  can be challenging. Instead, we crawl a set of text prompts  $\tilde{\mathcal{C}}_k$  expressing similar negative concepts from social media platforms, aiming to elicit the  $k$ -th negative concept in the generated images. We then rely on the zero-shot classification's capability of CLIP to encode the negative text prompts into multiple high-dimensional features. These prompts are then aggregated (clustered) into a single prototype prompt  $\tilde{c}_k^*$  representing the respective negative class:
$$
\tilde {\boldsymbol {c}} _ {k} ^ {*} = \operatorname {c l u s t e r} \left(\varphi \left(\tilde {\mathcal {C}} _ {k}\right)\right). \tag {4}
$$
We employ three distinct clustering methods to derive the prototype prompts. In the case of single-label to single-class refusals (e.g., ImageNet), we utilize the embedding of the corresponding label as the prototype prompt. For multiple-labels to multiple-class refusals (e.g., I2P datasets), the clustering center is computed using K-means, which then serves as the prototype prompt. As for multiple dependent concept refusals (as depicted in Figure 4), we combine all the concepts using commas or the word "and" to form the prototype prompt.
Prior studies on image editing [33] have shown that diffusion models utilize cross-attention layers to combine visual and textual features, resulting in the generation of spatial attention maps for each textual token. As shown in the top of Figure 2, the visual features of a noisy image  $z_{t}$  are projected to a query matrix via a linear layer  $\ell_{Q}(\cdot)$ , and  $Q = \ell_{Q}(z_{t})$ . Similarly, the textual feature  $\varphi (\pmb {c})$  is projected to a key matrix  $K = \ell_{K}(\varphi (\pmb {c}))$  and a value matrix  $V = \ell_{V}(\varphi (\pmb {c}))$  through linear layers  $\ell_K(\cdot),\ell_V(\cdot)$ . The attention maps  $M$  are then calculated as follows:
$$
M = \operatorname {S o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d}}\right) = \left[ \operatorname {a t t n} _ {1}, \operatorname {a t t n} _ {2}, \dots , \operatorname {a t t n} _ {l}, \dots , \operatorname {a t t n} _ {L} \right] \in \mathbb {R} ^ {C \times (H \times W) \times L}, \tag {5}
$$
where  $d = H \times W$  represents the dimension of the key and value projection layers.  $C$  and  $L$  denote the number of attention heads and tokens in a single sequence, respectively. Channel  $\mathrm{attn}_l \in \mathbb{R}^{C \times H \times W}$  can be squared and visualized as the  $l$ -th token in the text prompt  $c$ . As a result, negative concepts in  $\mathrm{attn}_l$  may appear in the weighted output features  $A = M \cdot V$ . We have noticed that negative concepts are frequently intertwined with normal concepts in the attention maps, which may explain why EBM and CDM are not as efficient. Furthermore, we lack prior knowledge about the user prompt, including which negative class it belongs to and where the negative token is located. This makes it difficult to use existing image editing methods that depend on explicit guidance signals for concept negation.
Attention Feature Refinement. We employ the prototype prompt generated using Equation (4) to retrieve the negative concepts present in the output features. Subsequently, using Equation (5), we calculate the negative attention maps for the prototype feature  $\tilde{c}_k^*$ :
$$
\widehat {M} ^ {k} = \operatorname {S o f t m a x} \left(\frac {Q \widehat {K} ^ {T}}{\sqrt {d}}\right) = \operatorname {S o f t m a x} \left(\frac {\ell_ {Q} (z _ {t}) \ell_ {K} (\tilde {\mathbf {c}} _ {k} ^ {*}) ^ {T}}{\sqrt {d}}\right). \tag {6}
$$
Intuitively,  $\widehat{M}^k$  highlights the  $k$ -th negative concepts present in the noisy image  $z_{t}$ . We proceed to obtain the negative features, denoted as  $\widehat{A}^k$ , through  $\widehat{A}^k = \widehat{M}^k \cdot \widehat{V}^k$ , where  $\widehat{V}^k = \ell_V(\tilde{\mathbf{c}}_k^*)$ . Finally, to eliminate the identified negative features, we compute the refined attention features  $A^*$  by:
$$
A ^ {*} = A - \sum_ {k} \sigma_ {k} \cdot \widehat {A} ^ {k}, \tag {7}
$$
Figure 3: Understanding the mechanism behind PROTORE's success. By using user prompts (represented by the yellow arrow) and prototype negative prompts (represented by the green arrow), we guide the diffusion process of Gaussian noise toward the intended space. We analyze the effectiveness of attention feature refinement in two common generation scenarios: benign generation and unsafe generation. These scenarios are determined by the relationship between user prompts and negative concepts. Our proposed attention feature refinement technique (indicated by the red dashed arrow) successfully redirects the diffusion process away from the negative space, ensuring convergence towards the normal space in both scenarios.
where coefficient  $\sigma_{k}$  controls the refinement step size.
The resulting attention feature refinement progress is intuitively visualized in Figure 3. In benign image generation scenarios (a) and (b), user prompts remain dissociated from negative concepts. We can observe that the refined diffusion process (indicated by the red dashed arrow) gradually converges to normal space, thereby affirming that attention feature refinement does not interfere with the generation of user-specified images. In contrast, the user prompts associated with unsafe image-generation scenarios involve negative concepts. In such cases, refinement of the diffusion process leads to a gradual shift away from the negative space, facilitating the effective negation of undesirable concepts from the generated images. In order to facilitate comprehension, two specific cases are examined in (b) and (d) where user prompts and negative concepts are either opposite or identical, respectively. Sub-figure (b) depicts a scenario in which the angle between the user and negative guidance is approximately 180 degrees. Under this circumstance, attention feature refinement modifies the diffusion process by increasing the step size along the user prompt direction, with no disturbance to image generation. In contrast, in sub-figure (d), the angle between the user and negative guidance approaches zero degrees. In this case, attention feature refinement directs the diffusion process away from negative guidance along the user guidance or in the opposite direction. Our intuition is verified by the experimental results and the discussion of method limitations.