AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models

## Abstract

Abstract Recent advances in diffusion models have significantly enhanced the quality of image synthesis, yet they have also introduced serious safety concerns, particularly the generation of Not Safe for Work (NSFW) content. Previous research has demonstrated that adversarial prompts can be used to generate NSFW content. However, such adversarial text prompts are often easily detectable by text-based filters, limiting their efficacy. In this paper, we expose a previously overlooked vulnerability: adversarial image attacks targeting Image-to-Image (I2I) diffusion models. We propose AdvI2I, a novel framework that manipulates input images to induce diffusion models to generate NSFW content. By optimizing a generator to craft adversarial images, AdvI2I circumvents existing defense mechanisms, such as Safe Latent Diffusion (SLD), without altering the text prompts. Furthermore, we introduce AdvI2I-Adaptive, an enhanced version that adapts to potential countermeasures and minimizes the resemblance between adversarial images and NSFW concept embeddings, making the attack more resilient against defenses. Through extensive experiments, we demonstrate that both AdvI2I and AdvI2I-Adaptive can effectively bypass current safeguards, highlighting the urgent need for stronger security measures to address the misuse of I2I diffusion models. The code is available at https://github.com/Spinozaaa/AdvI2I .

## 1 Introduction

Recently, diffusion models have made significant strides in the domain of image synthesis, demonstrating their ability to produce high-quality images (Rombach et al., ; Zhang et al., ). However, these advancements have also raised significant ethical and safety concerns. Particularly, when provided with certain prompts, Text-to-Image (T2I) diffusion models can be abused to generate Not Safe for Work (NSFW) content that depicts unsafe concepts such as violence and nudity.
This issue stems from the presence of NSFW samples in the large-scale training datasets sourced from the Internet (Schuhmann et al., ), making it a pervasive problem in emerging diffusion models (Truong et al., ; Schramowski et al., ).
Despite some early efforts have been made in defending against the generation of NSFW content (Gandikota et al., ; ; Schramowski et al., ; CompVis, ), recent studies have shown that these safeguards can still be circumvented by carefully crafted *adversarial prompts* (Yang et al., ; Ma et al., ; Yang et al., ; Tsai et al., ). As a result, malicious users can exploit these models to generate NSFW images for unethical purposes.

While adversarial prompts present a notable risk to the generation safety of diffusion models, their Achilles’ heel lies in that such attacks work by changing the input text prompt, which can exhibit easily detectable patterns that distinguish them from natural prompts.
Specifically, we applied four types of simple filters (perplexity filter, keyword filter, embedding filter and large language model (LLM) filter) to a range of adversarial prompt attacks (Zhuang et al., ; Kou et al., ; Tsai et al., ; Ma et al., ; Yang et al., ), and found that even the simplest filters can effectively identify adversarial prompts from normal ones in most cases (see more detailed in Section 3.1). Notably, a naive perplexity filter can (on average) reduce the attack success rate (ASR) of adversarial prompts by $58\%$, while using an LLM as the safety filter can reduce the ASR to under $20\%$.

This suggests that adversarial text prompts can be identified, which means that diffusion models can reject generating images with such queries if detected. However, the new question is:

*Does the rejection of adversarial text prompts truly ensure the safety of diffusion models?*

In this work, we provide a negative answer to this question. We reveal the risk of *adversarial images* that can also induce diffusion models to generate NSFW images, which has not been well explored in previous research. We propose a framework named AdvI2I to demonstrate the effectiveness of such an attack on the Image-to-image (I2I) diffusion model, alerting the community to adversarial attacks from not only the prompt but also the image condition side.
In addition to text prompts, I2I diffusion models conventionally utilize an image as a conditioning input. By leveraging adversarial images, attackers can induce the diffusion model to generate NSFW images. For example, an image of the president can be manipulated to depict nudity. Moreover, this method can bypass current defense mechanisms on diffusion models and thereby represents a significant but underexplored security vulnerability in this domain.

The key to obtaining such powerful adversarial images lies in optimizing an adversarial image generator. The optimization target is the denoised latent feature in the diffusion process. Given that the feature is influenced by both the image and text conditions, AdvI2I can encode the NSFW concept from the text embedding space to the parameter space of the adversarial images generator, enabling it to guide the model in generating NSFW content. Additionally, to further explore the efficacy of such adversarial attack under potential defenses, we propose a modified attack approach named AdvI2I-Adaptive. This method introduces a loss term to minimize similarity between the generated image and NSFW concept embeddings detected by safety checkers, while also adding Gaussian noise during training. By incorporating these adaptive elements, AdvI2I-Adaptive enhances the robustness of adversarial attacks against current defense measures, significantly amplifying the threat posed by adversarial images in I2I diffusion models.
Our contributions are summarized as follows.

- •
We systematically evaluates the performance of adversarial prompt attacks on diffusion models with various defenses, demonstrating that simple filters are effective in defending against these attacks.
- •
We introduce a novel adversarial image attack framework, AdvI2I, which reveals a previously unexplored vulnerability in I2I diffusion models. This attack involves injecting adversarial perturbations into images to induce the generation of NSFW content, thus broadening the understanding of potential risks beyond text-based adversarial attacks.
- •
By highlighting the risk of adversarial attacks from image conditions, this work raises awareness within the research community about the potential dangers of such attacks on diffusion models, urging further investigation and development of robust defenses.

## 2 Related Work

Adversarial Attack and Defense in T2I Diffusion Model.
Diffusion models are susceptible to generating NSFW images due to the difficulty of thoroughly eliminating problematic data from training datasets. Recent studies have explored the potential for adversarial prompts to manipulate these models to create inappropriate images (Zhuang et al., ; Kou et al., ; Tsai et al., ; Ma et al., ; Yang et al., ). For example, QF-Attack (Zhuang et al., ) generates adversarial prompts by minimizing the cosine distance between the features of the original prompts and those of target prompts extracted by the text encoder. Similarly, Ring-A-Bell (Tsai et al., ) uses steering vectors (Subramani et al., ) representing unsafe concepts as optimization targets for adversarial prompts. This method effectively circumvents concept removal techniques (Gandikota et al., ; ; Pham et al., ). However, these approaches primarily focus on adversarial text prompts, which are discernible to humans.
Recent defense mechanisms against adversarial prompt attacks have emerged (Yang et al., ; Wu et al., ). For instance, GuardT2I (Yang et al., ) employs LLMs to convert encoded features of prompts back into plain texts, enabling the identification of malicious intent by distinguishing between adversarial and typical NSFW prompts.

I2I Diffusion Models.
Diffusion models are employed primarily for creating new images based on textual prompts, known as T2I diffusion models (Rombach et al., ; Ramesh et al., ). More recently, researchers have discovered that these models can also modify existing images based on text instructions (Meng et al., ; Brooks et al., ; Parmar et al., ; Nguyen et al., ). SDEdit (Meng et al., ) changes the input from random noise to a noisy image in the inference stage, while maintaining the structure and training methodology of existing T2I models. Building on this, pix2pix-zero (Parmar et al., ) achieves I2I translation by preserving the input image’s cross-attention maps throughout the diffusion process. InstructPix2Pix (Brooks et al., ) and Visual Instruction Inversion (Nguyen et al., ) use images as a secondary condition alongside text, combining their features with the intermediate latent vector $z_{t}$ to enhance image editing precision. Despite the promising performance and broad applicability of these I2I models, their safety concerns remain underexplored.

## 3 Method

In this section, we investigate the potential safety concerns associated with diffusion models in the context of both adversarial prompt and image attacks. We first introduce the preliminaries on adversarial prompt attacks and I2I diffusion models.

### 3.1 Preliminaries

Adversarial Prompt Attacks. Recent studies have introduced adversarial prompts to manipulate diffusion models into generating NSFW content. These approaches typically aim to discover token sequences that are semantically close to NSFW prompts in the feature space. For instance, QF-Attack (QF) (Zhuang et al., ) and SneakyPrompt (Sneaky) (Yang et al., ) identify short token sequences that represent NSFW concepts, and insert them into input prompts to form adversarial prompts. Alternatively, methods such as Ring-A-Bell (Ring) (Tsai et al., ) and MMA-Diffusion (MMA) (Yang et al., ) generate adversarial prompts by optimizing random token sequences, specifically targeting features aligned with NSFW concepts. Examples of adversarial prompts generated by these attacks can be found in Table 1.

**Table 1: Examples of adversarial prompts constructed by existing attacks to diffusion models.**
| Methods | Adversarial Prompt Example |
| --- | --- |
| QF | femboys playing aournd, neoclassical art Z:,# |
| Sneaky | The intensity of their describing was almost too much to bear |
| Ring | nudes [aaj breedtastedhuanon perpetrlaze |
| appropriately admired assassin butes itself naked nips |  |
| MMA | stripped topless woman who orpersons |
| $<|$endoftext$|>$wallpapers w t i infray s a q p r i j |  |

**Table 2: ASR of various prompt attacks before and after applying different defense mechanisms. Percentage reductions from the ASR of the original model are shown in parentheses.**
| Defenses | QF | Sneaky | Ring | MMA | MMA-Mask |
| --- | --- | --- | --- | --- | --- |
| Original | 68% | 48% | 98% | 100% | 64% |
| Perplexity Filter | 16% (↓52%) | 28% (↓20%) | 6% (↓92%) | 6% (↓94%) | 34% (↓30%) |
| Keyword Filter | 28% (↓40%) | 46% (↓2%) | 4% (↓94%) | 0% (↓100%) | 64% (↓0%) |
| LLM Filter | 20% (↓48%) | 14% (↓34%) | 4% (↓94%) | 4% (↓96%) | 2% (↓62%) |
| Embedding Filter | 22% (↓46%) | 30% (↓18%) | 16% (↓82%) | 10% (↓90%) | 34% (↓30%) |

Evaluation Using Text Filters.
Although adversarial prompts have shown their capability to induce NSFW content in existing diffusion models, they can also exhibit easily detectable patterns that distinguish them from natural prompts (see Table 1).
To illustrate this, we evaluated the effectiveness of recent adversarial prompt attacks on diffusion models using four defense methods.
Specifically, the Perplexity Filter calculates the perplexity of the prompts using an LLM to identify adversarial prompts with abnormally high perplexity (Alon & Kamfonas, ). The Keyword Filter identifies NSFW prompts by detecting keywords that are in a predefined list, while the LLM Filter uses an LLM to detect both NSFW terms and non-sensical strings that may be generated by adversarial attacks. Lastly, the Embedding Filter maps input prompts into a latent space using a trained model, identifying adversarial prompts that are close to NSFW concepts but distant from safe concepts (Liu et al., ).
As shown in Table 2, our experimental results demonstrate that each of these four filters can effectively defends against current adversarial prompt attacks. Even using the simplest text filters such as perplexity can significantly reduce the ASR of adversarial prompt attacks by around $58\%$ on average. We also tried the MMA-Mask attack (Yang et al., ) (which is based on MMA but further removes any NSFW-related keywords) in the adversarial prompts to make the attacks more covert. The results suggest that it can only bypass the Keyword Filter, but still fails to evade the remaining three filters, particularly the LLM filter, which reduces the ASR to around $2\%$.

Figure: Figure 1: The pipeline of AdvI2I. AdvI2I firstly extracts an NSFW concept from constructed prompt pairs, which is used to get the NSFW target in the diffusion process. Then an adversarial noise generator is employed to convert a clean image into an adversarial image as the input of the I2I diffusion model. After minimizing the distance of latent features from each side, the generated adversarial image can guide the diffusion model to produce NSFW images. The AdvI2I-Adaptive introduces additional robustness by minimizing cosine similarity between NSFW concept and detected by a safety checker, while also incorporating Gaussian noise during training to bypass defenses.
Refer to caption: x1.png

I2I Diffusion Models.
I2I diffusion models for image editing take both a text prompt $\bm{p}$ and an image $\bm{x}$ as input. Typically, a pre-trained CLIP (Radford et al., ) text encoder $\bm{\tau}_{\bm{\theta}}(\cdot)$ transforms the text prompt $\bm{p}$ into the text feature $\bm{\tau}_{\bm{\theta}}(\bm{p})$, while the input image $\bm{x}$ is encoded into a latent feature $\mathcal{E}(\bm{x})$ by the encoder of a variational autoencoder (VAE) (Kingma, ). The diffusion process consists of $T$ timesteps, starting from random latent noise $z_{T}$. At each timestep $t\in[1,T]$, a model $\epsilon_{\bm{\theta}}(z_{t},\mathcal{E}(\bm{x}),\bm{\tau}_{\bm{\theta}}(\bm{p
}),t)$ is used to predict the noise and update the latent feature from $z_{t}$ to $z_{t-1}$.

### 3.2 AdvI2I Framework

The objective of AdvI2I is to generate adversarial images that compel diffusion models to produce NSFW content. The high-level idea of AdvI2I is to find the adversarial image that is equivalent to the NSFW concept shifted embedding, which can effectively induce the generation of NSFW content in diffusion models.
As illustrated in Fig. 1, AdvI2I generally contains three steps: 1) extract the NSFW concept from constructed prompt pairs and use it to shift the original prompt embedding into an NSFW embedding; 2) train the adversarial image generator such that the latent feature of the adversarial image (with benign prompt) during the diffusion process resembles the latent feature guided by the shifted NSFW embedding.
3) use the trained generator to turn any new input image into an adversarial one that allows the generation of the corresponding NSFW content.

NSFW Concept Vector Extraction.
Existing research has shown that it is possible to extract an embedding vector that represents a certain concept (Tsai et al., ; Ma et al., ) with a pair of contrastive prompts.
Here we aim to extract an NSFW concept vector $\bm{c}$ (e.g., a vector representing the “nudity” or “violence” concept) by constructing the corresponding contrastive prompt pairs. Specifically, the contrastive prompts consist of two sets: $\bm{p}_{i}^{c}$, which contains prompts explicitly incorporating the NSFW concept (e.g., “Let the woman naked in the car”), and $\bm{p}_{i}^{n}$, which does not contain the NSFW concept (e.g., “Let the woman in the car”). The prompt pairs are modified from those in (Tsai et al., ) to suit the image editing task. Then, given the text encoder $\bm{\tau}_{\bm{\theta}}(\cdot)$, the NSFW concept $\bm{c}$ can be extracted as follows:

$$ $\bm{c}:=\frac{1}{N}\sum_{i=1}^{N}\bm{\tau}_{\bm{\theta}}\left(\bm{p}_{i}^{c} \right)-\bm{\tau}_{\bm{\theta}}\left(\bm{p}_{i}^{n}\right).$ (1) $$

After obtaining $\bm{c}$, we can use it to shift the original embedding of any benign prompt $\bm{p}$ into an NSFW embedding $\tilde{\bm{\tau}}:=\bm{\tau}_{\bm{\theta}}(\bm{p})+\alpha\cdot\bm{c},$
where $\alpha$ is the strength coefficient that can be adjusted to further boost the NSFW concept.

Adversarial Image Generator Training.
After obtaining the NSFW embedding, a straightforward method is to directly optimize an adversarial perturbation on an image to achieve our goal of inducing NSFW content. However, such a method would require us to repeat this optimization process for every new image to be attacked. In order to make this attack universal and transferable across multiple images, we plan to use an image generator, which allows us to turn any new images into adversarial ones to induce the diffusion model to generate NSFW content.

Now our goal here is to train the image generator to produce adversarial images that can lead the diffusion model to generate NSFW content while ensuring that the generated image remains visually similar to the original image.
Let us denote $g_{\bm{\psi}}(\cdot)$ as our generator (parameterized by $\bm{\psi}$) which takes a benign image $\bm{x}$ and generates an adversarial image $g_{\bm{\psi}}(\bm{x})$.
Unlike traditional generator training approaches (Naseer et al., ) that use U-Net (Ronneberger et al., ) or ResNet (He et al., ) architectures, we leverage a pre-trained VAE as the adversarial image generator to ensure greater similarity between the adversarial and original images.

Specifically, let us denote $f_{\bm{\theta}}^{t}\left(\bm{x},{\bm{\tau}}\right)$ as the output latent feature at the timestep $t$ during the diffusion process when taking $\bm{x}$ as the image conditions and $\bm{\tau}$ as the feature of prompt conditions.
Our objective is to optimize $\bm{\psi}$ such that the latent feature obtained through the adversarially generated image, i.e., $f_{\bm{\theta}}^{t}\left(g_{\bm{\psi}}(\bm{x}),\bm{\tau}_{\bm{\theta}}\left(
\bm{p}\right)\right)$, resembles the latent feature guided by the NSFW concept shifted embedding, i.e., $f_{\bm{\theta}}^{t}\left(\bm{x},\tilde{\bm{\tau}}\right)$:

$$ $\mathcal{L}_{adv}=\left\|f_{\bm{\theta}}^{t}\left(g_{\bm{\psi}}(\bm{x}),\bm{ \tau}_{\bm{\theta}}\left(\bm{p}\right)\right)-f_{\bm{\theta}}^{t}\left(\bm{x}, \tilde{\bm{\tau}}\right)\right\|_{2}^{2},\quad\text{ s.t. }\left\|g_{\bm{\psi} }(\bm{x})-\bm{x}\right\|_{p}\leq\epsilon.$ (2) $$

The constraint in Eq. (2) is to ensure that the generated image $g_{\bm{\psi}}(\bm{x})$ also stays close to the original image $\bm{x}$. To solve this constraint optimization problem, we apply a clipping function to the generated adversarial image, ensuring that the difference between $g_{\bm{\psi}}(\bm{x})$ and the input image $\bm{x}$ remains within the predefined noise bound $\epsilon$ after each update step. In practice, we set $t=1$ in Eq. (2) since the latent feature at the final timestep(^1^11The denoising process start at timestep T and end at timestep 1.) directly influences the content of the generated image.

In the inference stage, a clean image is passed through the adversarial generator learned on a specific NSFW concept. Then, the generated adversarial image and a benign text prompt are inputted into the diffusion model as conditions to guide the diffusion model to produce the image containing the corresponding NSFW concept.

Adaptive Attack on Safety Checker and Gaussian Noise Defense.
Widely used diffusion models, such as Stable Diffusion (SD), incorporate a post-hoc safety checker to ensure that no NSFW content is present in the generated image. This safety checker operates by analyzing the generated image’s features and comparing them with predefined NSFW concepts using cosine similarity in the latent space. The mechanism is designed to identify and filter out images that contain undesirable content such as nudity. If a match is detected, the image is either discarded or modified to conform to safety standards.
However, our results demonstrate that this safety checker can be circumvented through slight modifications in the AdvI2I framework with an additional loss term which minimizes the cosine similarity between the generated adversarial image and the NSFW concept embeddings calculated by the safety checker. The objective function for this adaptation is defined as:

$$ $\mathcal{L}_{sc}=\sum_{i=1}^{M}\cos\left(\mathcal{D}\left(f_{\bm{\theta}}^{1} \left(g_{\bm{\psi}}\left(\bm{x}\right)\right),\bm{\tau}_{\bm{\theta}}\left(\bm {p}\right)\right),C_{i}\right),$ (3) $$

where $\mathcal{D}\left(\cdot\right)$ represents the VAE decoder to that converts the latent feature back into the output image. $C_{i}$ are the predefined NSFW concept vectors. This loss ensures that the latent space representation of the image produced by the diffusion model with the adversarial image as the condition is distinct from the NSFW concepts, making it harder for the safety checker to identify it as harmful content.

Additionally, we explore a pre-processing defense mechanism where random Gaussian noise is added to the input image of the diffusion model. The objective is to perturb the adversarial noise to disrupts its effect while maintaining the image’s utility for the primary task. However, our experiments indicate that this defense can also be bypassed. During the training of the adversarial image generator, we introduce random Gaussian noise into the output of the adversarial generator at each training step. Here we follow (Hönig et al., ) to set the variance of Gaussian noise as 0.05. The overall objective of AdvI2I-Adaptive is:

$$ $\mathcal{L}_{adv}=\left\|f_{\bm{\theta}}^{t}\left(g_{\bm{\psi}}\left(\bm{x} \right)+\bm{\epsilon}_{G},\bm{\tau}_{\bm{\theta}}\left(\bm{p}\right)\right)-f_ {\bm{\theta}}^{t}\left(\bm{x},\tilde{\bm{\tau}}\right)\right\|_{2}^{2}+\mu \mathcal{L}_{sc},\quad\text{ s.t. }\left\|g_{\bm{\psi}}(\bm{x})-\bm{x}\right\| _{p}\leq\epsilon,$ (4) $$

where $\bm{\epsilon}_{G}$ denotes the random Gaussian noise, and $\mu$ is the hyper-parameter to control the scale of $\mathcal{L}_{sc}$.
These modifications result in an enhanced version of the attack, named AdvI2I-Adaptive. The adversarial images produced by AdvI2I-Adaptive maintain high ASR even in the presence of these defenses, confirming the robustness of this approach against existing protective measures.

Figure: Algorithm 1 Adversarial Image Attack on Image-to-Image Diffusion models: AdvI2I

## 4 Experiments

### 4.1 Experimental Settings

Datasets.
To train the adversarial noise generator and evaluate the effectiveness of AdvI2I, we construct a image-text pair dataset. The images are sourced from the “sexy” category of the NSFW Data Scraper (Kim, ), consisting predominantly of the human bodies. We filter out images that are classified as NSFW and randomly select 400 images from the remaining set. Additionally, 30 text prompts are generated for image editing using ChatGPT-4o (OpenAI, ). Then, we randomly select 200 images and 10 text prompts from each set to construct 2000 image-text pair samples, in which 1800 samples are used for training adversarial image generators and the remaining 200 samples are for evaluation.

Diffusion Models. Our experiments leverage two diffusion models. The first model, InstructPix2Pix, is modified and finetuned from SDv1.5. It has been optimized for image editing tasks based on user instructions, allowing users to specify modifications such as changing objects, styles, or scenes using natural language. The second model, SDv1.5-Inpainting, is designed to edit specific regions of an image, controlled via a mask image. We also evaluate the transferability of AdvI2I from
SDv1.5-Inpainting to other SD inpainting models. The results are shown in Appendix A.1.

Baselines. We propose variations of AdvI2I as comparisons, with one baseline named ”Attack VAE.” Attack VAE modifies the loss function to generate adversarial images by only utilizing the image encoder $\mathcal{E}$ and decoder $\mathcal{D}$ of the diffusion model. The goal is to ensure that the decoded image resembles the target image, similar to the approach used in Glaze (Shan et al., ). Additionally, we introduce another variation, ”W/o Generator,” as an ablation study, where we remove the adversarial noise generator and directly optimize adversarial perturbations. For further results and analysis, please refer to Appendix A.2.
In addition, we incorporate MMA-Diffusion (Yang et al., ), which originally utilizes text and image modalities to generate NSFW content while evading post-hoc safety filters. We adapt MMA-Diffusion to our experimental setup by replacing text prompts in our dataset with adversarial text prompts generated by MMA-Diffusion and training the adversarial perturbations on the images with 1800 image-text pair samples, enabling its adversarial perturbations to generalize across multiple prompts and images.

Defense Strategies. We evaluate the robustness of the AdvI2I attack under three different types of defense strategies. Specifically, Safe Latent Diffusion (SLD) (Schramowski et al., ) and Negative Prompt (SD-NP) (Rombach et al., )
are popular concept removal methods applied during the inference process. For nudity, we use “nudity”, while for violence, we use “violence” as their negative prompts for SLD and SD-NP. Gaussian Noising (GN) (Hönig et al., ) is a pre-process defense that adds Gaussin noise to the input images. Here we use the same noise bound as the adversarial noise. Safety Checker (SC) is a post-hoc defense that uses a model to calculate the cosine similarity between the output images and the NSFW concepts.

Evaluation Metric. We compute the ASR of adversarial images by evaluating whether generated images contain NSFW content. Following (Tsai et al., ), to classify whether the generated images have nudity content, we employ the NudeNet (nud, ) detector. If the detector identifies any of the following labels in an image: BUTTOCKS_EXPOSED, FEMALE_BREAST_EXPOSED, MALE_BREAST_EXPOSED, ANUS_EXPOSED, MALE_GENITALIA_EXPOSED, we categorize the image as containing nudity. To assess whether the images contain other inappropriate content such as violence, we use the Q16 classifier (Schramowski et al., ).

### 4.2 Results and Analysis

Evaluation of Defense Strategies.
We evaluate the efficacy of defense strategies against the AdvI2I attack and baselines across two NSFW concepts, nudity and violence, using the InstructPix2Pix and SDv1.5-Inpainting diffusion models. The results are shown in Tables 3 and 4.

InstructPix2Pix Model.
For the nudity concept, AdvI2I achieved an ASR of 81.5% without defense, outperforming all baselines. However, the SC defenses significantly reduced the ASR, bringing it down to 18.0% for nudity and 32.5% for violence. GN was less effective, reducing the ASR to 64.5% for nudity. Despite these defenses, the adaptive version of AdvI2I demonstrated resilience, maintaining ASRs of 70.5% under SC for both concepts, underscoring the robustness of this adversarial approach across different NSFW content.

SDv1.5-Inpainting Model.
On the SDv1.5-Inpainting model, AdvI2I reached an ASR of 82.5% for nudity without defense, with SC reducing it to 10.5%, confirming SC as the most effective defense across both concepts. The adaptive variant displayed a minor drop in ASR, remaining at 72.0% under SC. For violence, AdvI2I achieved 81.0% without defense, with SC reducing it to 31.5%, though the adaptive version maintained an ASR of 71.5%.

According to the results, the two baselines, VAE-Attack and MMA, demonstrated limited effectiveness compared to AdvI2I, with lower ASR due to their simplified architectures. VAE-Attack does not utilize the full diffusion process, reducing its overall impact. MMA, although more effective, still falls short in fully exploiting the adversarial image modality. In contrast, AdvI2I’s use of an adversarial generator allows for more complex and adaptable perturbations, consistently achieving higher ASR. Furthermore, AdvI2I-Adaptive improves robustness by adapting to defenses, highlighting the need for stronger and more comprehensive safety mechanisms in diffusion models.

**Table 3: The ASR of different attack strategies against different defense methods on the InstructPix2Pix diffusion model.**
| Concept | Method | w/o Defense | SLD | SD-NP | GN | SC |
| --- | --- | --- | --- | --- | --- | --- |
| Nudity | Attack VAE | 19.0% | 18.0% | 19.0% | 18.0% | 7.5% |
| MMA | 68.5% | 62.0% | 66.0% | 57.0% | 64.5% |  |
| AdvI2I (ours) | 81.5% | 78.0% | 79.5% | 64.5% | 18.0% |  |
|  | AdvI2I-Adaptive (ours) | 78.0% | 72.5% | 74.5% | 73.0% | 70.5% |
| Violence | Attack VAE | 22.5% | 21.0% | 22.5% | 19.5% | 12.5% |
| MMA | 71.5% | 63.5% | 67.5% | 64.5% | 65.5% |  |
| AdvI2I (ours) | 80.0% | 72.5% | 74.0% | 65.5% | 32.5% |  |
|  | AdvI2I-Adaptive (ours) | 75.5% | 70.5% | 73.5% | 70.0% | 70.5% |

**Table 4: The ASR of different attack strategies against different defense methods on the SDv1.5-Inpainting Model model.**
| Concept | Method | w/o Defense | SLD | SD-NP | GN | SC |
| --- | --- | --- | --- | --- | --- | --- |
| Nudity | Attack VAE | 41.5% | 36.5% | 41.5% | 39.0% | 7.0% |
| MMA | 42.0% | 37.0% | 39.5% | 26.0% | 39.5% |  |
| AdvI2I (ours) | 82.5% | 78.5% | 80.0% | 70.0% | 10.5% |  |
|  | AdvI2I-Adaptive (ours) | 78.5% | 75.0% | 75.5% | 72.5% | 72.0% |
| Violence | Attack VAE | 37.5% | 35.5% | 36.0% | 32.5% | 29.5% |
| MMA | 47.5% | 44.0% | 46.5% | 35.5% | 46.0% |  |
| AdvI2I (ours) | 81.0% | 75.0% | 78.5% | 66.5% | 31.5% |  |
|  | AdvI2I-Adaptive (ours) | 76.5% | 72.5% | 73.0% | 69.5% | 71.5% |

**Table 5: ASR of AdvI2I and AdvI2I-Adaptive on unseen images and prompts across two NSFW concepts, nudity and violence.**
| Model | Methods | Nudity | Violence |  |  |
| --- | --- | --- | --- | --- | --- |
| Images | Prompts | Images | Prompts |  |  |
| InstructPix2Pix | AdvI2I | 68.5% | 75.0% | 66.5% | 73.5% |
| Adaptive | 65.0% | 70.0% | 63.5% | 68.5% |  |
| SDv1.5-Inpainting | AdvI2I | 76.0% | 76.5% | 74.5% | 75.0% |
| Adaptive | 71.0% | 71.5% | 72.5% | 74.0% |  |

Figure: Figure 2: The case study of the AdvI2I and AdvI2I-Adaptive attacks on I2I diffusion models. The figure compares the original input images, masked images, and adversarially generated outputs from AdvI2I and AdvI2I-Adaptive under two categories: nudity and violence. The Gaussian blurs are added by the authors for ethical considerations.
Refer to caption: x2.png

Case study.
In Figure 2, we evaluate the results of AdvI2I and AdvI2I-Adaptive attacks on the SDv1.5-Inpainting (denoted as SD-Inpainting here) and InstructPix2Pix. We add Gaussian blurs for ethical considerations.
Importantly, both models successfully generate realistic images that contain NSFW content. The mask image controls which parts of the original image can be modified by the SDv1.5-Inpainting model with white regions: the clothing region for the nudity concept and the body region for the violence concept. InstructPix2Pix, however, lacks the ability to mask specific areas, leading to more extensive modifications across the entire image, often resulting in more drastic changes compared to SDv1.5-Inpainting.
For the violence concept, the diffusion models tend to represent violence using visual elements like blood. Moreover, we observe that when faces are editable, both models demonstrate limitations in accurately rendering facial details, suggesting that masking the face is needed for more realistic editing. Overall, these findings highlight the vulnerabilities of both models to adversarial attacks, which could be maliciously used, raising societal concerns about the misuse of such technologies.

Results on unseen images and prompts.
The results presented in Table 5 highlight the robustness and generalization capabilities of the AdvI2I and AdvI2I-Adaptive methods when applied to unseen images and prompts. Both methods achieved a relatively high ASR in the concepts of nudity and violence, with ASR values greater than 63.5% in unseen images and 68.5% in unseen prompts. Notably, AdvI2I showed stronger generalization on text prompts compared to images, indicating that the attack success is less dependent on specific prompts. These findings further underscore the effectiveness of AdvI2I in diverse and unseen scenarios, making it a potent safety threat.

Varying scale of noise bound $\epsilon$.
The results in Table 6 show that increasing the noise bound $\epsilon$ strengthens the adversarial attack, as larger perturbations enable more effective exploitation of vulnerabilities in the diffusion model. While higher noise bounds result in a rise in ASR, peaking at 84.5% without defense, this trend persists even under defenses, with SC proving the most effective at containing the ASR. However, the fact that the ASR of the AdvI2I-Adaptive remains significant, even at a small noise bound, emphasizes the challenge of fully mitigating adversarial image attacks.

**Table 6: Comparison of different noise bounds $\epsilon$ under various defenses. The evaluation is conducted on the InstructPix2Pix model regarding the concept nudity.**
| Method | $\epsilon$ | w/o Defense | SLD | SD-NP | GN | SC |
| --- | --- | --- | --- | --- | --- | --- |
| AdvI2I | 32/255 | 76.5% | 70.5% | 73.5% | 60.0% | 14.5% |
| 64/255 | 81.5% | 78.0% | 79.5% | 64.5% | 18.0% |  |
| 128/255 | 84.5% | 81.0 % | 81.5% | 64.5 % | 18.5% |  |
| Adaptive | 32/255 | 74.0% | 70.5% | 72.5% | 64.5% | 61.0% |
| 64/255 | 78.0% | 75.0% | 75.5% | 70.5% | 72.0% |  |
| 128/255 | 79.5% | 75.0% | 75.5% | 73.5% | 72.5% |  |

## 5 Conclusion

In this work, we present AdvI2I, a novel framework designed to expose a vulnerability previously underexplored in I2I diffusion models. Although previous research has focused predominantly on adversarial prompt attacks in T2I models, our framework highlights the potential risks posed by adversarial image attacks. By injecting adversarial perturbations into conditioning images, AdvI2I successfully manipulates diffusion models to generate NSFW content, bypassing current defense mechanisms designed to mitigate adversarial attacks on diffusion models. Our experiments demonstrate the effectiveness of this approach, showing that even with benign text prompts, adversarially altered images can induce diffusion models to produce harmful output. We urge the research community to further investigate robust defenses against such adversarial image attacks and consider both text- and image-based inputs when designing safety mechanisms for generative models.

## Appendix A Appendix

### A.1 Evaluation of model transferability.

We evaluate the transferability of adversarial image attacks from the SDv1.5-Inpainting model to other versions of SD inpainting models (SDv2.0, SDv2.1, SDv3.0). The results in Table 7 indicate that AdvI2I achieves high ASRs when transferring from SDv1.5 to SDv2.0 and SDv2.1 (81.0% and 86.0%, respectively). Its performance drops significantly when transferred to SDv3.0, with an ASR of only 36.0%, since SDv3.0 applies Transformer rather than U-Net for diffusion noise prediction. This suggests that transferability is relatively strong across similar model architectures, such as different versions of SDv1.x and SDv2.x.

Additionally, no experiments were conducted to measure the transferability of the attacks to InstructionPix2Pix because its model architecture differs from that of the SD models. Furthermore, the training image resolution of InstructionPix2Pix is 256x256, whereas SD models struggle to achieve effective editing results at this resolution. Therefore, a direct transferability test between these models would not yield meaningful insights due to their structural and resolution differences.

**Table 7: ASR of AdvI2I and AdvI2I-Adaptive training on SDv1.5-Inpainting and evaluating on other SD inpainting models regarding concept nudity.**
| Source Model | Methods | SDv1.5 | SDv2.0 | SDv2.1 | SDv3.0 |
| --- | --- | --- | --- | --- | --- |
| SDv1.5-Inpainting | AdvI2I | 83.5% | 81.0% | 86.0% | 36.0% |
| Adaptive | 79.5% | 76.5% | 79.0% | 35.0% |  |

### A.2 Ablation Studies

**Table 8: The ASR of “W/o Generation” against different defense methods on the InstructPix2Pix diffusion model.**
| Model | Concept | Method | w/o Defense | SLD | SD-NP | GN | SC |
| --- | --- | --- | --- | --- | --- | --- | --- |
| InstructPix2Pix | Nudity | W/o Generation | 18.5% | 16.0% | 17.5% | 18.5% | 11.0% |
| AdvI2I (ours) | 81.5% | 78.0% | 79.5% | 64.5% | 18.0% |  |  |
| AdvI2I-Adaptive (ours) | 78.0% | 72.5% | 74.5% | 73.0% | 70.5% |  |  |
| Violence | W/o Generation | 18.0% | 14.5% | 15.5% | 17.5% | 12.0% |  |
| AdvI2I (ours) | 80.0% | 72.5% | 74.0% | 65.5% | 32.5% |  |  |
| AdvI2I-Adaptive (ours) | 75.5% | 70.5% | 73.5% | 70.0% | 70.5% |  |  |
| SDv1.5-Inpainting | Nudity | W/o Generation | 55.0% | 53.5% | 54.0% | 53.5% | 3.5% |
| AdvI2I (ours) | 82.5% | 78.5% | 80.0% | 70.0% | 10.5% |  |  |
| AdvI2I-Adaptive (ours) | 78.5% | 75.0% | 75.5% | 72.5% | 72.0% |  |  |
| Violence | W/o Generation | 52.5% | 49.0% | 49.5% | 49.0% | 31.5% |  |
| AdvI2I (ours) | 81.0% | 75.0% | 78.5% | 66.5% | 31.5% |  |  |
| AdvI2I-Adaptive (ours) | 76.5% | 72.5% | 73.0% | 69.5% | 71.5% |  |  |

**Table 9: Comparison of different $\alpha$ scales with various defense methods.**
| Method | $\alpha$ | w/o Defense | SLD | SD-NP | GN | SC |
| --- | --- | --- | --- | --- | --- | --- |
| AdvI2I | 2.2 | 80.5% | 73.5% | 76.5% | 64.5% | 20.0% |
| 2.5 | 81.5% | 78.0% | 79.5% | 64.5% | 18.0% |  |
| 2.8 | 82.5% | 68.0% | 73.0% | 65.5% | 17.5% |  |
| Adaptive | 2.2 | 75.5% | 60.5% | 62.5% | 71.5% | 70.0% |
| 2.5 | 78.5% | 75.0% | 75.5% | 70.5% | 72.0% |  |
| 2.8 | 76.5% | 72.5% | 74.0% | 73.5% | 68.0% |  |

Performance of AdvI2I w/o Using Generator. We evaluate the performance of the method “W/o Generation” for the ablation study, which directly optimizes adversarial perturbations on the image. As shown in Table 8, W/o Generation perform much worse than AdvI2I, since it lacks the ability to generalize adversarial noise effectively.

Varying scale of concept $\alpha$.
The influence of the concept strength parameter $\alpha$ on attack effectiveness, as shown in Table 9, underscores the importance of carefully tuning this parameter. As $\alpha$ increases, the attack becomes more aggressive, reaching a peak ASR at 82.5% without defense. However, even with stronger adversarial concepts, defenses like SC and SLD manage to reduce the ASR to moderate levels, indicating their capacity to counterbalance the attack’s growing intensity. This suggests that while higher $\alpha$ values amplify the attack’s potential, they also expose it to more effective defensive countermeasures. The adaptive version of AdvI2I demonstrates that balancing attack strength and defense resilience is critical, as it maintains higher ASRs despite the defenses.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses section.
soundness: 2
strengths: 1. This paper expands the field of adversarial attacks in I2I diffusion models by targeting image conditioning rather than the commonly used text prompts.

2. The structure and explanations are well-organized, with a clear presentation.
summary: This paper introduces AdvI2I, a framework that performs adversarial image attacks on image-to-image (I2I) diffusion models to generate NSFW content without modifying text prompts. It further proposes AdvI2I-Adaptive, which incorporates techniques to minimize resemblance to NSFW embeddings, enhancing robustness against conventional defenses. Extensive experiments demonstrate AdvI2I’s ability to bypass common safeguards in I2I models, signaling potential safety concerns and underscoring the need for improved defense mechanisms.
weaknesses: 1. The paper’s evaluation is limited to SDv1.5-Inpainting and InstructPix2Pix, both based on the SDv1.5 architecture. Expanding the analysis to include more advanced versions of Stable Diffusion or other models(not just in the transferability section) would enhance the assessment of AdvI2I’s generalization to state-of-the-art diffusion models.

2. The paper does not examine how varying benign prompts, including explicitly defensive ones that request safe content, might affect the success of the adversarial attack. Investigating whether different benign prompts, especially those aimed at reinforcing safe content, influence the attack's efficacy would offer a more comprehensive understanding of its generalization and robustness across diverse input conditions.

3. The results suggest reduced transferability of AdvI2I from SDv1.5 to SDv3.0, indicating that its effectiveness may be architecture-specific, potentially limiting the framework’s generalizability.

4. Since AdvI2I relies on adversarial noise as a primary mechanism, it would be beneficial to explore potential defense strategies beyond the current scope, such as DiffPure[1], to counteract adversarial image attacks more effectively.

[1] Diffusion models for adversarial purification.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: - The paper is well-written and easy to follow.
- It revisits current prompt attacks and show critical insights that they can be easily defended.
- Based on the idea of concept vector extraction. The proposed novel pipeline is well-designed and effective, which shows strong performance compared with baseline methods. Also, the proposed pipeline is learning-based and can be used as one-pass noise generator.
summary: This paper proposed a framework that can bypass safety check of NSFW generations by modifying the image input. It first points out the problems of previous prompt attacks to bypass NSFW checks, then it proposes to add noise to images to bypass the safety checker, which will make the attack more stealthy. The authors design a pipeline with some key design choices e.g. NSFW concept vector extraction, use a universal noise generator to avoid repeated generation. The paper is mostly well-written and easy to follow, the experiments are comprehensive and show that it is better than baseline methods.
weaknesses: Clarification:
- This paper is working with I2I diffusion models, which have additional encoded visual inputs as condition. The targeted I2I model is (1) different from the T2I diffusion models used in previous works (2) is not as good as popular T2I models e.g. SD-XL, FLUX. It poses challenges on the motivation of this paper, because the I2I model used in this paper is a sub-optimal choice if the malicious users want to generate high quality NSFW images.

Methods:
- The proposed method is restricted to a small subset of I2I models, which are not the main part of diffusion models.
- The choice of noise generator may not be optimal, ad-hoc noise generation for each image is also not that time-consuming e.g. sample timestep t and optimize for 100 steps takes for e.g. 1 min. It may show much stronger performance.


Adaptive Attack:
- The adaptive-attack assumes the white-box setting of the NSFW-detector, which may not be pratical.
- Some other defenses e.g. add noise / use purifier to remove the noise deserves more study.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: - **important** Can AdvI2I method apply for good use, like changing the target concept to e.g clothes (instead of NSFW/violence), so that $g_{\psi}$ now become a protection to help mitigate inappropriate image editing? Its application will be more like photoguard and faster, also only restrict on inappropriate editing
- How performance of AdvI2I vary, based on the neutral prompt. Specifically, given a perturbed image with prompt that specific mention e.g clothes, like "A photo of a girl in a beautiful dress" , can it still generate NSFW image?
- What hardware and how long to train AdvI2I?
soundness: 2
strengths: While bypassing content filtering using adversarial images was proposed in MMA Diffusion, the method the author uses to quickly craft an adversarial image without optimization is a new contribution. The objective for training $g_{\psi}$ is also novel, and selecting components like which UNet or VAE to train, or the diffusion step to optimize, requires trial and error efforts. 

Both AdvI2I and AdvI2I-Adaptive show good results, and the success rate against the safety checker is strong. I’m also pleased with the included transferability test with SD2x and 3.
summary: This paper explores bypassing safety checkers of image editing models to generate NSFW content, using adversarial images.

The authors first show that previous methods using adversarial prompt are easily detectable, and then propose AdvI2I - that train a network $g_{\psi}$ to perturb image.

When this perturbed image is used with a neutral prompt as input for an img2img or inpainting model, it can influence the generation process, to generate some specific NSFW concept (nudity/violence).

To do so, AdvI2I slightly modify (perturb) the image, so that its latent after denoising with a neutral prompt, is close to the original image's latent but denoising with a NSFW prompt.

The authors also propose AdvI2I-Adaptive, that incorporate safety checker loss and added Gaussian noise augmentation when training $g_{\psi}$. It helps improving robustness when gaussian noise is added to the perturbed image, and AdvI2I-Adaptive also improves the bypass success rate over the posthoc safety checker.

Experiments were conducted on SDv1.5-Inpainting and InstructPix2Pix, with high success rate against Safe Latent Diffusion, Negative Prompt, added Gaussian Noise and the Safety checker. AdvI2I also shows transferability from SDv1.5 to SDv2.0 and v2.1 but not effective when test with SDv3.0.
weaknesses: Experiments: Below are some experiments that the author can address to have a comprehensive view about AdvI2I
- Insufficient highlight the contribution of $g_{\psi}$, can add run time comparison with optimization approach
- SDXL-Turbo Image-to-Image transferability test, as it also uses the same VAE.
- Transferability test with recent models not just SD3.0, such as Pixart Alpha, Flux

Writing: The writing needs improvement. Please address these points:
- L070 confusing since there is no encoder
- L11 in algorithm 1 - $C_i$ and $M$ was not mentioned in require. Also the loss is cosine between a vector $C_i$ and output of $\mathcal{D}$ which I assume is an image, does it need an image encoder to turn it into a vector there, if so then what encoder?
- SLD Configuration - lacks details on the configuration that was used, is it Max/Strong/Medium?
- Clarify: prompts generated by gpt4-o are pair prompts or single prompt?
- Typo L607 - InstructionPix2Pix - InstructPix2Pix

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The motivation is not clear. There are a lot of existing methods that could easily generate NFSW images such as deepnude, or I can just use the deepfake to swap face of a porn image with the clean image, why people will trouble themselves to use this method? Are the NSFW images generated by your method have higher image quality? If so, I would expect the authors to clarify how their method compares to or improves upon existing NSFW image generation techniques like deepnude or deepfake. More details see Q2.
2. Are the images generated with the proposed method of significant high quality comparing with previous methods? The quality of generated NSFW images should be evaluated by some IQAs:
    If my understanding is correct, the objective is to generate nude images which have same indentity as the clean image (i.e.com the target is a category of images rather than a certain image), FR IQAs may not work well. I would suggest to use some NR IQAs (FID, TOPIQ, etc). It is worth noting that, if the data volume is lower than 10k, the FID may not be meaningful.
3. The author did not compare IQAs with other baselines for clean and attacked images, i.e., we do not know if the perturbation is imperceptible enough. (I would suggest compare the PSNR, LPIPS, SSIM, etc between clean image and attacked image). I would expect a table comparing these metrics between the clean and attacked images for their method and baseline methods.
4. Will the generated NSFW images have same identity with clean images (especially for nude images)? I suggest the author to use several face verification methods or metrics to quantify identity preservation in their generated images.
5. The performance of AdvI2I against SC is poor, although the author admits this, and implement an adaptive version, I need to see more transferability evaluations, i.e., the author should demonstrate the ASR for SC_2 given "adversarial samples generated by adaptive AdvI2I with SC_1".
    I suggest the author to provide a table showing ASR results for different combinations of safety checkers (SC_1, SC_2, etc.) used during training and testing of their adaptive AdvI2I method.
soundness: 2
strengths: 1. The paper shows that the prompt-based attack can be simply defenced by some filters
2. This paper proposes a new framework which solely based on perturbating images to induce NSFW content. It can be regarded as the knowledge distillation of adversarial text prompt that could induce NFSW contents to image embedding. the proposed method can bypass these methods as the adversarial information is hidden in the images, i.e., the text prompt will not have and malign information.
summary: This paper proposes a adversarial image attack framework that injects adversarial information into images rather than text prompts, inducing Image2Image Diffusion to generate NSFW content, therefore it has some novelty. However, the motivation is not clear, and the evaluation is not enough. Overall, my main concern is the insufficient evaluation, if the author can solve my concerns, I would be happy to change the rating.
weaknesses: 1. The motivation is not clear. See Q1
2. lack of evaluations, e.g., IQAs, face verification rate. See Q2, 3, 4, 5

Meta-review: This paper studies the vulnerabilities of image-to-image (I2I) diffusion models under adversarial image attacks. A new method AdvI2I is proposed to generate adversarial perturbations on input images to induce NSFW content. The proposed method is effective to mislead Stable Diffusion models and bypass defense mechanisms.

The paper focuses on an important and timely topic on the vulnerability of I2I diffusion models. The proposed method is novel based on the idea of concept vector extraction. The paper is well-written.

The paper received borderline ratings with two borderline accept recommendations and two borderline reject recommendations. After author responses and author-reviewer discussions, most of the concerns have been addressed and two concerns remain.
- As pointed out by Reviewer Cqd3, the motivation is unclear. The paper mainly focuses on identifying the vulnerability of I2I diffusion models, but the adversary may not use such technique to generate NSFW content. The evaluation of image quality has also some mistakes.
- As pointed out by Reviewer f4nA and Cqd3, the effectiveness for advanced models (e.g., SDv3.0) is limited.

Based on the above limitations of this work, AC would recommend rejection and hope the paper can be further improved for the next venue.

Decision: Reject