SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhaoyu Chen $^{1,2,*}$ ; Bo Li $^{2,\dagger}$ , Shuang Wu $^{2}$ , Kaixun Jiang $^{1}$ , Shouhong Ding $^{2}$ , Wenqiang Zhang $^{1,3,\dagger}$
$^{1}$ Academy for Engineering and Technology, Fudan University  
 $^{2}$ Youtu Lab, Tencent  $^{3}$ School of Computer Science, Fudan University {zhaoyuchen20, wqzhang}@fudan.edu.cn, njumagiclibo@gmail.com, {calvinwu, ericshding}@tencent.com
Unrestricted adversarial attacks typically manipulate the semantic content of an image (e.g., color or texture) to create adversarial examples that are both effective and photorealistic, demonstrating their ability to deceive human perception and deep neural networks with stealth and success. However, current works usually sacrifice unrestricted degrees and subjectively select some image content to guarantee the photorealism of unrestricted adversarial examples, which limits its attack performance. To ensure the photorealism of adversarial examples and boost attack performance, we propose a novel unrestricted attack framework called Content-based Unrestricted Adversarial Attack. By leveraging a low-dimensional manifold that represents natural images, we map the images onto the manifold and optimize them along its adversarial direction. Therefore, within this framework, we implement Adversarial Content Attack (ACA) based on Stable Diffusion and can generate high transferable unrestricted adversarial examples with various adversarial contents. Extensive experimentation and visualization demonstrate the efficacy of ACA, particularly in surpassing state-of-the-art attacks by an average of  $13.3 - 50.4\%$  and  $16.8 - 48.0\%$  in normally trained models and defense methods, respectively.
Deep neural networks (DNNs) have significantly progressed in many tasks [18, 7]. However, with the rise of adversarial examples, the robustness of DNNs has been dramatically challenged [16]. Adversarial examples show the vulnerability of DNNs and expose security vulnerabilities in many security-sensitive applications. To avoid potential risks and further research the robustness of DNNs, it is of great value to expose as many "blind spots" of DNNs as possible at the current research stage.
Nowadays, various methods are proposed to generate adversarial examples [4, 5, 6]. To maintain human visual imperceptibility and images' photorealism, adversarial perturbations within the constraint of  $l_{p}$  norm are generated by these adversarial attacks. However, it is well known that the adversarial examples generated under  $l_{p}$  norm have obvious limitations: firstly, they are not ideal in terms of perceptual similarity and are still easily perceptible by humans [24, 23, 62]; secondly, these adversarial perturbations are not natural enough and have an inevitable domain shift with the noise in the natural world, resulting in the adversarial examples being different from the hard examples that appear in the real world [64]. In addition, current defense methods against  $l_{p}$  norm adversarial examples overestimate their abilities, known as the Dunning-Kruger effect [28]. It can effectively defend against  $l_{p}$  norm adversarial examples but is not robust enough when facing new and unknown attacks [25]. Therefore, unrestricted adversarial attacks are beginning to emerge, using unrestricted but natural changes to replace small  $l_{p}$  norm perturbations, which are more practically meaningful.
*This work was done when Zhaoyu Chen was an intern at Youtu Lab, Tencent.
$^\dagger$  indicates corresponding authors.
Existing unrestricted adversarial attacks generate adversarial examples based on image content such as shape, texture, and color. Shape-based unrestricted attacks [56, 1] iteratively apply small deformations to the image through a gradient descent step. Then, texture-based unrestricted attacks [2, 40] are introduced, which manipulate an image's general attributes (texture or style) to generate adversarial examples. However, texture-based attacks result in unnatural results and have low adversarial transferability. Researchers then discover that manipulating pixel values along dimensions generates more natural adversarial examples, leading to the rise of color-based unrestricted attacks [20, 30, 2, 61, 47, 60]. Nonetheless, color-based unrestricted attacks tend to compromise flexibility in unconstrained settings to guarantee the photorealism of adversarial examples. They are achieved either through reliance on subjective intuition and objective metrics or by implementing minor modifications, thereby constraining their potential for adversarial transferability.
Considering the aforementioned reasons, we argue that an ideal unrestricted attack should meet three criteria: i) it needs to maintain human visual imperceptibility and the photorealism of the images; ii) the attack content should be diverse, allowing for unrestricted modifications of image contents such as texture and color, while ensuring semantic consistency; iii) the adversarial examples should have a high attack performance so that they can transfer between different models. However, there is still a substantial disparity between the current and ideal attacks.
To address this gap, we propose a novel unrestricted attack framework called Content-based Unrestricted Adversarial Attack. Firstly, we consider mapping images onto a low-dimensional manifold. This low-dimensional manifold is represented by a generative model and expressed as a latent space. This generative model is trained on millions of natural images, possessing two characteristics: i) sufficient capacity to ensure the photorealism of generated images; ii) well-alignment of image contents with latent space ensures a diversity of content. Subsequently, more generalized images can be generated by walking along the low-dimensional manifold. Optimizing the adversarial objective on this latent space allows us to achieve more diverse adversarial contents. In this paper, we propose Adversarial Content Attack (ACA) utilizing the diffusion model as a low-dimensional manifold. Specifically, we employ Image Latent Mapping (ILM) to map images onto the latent space, and utilize Adversarial Latent Optimization (ALO) to optimize the latents, thereby generating unrestricted adversarial examples with high transferability. In conclusion, our main contributions are:
- We propose a novel attack framework called Content-based Unrestricted Adversarial Attack, which utilizes high-capacity and well-aligned low-dimensional manifolds to generate adversarial examples that are more diverse and natural in content.
- We achieve an unrestricted content attack, known as the Adversarial Content Attack. By utilizing Image Latent Mapping and Adversarial Latent Optimization techniques, we optimize latents in a diffusion model, generating high transferable unrestricted adversarial examples.
- The effectiveness of our attack has been validated through experimentation and visualization. Notably, we have achieved a significant improvement of  $13.3\sim 50.4\%$  over state-of-the-art attacks in terms of adversarial transferability.
2 Background and Preliminary
Problem Definition. For a deep learning classifier  $\mathcal{F}_{\theta}(\cdot)$  with parameters  $\theta$ , we denote the clean image as  $x$  and the corresponding true label as  $y$ . Formally, unrestricted adversarial attacks aim to create imperceptible adversarial perturbations (such as image distortions, texture or color modifications, etc.) for a given input  $x$  to generate an adversarial example  $x_{adv}$  that can mislead the classifier  $\mathcal{F}_{\theta}(\cdot)$ :
$$
\max  _ {x _ {a d v}} \mathcal {L} \left(\mathcal {F} _ {\theta} \left(x _ {a d v}\right), y\right), \quad \text {s . t .} x _ {a d v} \text {i s n a t u r a l}, \tag {1}
$$
where  $\mathcal{L}(\cdot)$  is the loss function. Because existing unrestricted attacks are limited by their attack contents, it prevents them from generating sufficiently natural adversarial examples and restricts their attack performance on different models. We hope that unrestricted adversarial examples are more natural and possess higher transferability. Therefore, we consider a more challenging and practical black-box setting to evaluate the attack performance of unrestricted adversarial examples. In contrast to the white-box setting, the black-box setting has no access to any information about the target model (i.e., architectures, network weights, and gradients). It can only generate adversarial examples by using a substitute model  $\mathcal{F}_{\phi}(\cdot)$  and exploiting their transferability to fool the target model  $\mathcal{F}_{\theta}(\cdot)$ .
Content-based Unrestricted Adversarial Attack. Existing unconstrained attacks tend to modify fixed content in images, such as textures or colors, compromising flexibility in unconstrained settings to ensure the photorealism of adversarial examples. For instance, ColorFool [47] manually selects the parts of the image that are sensitive to human perception in order to modify the colors, and its effectiveness is greatly influenced by human intuition. Natural Color Fool [60], utilizes ADE20K [63] to construct the distribution of color distributions, resulting in its performance being restricted by the selected dataset. These methods subjectively choose the content to be modified for minor modifications, but this sacrifices the flexibility under unrestricted settings and limits the emergence of more "unrestricted" attacks. Taking this into consideration, we contemplate whether it is possible to achieve an unrestricted adversarial attack that can adaptively modify the content of an image while ensuring the semantic consistency of the image.
An ideal unrestricted attack should ensure the photorealism of adversarial examples, possess diverse adversarial contents, and exhibit potential attack performance. To address this gap, we propose a novel unrestricted attack framework called Content-based Unrestricted Adversarial Attack. Within this framework, we assume that natural images can be mapped onto a low-dimensional manifold by a generative model. As this low-dimensional manifold is well-trained on natural images, it naturally ensures the photorealism of the images and possesses the rich content present in natural images. Once we map an image onto a low-dimensional manifold, moving it along the adversarial direction on the manifold yields an unrestricted adversarial example. We argue that such a framework is closer to the ideal of unrestricted adversarial attacks, as it inherently guarantees the photorealism of adversarial examples and rich image content. Moreover, since the clas
sifier itself fits the distribution of this low-dimensional manifold, adversarial examples generated along the manifold have more adversarial contents and the potential for strong attack performance, as shown in Figure 1.
Figure 1: Adversarial examples are generated along the adversarial direction of the low-dimensional manifold of natural images. This manifold represents many contents of natural images, so the generated unrestricted adversarial examples combine multiple adversarial contents (shape, texture and color).
Naturally, selecting a low-dimensional manifold represented by a generative model necessitates careful consideration. There are two characteristics taken into account: i) sufficient capacity to ensure photorealism in the generated images; and ii) well-alignment ensures that image attributes are aligned with the latent space, thereby promoting diversity in content generation. Recently, diffusion models have emerged as a leading approach for generating high-quality images across varied datasets, frequently outperforming GANs [9]. However, several large-scale text-to-image diffusion models, includingImagen [44],DALL-E2 [41],and Stable Diffusion [42], have only recently come to the fore, exhibiting unparalleled semantic generation capabilities. Considering the trade-off between computational cost and high-fidelity image generation, we select Stable Diffusion as the low-dimensional manifold in this paper. It is based on prompt input and is capable of generating highly realistic natural images that conform to the semantics of the prompts.
3 Adversarial Content Attack
Based on the aforementioned framework and the full utilization of the diffusion model's capability, we achieve the unrestricted content-based attack known as Adversarial Content Attack (ACA), as shown in Figure 2. Specifically, we first employ Image Latent Mapping (ILM) to map images onto the latent space represented by this low-dimensional manifold. Subsequently, we introduce an Adversarial Latent Optimization (ALO) technique that moves the latent representations of images along the adversarial direction on the manifold. Finally, based on iterative optimization, ACA can generate highly transferable unrestricted adversarial examples that appear quite natural. The
Figure 2: Pipeline of Adversarial Content Attack. First, we use Image Latent Mapping to map images into latent space. Next, Adversarial Latent Optimization is used to generate adversarial examples. Eventually, the generated adversarial examples can fool the target classifier.
algorithm for ACA is presented in Algorithm 1, and we further combine the diffusion model to design the corresponding mapping and optimization methods.
3.1 Image Latent Mapping
For the diffusion model, the easiest image mapping is the inverse process of DDIM sampling [9, 48] with the condition embedding  $\mathcal{C} = \psi(\mathcal{P})$  of prompts  $\mathcal{P}$ , based on the assumption that the ordinary differential equation (ODE) process can be reversed in the limit of small steps:
$$
z _ {t + 1} = \sqrt {\frac {\alpha_ {t + 1}}{\alpha_ {t}}} z _ {t} + \sqrt {\alpha_ {t + 1}} \left(\sqrt {\frac {1}{\alpha_ {t + 1}} - 1} - \sqrt {\frac {1}{\alpha_ {t}} - 1}\right) \cdot \epsilon_ {\theta} \left(z _ {t}, t, \mathcal {C}\right), \tag {2}
$$
where  $z_0$  is the given real image, a schedule  $\{\beta_0,\dots,\beta_T\} \in (0,1)$  and  $\alpha_{t} = \prod_{1}^{t}(1 - \beta_{i})$ . In general, this process is the reverse direction of the denoising process ( $z_0\to z_T$  instead of  $z_{T}\rightarrow z_{0}$ ), which can map the image  $z_0$  to  $z_{T}$  in the latent space. Image prompts are automatically generated using image caption models (e.g., BLIP v2 [31]). For simplicity, the encoding of the VAE is ignored.
Text-to-image synthesis usually emphasizes the effect of the prompt. Therefore, a classifier-free guidance technique [19] is proposed. Its prediction is also performed unconditionally, which is then extrapolated with the conditioned prediction. Given  $w$  as the guidance scale parameter and  $\varnothing = \psi(\mathbf{\theta}^{\prime \prime \prime})$  as the embedding of a null text, the classifier-free guidance prediction is expressed by:
$$
\tilde {\epsilon} _ {\theta} \left(z _ {t}, t, \mathcal {C}, \varnothing\right) = w \cdot \epsilon_ {\theta} \left(z _ {t}, t, \mathcal {C}\right) + (1 - w) \cdot \epsilon_ {\theta} \left(z _ {t}, t, \varnothing\right), \tag {3}
$$
where  $w = 7.5$  is the default value for Stable Diffusion. However, since the noise is predicted by the model  $\epsilon_{\theta}$  in the inverse process of DDIM sampling, a slight error is incorporated in every step. Due to the existence of a large guidance scale parameter  $w$  in the classifier-free guidance technique, slight errors are amplified and lead to cumulative errors. Consequently, executing the inverse process of DDIM sampling with classifier-free guidance not only disrupts the Gaussian distribution of noises but also induces visual artifacts of unreality [37].
To mitigate cumulative errors, we follow [37] and optimize a null text embedding  $\varnothing_{t}$  for each timestamp  $t$ . First, the inverse process of DDIM sampling with  $w = 1$  outputs a series of consecutive latents  $\{z_0^*, \dots, z_T^*\}$  where  $z_0^* = z_0$ . Then, we conduct the following optimization with  $w = 7.5$  (the default value for Stable Diffusion) and  $z_{T} = z_{t}$  during  $N$  iterations for the timestamps  $t = \{T, \dots, 1\}$ :
$$
\min  _ {\varnothing_ {t}} \left| \left| z _ {t - 1} ^ {*} - z _ {t - 1} \left(\bar {z} _ {t}, t, \mathcal {C}, \varnothing_ {t}\right) \right| \right| _ {2} ^ {2}, \tag {4}
$$
$$
z _ {t - 1} \left(\bar {z} _ {t}, t, \mathcal {C}, \varnothing_ {t}\right) = \sqrt {\frac {\alpha_ {t - 1}}{\alpha_ {t}}} \bar {z} _ {t} + \sqrt {\alpha_ {t - 1}} \left(\sqrt {\frac {1}{\alpha_ {t - 1}} - 1} - \sqrt {\frac {1}{\alpha_ {t}} - 1}\right) \cdot \tilde {\epsilon} _ {\theta} \left(z _ {t}, t, \mathcal {C}, \varnothing_ {t}\right). \tag {5}
$$
At the end of each step, we update  $z_{t-1}^{-}$  to  $z_{t-1}(\bar{z}_t, t, \mathcal{C}, \varnothing_t)$ . Finally, we obtain the latent of the given image in the low-dimensional manifold, consisting of the noise  $\bar{z}_T$ , the null text embedding  $\varnothing_t$ , and the text embedding  $\mathcal{C} = \psi(\mathcal{P})$ . Note that compared to other strategies [12, 43, 26, 54], the current strategy is simple and effective and does not require fine-tuning to obtain high-quality image reconstruction. Next, we exploit this latent to generate unrestricted adversarial examples.
3.2 Adversarial Latent Optimization
In this section, we propose an optimization method for latents to maximize the attack performance on unrestricted adversarial examples. In the latent space of a given image after ILM, the null text embedding  $\varnothing_{t}$  ensures the quality of the reconstructed image, while the text embedding  $\mathcal{C}$  ensures the semantic information of the image. Therefore, optimizing both embeddings may not be ideal. Considering that the noise  $z_{T}$  largely represents the image's information in the latent space, we choose to optimize it instead. However, this optimization is still challenged by complex gradient calculations and the overflow of the value range.
Based on the latents generated by ILM, we define the denoising process of diffusion models as  $\Omega(\cdot)$  through Equation 5, and it involves  $T$  iterations:
$$
\Omega \left(z _ {T}, T, \mathcal {C}, \left\{\varnothing_ {t} \right\} _ {t = 1} ^ {T}\right) = z _ {0} \left(z _ {1} \left(\dots , \left(z _ {T - 1}, T - 1, \mathcal {C}, \varnothing_ {T - 1}\right), \dots , 1, \mathcal {C}, \varnothing_ {1}\right), 0, \mathcal {C}, \varnothing_ {0}\right). \tag {6}
$$
Therefore, the reconstructed image is denoted as  $\bar{z}_0 = \Omega (z_T,T,\mathcal{C},\{\varnothing_t\})$ . The computational process of VAE is disregarded herein, as it is differentiable. Combining Equation 7, our adversarial objective optimization is expressed by:
$$
\max  _ {\delta} \mathcal {L} \left(\mathcal {F} _ {\theta} \left(\bar {z} _ {0}\right), y\right), \quad s. t. \| \delta \| _ {\infty} \leq \kappa , \bar {z} _ {0} = \Omega \left(z _ {T} + \delta , T, \mathcal {C}, \{\varnothing_ {t} \}\right) \text {a n d} \bar {z} _ {0} \text {i s n a t u r a l}, \tag {7}
$$
where  $\delta$  is the adversarial perturbation on the latent space. Our loss function consists of two parts: i) cross-entropy loss  $\mathcal{L}_{ce}$ , which mainly guides adversarial examples toward misclassification. ii) mean square error loss  $\mathcal{L}_{mse}$  mainly guides the generated adversarial examples to be as close as possible to clean images on  $l_{2}$  distance. Therefore, the total loss function  $\mathcal{L}$  is expressed as:
$$
\mathcal {L} \left(\mathcal {F} _ {\theta} \left(\bar {z} _ {0}\right), y, z _ {0}\right) = \mathcal {L} _ {c e} \left(\mathcal {F} _ {\theta} \left(\bar {z} _ {0}\right), y\right) - \beta \cdot \mathcal {L} _ {m s e} \left(\bar {z} _ {0}, z _ {0}\right), \tag {8}
$$
where  $\beta$  is 0.1 in this paper. The loss function  $\mathcal{L}$  aims to maximize the cross-entropy loss and minimize the  $l_{2}$  distance between the adversarial example  $\bar{z}_0$  and the clean image  $z_0$ .
To ensure the consistency of  $z_0$  and  $\bar{z}_0$ , we assume that  $\delta$  does not change the consistency when  $\delta$  is extremely small, i.e.,  $||\delta||_{\infty} \leq \kappa$ . The crux pertains to determining the optimal  $\delta$  that yields the maximum classification loss. Analogous to conventional adversarial attacks, we employ gradient-based techniques to estimate  $\delta$  through:  $\delta \simeq \eta \nabla_{z_T} \mathcal{L}\left(\mathcal{F}_{\theta}(\bar{z}_0), y\right)$ , where  $\eta$  denotes the magnitude of perturbations that occur in the direction of the gradient. To expand  $\nabla_{z_T} \mathcal{L}\left(\mathcal{F}_{\theta}(\bar{z}_0), y\right)$  by the chain rule, we can have these derivative terms as follows:
$$
\nabla_ {z _ {T}} \mathcal {L} \left(\mathcal {F} _ {\theta} \left(\bar {z} _ {0}\right), y\right) = \frac {\partial \mathcal {L}}{\partial \bar {z} _ {0}} \cdot \frac {\partial \bar {z} _ {0}}{\partial z _ {1}} \cdot \frac {\partial z _ {1}}{\partial z _ {2}} \dots \frac {\partial z _ {T - 1}}{\partial z _ {T}}. \tag {9}
$$
Skip Gradient. After observing the items, we find that although each item is differentiable, it is not feasible to derive the entire calculation graph. First, we analyze the term  $\frac{\partial\mathcal{L}}{\partial z_0}$ , which represents the derivative of the classifier with respect to the reconstructed image  $\bar{z}_0$  and provides the adversarial gradient direction. Then, for  $\frac{\partial z_t}{\partial z_{t + 1}}$ , each calculation of the derivative represents the calculation of a backpropagation. Furthermore, a complete denoising process accumulates  $T$  calculation graphs, resulting in memory overflow (similar phenomena are also found in [45]). Therefore, the gradient of the denoising process cannot be directly calculated.
Fortunately, we propose a skip gradient to approximate  $\frac{\partial z_0}{\partial z_T} = \frac{\partial z_0}{\partial z_1} \cdot \frac{\partial z_1}{\partial z_2} \cdots \frac{\partial z_{T-1}}{\partial z_T}$ . Recalling the diffusion process, the denoising process aims to eliminate the Gaussian noise added in DDIM sampling [48, 9, 42]. DDIM samples  $z_t$  at any arbitrary time step  $t$  in a closed form using reparameterization trick:
$$
z _ {t} = \sqrt {\alpha_ {t}} z _ {0} + \sqrt {1 - \alpha_ {t}} \varepsilon , \quad \varepsilon \sim \mathcal {N} (0, I). \tag {10}
$$
Consequently, we perform a manipulation by rearranging Equation 10 to obtain  $z_0 = \frac{1}{\sqrt{\alpha_t}} z_t - \sqrt{\frac{1 - \alpha_t}{\alpha_t}} \varepsilon$ . Hence, we further obtain  $\frac{\partial z_0}{\partial z_t} = \frac{1}{\sqrt{\alpha_t}}$ . In Stable Diffusion, timestep  $t$  is at most 1000, so  $\lim_{t \to 1000} \frac{\partial z_0}{\partial z_t} = \lim_{t \to 1000} \frac{1}{\sqrt{\alpha_t}} \approx 14.58$ . In summary,  $\frac{\partial z_0}{\partial z_t}$  can be regarded as a constant  $\rho$  and Equation 9 can be re-expressed as  $\nabla_{z_T} \mathcal{L}\left(\mathcal{F}_{\theta}(\bar{z}_0), y\right) = \rho \frac{\partial \mathcal{L}}{\partial \bar{z}_0}$ . In summary, skip gradients approximate the gradients of the denoising process while reducing the computation and memory usage.
Differentiable Boundary Processing. Since the diffusion model does not explicitly constrain the value range of  $\bar{z}_0$ , the modification of  $z_{T}$  may cause the value range to be exceeded. So we introduce
Algorithm 1 Adversarial Content Attack