SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qizhang Li $^{1,2}$ , Yiwen Guo $^{3*}$ , Wangmeng Zuo $^{1*}$ , Hao Chen $^{4}$
<sup>1</sup>Harbin Institute of Technology, <sup>2</sup>Tencent Security Big Data Lab, <sup>3</sup>Independent Researcher, <sup>4</sup>UC Davis {liqizhang95,guoyiwen89}@gmail.com wmzuo@hit.edu.cn chen@ucdavis.edu
Intermediate-level attacks that attempt to perturb feature representations following an adversarial direction drastically have shown favorable performance in crafting transferable adversarial examples. Existing methods in this category are normally formulated with two separate stages, where a directional guide is required to be determined at first and the scalar projection of the intermediate-level perturbation onto the directional guide is enlarged thereafter. The obtained perturbation deviates from the guide inevitably in the feature space, and it is revealed in this paper that such a deviation may lead to sub-optimal attack. To address this issue, we develop a novel intermediate-level method that crafts adversarial examples within a single stage of optimization. In particular, the proposed method, named intermediate-level perturbation decay (ILPD), encourages the intermediate-level perturbation to be in an effective adversarial direction and to possess a great magnitude simultaneously. In-depth discussion verifies the effectiveness of our method. Experimental results show that it outperforms state-of-the-arts by large margins in attacking various victim models on ImageNet (+10.07% on average) and CIFAR-10 (+3.88% on average). Our code is at https://github.com/qizhangli/ILPD-attack.
Adversarial examples can be crafted without getting access to essential information (e.g., architecture and parameters) of the victim models [34]. Due to their applicability of assessing AI safety in many real-world deployment environments, recent years have witnessed a steady increase in the study of black-box adversarial examples [29, 4, 11, 46, 8, 16]. A considerable proportion of black-box attacks rely on transferability of adversarial examples, i.e., a phenomenon where adversarial examples crafted targeting a white-box substitute model can also fool unknown victims with a moderate success rate.
Despite being crucial to many attacks, the adversarial transferability has not been fathomed out. Some work attempts to regulate the intermediate-level perturbation (discrepancy between the intermediate-level representation of adversarial examples and that of benign examples) to show a large magnitude and align with an directional guided that can effectively maximize the prediction loss of a model [20, 26, 43, 52]. To achieve this, they choose to maximize the scalar projection of the intermediate-level perturbation onto a directional guide. The determination of the directional guide of such methods are various, and each was well-motivated. Though effective, such an objective inevitably deviates the intermediate-level perturbation from the well-motivated directional guide (as will be discussed in Section 3.1), and it may deteriorate the attack performance on the substitute model and lead to sub-optimal transferability despite showing a large perturbation magnitude in the middle layer.
In this paper, we propose a method that encourages the intermediate-level perturbation to possess a greater magnitude than a directional guide by its nature and to be in the same adversarial direction as that of the guide. This is achieved by introducing intermediate-level perturbation decay (ILPD)
* Yiwen Guo and Wangmeng Zuo lead the project. Correspondence to: Wangmeng Zuo (wmzuo@hit.edu.cn).
in a single stage of optimization. Extensive experiments were conducted to show that our ILPD outperforms existing state-of-the-arts by large margins in attacking various victim models on CIFAR10 and ImageNet. Moreover, such a method can further be readily integrated into a variety of existing transfer-based attacks.
Adversarial examples can be generated in white-box and black-box settings.
White-box attacks. In white-box settings, the architecture and parameters of the victim model are known to the adversary, and the gradient of the victim models can therefore be directly utilized to generate adversarial examples. Given a victim model  $f: \mathbb{R}^n \to \mathbb{R}^c$  which has been trained to classify any input, a typical way of generating adversarial examples is to maximize the prediction loss  $L(f(\mathbf{x} + \Delta \mathbf{x}), y)$  of  $f$  while constraining the  $\ell_p$  norm of the perturbation  $\Delta \mathbf{x}$  to a benign input  $\mathbf{x}$ , i.e.,  $\max_{\| \Delta \mathbf{x} \|_p \leq \epsilon} L(f(\mathbf{x} + \Delta \mathbf{x}), y)$ , where  $\epsilon$  is the perturbation budget and  $y$  is the ground-truth. FGSM [14] derives a single gradient step to obtain the perturbation  $\Delta \mathbf{x} = \epsilon \cdot \mathrm{sign}(\nabla_{\mathbf{x}} L(f(\mathbf{x}), y))$  in an  $\ell_{\infty}$  setting. If not otherwise mentioned, we denote  $L(f(\cdot), \cdot)$  as  $L_f(\cdot, \cdot)$  for simplicity of notation. For performing stronger attacks, iterative methods have also been proposed, e.g., I-FGSM [25] and PGD [32] which update the perturbation at the  $j$ -th iteration as
$$
\boldsymbol {\Delta} \mathbf {x} _ {j} = \operatorname {C l i p} _ {\boldsymbol {\Delta} \mathbf {x}} ^ {\epsilon} (\boldsymbol {\Delta} \mathbf {x} _ {j - 1} + \eta \cdot \operatorname {s i g n} (\nabla_ {\mathbf {x}} L _ {f} (\mathbf {x} + \boldsymbol {\Delta} \mathbf {x} _ {j - 1}, y))), \tag {1}
$$
where  $\eta$  is the update step size and the  $\mathrm{Clip}_{\Delta \mathbf{x}}^{\epsilon}$  operation ensures that the norm of perturbation does not exceed  $\epsilon$ .
Black-box attacks. Issuing attacks in black-box settings is much more challenging than in white-box settings due to the lack of information of the victim model. In general, neither the architecture nor the parameter of the victim model is accessible in the black-box settings, and the attacker can only obtain the prediction confidence or even only the predicted label, providing any input. Two lines of methods have been developed for crafting adversarial examples under such circumstances, i.e., query-based methods and transfer-based methods. Query-based methods advocate searching along promising directions [4, 5, 49, 48] or estimating the gradient of victim models via zeroth-order optimization [6, 22, 23, 41], while transfer-based methods train substitute models to craft adversarial examples and can be query-free.
Adversarial transferability. As the core of transfer-based attacks, the transferability of adversarial examples from substitute models to unknown victim models has been studied widely over recent years, and a variety of methods have been proposed to enhance it. Related to our work, some previous work also achieves improved transferability by modifying the gradient computation on substitute models. For instance, Guo et al. [15] advocated removing non-linear activations during back-propagation, especially in later layers of DNNs. Wu et al. [45] suggested to take gradients more through skip connections, if exist, on the substitute models. In particular, intermediate-level representations were taken special care of in some prior arts [20, 26, 16, 43, 52].
3 Perturbation Decay in the Feature Space
We revisit some existing attacks that operate on intermediate-level representations and identify their possible limitations in Section 3.1 and propose a novel method to improve them in Section 3.2.
3.1 Revisiting Intermediate-level Attacks
As transfer-based attacks, intermediate-level attacks are performed with at least a substitute model. Given a model input  $\mathbf{x}$ , the output of an  $\alpha$ -layer substitute model can be simplified as  $f(\mathbf{x}) = W_{\alpha}^{T}\phi_{\alpha -1}(W_{\alpha -1}^{T}\ldots \phi_{1}(W_{1}^{T}\mathbf{x}))$ , where, for  $i\in \{1,\dots ,\alpha \}$ ,  $\phi_i$  is an activation function, and  $W_{i}\in \mathbb{R}^{n_{i - 1}\times n_{i}}$  parameterizes a layer in the model. Assume that the input is of  $n$  dimensions and there are  $c$  classes for prediction, then we have  $n_0 = n$ , and  $n_\alpha = c$ . Existing intermediate-level attacks decompose the model into a  $\beta$ -layer feature extractor  $h$  and an  $(\alpha -\beta)$ -layer classifier  $g$ , i.e., we can formulate the model as  $f(\mathbf{x}) = g(h(\mathbf{x}))$  for any  $\mathbf{x}$ . A motivation of intermediate-level attacks is that different models might share the same low-level representations in  $h$  and it suffices to operate on the output layer of  $h$  [20, 26].
A typical work of intermediate-level attack is ILA [20]. It attempts to regulate the intermediate-level perturbation (i.e., discrepancy between the intermediate-level representation of adversarial examples and that of benign examples) to show a large norm and to be close to a directional guide. The directional guide in ILA is chosen as the discrepancy from benign representations that could be reached in the output space of  $h$  and leads to maximum prediction loss of  $f$ , given the perturbation budget. Since white-box attacks all devote to achieving maximal prediction loss, ILA can be performed with simply a  $u$ -step baseline attack (e.g., I-FGSM) and the intermediate-level perturbation achieved by the baseline attack as the directional guide.
Let  $\mathbf{x} + \Delta \mathbf{x}_j$  be the adversarial input achieved at the  $j$ -th step of the baseline attack, we denote  $\mathbf{z}_h^{(j)} := h(\mathbf{x} + \Delta \mathbf{x}_j)$  as the intermediate-level representation of  $\mathbf{x} + \Delta \mathbf{x}_j$ . Further, denote by  $\mathbf{v}$  the directional guide, then we can formulate it as  $\mathbf{v} = (\mathbf{z}_h^{(u)} - \mathbf{z}_h^{(0)})$ .
ILA maximizes the scalar projection, which is
$$
\max  _ {\| \boldsymbol {\Delta} \mathbf {x} \| _ {p} \leq \epsilon} \mathbf {v} ^ {T} \left(h (\mathbf {x} + \boldsymbol {\Delta} \mathbf {x}) - h (\mathbf {x})\right), \tag {2}
$$
where  $1 / \| \mathbf{v}\| _2$  is omitted as it does not affect the optimal solution. With such a learning objective, the norm of the vector projection is maximized if  $\mathbf{v}$  and  $h(\mathbf{x} + \Delta \mathbf{x}) - h(\mathbf{x})$  form an acute angle. Since the direction of intermediate-level perturbation is not restricted to be in exactly the same direction as that of the directional guide, the obtained intermediate-level perturbation often deviates from the directional guide in practice. Yet, the adversarial perturbations are generally not robust enough [1], and such directional deviations normally leads to decrease in the prediction loss of the substitute model, even with large perturbation magnitude in the middle layer.
A quick experiment was conducted to examine the impact of such directional deviations. The experiment was conducted on CIFAR-10 [24] using a VGG-19 [36] as the substitute model and another VGG-19 with the same  $h$  as the victim model, just to simplify the analyses. The directional guide v of ILA was obtained by 10-steps I-FGSM following prior work [26, 16, 27]. We evaluated the cross-entropy prediction loss of the models with various intermediate-level perturbations to demonstrate how the attack performance can be affected by the scalar projection (onto v) and the directional deviation (from v). Somewhat unsurprisingly, as depicted in Figure 1, when fixing the direction of intermediate-level perturbations, the perturbations with larger magnitude leads to more significant attack per
(a) Substitute model
Figure 1: How the prediction loss of (a) the substitute model and (b) the victim model vary with the scalar projection onto  $\mathbf{v}$  and the direction deviation from  $\mathbf{v}$  (i.e., the cosine similarity between  $\mathbf{v}$  and  $h(\mathbf{x} + \Delta \mathbf{x}) - h(\mathbf{x}))$ . The purple box represents the results achieved by ILA.
(b) Victim model
formance on both the substitute model and the victim model. More enticingly, with intermediate-level perturbations showing similar scalar projection, those deviating more from the directional guide leads to lower prediction loss on the victim model and hence worse black-box attack performance. As illustrated in the figure, the ILA perturbation only slightly deviates from the directional guide  $\mathbf{v}$  (with a cosine similarity of roughly 0.90), yet its attack performance is already inferior to that of the intermediate-level perturbations along the direction of  $\mathbf{v}$  with even smaller magnitude. Follow-up work of ILA, e.g., [26, 16], mostly focuses on developing more powerful directional guides, yet the deviation remains and the same problem persists.
3.2 Our Solution
Given Figure 1, one may consider that a straightforward improvement of the intermediate-level attacks is to instead maximize the magnitude of (intermediate-level) perturbation only along the direction of  $\mathbf{v}$ . Nevertheless, most existing intermediate-level attacks consists of two separate stages which sequentially seeks  $\mathbf{v}$  and maximizes perturbation magnitude, and they fail to take into account whether there exists a larger intermediate-level perturbation along the direction of  $\mathbf{v}$ , given the perturbation budget in the input space of  $f$ .
In this context, we aim to develop a novel method that seeks intermediate-level perturbations to be in an adversarial direction and to possess larger norms (or say larger magnitudes) compared to a directional guide in the same direction simultaneously. As with previous methods, we consider a direction to be adversarial if a guide along this direction achieves large prediction loss by itself when being fed to the substitute classifier  $g$ . The intermediate-level perturbation with a relatively larger magnitude can simply be obtained by amplifying the directional guide by  $\gamma \times$  in the middle layer. However, such an intermediate-level perturbation may not be reachable with a limited perturbation budget, i.e.,  $\| \Delta \mathbf{x} \|_p \leq \epsilon$ , in the input space.
In light of all these considerations, we turn to the "dual" optimization problem and opt to maximize the prediction loss of  $g$  with a  $\gamma \times$  "decayed" intermediate-level perturbation (i.e.,  $(h(\mathbf{x} + \pmb{\Delta}\mathbf{x}) - h(\mathbf{x})) / \gamma$  which is regarded as the directional guide  $\mathbf{v}$ ). That is,
$$
\max  _ {\| \boldsymbol {\Delta} \mathbf {x} \| _ {p} \leq \epsilon} L (g (h (\mathbf {x}) + \frac {1}{\gamma} (h (\mathbf {x} + \boldsymbol {\Delta} \mathbf {x}) - h (\mathbf {x}))), y) = \max  _ {\| \boldsymbol {\Delta} \mathbf {x} \| _ {p} \leq \epsilon} L (g (\frac {1}{\gamma} h (\mathbf {x} + \boldsymbol {\Delta} \mathbf {x}) + (1 - \frac {1}{\gamma}) h (\mathbf {x})), y). \tag {3}
$$
Since the method seems decays the intermediate-level perturbation from the benign features, we call it intermediate-level perturbation decay (ILPD). Apparently, setting  $\gamma = 1$  means attack without the perturbation modification and it makes the method equivalent to the baseline multi-step attack. While, enforcing  $\gamma > 1$  leads to decayed perturbation as desired. The operation of  $h(\mathbf{x} + \Delta \mathbf{x}) / \gamma + (\gamma - 1)h(\mathbf{x}) / \gamma)$  resembles the mixup augmentation [51], except there is no randomness for  $\gamma$  and  $\gamma$  is suggested to be relatively large to encourage larger perturbation magnitude and larger loss.
Figure 2 demonstrates the performance of our ILPD, together with that of ILA. Solid dots  $\bullet ,\bullet ,\dots ,\bullet$  represent achievable performance of our ILPD with  $1 / \gamma \in \{1.0,0.9,\dots ,0.1\}$  while the blue cross mark  $\times$  represents the performance of ILA in the figure. From the vertical axis, it can be seen that the ILPD adversarial examples are capable of tricking the victim model into showing significantly larger prediction loss (4.08->4.54), despite possessing lower perturbation magnitude in the feature space, comparing with ILA examples. Given  $\gamma$  in the concerned range, ILPD-obtained perturbation directions are very effective in the sense of being adversarial. See the shaded area which illustrates how the attack performance varies with the perturbation magnitude along with these directions in the middle layer.
Figure 2: Comparing the achieved attack performance and the intermediate-level perturbation magnitude of ILA and our ILPD. Darker colors of the solid dots indicate larger values of  $\gamma$ . Best viewed in color.
It is worth noting that we normally suggest a relatively large  $\gamma$  with  $\gamma \geq 2$  (i.e.,  $1 / \gamma \leq 0.5$ ). The formulation of ILPD leads to a  $\gamma \times$  larger perturbation magnitude anyway, than that of  $(h(\mathbf{x} + \Delta \mathbf{x}) - h(\mathbf{x})) / \gamma$  which is regarded as the directional guide  $\mathbf{v}$ . However, given  $g$  being non-linear in practice, a  $\gamma \times$  amplified intermediate-level perturbation does not truly lead to  $\gamma \times$  larger prediction loss of  $g$ . In particular, the obtained loss is more difficult to be guaranteed especially when  $\gamma$  is extremely large, e.g.,  $\gamma \gg 10$  or, equivalently,  $1 / \gamma \ll 0.1$ . Therefore, we simply apply with  $0.1 \leq 1 / \gamma \leq 0.5$  in our main experiments in Section 5. Comprehensive ablation studies are deferred to Section 5.4.
4 Analysis from the Gradient Alignment Perspective
Recall that the diversity in input gradients across models is the main obstacle to adversarial transferability, thus, in this section, we analyze ILPD from the perspective of gradient alignment between the substitute and victim models, in order to shed more light on its effectiveness.
4.1 Impacting Factors of Gradient Alignment
Aiming at compromising an unknown victim model  $f_{\mathrm{t}}$ , a baseline transfer-based attack (e.g., I-FGSM) can generate adversarial example on a white-box substitute model  $f$ . In fact, if  $\nabla_{\mathbf{x}}L_{f}(\mathbf{x} + \Delta \mathbf{x}_{j},y)\approx$
(a) Similarity of input gradients
(b) Success rate
Figure 3: How (a) the cosine similarity of input gradients and (b) success rate vary with I-FGSM iterations, using tens pairs of substitute and victim models. The shaded areas indicate the standard deviation of the pairs. Best viewed in color.
$\nabla_{\mathbf{x}}L_{f_{\mathrm{t}}}(\mathbf{x} + \Delta \mathbf{x}_j,y)$ , then the obtained adversarial examples will be as powerful on the victim model. Yet, the input gradients are often not well-aligned, and, as depicted in Figure 3(a), after some initial iterations, the input gradient of  $L_{f}(\mathbf{x} + \Delta \mathbf{x}_{j},y)$  and that of  $L_{f_{\mathrm{t}}}\left(\mathbf{x} + \Delta \mathbf{x}_{j},y\right)$  may be increasingly different as  $j$  increases, which is probably a major obstacle to the adversarial transferability of multi-step attacks.
Let us delve deep into this. For simplicity of notations, the activation functions  $\phi_{i}$  is restricted to ReLU. Then, the input gradient  $\nabla_{\mathbf{x}}L_{f}(\mathbf{x} + \Delta \mathbf{x}_{j},y)$  can be written as the product of three terms:
$$
\begin{array}{l} \nabla_ {\mathbf {x}} L _ {f} (\mathbf {x} + \pmb {\Delta} \mathbf {x} _ {j}, y) = \nabla_ {\mathbf {x}} h (\mathbf {x} + \pmb {\Delta} \mathbf {x} _ {j}) \nabla_ {\mathbf {z} _ {h} ^ {(j)}} L (g (\mathbf {z} _ {h} ^ {(j)}), y) \\ = \nabla_ {\mathbf {x}} h (\mathbf {x} + \boldsymbol {\Delta} \mathbf {x} _ {j}) \left(W _ {\beta + 1} \prod_ {i = \beta + 1} ^ {\alpha - 1} D _ {i} \left(\mathbf {z} _ {h} ^ {(j)}\right) W _ {i + 1}\right) \nabla_ {\mathbf {z} _ {g} ^ {(j)}} L \left(\mathbf {z} _ {g} ^ {(j)}, y\right), \tag {4} \\ \end{array}
$$
following the chain rule, where  $D_{i}(\mathbf{z}_{h}^{(j)})$  provides a binary matrix whose diagonal entry is set to 1 if it corresponds to a nonzero activation within the  $i$ -th layer, and it is set to 0 if otherwise. If there exist max pooling layers, they can be similarly formulated, and the gradient through these layers can also be given as a binary matrix in  $\{D_{i}\}$ .
The input gradient of a victim model  $f_{\mathrm{t}}$  (i.e.,  $\nabla_{\mathbf{x}}L_{f_{\mathrm{t}}}(\mathbf{x} + \Delta \mathbf{x}_{j},y)$ ) can be similarly formulated as in Eq. (4). The diversity between  $\nabla_{\mathbf{x}}L_{f}(\mathbf{x} + \Delta \mathbf{x}_{j},y)$  and  $\nabla_{\mathbf{x}}L_{f_{\mathrm{t}}}(\mathbf{x} + \Delta \mathbf{x}_{j},y)$  probably comes from some difference in the activation masks in  $\{D_{i}\}$  and/or diversity in the gradients of  $L$  w.r.t. the logits (i.e.,  $\nabla_{\mathbf{z}_g}L(\mathbf{z}_g,y)$  where  $j$  is dropped for simplicity of notation). Note that the gradients of  $h$  are not concerned as it has been assumed in prior work that different models might share a similar  $h$ .
By replacing 1) activation masks in  $\{D_i\}$  and/or 2) the gradient of  $L$  w.r.t. the logits (i.e.,  $\nabla_{\mathbf{z}_g}L(\mathbf{z}_g,y)$ ) with those obtained on the victim model, we are capable of analyzing how much impact they actually make. An experiment is thus conducted, using ten pairs of substitute and victim models with the VGG-19 architecture. Figure 3 shows that the transferability can be remarkably improved by replacing any of them. Replacing  $\nabla_{\mathbf{z}_g}L(\mathbf{z}_g,y)$  seems more effective at initial iterations while replacing the activation masks leads to slightly more improvement at the end. Given the cross-entropy loss which evaluates the discrepancy between the prediction probabilities and the one-hot ground-truth vector  $\mathbf{y}$ , we have  $\nabla_{\mathbf{z}_g}L(\mathbf{z}_g,y) = \mathbf{p} - \mathbf{y}$  where  $\mathbf{p} = s(\mathbf{z}_g)$  is a vector containing all prediction probabilities and  $s$  is the softmax function. That said, at initial iterations, the difference in  $\mathbf{p}$  between the models is a more influential factor, and, at later iterations, the difference in  $\{D_i\}$  becomes dominant.
As the adversarial example is optimized on the substitute model, the prediction of a I-FGSM example is very likely to be less incorrect on the victim model even at the first iteration. Difference in  $\mathbf{p}$  across models leads to difference in the input gradient, and it is possible that the difference becomes more and more severe, forming a vicious circle in some sense.
4.2 Analysis of ILPD and Some Other Attacks
Recall that the input gradient obtained by ILPD at the  $j$ -th iteration is
$$
\nabla_ {\mathbf {x}} h (\mathbf {x} + \Delta \mathbf {x} _ {j}) \nabla_ {\tilde {\mathbf {z}} _ {h} ^ {(j)}} L (g \left(\tilde {\mathbf {z}} _ {h} ^ {(j)}\right), y) \tag {5}
$$
(b) Success rate
Figure 4: How (a) the cosine similarity of input gradients and (b) success rate change along with I-FGSM and ILPD iterations, using ten pairs of substitute and victim models. Shaded areas indicate the standard deviation. Best viewed in color.
in which  $\tilde{\mathbf{z}}_h^{(j)} = \frac{1}{\gamma}\mathbf{z}_h^{(j)} + (1 - \frac{1}{\gamma})\mathbf{z}_h^{(0)}$ . It can be regarded as replacing  $\nabla_{\mathbf{z}_h^{(j)}}L(g(\mathbf{z}_h^{(j)}),y)$  in Eq. (4) with the gradient of  $L$  w.r.t. to some less adversarial intermediate-level perturbation  $\tilde{\mathbf{z}}_h^{(j)}$ . Less adversarial intermediate-level perturbations lead to less incorrect predictions, which probably aligns better with that of the victim model.
Figure 4 shows that ILPD leads to more aligned input gradient across models, especially after the first 20 iterations. The attack success rates are compared in Figure 4(b), and it can be seen that, thanks to the improved gradient alignment, ILPD is capable of generating more transferable adversarial examples in comparison to I-FGSM, achieving  $+4.00\%$  absolute gains at the 100-th iteration. More detailed analyses show that the ILPD examples lead to activation masks and prediction probabilities better aligned with those of the victim model.
In addition to our ILPD, some other intermediate-level methods can also be seen as beneficial to gradient alignment. In particular, for ILA [20], we have