SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junqi Gao\*,1, Biqing Qi\*,2,3,4, Yao Li†, 1, Zhichang Guo†, Dong Li†, Yuming Xing†, Dazhi Zhang†
$^{1}$ School of Mathematics, Harbin Institute of Technology
$^{2}$ Department of Control Science and Engineering, Harbin Institute of Technology
$^{3}\mathrm{C}^{3}\mathrm{I}$ , Department of Electronic Engineering, Tsinghua University
4Frontis.AI
* Equal contribution, † Corresponding author.
{gjunqi97, qibiqing7, mathgzc, arvinlee826}  $@$  gmail.com
{yaoli0508, xyuming, zhangdazhi}  $@$  hit.edu.cn
The transferability of adversarial perturbations provides an effective shortcut for black-box attacks. Targeted perturbations have greater practicality but are more difficult to transfer between models. In this paper, we experimentally and theoretically demonstrated that neural networks trained on the same dataset have more consistent performance in High-Sample-Density-Regions (HSDR) of each class instead of low sample density regions. Therefore, in the target setting, adding perturbations towards HSDR of the target class is more effective in improving transferability. However, density estimation is challenging in high-dimensional scenarios. Further theoretical and experimental verification demonstrates that easy samples with low loss are more likely to be located in HSDR. Perturbations towards such easy samples in the target class can avoid density estimation for HSDR location. Based on the above facts, we verified that adding perturbations to easy samples in the target class improves targeted adversarial transferability of existing attack methods. A generative targeted attack strategy named Easy Sample Matching Attack (ESMA) is proposed, which has a higher success rate for targeted attacks and outperforms the SOTA generative method. Moreover, ESMA requires only  $5\%$  of the storage space and much less computation time comparing to the current SOTA, as ESMA attacks all classes with only one model instead of seperate models for each class. Our code is available at https://github.com/gjq100/ESMA
Deep learning models exhibits substantial computational capacity in many downstream tasks, but are vulnerable to adversarial attacks [1, 2]. Such attacks tend to be transferable [3, 4] as well as real-world achievable [5], which makes it implementable in black-box scenarios. Targeted attacks are known to be more difficult to transfer [4, 6] compared with non-targeted attacks.
Intuitively, directions and transferability of adversarial attacks are closely related, but their relationship is rarely discussed. [7] found that samples in low density regions of ground truth distribution are more susceptible to adversarial attacks. They also verified that adversarial perturbations that aligned with the low density direction of ground truth distribution can lead to better transferability. For targeted attack scenarios, the direction that can bring more transferability has not been fully researched yet is not clear.
Findings and arguments. In non-targeted scenarios, directing towards low-density regions of the ground-truth distribution can improve adversarial transferability [7]. However, this approach is not direct enough for targeted attacks. As shown in Figure 1, perturbations pointing at High-Sample-Density-Regions (HSDR) of the target domain are more effective than that pointing at low-density regions of the ground-truth distribution, it perturbs samples more directly to the target discrimination region. Moreover, we demonstrate and theoretically prove that neural networks tend to have more consistent outputs in the HSDR of each class, which means that perturbations pointing to the HSDR of the target class are also transferable between different models. However, density estimation for samples in high dimension is challenging.
Fortunately, by the definitions of hard samples and easy samples in [8], we found a fact that helps, in each category, easy samples with smaller losses are more likely to locate in HSDR. We provide theoretical and experimental assurance for this fact. Based on this fact, we can directly perturbate towards such easy samples of the target domain to improve transferability without density estimation.
Related works. Adversarial attacks can be divided into white-box attacks and black-box attacks. In the white-box setting, attackers have access to the model's structure and parameters, while in the black-box setting, they have no such information but only access to the input and output of the model. This is typically the scenario encountered in real-world situations. Black-box attacks include query-based attacks [9, 10, 11] and transfer-based attacks [12, 13, 14]. Conducting too many queries is impractical in real-world applications. In contrast, transfer-based attacks are more feasible as they do not require queries. Transfer-based attacks often require the use of a white-box surrogate model to produce adversarial perturbations. In terms of the way adversarial perturbations are generated, there are
Figure 1: A schematic example of our motivation, plotting the probability density (darker the color represents larger the density) and samples for three populations (orange, cyan, and green). The black line indicates the Bayesian discriminant boundary.
two approaches: iterative instance-specific methods and generative methods. Iterative instance-specific methods utilize model gradients to iteratively add perturbations to specified samples (e.g., FGSM [3], C&W [5], PGD [15]). To enhance the transferability of adversarial samples, subsequent work combines these methods with various techniques, such as introducing momentum [12, 13] and considering input transformations [13, 14, 16, 17] during iterations, training auxiliary classifiers [18, 19], or substituting different loss functions [20, 21, 22]. However, instance-specific methods are primarily designed for non-targeted scenarios, often lacking effectiveness in targeted settings. Although relatively good results have been achieved for targeted attacks [20, 21, 23], instance-specific methods still require iteratively creating perturbations for each specified sample, while generators trained for generating adversarial perturbations can be generalized on more samples after training [24, 25, 26, 27].
The leading perturbation generative method currently is TTP [24], which employs a target-specific GAN to align clean and augmented data from the source domain with the target domain data. However, TTP necessitate training a dedicated generator for each class. This significantly increases storage requirements and training time. In our work, we design a generator for simultaneous attacks on all target classes, which uses class embedding information for each target class. To construct these embeddings, we adopt techniques similar to SNE [28] to align the surrogate model's output logits for target classes with the generator's embeddings. This enables latent features learned by the surrogate model to guide the construction of well-structured class embeddings. Building on our findings, we train a multi-class perturbation generator to simultaneously perturb the source domain samples towards easy samples with low loss of each target class. Experiments on the ImageNet dataset shown that our method can obtain a better target transfer success rate than TTP, while requiring much less storage. Our method also has certain advantages in training time.
Therefore, our contribution can be summarized as follows:
- We found that deep learning models have more consistent outputs in HSDR of each class, as demonstrated theoretically and through experiments (Section 2.1). This implies that adding
adversarial perturbations pointing to the HSDR of the target class results in better targeted adversarial transferability.
- We experimentally and theoretically verified that easy samples with low loss in early-stopping models are likely to be located in HSDR, which allows us to directly add perturbations pointing to such samples of the target class to improve targeted adversarial transferability (Section 2.2). This avoids density estimation of high-dimensional samples, which is challenging and computationally expensive.
- We introduced the Easy Sample Matching Attack (ESMA) (Section 3). ESMA achieves a higher targeted transfer success rate compared to SOTA generative attacks TTP. Furthermore, it only needs one model to perform attacks for all target classes (Section 4), which requires much less storage space than TTP (only about 1/20 of the storage space in 10 targets case), and requires less training time.
Our findings and conclusions can not only provide guidance for target transfer attacks, but more importantly, they reveal the consistency of deep learning models in the HSDR, and the correlation between sample difficulty and local sample density. Not only that, the design of our multi-target perturbation generative model can provide a new reference for subsequent related research.
2 Main conclusions
In this section, we combine illustrative experiments and theoretical proofs to illustrate our two conclusions in turn, and we use the following notations and definitions.  $\mathcal{Z} = \mathcal{X}\times \mathcal{Y}\in$ $\mathbb{R}^d\times \mathbb{R}$  is the sample space. The i.i.d. dataset  $S = \{x_{i},y_{i}\}_{i = 1}^{n}$  consists of  $n$  sample pairs  $(x_{i},y_{i}),1\leq i\leq n$  . Given  $y$  , the conditional distribution of  $x$  is  $\mathcal{D}_{x|y}$  . Feature mapping  $f:\mathcal{X}\to \mathbb{R}^{K}$  , where  $K$  is the number of class, the class of feature mapping  $f\in \mathcal{F}$  . Specifically, the parametrized class  $\mathcal{F}_{\boldsymbol{w}}:= \left\{f_{\boldsymbol{w}}:\mathcal{X}\to \mathbb{R}^{K},\boldsymbol {w}\in \mathcal{W}\right\}$  , where  $\mathcal{W}$  is the parameter space. Define  $\mathcal{F}_w^S\coloneqq \left\{f_w^S = S\circ f_w:f_w\in \mathcal{F}_w,S\circ f_w(x) = S(f_w(x))\right\}$ $s$  denotes Softmaxtransformation. The Softmax-Cross-Entropy loss is denoted as  $\ell_{sce}(\cdot ,\cdot):\mathbb{R}^K\times \mathbb{R}\rightarrow \mathbb{R}$  . Let  $S_{j}\coloneqq \{(x,y)\in S:y = j\} ,\mathcal{I}_{j}\coloneqq \left\{i:(x_{i},y_{i})\in S^{j}\right\} ,\mathcal{C}_{j}\coloneqq \{x\in \mathcal{X}:(x,y)\in \mathcal{Z},y = j\}$
Definition 1  $((j, x_0, r)$ -Local sample density) Given a class  $j \in [K]$ ,  $(x_0, y_0) \in S_j$ , the  $(j, x_0, r)$ -Local sample density:
$$
\rho_ {(j, x _ {0}, r)} = \frac {\sum_ {i \in \mathcal {I} _ {j}} \mathbb {1} (x _ {i} \in \mathcal {B} (x _ {0} , r))}{\operatorname {v o l} \mathcal {B} (x _ {0} , r)}
$$
where  $\mathcal{B}(x_0,r)\coloneqq \left\{x\in \mathbb{R}^d:\| x - x_0\| \leq r\right\}$ ,  $\operatorname {vol}\mathcal{B}(x_0,r)$  denotes the volume of  $\mathcal{B}(x_0,r)$ .
Now we use the notation  $\mathcal{I}_{(j,x_0,r)} = \{i:(x_i,y_i)\in S_j,x_i\in \mathcal{B}(x_0,r),(x_0,y_0)\in S_j)\}$ $\mathcal{C}_{(j,x_0,r)} =$ $\{x:x\in \mathcal{C}_j,x_i\in \mathcal{B}(x_0,r),(x_0,y_0)\in S_j)\}$  . For simplicity, we denote  $\ell_{sce}(f_{\pmb{w}}(x),y)$  as  $\ell (\pmb {w},x)$  and define the local empirical risk  $R_{(j,x_0,r)}(\pmb {w}) = \frac{1}{|\mathcal{I}_{(j,x_0,r)}|}\sum_{i\in \mathcal{I}_{(j,x_0,r)}}\ell (\pmb {w},x_i).$
2.1 The Output Consistency of Different Deep Learning Models in HSDR
We conduct an experiment to explain why samples in low-density regions of ground-truth distribution are vulnerable to attacks, and verify the consistency of the output of the deep learning model in HSDR. A dataset consist of 200 samples is constructed by sampling from two 2-d Gaussian distributions with equal probability, which is then used to train a neural network for classification. Subsequently, we plot the discriminant region of the Bayes' criterion (which has the minimum error rate) for the known ground truth prior distribution and compared it with the discriminant region of the trained classifier. In addition, we train two other neural networks with different structures and parameter quantities, and plot the output differences of the three neural networks (Figure 2). The discriminant region in Figure 2(a) can reach the Bayesian error rate. The intersection of the two population distributions in the middle of the probability density curve (as shown in Figure 2(b)) belongs to the low-density region of the ground-truth distribution and also to the misclassified region. In such a region, even with minimal expected error, the Bayesian discriminant criterion will classify samples with relatively small probability density of a single population into another class. These samples can be thought of as "outliers". The trained NN classifier discriminates samples into their original categories, causing a
(b)
Figure 2: (a): Bayesian discriminant region, darker the color indicate higher the confidence probability. (b): Classifier discriminant region, the probability density curves of the two population distributions are plotted, the white part represents the low-density region of ground truth joint distribution, and we boxed out the small pits in the Bayesian misclassified region. (c): Classifier discriminant region with samples. We boxed an outlier. (d): Output differences between three different classifiers, darker purple indicates greater difference in output between different classifiers
difference in decision boundaries from the Bayesian prior classifier and creating small pits as shown in Figure 2(a). Most of the samples around the pit (Figure 2(c)) belong to another class due to its lower population density compared to the other class. Hence, perturbing the samples from this pit (i.e. outliers) towards the discriminant region of the other class becomes easier. If other trained neural networks can accurately classify such outliers, they will also generate similar pits. By adding perturbations pointing these pits to samples from another class, they will be perturbed into such pits, enabling the transfer of adversarial samples between different classifiers, which explained the findings of [7].
However, as we stated in Figure 1, perturbations towards the low-density regions of the ground-truth distribution are not direct enough for targeted attacks, while perturbations towards the HSDR of the target class are more direct. Combining this with Figure 2(d), we observe that different classifiers have more consistent outputs in the HSDR. With Theorem 1, we theoretically verified this, which also implies that perturbing samples to the HSDR of the target class leads to better targeted adversarial transferability.
Theorem 1 (Local output consistency) For a target class  $j \in [K]$ , and two different parametrized class  $\mathcal{F}_{\boldsymbol{w}_1} := \{f_{\boldsymbol{w}_1} : \boldsymbol{w}_1 \in \mathcal{W}_1\}$ ,  $\mathcal{F}_{\boldsymbol{w}_2} := \{f_{\boldsymbol{w}_2} : \boldsymbol{w}_2 \in \mathcal{W}_2\}$ , assume that  $\frac{1}{|\mathcal{I}_{(j, x_0, r)}|} \left| \sum_{i \in \mathcal{T}_{(j, x_0, r)}} \left( f_{\boldsymbol{w}_1}^{\mathcal{S}_k}(x_i) - f_{\boldsymbol{w}_2}^{\mathcal{S}_k}(x_i) \right) \right| \leq \gamma$ , then for any sample  $(x_0, y_0) \in S_j$ , in the neighborhood  $\mathcal{B}(x_0, r)$ , with probability at least  $1 - \delta$ , the following holds:
$$
\begin{array}{l} \left\| \mathbb {E} _ {x \sim \mathcal {D} _ {x | y}} \left[ f _ {\boldsymbol {w} _ {1}} ^ {\mathcal {S}} (x) - f _ {\boldsymbol {w} _ {2}} ^ {\mathcal {S}} (x) \mid x \in \mathcal {C} _ {(j, x _ {0}, r)} \right] \right\| _ {\infty} \\ \leq \mathcal {O} \left(\sqrt {\frac {K d ^ {d / 2}}{\rho_ {(j , x _ {0} , r)} 2 ^ {d} r ^ {d}}} \log^ {2} \left(r ^ {d} \sqrt {\rho_ {(j , x _ {0} , r)}}\right) + \sqrt {\frac {d ^ {d / 2} \log (2 K / \delta)}{\rho_ {(j , x _ {0} , r)} 2 ^ {d + 1} r ^ {d}}}\right) + \gamma . \\ \end{array}
$$
Remark. Theorem 1 suggests that different models have a more consistent output near the samples in HSDR, specifically, the difference between the outputs has a bound of order  $\tilde{O}\left(\sqrt{\frac{Kd^{d/2}}{\rho(j,x_0,r)2^d r^d}}\right)$ . When the number of classes and dimensions are fixed, a larger sample density results in more consistent performance, a weaker consistency within smaller neighborhoods. Note that this local consistency is weakened as the dimension increases, but the relative output consistency between HSDR and LSDR does not change.
However, a very practical problem is that sample points in high-dimensional space tend to be very discrete due to the curse of dimensionality [29], which makes it difficult to find a suitable local neighborhood size to calculate the local density of samples, moreover, in the case of large datasets, calculating local density can be computationally expensive. But the conclusion of the next section can help us find the sample located in the HSDR using early-stopping models without directly calculating the local density of the sample.
2.2 Correlation between Local Sample Density and Sample Difficulty
In this section we illustrate the correlation between sample density and sample difficulty. Hard samples have attracted much attention in various tasks because of their significance for training
convergence and model generalization [8, 30, 31, 32]. [8] proposed that for the convergent model, the difficulty of the sample can be measured by the loss gradient norm of the sample, easy samples has a relatively small loss gradient norm, while the loss gradient norm of difficult samples is relatively large, especially, those with too large gradient norm may be outliers.
We use the following theoretical analysis to illustrate that for a trained classifier, the samples in the HSDR tend to have a smaller local empirical risk. At the same time, smaller loss gradient norms ensure that losses are more consistent in small neighborhoods, which implies those samples with both smaller loss and loss gradient norms guarantee less local empirical risk in the neighborhood where the sample is located. To illustrate the following conclusions, we make several mild assumptions:
Assumption 1 (Smoothness assumption)  $\ell(w, x)$  satisfies the following conditions of Lipschitz continuous gradient:
$$
\begin{array}{l} \left\| \nabla_ {\boldsymbol {w}} \ell \left(\boldsymbol {w} _ {1}, x\right) - \nabla_ {\boldsymbol {w}} \ell \left(\boldsymbol {w} _ {2}, x\right) \right\| \leq L _ {1} \left\| \boldsymbol {w} _ {1} - \boldsymbol {w} _ {2} \right\|, \forall \boldsymbol {w} _ {1}, \boldsymbol {w} _ {2} \in \mathcal {W}, \\ \left\| \nabla_ {x} \ell (\boldsymbol {w}, x _ {1}) - \nabla_ {x} \ell (\boldsymbol {w}, x _ {2}) \right\| \leq L _ {2} \left\| x _ {1} - x _ {2} \right\|, \forall x _ {1}, x _ {2} \in \mathcal {X}. \\ \end{array}
$$
Assumption 2  $\| \nabla_{\boldsymbol{w}}\ell (\boldsymbol {w},x)\| \leq G$  for all  $\boldsymbol {w}\in \mathcal{W}$
Assumption 3 (Polyak-Lojasiewicz Condition)  $R_{(j,x_0,r)}(\pmb{w})$  satisfies the PL-condition:
$$
\frac {1}{2} \left\| \nabla_ {\boldsymbol {w}} R _ {(j, x _ {0}, r)} (\boldsymbol {w}) \right\| ^ {2} \geq \mu \left(R _ {(j, x _ {0}, r)} (\boldsymbol {w}) - R _ {(j, x _ {0}, r)} ^ {*}\right),
$$
where  $R_{(j,x_0,r)}(\pmb{w}) = \frac{1}{|\mathcal{I}_{(j,x_0,r)}|}\sum_{i\in \mathcal{I}_{(j,x_0,r)}}\ell (\pmb{w},x_i)$  is the local empirical risk,  $R_{(j,x_0,r)}^* = \inf_{\pmb{w}\in \mathcal{W}}R_{(j,x_0,r)}(\pmb{w})$
Assumption 4 For  $\forall i\in \mathcal{I}_{(j,x_0,r)}$ , the following holds:
$$
\left\langle \nabla_ {\boldsymbol {w}} R _ {(j, x _ {0}, r)} (\boldsymbol {w}), \nabla_ {\boldsymbol {w}} \ell (\boldsymbol {w}, x _ {i}) \right\rangle \geq \beta \left\| \nabla_ {\boldsymbol {w}} R _ {(j, x _ {0}, r)} (\boldsymbol {w}) \right\| ^ {2}
$$
Assumption 1 and 2 were made in [33], [34] and [35], where Assumption 2 can actually be derived from Assumption 1 when the input space is bounded, which is usually satisfied. Even a non-convex function can still satisfy Assumption 3 [34], the inequality in Assumption 3 implies that all stationary points are global minimum [36], which is proved in recently works [37, 38] for over-parameterized DNNs. Assumption 4 is reasonable when local neighborhood radius  $r$  is small.
Algorithm 1 Mini-batch SGD
Require: Initialized weights  $\boldsymbol{w}^{1}$ , total steps  $T$ , sample set  $S$ , batch size  $M$  and step size  $\eta_{t}$ .
1: for  $t = 1 \gets T$  do
2: Randomly sample  $M$  different samples  $B^t$  from  $S$ , where batch size  $|B_t| = M$ , corresponding indicator set denote as  $\mathcal{I}_{B^t}$ .
3:  $\pmb{w}^{t + 1} = \pmb{w}^t -\frac{1}{M}\sum_{i\in \mathcal{I}_{B^t}}\nabla_{\pmb{w}}\ell (\pmb{w}^t,x_i)$
4: end for
Return:  $\boldsymbol{w}^{T + 1}$
Consider the local empirical risk under Algorithm 1, we use Theorem 2 to illustrate the convergence rate relies on local density.
Theorem 2 (Optimization relies on local density) Given a learnable parametrized class  $\mathcal{F}_{\boldsymbol{w}} := \{f_{\boldsymbol{w}} : \mathcal{X} \to \mathbb{R}^{K}, \boldsymbol{w} \in \mathcal{W}\}$ , let  $\boldsymbol{w}^t$  updated by Algorithm 1, under Assumption 1, 2, 3 and 4, set  $\eta_t = \frac{1}{\beta \mu t}$  and  $T \leq \frac{L_1}{2\beta \mu}$ , then with probability at least  $1 - \delta$ , holds
$$
R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t + 1}) - R _ {(j, x _ {0}, r)} ^ {*} \leq \left(1 - \frac {2}{t} \tau (\rho_ {(j, x _ {0}, r)})\right) (R _ {(j, x _ {0}, r)} (\boldsymbol {w} ^ {t}) - R _ {(j, x _ {0}, r)} ^ {*}) + o \left(\frac {1}{t ^ {2}}\right),
$$
where  $\tau \left(\rho_{(j,x_0,r)}\right) = \max \left\{\left(\frac{\rho_{(j,x_0,r)}\pi^{d / 2}r^d}{\Gamma\left(\frac{d}{2} + 1\right)M} -\sqrt{\frac{\ln(T / \delta)}{2M}}\right),0\right\}$ , which is a non-descending function of  $\rho_{(j,x_0,r)}$ .
According to theorem 2, the local empirical risk  $R_{(j,x_0,r)}(\pmb{w})$  will reach a more faster convergence rate in HSDR and, relatively, a relatively slower convergence rate in LSDR, thus for early-stopping models, they have a lower local empirical risk in HSDR. Combined with the following Proposition 1, we show that a smaller gradient norm guarantees a smaller local empirical risk. For overfitting models, the local empirical risk of each neighborhood where samples are located may be very small, but for early-stopping models, this relativity of local empirical risk with respect to local sample density can be maintained.
Proposition 1 Under Assumption 1, given any  $(x_i,y_i)\in S$ , for any  $x\in \mathcal{X}$  that satisfies  $\| x_{i} - x\| \leq r$ , the following holds:
$$
\frac {\left| \ell (\boldsymbol {w} , x _ {i}) - \ell (\boldsymbol {w} , x) \right|}{r} - \frac {3 L _ {2} r}{2} \leq \| \nabla_ {x} \ell (\boldsymbol {w}, x _ {i}) \| \leq \frac {\ell (\boldsymbol {w} , x _ {i})}{r} + \frac {3 L _ {2} r}{2}.
$$
Remark. Proposition 1 suggests that minimizing the loss leads to a smaller norm of the loss gradient. However, this constraint becomes less tight as the neighborhood radius  $r$  decreases. Compared with the loss itself, the loss gradient norm further guarantees the proximity of local loss, especially when  $r$  is small. Therefore, samples with smaller loss and smaller loss gradient norm are more likely to be in a neighborhood with smaller local empirical risk.
Using the above conclusion, for an early-stopping model, samples with smaller losses and loss gradient norms tend to be located in HSDR. Still in the example in the previous section, we train three models with early-stopping, and the learning rate adjusted with the number of steps, and then plotted Figure 3. The model has more consistent outputs in HSDR. When the loss and gradient norms of a sample are small, the region where the sample is located has smaller local empirical risk. Samples in HSDR have smaller local empirical risk, which is consistent with our theoretical analysis. Additionally, the rightmost graph of Figure 3 shows that samples with smaller loss and gradient norms often locate in HSDR, validating our conclusions.
Therefore, we can determine whether a sample is more likely to located in HSDR or LSDR by evaluating whether it has smaller loss and gradient norms simultaneously. This eliminates the need to calculate local sample densities to find samples in HSDR.