SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yian Deng
Manchester, UK, M13 9PL
yian.deng@manchester.ac.uk
Tingting Mu
tingting.mu@manchester.ac.uk
The strategy of ensemble has become popular in adversarial defense, which trains multiple base classifiers to defend against adversarial attacks in a cooperative manner. Despite the empirical success, theoretical explanations on why an ensemble of adversarially trained classifiers is more robust than single ones remain unclear. To fill in this gap, we develop a new error theory dedicated to understanding ensemble adversarial defense, demonstrating a provable 0-1 loss reduction on challenging sample sets in adversarial defense scenarios. Guided by this theory, we propose an effective approach to improve ensemble adversarial defense, named interactive global adversarial training (iGAT). The proposal includes (1) a probabilistic distributing rule that selectively allocates to different base classifiers adversarial examples that are globally challenging to the ensemble, and (2) a regularization term to rescue the severest weaknesses of the base classifiers. Being tested over various existing ensemble adversarial defense techniques, iGAT is capable of boosting their performance by up to  $17\%$  evaluated using CIFAR10 and CIFAR100 datasets under both white-box and black-box attacks.
Many contemporary machine learning models, particularly the end-to-end ones based on deep neural networks, admit vulnerabilities to small perturbations in the input feature space. For instance, in computer vision applications, a minor change of image pixels computed by an algorithm can manipulate the classification results to produce undesired predictions, but these pixel changes can be imperceptible to human eyes. Such malicious perturbations are referred to as adversarial attacks. These can result in severe incidents, e.g., medical misdiagnose caused by unauthorized perturbations in medical imaging [43], and wrong actions taken by autonomous vehicles caused by crafted traffic images [34].
The capability of a machine learning model to defend adversarial attacks is referred to as adversarial robustness. A formal way to quantify such robustness is through an adversarial risk, which can be intuitively understood as the expectation of the worst-scenario loss computed within a local neighborhood region around a naturally sampled data example [40]. It is formulated as below:
$$
R _ {a d v} (\mathbf {f}) = \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D}} \left[ \max  _ {\mathbf {z} \in \mathcal {B} (\mathbf {x})} \ell \left(\hat {y} \left(\mathbf {f} (\mathbf {z})\right), y\right) \right], \tag {1}
$$
where  $\mathcal{B}(\mathbf{x})$  denotes a local neighborhood region around the example  $\mathbf{x}$  sampled from a natural data distribution  $\mathcal{D}$ . The neighbourhood is usually defined based on a selected norm, e.g., a region containing any  $\mathbf{z}$  satisfying  $\| \mathbf{z} - \mathbf{x}\| \leq \varepsilon$  for a constant  $\varepsilon > 0$ . In general,  $\ell (\hat{y},y)$  can be any loss function quantifying the difference between the predicted output  $\hat{y}$  and the ground-truth output  $y$ . A
learning process that minimizes the adversarial risk is referred to as adversarially robust learning. In the classification context, a way to empirically approximate the adversarial robustness is through a classification error computed using a set of adversarial examples generated by applying adversarial attacks to the model [4]. Here, an adversarial attack refers to an algorithm that usually builds on an optimization strategy, and it produces examples perturbed by a certain strength so that a machine learning model returns the most erroneous output for these examples [10, 24].
On the defense side, there is a rich amount of techniques proposed to improve the model robustness [10, 1]. However, a robust neural network enhanced by a defense technique would mostly result in a reduced accuracy for classifying the natural examples [52]. A relation between the adversarial robustness and the standard accuracy has been formally proved by Tsipras et al. [38]: "Any classifier that attains at least  $1 - \delta$  standard accuracy on a dataset  $\mathcal{D}$  has a robust accuracy at most  $\delta p / (1 - p)$  against an  $L_{\infty}$ -bounded adversary with  $\varepsilon \geq 2\eta$ ," where  $\varepsilon$  is the  $L_{\infty}$  bound indicating the attack strength, while  $\eta$  is sufficiently large and  $p \geq 0.5$  and they both are used for data generation. This statement presents a tradeoff between the adversarial robustness and natural accuracy. There has been a consistent effort invested to mitigate such tradeoff by developing defense techniques to improve robust accuracy without sacrificing much the natural accuracy.
Ensemble defense has recently arisen as a new state of the art [4, 25]. The core idea is to train strategically multiple base classifiers to defend the attack, with the underlying motivation of improving the statistical stability and cooperation between base models. Existing effort on ensemble defense has mostly been focused on demonstrating performance success for different algorithmic approaches. It is assumed that training and combining multiple base models can defend better adversarial attacks as compared to training a single model. Although being supported by empirical success, there is few research that provides rigorous understanding to why this is the case in general. Existing results on analyzing generalized error for ensemble estimators mostly compare the error of the ensemble and the averaged error of the base models, through, for instance, decomposition strategies that divide the error term into bias, variance, co-variance, noise and/or diversity terms [45, 7, 39]. It is not straightforward to extend such results to compare the error of an ensemble model and the error of a model without an ensemble structure.
To address this gap, we develop a new error theory (Theorem 4.1) dedicated to understanding ensemble adversarial defense. The main challenge in mitigating the tradeoff between the adversarial robustness and natural accuracy comes from the fact that adversarial defence techniques can reduce the classifier's capacity of handling weakly separable example pairs that are close to each other but from different classes. To analyse how ensemble helps address this particular challenge, we derive a provable error reduction by changing from using one neural network to an ensemble of two neural networks through either the average or max combiner of their prediction outputs.
Although ensemble defense can improve the overall adversarial robustness, its base models can still be fooled individually in some input subspaces, due to the nature of the collaborative design. Another contribution we make is the proposal of a simple but effective way to improve each base model by considering the adversarial examples generated by their ensemble accompanied by a regularization term that is designed to recuse the worst base model. We experiment with the proposed enhancement by applying it to improve four state-of-the-art ensemble adversarial defense techniques. Satisfactory performance improvement has been observed when being evaluated using CIFAR-10 and CIFAR-100 datasets [51] under both white-box and black-box attacks.
Adversarial Attack: Typical attack techniques include white-box and black-box attacks. The white-box adversary has access to the model, e.g., the model parameters, gradients and formulation, etc, while the black-box adversary has limited knowledge of the model, e.g., knowing only the model output, and therefore is closer to the real-world attack setting [5]. Representative white-box attacks include Deepfool [27] and the projected gradient descent (PGD) [26]. The fast gradient sign method (FGSM) [18] is a simplified one-iteration version of PGD. The momentum iterative method (MIM) [15] improves the FGSM attack by introducing the momentum term. Another two commonly-used and cutting-edge white-box attacks are the Carlini-Wagner (CW) attack [8] that computes the perturbation by optimizing the perturbation scales, and the Jacobian-based saliency map attack (JSMA) [31] that perturbs only one pixel in each iteration. Representative black-box attacks include
the square attack (SA) [3], the SignHunter attack [2] and the simple black-box attack (SimBA) [19], among which SA can generate stronger attacks but is more computationally expensive. A thorough survey on adversarial attacks is provided by Akhtar et al. [1]. AutoAttack [12] encapsulates a selected set of strong white and black-box attacks and is considered as the state-of-the-art attack tool. In general, when evaluating the adversarial robustness of a deep learning model, it should be tested under both the white and black-box attacks.
Adversarial Defense: Adversarial training is the most straightforward and commonly used defense technique for improving adversarial robustness. It works by simply expanding the training data with additional adversarial examples. The initial idea was firstly adopted by Christian et al. [11] to train robust neural networks for classification, later on, Madry et al. [26] used adversarial examples generated by the PGD attack for improvement. Overall, this strategy is very adaptive and can be used to defend any attack, but it is on the expense of consuming more training examples. It has been empirically observed that adversarial training can reduce loss curvatures in both the input and model parameter spaces [17, 16], and this reduces the adversarial robustness gap between the training and testing data [52, 47]. These findings motivate the development of a series of regularization techniques for adversarial defense, by explicitly regularizing the curvature or other relevant geometric characteristics of the loss function. Many of these techniques can avoid augmenting the training set and are computationally cheaper. Typical regularization techniques that attempt to flatten the loss surface in the input space include the curvature regularization (CURE) [28], local linearity regularization (LLR) [32], input gradient regularization [29] and Lipschitz regularization [41]. There are also techniques to flatten the loss in the model parameter space, such as TRADES [52], misclassification aware adversarial training (MART) [44], robust self-training (RST) [9, 33], adversarial weight perturbation (AWP) [47], and HAT [46], etc. Alternative to adversarial training and regularization, other defense strategies include pruning [35], pre-training [22], feature denoising [48], domain adaptation [36] and ensemble defense [25, 37], etc. Croce et al. [13] has provided a summary of recent advances.
Ensemble Adversarial Defense: Recently, ensemble has been actively used in adversarial defense, showing promising results. One core spirit behind ensemble is to encourage diversity between base models in order to achieve an improved ensemble prediction [45, 7, 39]. Therefore, the advances are mostly focused on designing effective ensemble diversity losses in training to improve adversarial robustness. For instance, the adaptive diversity promoting (ADP) method [30] uses Shannon entropy for uncertainty regularization and a geometric diversity for measuring the difference between the predictions made by base classifiers. The transferability reduced smooth (TRS) method [50] formulates the diversity term based on cosine similarities between the loss gradients of base models, meanwhile increases the model smoothness via an  $l_{2}$ -regularization of these gradients during training. The similar idea of exploiting loss gradients was proposed earlier in the gradient alignment loss (GAL) method [23]. The conditional label dependency learning (CLDL) method [42] is a latest improvement over GAL and TRS, which measures the diversity using both the predictions and loss gradients of the base models. However, the ensemble nature of encouraging diversity can cause vulnerability for some base models over certain subsets of adversarial examples [28]. In practice, this can limit the overall robustness of the ensemble when the other base models are not strong enough to correct the weak ones. To address this, the diversifying vulnerabilities for enhanced robust generation of ensembles (DVERGE) [49] proposes a vulnerability diversity to encourage each base model to be robust particularly to the other base models' weaknesses. The latest development for improving ensemble defense, known as synergy-of-experts (SoE) [14], follows a different research path. For each input, it adaptively selects a base model with the largest confidence to make the final prediction instead of combining all, for which the supporting algorithm and theory have been developed. Some surveys on ensemble adversarial attacks and defense can be found in He et al. [21], Lu et al. [25].
Bold capital and lower-case letters, e.g.,  $\mathbf{X}$  and  $\mathbf{x}$ , denote matrices and vectors, respectively, while lower-case letters, e.g.,  $x$ , denote scalars. The  $i$ -th row and column of a matrix  $\mathbf{X}$  are denoted by  $\mathbf{x}_i$  and  $\mathbf{x}^{(i)}$ , respectively, while  $x_{i,j}$  and  $x_{i}$  the elements of  $\mathbf{X}$  and  $\mathbf{x}$ . A classification dataset  $D = \{(\mathbf{x}_i,y_i)\}_{i=1}^n$  includes  $n$  examples, which are referred to as natural examples, with  $\mathbf{x}_i \in \mathcal{X} \subset \mathbb{R}^d$  (feature vector) and  $y_i \in [C] = \{1,2,\ldots,C\}$  (class label). We sometimes express the label of an example as  $y(\mathbf{x})$  or  $y_{\mathbf{x}}$ . Storing  $\mathbf{x}_i$  as a row of  $\mathbf{X} \in \mathbb{R}^{n \times d}$  and  $y_i$  an element of  $\mathbf{y} \in \mathbb{R}^n$ , we also denote this dataset by  $D = (\mathbf{X},\mathbf{y})$ . The classifier  $\mathbf{f}: \mathcal{X} \to [0,1]^C$  outputs class probabilities
usually computed by a softmax function. Given the computed probability  $f_{c}$  for the  $c$ -th class,  $\hat{y}_{\mathbf{f}}(\mathbf{x}) = \arg \max_{c \in [C]} f_{c}(\mathbf{x})$  predicts the class. For a neural network, we denote by  $\mathbf{W}^{(l)}$  the weight matrix connecting the  $l$ -th and the  $(l-1)$ -th layers and by  $w_{i,j}^{(l)}$  its  $ij$ -th element. The  $L_{2}$ -norm  $\| \cdot \|_{2}$  is used to compute the vector length, while the  $L_{\infty}$ -norm  $\| \cdot \|_{\infty}$  to generate adversarial attacks. Concatenation of two sets is denoted by the symbol  $\cup$ .
We focus on classification by minimizing a classification loss  $\ell(\mathbf{f}(\mathbf{x}), y_{\mathbf{x}})$ , and adapt it to  $\ell(\mathbf{f}(\mathbf{X}), \mathbf{y})$  for the whole dataset. Also, we use  $\ell_{CE}$  to emphasize the cross-entropy loss. The loss gradient is  $\nabla \ell(\mathbf{f}(\mathbf{x}), y_{\mathbf{x}}) = \frac{\partial \ell(\mathbf{f}(\mathbf{x}), y_{\mathbf{x}})}{\partial \mathbf{x}}$ . A cheap way to estimate the loss curvature is by finite difference approximation [28], e.g., the following curvature measure based on  $L_2$ -norm:
$$
\lambda_ {\mathbf {f}} (\mathbf {x}, \delta) = \frac {\left\| \nabla \ell \left(\mathbf {f} \left(\mathbf {x} + \delta\right) , y _ {\mathbf {x}}\right) - \nabla \ell \left(\mathbf {f} (\mathbf {x}) , y _ {\mathbf {x}}\right) \right\| _ {2}}{\left\| \boldsymbol {\delta} \right\| _ {2}}, \tag {2}
$$
where  $\delta \in \mathbb{R}^d$  is a perturbation. It measures how a surface bends at a point by different amounts in different directions. An adversarial example  $\tilde{\mathbf{x}} = \phi (\mathbf{f},\mathbf{x},A)$  is generated by attacking the classifier  $\mathbf{f}$  using an attack algorithm  $A$  on a natural example  $\mathbf{x}$ . It is further adapted to  $\tilde{\mathbf{X}} = \phi (\mathbf{f},\mathbf{X},A)$  for the set of adversarial examples each generated from a natural example in  $\mathbf{X}$ . The quantity  $\delta (\mathbf{f},\mathbf{x},A)$ $= \phi (\mathbf{f},\mathbf{x},A) - \mathbf{x}$  is referred to as the adversarial perturbation of  $\mathbf{x}$ , simplified to  $\delta_{\mathbf{x}} = \tilde{\mathbf{x}} -\mathbf{x}$ . To control the perturbation strength, we restrict  $\| \delta_{\mathbf{x}}\|_{\infty}\leq \varepsilon$  for some  $\varepsilon >0$ , which results in the following adversarial example formulation, as
$$
\phi_ {\varepsilon} (\mathbf {f}, \mathbf {x}, A) = \min  \left(\max  \left(\phi (\mathbf {f}, \mathbf {x}, A), \mathbf {x} - \varepsilon\right), \mathbf {x} + \varepsilon\right), \tag {3}
$$
where both  $\min (\cdot ,\cdot)$  and  $\max (\cdot ,\cdot)$  are element-wise operators comparing their inputs.
4 An Error Theory for Adversarial Ensemble Defense
In adversarial ensemble defense, a widely accepted research hypothesis is that training and combining multiple base classifiers can improve adversarial defense as compared to training a single classifier. However, this hypothesis is mostly supported by empirical successes and there is a lack of formal theoretical justification. In this work, we seek theoretical evidence, proving that, when using multi-layer perceptrons (MLPs) for classification, classification error reduces when applying adversarial defence to the base MLPs of an ensemble as compared to a single MLP, under assumptions feasible in practice. The following theorem formalizes our main result.
Theorem 4.1. Suppose  $\mathbf{h},\mathbf{h}^0,\mathbf{h}^1\in \mathcal{H}:\mathcal{X}\to [0,1]^C$  are  $C$ -class  $L$ -layer MLPs satisfying Assumption 4.2. Given a dataset  $D = \{(\mathbf{x}_i,y_i)\}_{i = 1}^n$ , construct an ambiguous pair set  $A(D)$  by Definition 4.3. Assume  $\mathbf{h},\mathbf{h}^0,\mathbf{h}^1$  are acceptable classifiers for  $A(D)$  by Assumption 4.4. Given a classifier  $\mathbf{f}\in \mathcal{H}:\mathcal{X}\rightarrow \mathbb{R}^{C}$  and a dataset  $D$ , assess its classification error by 0-1 loss, as
$$
\hat {\mathcal {R}} _ {0 / 1} (D, \mathbf {f}) = \frac {1}{| D |} \sum_ {\mathbf {x} \in D} 1 \left[ f _ {y _ {\mathbf {x}}} (\mathbf {x}) <   \max  _ {c \neq y _ {\mathbf {x}}} f _ {c} (\mathbf {x}) \right], \tag {4}
$$
where  $1[true] = 1$  while  $1[false] = 0$ . For an ensemble  $\mathbf{h}_e^{(0,1)}$  of two base MLPs  $\mathbf{h}^0$  and  $\mathbf{h}^1$  through either an average or a max combiner, i.e.,  $\mathbf{h}_e^{(0,1)} = \frac{1}{2} (\mathbf{h}^0 + \mathbf{h}^1)$  or  $\mathbf{h}_e^{(0,1)} = \max(\mathbf{h}^0, \mathbf{h}^1)$ , it has a lower empirical 0-1 loss than a single MLP for classifying ambiguous examples, such as
$$
\mathbb {E} _ {a \sim A (D)} \mathbb {E} _ {\mathbf {h} ^ {0}, \mathbf {h} ^ {1} \in \mathcal {H}} \left[ \hat {\mathcal {R}} _ {0 / 1} \left(a, \mathbf {h} _ {e} ^ {(0, 1)}\right) \right] <   \mathbb {E} _ {a \sim A (D)} \mathbb {E} _ {\mathbf {h} \in \mathcal {H}} \left[ \hat {\mathcal {R}} _ {0 / 1} (a, \mathbf {h}) \right]. \tag {5}
$$
We prove the result for MLPs satisfying the following assumption.
Assumption 4.2 (MLP Requirement). Suppose a  $C$ -class  $L$ -layer MLP  $\mathbf{h}:\mathbb{R}^d\to [0,1]^C$  expressed iteratively by
$$
\mathbf {a} ^ {(0)} (\mathbf {x}) = \mathbf {x}, \tag {6}
$$
$$
\mathbf {a} ^ {(l)} (\mathbf {x}) = \sigma \left(\mathbf {W} ^ {(l)} \mathbf {a} ^ {(l - 1)} (\mathbf {x})\right), l = 1, 2, \dots , L - 1, \tag {7}
$$
$$
\mathbf {a} ^ {(L)} (\mathbf {x}) = \mathbf {W} ^ {(L)} \mathbf {a} ^ {(L - 1)} (\mathbf {x}) = \mathbf {z} (\mathbf {x}), \tag {8}
$$
$$
\mathbf {h} (\mathbf {x}) = \operatorname {s o f t m a x} (\mathbf {z} (\mathbf {x})), \tag {9}
$$
where  $\sigma(\cdot)$  is the activation function applied element-wise, the representation vector  $\mathbf{z}(\mathbf{x}) \in \mathbb{R}^C$  returned by the  $L$ -th layer is fed into the prediction layer building upon the softmax function. Let  $w_{s_{l+1}, s_l}^{(l)}$  denote the network weight connecting the  $s_l$ -th neuron in the  $l$ -th layer and the  $s_{l+1}$ -th neuron in the  $(l+1)$ -th layer for  $l \in \{1, 2, \ldots, L\}$ . Define a column vector  $\mathbf{p}^{(k)}$  with its  $i$ -th element computed from the neural network weights and activation derivatives, as  $p_i^{(k)} = \sum_{s_L} \frac{\partial a_{s_L}^{(L-1)}(\mathbf{x})}{\partial x_k} w_i^{(L)}$ , for  $k = 1, 2, \ldots, d$  and  $i = 1, 2, \ldots, C$ , also a matrix  $\mathbf{P_h} = \sum_{k=1}^d \mathbf{p}^{(k)} \mathbf{p}^{(k)^T}$  and its factorization  $\mathbf{P_h} = \mathbf{M_h} \mathbf{M_h}^T$  with a full-rank factor matrix  $\mathbf{M_h}$ . For constants  $\lambda, B > 0$ , suppose the following holds for  $\mathbf{h}$ :
1. Its cross-entropy loss curvature measured by Eq. (2) satisfies  $\lambda_{\mathbf{h}}(\mathbf{x},\pmb {\delta})\leq \tilde{\lambda}$
2. The factor matrix satisfies  $\| \mathbf{M_h}\| _2\leq B_0$  and  $\left\| \mathbf{M_h}^\dagger \right\| _2\leq B$  , where  $\| \cdot \| _2$  denotes the vector induced  $l_{2}$  -norm for matrix.
We explain the feasibility of the above MLP assumptions in the end of this section.
Although adversarial defense techniques can improve adversarial robustness, new challenges arise in classifying examples that are close to each other but from different classes, due to the flattened loss curvature for reducing the adversarial risk. We refer to a pair of such challenging examples as an ambiguous pair. Our strategy of proving improved performance for adversarial defense is to (1) firstly construct a challenging dataset  $A(D)$  comprising samples from these pairs, which is referred to as an ambiguous pair set, and then (2) prove error reduction over  $A(D)$ . To start, we provide formal definitions for the ambiguous pair and set.
Definition 4.3 (Ambiguous Pair). Given a dataset  $D = \{(\mathbf{x}_i, y_i)\}_{i=1}^n$  where  $\mathbf{x}_i \in \mathcal{X}$  and  $y_i \in [C]$ , an ambiguous pair contains two examples  $a = ((\mathbf{x}_i, y_i), (\mathbf{x}_j, y_j))$  satisfying  $y_i \neq y_j$  and
$$
\left\| \mathbf {x} _ {i} - \mathbf {x} _ {j} \right\| _ {2} \leq \frac {1}{J B \sqrt {C (\tilde {\lambda} ^ {2} - \xi)}}, \tag {10}
$$
where  $J > 2$  is an adjustable control variable,  $\tilde{\lambda}$ ,  $B$  and  $\xi \leq \tilde{\lambda}^2$  are constants associated with the MLP under Assumption 4.2. The ambiguous pair set  $A(D)$  contains all the ambiguous pairs existing in  $D$ , for which  $J$  is adjusted such that  $A(D) \neq \emptyset$ .
In Theorem 4.1, we are only interested in classifiers that do not fail too badly on  $A(D)$ , e.g., having an accuracy level above  $42.5\%$ . Comparing poorly performed classifiers is not very meaningful, also the studied situation is closer to practical setups where the starting classifiers for improvement are somewhat acceptable. Such a preference is formalized by the following assumption:
Assumption 4.4 (Acceptable Classifier). Suppose an acceptable classifier  $\mathbf{f}:\mathbb{R}^d\to [0,1]^C$  does not perform poorly on the ambiguous pair set  $A(D)$  associated with a control variable  $J$ . This means that, for any pair  $a = ((\mathbf{x}_i,y_i),(\mathbf{x}_j,y_j))\in A(D)$  and for any example  $(\mathbf{x}_i,y_i)$  from the pair, the following holds:
1. With a probability  $p \geq 42.5\%$ , the classifier correctly classifies  $(\mathbf{x}_i, y_i)$  by a sufficiently large predicted score, i.e.,  $f_{y_i}(\mathbf{x}_i) \geq 0.5 + \frac{1}{J}$ , while wrongly classifies the other example  $\mathbf{x}_j$  to  $y_i$  by a less score, i.e.,  $f_{y_i}(\mathbf{x}_j) \leq 0.5 + \frac{1}{J}$ .
2. When the classifier predicts  $(\mathbf{x}_i, y_i)$  to class  $\hat{y}_i$ , the predicted scores for the other classes excluding  $y_i$  are sufficiently small, i.e.,  $f_c(\mathbf{x}_i) \leq \frac{1 - f_{\hat{y}_i}(\mathbf{x}_i)}{C - 1}$  for  $c \neq y_i, \hat{y}_i$ .
Proof for Theorem 4.1 together with a toy illustration example is provided in supplementary material.