SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption Discussion. Assumption 4.2 is feasible in practice. Reduced loss curvature is a natural result from adversarial defense, particularly for adversarial training and regularization based methods [16, 17] as mentioned in Section 2. Regarding its second part determined by neural network weights and activation derivatives, common training practices like weight regularization and normalization help prevent from obtaining overly inflated elements in  $\mathbf{P_h}$ , and thus bound  $\| \mathbf{M_h}\| _2$  and  $\left\| \mathbf{M_h}^\dagger \right\| _2$ . Following Definition 4.3, the ambiguous pair  $a = ((\mathbf{x}_i,y_i),(\mathbf{x}_j,y_j))$  is constructed to
let the classifier struggle with classifying the neighbouring example, e.g.,  $(\mathbf{x}_j, y_j)$ , when it is able to classify successfully, e.g.,  $(\mathbf{x}_i, y_i)$ . Consequently, the success of classifying  $(\mathbf{x}_i, y_i)$  is mostly accompanied with a failure of classifying  $(\mathbf{x}_j, y_j)$  into  $y_i$ , and vice versa. In Assumption 4.4, for an acceptable classifier, the first part assumes its failure is fairly mild, while the second part assumes its struggle is between  $y_i$  and  $y_j$ . As shown in our proof of Theorem 4.1, in order for Eq. (5) to hold, a polynomial inequality of the probability  $p$  needs to be solved, providing a sufficient condition on achieving a reduced ensemble risk, i.e.,  $p \geq 42.5\%$ . Later, we conduct experiments to examine how well some assumptions are met by adversarially trained classifiers and report the results in supplementary material.
5 iGAT: Improving Ensemble Mechanism
Existing ensemble adversarial defense techniques mostly base their design on a framework of combining classification loss and diversity for training. The output of each base classifier contains the probabilities of an example belonging to the  $C$  classes. For an input example  $\mathbf{x} \in \mathcal{X}$ , we denote its output from the  $i$ -th base classifier by  $\mathbf{h}^i(\mathbf{x}) = [h_1^i(\mathbf{x}), \dots, h_C^i(\mathbf{x})]$  for  $i \in [N]$ , where  $N$  denotes the number of used base classifiers. Typical practice for combining base predictions includes the averaging, i.e.,  $\mathbf{h}(\mathbf{x}) = \frac{1}{N} \sum_{i=1}^{N} \mathbf{h}^i(\mathbf{x})$ , or the max operation, i.e.,  $h_j(\mathbf{x}) = \max_{i \in [N]} (h_j^i(\mathbf{x}))$ . Without loss of generality, we denote the combiner by  $\mathbf{h} = \mathbf{c}(\mathbf{h}^1, \dots, \mathbf{h}^N)$ . To train the base classifiers, we exemplify an ensemble loss function using one training example  $(\mathbf{x}, y_{\mathbf{x}})$ , as below
$$
L _ {E} (\mathbf {x}, y _ {\mathbf {x}}) = \underbrace {\sum_ {i = 1} ^ {N} \ell \left(\mathbf {h} ^ {i} (\mathbf {x}) , y _ {\mathbf {x}}\right)} _ {\text {c l a s s i f i c a t i o n l o s s}} + \omega \operatorname {R e g} (\mathbf {h} (\mathbf {x})) + \gamma \operatorname {D i v e r s i t y} \left(\mathbf {h} ^ {1} (\mathbf {x}), \mathbf {h} ^ {2} (\mathbf {x}), \dots , \mathbf {h} ^ {N} (\mathbf {x}), y _ {\mathbf {x}}\right)), \tag {11}
$$
where  $\omega, \gamma \geq 0$  are hyperparameters. An example choice for regularization is the Shannon entropy of the ensemble  $\mathbf{h}(\mathbf{x})$  [30]. Significant research effort has been invested to diversity design, for which it is optional whether to use the class information in diversity calculation. In the first section of supplementary material, we briefly explain four ensemble adversarial defense techniques highlighting their loss design strategies. These include ADP [30], CLDL [42], DVERGE [49] and SoE [14], and they are used later in Section 6 to test our proposed enhancing approach.
Despite the effort in diversity design that encourages better collaboration between base classifiers, it is unavoidable for some base classifiers to struggle with classifying examples from certain input subspaces. There are intersected subspaces that all the base classifiers are not good at classifying. To address this, we propose an interactive global adversarial training (iGAT) approach. It seeks support from adversarial examples globally generated by the ensemble and distributes these examples to base classifiers with a probabilistic strategy empirically proven effective. Additionally, it introduces another regularization term to improve over the severest weakness of the base classifiers. Below we describe our proposal in detail.
5.1 Distributing Global Adversarial Examples
We aim at improving adversarial robustness over intersected feature subspaces which are hard for all base classifiers to classify. These regions can be approximated by global adversarial examples generated by applying adversarial attacks to the ensemble, which are
$$
\left(\tilde {\mathbf {X}}, \tilde {\mathbf {y}}\right) = \left(\phi_ {\varepsilon} \left(\mathbf {c} \left(\mathbf {h} ^ {1}, \dots , \mathbf {h} ^ {N}\right), \mathbf {X}, A\right), \mathbf {y}\right), \tag {12}
$$
where rows of  $\tilde{\mathbf{X}}$  store the feature vectors of the generated adversarial examples. For instance, the FGSM attack can be used as  $A$ . Instead of feeding the same full set of adversarial examples to train each base classifier, we distribute different examples to different base classifiers, to improve performance and to reduce training time. The generated examples are divided into  $N$  groups according to their predicted class probabilities. The  $i$ -th group  $\left(\tilde{\mathbf{X}}^i,\tilde{\mathbf{y}}^i\right)$  is used to train the  $i$ -th base classifier, contributing to its classification loss.
Our core distributing strategy is to encourage each base classifier to keep improving over regions that they are relatively good at classifying. This design is motivated by our theoretical result. We have proved in Theory 4.1 an error reduction achieved by the ensemble of base MLPs that satisfy
the acceptability Assumption 4.4. This assumption is partially examined by whether the classifier returns a sufficiently high prediction score for the correct class or low scores for most of the incorrect classes for some challenging examples. By keeping assigning each base classifier new challenging examples that they are relatively good at classifying, it encourages Assumption 4.4 to continue to hold. In Section 6.3, we perform ablation studies to compare our proposal with a few other distributing strategies, and the empirical results also verify our design. Driven by this strategy, we propose one hard and one soft distributing rule.
Hard Distributing Rule: Given a generated adversarial example  $(\tilde{\mathbf{x}},y) = \left(\left(\tilde{\mathbf{X}}\right)_k,\tilde{y}_k\right)$ , the following rule determines which base classifier to assign it:
$$
\text {I f} h _ {y} ^ {i} (\tilde {\mathbf {x}}) > \max  _ {j \neq i, j \in [ N ]} h _ {y} ^ {j} (\tilde {\mathbf {x}}), \text {a s s i g n} (\tilde {\mathbf {x}}, y) \text {t o} \left(\tilde {\mathbf {X}} ^ {i}, \tilde {\mathbf {y}} ^ {i}\right). \tag {13}
$$
We refer to it as a hard distributing rule as it simply assigns examples in a deterministic way. The example is assigned to the base classifier that returns the highest predicted probability on its ground truth class.
Soft Distributing Rule: A hard assignment like the above can be sensitive to errors. Alternatively, we propose a soft distributing rule that utilizes the ranking of the base classifiers based on their prediction performance meanwhile introduces uncertainty. It builds upon roulette wheel selection [6], which is a commonly used genetic operator in genetic algorithms for selecting promising candidate solutions. Firstly, we rank in descending order the predicted probabilities  $\{h_y^i (\mathbf{x})\}_{i = 1}^N$  by all the base classifiers for the ground truth class, and let  $r_{\mathbf{x}}(\mathbf{h}^i)\in [N]$  denote the obtained ranking for the  $i$ -th base classifier. Then, we formulate a ranking-based score for each base classifier as
$$
p _ {i} = \frac {2 ^ {N - r _ {\mathbf {x}} \left(\mathbf {h} ^ {i}\right)}}{\sum_ {i \in [ N ]} 2 ^ {i - 1}}, \tag {14}
$$
and it satisfies  $\sum_{i\in [N]}p_i = 1$ . A more top ranked base classifier has higher score. Next, according to  $\{p_i\}_{i = 1}^N$ , we apply roulette wheel selection and distribute the example to the selected base classifier. Specifically, the selection algorithm constructs  $N$  intervals  $\{[a_i,b_i]\}_{i = 1}^N$  where  $a_1 = 0$ ,  $b_{1} = p_{1}$  also  $a_{i} = b_{i - 1}$  and  $b_{i} = a_{i} + p_{i}$  for  $i = 2,3,\ldots ,N$ . After sampling a number  $q\in (0,1]$  following a uniform distribution  $q\sim U(0,1)$ , check which interval  $q$  belongs to. If  $a_{i} < q\leq b_{i}$ , then the example is used to train the  $i$ -th base classifier. This enables to assign examples based on ranking but in a probabilistic manner in order to be more robust to errors.
5.2 Regularization Against Misclassification
We introduce another regularization term to address the severest weakness, by minimizing the probability score of the most incorrectly predicted class by the most erroneous base classifier. Given an input example  $(\mathbf{x}, y_{\mathbf{x}})$ , the proposed term is formulated as
$$
L _ {R} (\mathbf {x}, y _ {\mathbf {x}}) = - \delta_ {0 / 1} \left(\mathbf {c} \left(\mathbf {h} ^ {1} (\mathbf {x}), \dots , \mathbf {h} ^ {N} (\mathbf {x})\right), y _ {\mathbf {x}}\right) \log \left(1 - \max  _ {i = 1} ^ {C} \max  _ {j = 1} ^ {N} h _ {i} ^ {j} (\mathbf {x})\right). \tag {15}
$$
Here,  $\delta_{0/1}(\mathbf{f}, y) \in \{0, 1\}$  is an error function, where if the input classifier  $\mathbf{f}$  can predict the correct label  $y$ , it returns 0, otherwise 1. This design is also motivated by Assumption 4.4, to encourage a weak base classifier to perform less poorly on challenging examples so that its chance of satisfying the acceptability assumption can be increased.
5.3 Enhanced Training and Implementation
The proposed enhancement approach iGAT, supported by (1) the global adversarial examples generated and distributed following Section 5.1 and (2) the regularization term proposed in Section 5.2, can be applied to any given ensemble adversarial defense method. We use  $L_{E}$  to denote the original
ensemble loss as in Eq. (11), the enhanced loss for training the base classifiers becomes
$$
\begin{array}{l} \min  _ {\left\{\mathbf {h} ^ {i} \right\} _ {i = 1} ^ {N}} \underbrace {\mathbb {E} _ {\left(\mathbf {x} , y _ {\mathbf {x}}\right) \sim \left(\mathbf {x} , \mathbf {y}\right)} \left[ L _ {E} \left(\mathbf {x} , y _ {\mathbf {x}}\right) \right]} _ {\text {o r i g i n a l e n s e m b l e l o s s}} + \underbrace {\alpha \sum_ {i = 1} ^ {N} \mathbb {E} _ {\left(\mathbf {x} , y _ {\mathbf {x}}\right) \sim \left(\tilde {\mathbf {X}} ^ {i} , \tilde {\mathbf {y}} ^ {i}\right)} \left[ \ell_ {C E} \left(\mathbf {h} ^ {i} (\mathbf {x}), y _ {\mathbf {x}}\right) \right]} _ {\text {a d d e d g l o b a l a d v e r s a r i a l l o s s}} \tag {16} \\ + \underbrace {\beta \mathbb {E} _ {(\mathbf {x} , y _ {\mathbf {x}}) \sim (\mathbf {X} , \mathbf {y}) \cup \left(\tilde {\mathbf {X}} , \tilde {\mathbf {y}}\right)} \left[ L _ {R} \left(\mathbf {x} , y _ {\mathbf {x}}\right) \right]} _ {\text {a d d e d m i s c l a s f i c a t i o n r e g u l a r i z a t i o n}}, \\ \end{array}
$$
where  $\alpha, \beta \geq 0$  are hyper-parameters. In practice, the base classifiers are firstly trained using an existing ensemble adversarial defense technique of interest, i.e., setting  $\alpha = \beta = 0$ . If some pre-trained base classifiers are available, they can be directly used instead, and fine-tuned with the complete loss. In our implementation, we employ the PGD attack to generate adversarial training examples, as it is the most commonly used in existing literature and in practice.
6 Experiments and Results of iGAT
In the experiments, we compare with six state-of-the-art ensemble adversarial defense techniques including ADP [30], CLDL [42], DVERGE [49], SoE [14], GAL [30] and TRS [50]. The CIFAR-10 and CIFAR-100 datasets are used for evaluation, both containing 50,000 training and 10,000 test images [51]. Overall, ADP, CLDL, DVERGE and SoE appear to be the top performing methods, and we apply iGAT<sup>1</sup> to enhance them. The enhanced, referred to as iGAT<sub>ADP</sub>, iGAT<sub>CLDL</sub>, iGAT<sub>DVERGE</sub> and iGAT<sub>SoE</sub>, are compared with their original versions, and additionally GAL [30] and TRS [50].
6.1 Experiment Setting
We test against white-box attacks including PGD with 20 inner optimization iterations and CW with  $L_{\infty}$  loss implemented by Wu et al. [47], and the black-box SignHunter (SH) attack [2] with 500 maximum loss queries. In accordance with Carmon et al. [9], the CW attack is applied on 1,000 equidistantly sampled testing examples. We also test against the strongest AutoAttack (AA) [12], which encapsulates variants of the PGD attack and the black-box square attack [3]. All attack methods use the perturbation strength  $\varepsilon = 8 / 255$ .
For all the compared methods, an ensemble of  $N = 8$  base classifiers with ResNet-20 [20] backbone is experimented, for which results of both the average and max output combiners are reported. To implement the iGAT enhancement, the soft distributing rule from Eq. (14) is used. The two hyper-parameters are set as  $\alpha = 0.25$  and  $\beta = 0.5$  for SoE, while  $\alpha = 5$  and  $\beta = 10$  for ADP, CLDL and DVERGE, found by grid search. Here SOE uses a different parameter setting because its loss construction differs from the others, thus it requires a different scale of the parameter range for tuning  $\alpha$  and  $\beta$ . In practice, minor adjustments to hyper-parameters have little impact on the results. The iGAT training uses a batch size of 512, and multi-step leaning rates of  $\{0.01, 0.002\}$  for CIFAR10 and  $\{0.1, 0.02, 0.004\}$  for CIFAR100. Implementation of existing methods uses either their pre-trained models or their source code for training that are publicly available. Each experimental run used one NVIDIA V100 GPU plus 8 CPU cores.
6.2 Result Comparison and Analysis
We compare different defense approaches by reporting their classification accuracies computed using natural images and adversarial examples generated by different attack algorithms, and report the results in Table 1. The proposed enhancement has lifted the performance of ADP and DVERGE to a state-of-the-art level for CIFAR-10 under most of the examined attacks, including both the white-box and black-box ones. The enhanced DVERGE by iGAT has outperformed all the compared methods in most cases for CIFAR-100. In addition, we report in Table 2 the accuracy improvement obtained by iGAT for the studied ensemble defense algorithms, computed as their accuracy difference normalised by the accuracy of the original algorithm. It can be seen that iGAT has positively improved the baseline methods in almost all cases. In many cases, it has achieved an accuracy boost over  $10\%$ .
<sup>1</sup>The source codes and pre-trained models can be found at https://github.com/xqsi/iGAT.
Table 1: Comparison of classification accuracies in percentage reported on natural images and adversarial examples generated by different attack algorithms under  $L_{\infty}$ -norm perturbation strength  $\varepsilon = 8 / 255$ . The results are averaged over five independent runs. The best performance is highlighted in bold, the 2nd best underlined.
Average Combiner (%)	Max Combiner (%)
Natural	PGD	CW	SH	AA	Natural	PGD	CW	SH	AA
CIFAR10	TRS	83.15	12.32	10.32	39.21	9.10	82.67	11.89	10.78	37.12	7.66
GAL	80.85	41.72	41.20	54.94	36.76	80.65	31.95	27.80	50.68	9.26
SoE	82.19	38.54	37.59	59.69	32.68	82.36	32.51	23.88	41.04	18.37
iGATSoE	81.05	40.58	39.65	57.91	34.50	81.19	31.98	24.01	40.67	19.65
CLDL	84.15	45.32	41.81	55.90	37.04	83.69	39.34	32.80	51.63	15.30
iGATCLDL	85.05	45.45	42.00	58.22	37.14	83.73	40.84	34.55	51.70	17.03
DVERGE	85.12	41.39	43.40	57.33	39.20	84.89	41.13	39.70	54.90	35.15
iGATDVERGE	85.48	42.53	44.50	57.77	39.48	85.27	42.04	40.70	54.79	35.71
ADP	82.14	39.63	38.90	52.93	35.53	80.08	36.62	34.60	47.69	27.72
iGATADP	84.96	46.27	44.90	58.90	40.36	80.72	39.37	35.00	48.36	29.83
CIFAR100	TRS	58.18	10.32	10.12	15.78	6.32	57.21	9.98	9.23	14.21	4.34
GAL	61.72	22.04	21.60	31.97	18.01	59.39	19.30	13.60	24.73	10.36
CLDL	58.09	18.47	18.01	29.33	15.52	55.51	18.89	13.07	22.14	4.51
iGATCLDL	59.63	18.78	18.20	29.49	14.36	56.91	20.76	14.09	20.43	5.20
SoE	62.60	20.54	19.60	36.35	15.90	62.62	16.00	11.40	24.25	8.62
iGATSoE	63.19	21.89	19.70	35.60	16.16	63.02	16.02	11.45	23.77	8.95
ADP	60.46	20.97	20.55	30.26	17.37	56.20	17.86	13.70	21.40	10.03
iGATADP	60.17	22.23	20.75	30.46	17.88	56.29	17.89	14.10	21.47	10.09
DVERGE	63.09	20.04	20.01	32.74	17.27	61.20	20.08	15.30	27.18	12.09
iGATDVERGE	63.14	23.20	22.50	33.56	18.59	61.54	20.38	17.80	27.88	13.89
Table 2: Accuracy improvement in percentage by iGAT, i.e.  $\frac{\mathrm{iGAT - original}}{\mathrm{original}} \times  {100}\%$  ,reported on natural images and adversarial examples generated by different attack algorithms under  ${L}_{\infty }$  -norm perturbation strength  $\varepsilon  = 8/{255}$  .
Average Combiner	Max Combiner
Natural	PGD	CW	SH	AA	Natural	PGD	CW	SH	AA
CIFAR10	ADP	+3.43	+16.75	+15.42	+11.28	+13.59	+0.80	+7.51	+1.16	+1.40	+7.61
DVERGE	+0.42	+2.75	+2.53	+0.77	+0.71	+0.45	+2.21	+2.52	-0.20	+1.59
SoE	-1.39	+5.29	+5.48	-2.98	+5.57	-1.42	-1.63	+0.54	-0.90	+6.97
CLDL	+1.07	+0.29	+0.45	+4.15	+0.27	+0.05	+3.81	+5.34	+0.14	+11.31
CIFAR100	ADP	-0.48	+6.01	+0.97	+0.66	+2.94	+0.16	+0.17	+2.92	+0.33	+0.60
DVERGE	+0.08	+15.77	+12.44	+2.50	+7.64	+0.56	+1.49	+16.34	+2.58	+14.89
SoE	+0.94	+6.57	+0.51	-2.06	+1.64	+0.64	+0.13	+0.44	-1.98	+3.83
CLDL	+2.65	+1.68	+1.05	+0.55	-7.47	+2.52	+9.90	+7.80	-7.72	+15.30
Here are some further discussions on the performance. We observe that DVERGE and its iGAT enhancement perform proficiently on both CIFAR10 and CIFAR100, while ADP and its enhancement are less robust on CIFAR100. We attempt to explain this by delving into the algorithm nature of DVERGE and ADP. The ADP design encourages prediction disparities among the base models. As a result, each base model becomes proficient in classifying a subset of classes that the other base models may struggle with. However, a side effect of this is to discourage base models from becoming good at overlapping classes, which may become ineffective when having to handle a larger number of classes. The reasonably good improvement achieved by iGAT for ADP in Table 2 indicates that an addition of global adversarial examples is able to rescue such situation to a certain extent. On the other hand, in addition to encouraging adversarial diversity among the base models, DVERGE also aims at a stable classification so that each example is learned by multiple base models. This potentially makes it suitable for handling both large and small numbers of classes. Moreover, we also observe that the average combiner provides better performance than the max combiner in general.
Table 3: Results of ablation studies based on iGAT $_{\text{ADP}}$  using CIFAR-10 under the PGD attack. The results are averaged over five independent runs. The best performance is highlighted in bold.
Opposite Distributing	Random Distributing	Hard Distributing	β = 0	iGATADP
Natural (%)	82.45	83.05	83.51	83.45	84.96
PGD (%)	41.31	42.60	44.21	42.32	46.25
The reason can be that an aggregated prediction from multiple well-trained base classifiers is more statistically stable.
The key designs of iGAT include its distributing rule and the regularization term. We perform ablation studies to examine their effectiveness. Firstly, we compare the used soft distributing rule with three alternative distributing rules, including (1) a distributing rule opposite to the proposed, which allocates the adversarial examples to the base models that produce the lowest prediction score, (2) a random distributing rule by replacing Eq. (14) by a uniform distribution, and (3) the hard distributing rule in Eq. (13). Then, we compare with the setting of  $\beta = 0$  while keeping the others unchanged. This change removes the proposed regularization term. Results are reported in Table 3 using iGAT $_{\text{ADP}}$  with the average combiner, evaluated by CIFAR-10 under the PGD attack. It can be seen that a change or removal of a key design results in obvious performance drop, which verifies the effectiveness of the design.
7 Conclusion, Limitation and Future Work
We investigate the challenging and crucial problem of defending against adversarial attacks in the input space of a neural network, with the goal of enhancing ensemble robustness against such attacks while without sacrificing much the natural accuracy. We have provided a formal justification of the advantage of ensemble adversarial defence and proposed an effective algorithmic improvement, bridging the gap between theoretical and practical studies. Specifically, we have proven a decrease in empirical 0-1 loss calculated on data samples challenging to classify, which is constructed to simulate the adversarial attack and defence scenario, under neural network assumptions that are feasible in practice. Also, we have proposed the iGAT approach, applicable to any ensemble adversarial defense technique for improvement. It is supported by (1) a probabilistic distributing rule for selectively allocating global adversarial examples to train base classifiers, and (2) a regularization penalty for addressing vulnerabilities across all base classifiers. We have conducted thorough evaluations and ablation studies using the CIFAR-10 and CIFAR-100 datasets, demonstrating effectiveness of the key designs of iGAT. Satisfactory performance improvements up to  $17\%$  have been achieved by iGAT.
However, there is limitation in our work. For instance, our theoretical result is developed for only two base MLPs. We are in progress of broadening the scope of Theorem 4.1 by further relaxing the neural network assumptions, researching model architectures beyond MLPs and beyond the average/max combiners, and more importantly generalizing the theory to more than two base classifiers. Additionally, we are keen to enrich our evaluations using large-scale datasets, e.g., ImageNet. So far, we focus on exploiting curvature information of the loss landscapes to understand adversarial robustness. In the future, it would be interesting to explore richer geometric information to improve the understanding. Despite the research success, a potential negative societal impact of our work is that it may prompt illegal attackers to develop new attack methods once they become aware of the underlying mechanism behind the ensemble cooperation.
We thank the five NeurIPS reviewers for their very insightful and useful comments that help improve the paper draft. We also thank "The University of Manchester - China Scholarship Council Joint Scholarship" for funding Yian's PhD research.