PARL: Enhancing Diversity of Ensemble Networks to Resist Adversarial Attacks via Pairwise Adversarially Robust Loss Function
Authors: Manaar Alam
Shubhajit Datta
Debdeep Mukhopadhyay
Arijit Mondal
Partha Pratim Chakrabarti
Venue: ICLR 2022 Submitted (2022)
Keywords: Adversarial Attack
Ensemble-based Defence
Model Diversity

Abstract: The security of Deep Learning classifiers is a critical field of study because of the existence of adversarial attacks. Such attacks usually rely on the principle of transferability, where an adversarial example crafted on a surrogate classifier tends to mislead the target classifier trained on the same dataset even if both classifiers have quite different architecture. Ensemble methods against adversarial attacks demonstrate that an adversarial example is less likely to mislead multiple classifiers in an ensemble having diverse decision boundaries. However, recent ensemble methods have either been shown to be vulnerable to stronger adversaries or shown to lack an end-to-end evaluation. This paper attempts to develop a new ensemble methodology that constructs multiple diverse classifiers using a Pairwise Adversarially Robust Loss (PARL) function during the training procedure. PARL utilizes gradients of each layer with respect to input in every classifier within the ensemble simultaneously. The proposed training procedure enables PARL to achieve higher robustness with high clean example accuracy against black-box transfer attacks compared to the previous ensemble methods. We also evaluate the robustness in the presence of white-box attacks, where adversarial examples are crafted on the target classifier. We present extensive experiments using standard image classification datasets like CIFAR-10 and CIFAR-100 trained using standard ResNet20 classifier against state-of-the-art adversarial attacks to demonstrate the robustness of the proposed ensemble methodology.

Under review as a conference paper at ICLR 2022
PARL: ENHANCING DIVERSITY OF ENSEMBLE NET- WORKS TO RESIST ADVERSARIAL ATTACKS VIA PAIRWISE ADVERSARIALLY ROBUST Loss FUNCTION
Anonymous authors Paper under double-blind review
ABSTRACT
The security of Deep Learning classifiers is a critical field of study because of the existence of adversarial allacks. Such attacks usually rely the principle of transferability. where an adversarial example crafted on a surrogate classifier tends to mislead the target classifier trained the same dataset even both clas- sifiers have quite different architecture. Ensemble methods against adversarial attacks demonstrate that an adversarial example is less likely to mislead multiple classifiers in an ensemble having diverse decision boundaries. However, recent ensemble methods have either been shown to vulnerable to stronger adver- saries shown to lack end-to-end evaluation. This paper attempts develop a new ensemble methodology that constructs multiple diverse classifiers using a Pairwise Adversarially Robust Loss (PARL) function during the training proce- dure. PARL. utilizes gradients of each layer with respect to input in every classi- fier within the ensemble simultaneously. The proposed training procedure enables PARL to achieve higher robustness against black-box transfer attacks compared previous ensemble methods without adversely affecting the accuracy of clean examples. We also evaluate the robustness in the presence of white-box attacks, where adversarial examples are crafted using parameters of the target classifier. We present extensive experiments using standard image classification datasets like CIFAR-10 and CIFAR-IOK trained using standard ResNet20 classifier against state-of-the the-t-ttttttttt adversarial attacks to demonstrate the robustness of the proposed ensemble methodology.
INTRODUCTION
Deep learning (DL) algorithms have seen rapid growth in recent years because of their unprece- dented successes with near-human accuracies in a wide variety of challenging tasks starting from im- age classification (Szegedy et al. 2016), speech recognition (Amodei et al. 2016), natural language processing (Wu et 2016) to self-driving cars (Bojarski et 20.1)) While DI algorithms are ex- tremely efficient in solving complicated decision making tasks, they are vulnerable to well-crafted adversarial examples (slightly perturbed valid input with visually imperceptible noise) (Szegedy) et al. 2014). The widely-studied phenomenon of adversarial examples among the research commu- nity has produced numerous attack methodologies with varied complexity and effective deceiving strategy (Goodfellow et al. 2015) Kurakin et al. 2017 Madry et al., 2018 Moosavi-Dezfool et al. 2016 Papernot et al. 2017 An extensive range defenses against such attacks has been proposed in the literature, which generally falls into two categories. The first category enhances the training strategy of deep learning models to make them less vulnerable to adversarial examples by train- ing the models with different degrees of adversarially perturbed training data (Bastani et al. 2016 Huang et al. 2015) Jin et al., 2015| Zheng et al., (016) or changing the training procedure like gra- dient masking, defensive distillation, etc. (Gu & Rigazio, 2015, Papernot et al.) 2016 Rozsa et al.) 2016 Shaham et al. 2015). However, developing such defenses has been shown to be extremely challenging. Carlini & Wagner (2017b) demonstrated that these defenses are not generalized for all varieties of adversarial attacks but are constrained to specific categories. The process of training with adversarially perturbed data is hard, often requires models with large capacity, and suffers from significant loss on clean example accuracy. Moreover, AAthalye et al.] (2018) demonstrated that the changes in training procedures provide false sense of security. The second category intends to

---

Under review as a conference paper at ICLR 2022
detect adversarial examples by simply flagging them (Bhagoji al.) 2017 Feinman et al.
2017
Gong et al. 2017; Grosse et al. 2017: Metzen et al. 2017 Hendrycks & Gimpel| 2017 Li Li
2017). However, even detection of adversarial examples can be quite complicated task. Carlini & Wagner 2017a) illustrated with several experimentations that these detection techniques could
be efficiently bypassed by a strong adversary having partial or complete knowledge of the internal
working procedure.
While all the approaches mentioned above deal with standalone classifiers, in this paper, we utilize the advantage of an ensemble of classifiers instead of a single standalone classifier to resist adversar- ial attacks. The notion of using diverse ensembles to increase the robustness of classifier against adversarial examples has recently been explored in the research community. The primary motivation of using an ensemble-based defense that if multiple classifiers with similar decision boundaries perform the same task, the transferability property of DI. classifiers makes it easier for an adver- sary to mislead all the classifiers simultaneously using adversarial examples crafted on any of the classifiers. However, it will be difficult for an adversary to mislead multiple classifiers simultane- ously if they have diverse decision boundaries. Strauss et al. (2017) used various ad-hoc techniques such different random initializations, different neural network structures, bagging the input data, adding Gaussian noise while training for creating multiple diverse classifiers to form an ensemble. The resulting ensemble increases the robustness of the classification task even in the presence of ad- versarial examples. Tramer et al. (2018) proposed Ensemble Adversarial Training that incorporates perturbed inputs transferred from other pre-trained models during adversarial training decouple adversarial example generation from the parameters of the primary model. Grefenstette et al. (2018) demonstrated that ensembling two models and then adversarially training them incorporates more robustness in the classification task than single-model adversarial training and ensemble of two sep- arately adversarially trained models. Kariyappa & Qureshi (2019) proposed Diversity Training of an ensemble of models with uncorrelated loss functions using Grudient Alignment Loss metric to reduce the dimension of adversarial sub-space shared between different models and increase the robustness of the classification task. Pang al. (2019) proposed Adaptive Diversity Promoting regularizer to train an ensemble of classifiers that encourages the non non-maximal predictions in each member in the ensemble be mutually orthogonal, which degenerates the transferabilityy that aids in resisting adversarial examples. Yang et al. (020) proposed a methodology that isolates the ad- versarial vulnerability in each sub-model of ensemble by distilling non-robust features. While all these works attempt to enhance the robustness of a classification task even in the presence of adversarial examples, Adam et al. 20188) proposed a stochastic method to add Variational Autoen- coders between layers as a noise removal operator for creating combinatorial ensembles to limit the transferability and detect adversarial examples.
In order to enhance the robustness of a classification task and/or to detect adversarial examples, the ensemble-bar based approaches mentioned above employ different strategies while training the mod- els. These ensembles either lack end-to-end evaluations for complicated datasets or lack evaluation against more aggressive attack scenarios like the methods discussed by Dong et al. (2018) and Liu et al. (2017), which demonstrate that adversarial examples misleading multiple models in en- semble tend to be more transferable. In this work, our primary objective is to propose a systematic approach to enhance the classification robustness of an ensemble of classifiers against adversarial examples by developing diversity in the decision boundaries among all the classifiers within that ensemble. The diversity is obtained by simaltaneously considering the mutual dissimilarity in gradi- ents of each layer with respect to input in every classifier while training them. The diversity among the classifiers trained in such a way helps to degenerate the transferability of adversarial examples within the ensemble.
Motivation behind the Proposed Approach and Contribution: The intuition behind developing the proposed ensemble methodology is discussed in Figure [ which shows a case study using clas- sifiers trained on CIFAR-10 dataset without loss of generality. Figure Ia shows an input image of a 'frog'. Figure Ib shows the gradient of loss with respect to inpu il in classifier M Mprims and the gradient for another classifier Mssm with a similar deci- sion boundary as Mprim. and denoted as sim- The classifier Maim is trained using the same parameter settings as Mprim but with different random initialization. Figure Id the gradients for a classifier Mai with not similar decision boundary compared to Mprims and denoted as
denoted as V prim. Ic
Fundamental operation behind the creation of almost all adversarial examples
2

---

Under review as a conference paper at ICLR 2022
Ale
127.16
49.01
(b)
(d)
(e)
Figure 1: (a) Input image; (b) V prim: Gradient of loss the primary classifier; (c) sim: Gradient of loss in another classifier with similar decision boundaries; (d) V aii: Gradient of loss in classifier with not so similar decision boundaries but comparable accuracy; (e) Symbolic directions of all the gradients in higher dimensions. The gradients are computed with respect to the image shown in (a).
div. The classifiers Mprims Msims and Mase have similar classification accuracies. The method for obtaining such classifiers is discussed later in this paper. Figure [Ie shows relative symbolic directions among all the aforementioned gradients in higher dimension. The directions between a pair of gradients are computed using cosine similarity. We can observe that V prim and V sim lead to almost in the same directions, aiding adversarial examples crafted on Mprim transfer into Msin. However, adversarial examples crafted in Mprim will be difficult to transfer into Maiv as the directions of Frrm and dio are significantly different.
The principal motivation behind the proposed methodology is to introduce constraint for reducing the cosine similarity of gradients among each classifier in an ensemble while training them simulta- neously. Such a learning strategy with mutual cooperation intends to ensure that gradients between each pair of classifiers in the ensemble are as dissimilar as possible. We make the following contri- butions using the proposed ensemble training method:
We propose a methodology to increase diversity in the decision boundaries among all the classi- fiers within an ensemble to degrade the transferability of adversarial examples.
We propose a Pairwise Adversarially Robust Loss (PARL. function by utilizing the gradients of each layer with respect input of every classifier within the ensemble simultaneously for training them to produce such varying decision boundaries.
The proposed method can significantly improve the overall robustness of the ensemble against black-box transfer attacks without substantially impacting the clean example accuracy.
We evaluated the robustness of PARL with extensive experiments using two standard image classi- fication benchmark datasets ResNet20 architecture against state-of-the-arr adversarial attacks.
2 THREAT MODEL
We consider the following two threat models in this paper while generating adversarial examples.
Zero Knowledge Adversary (Az): The adversary Ag does not have access the target ensemble Mr but has access to a surrogate ensemble Mg trained with the same dataset. We term Ag as black-box adversary. The adversary Az crafts adversarial examples on Ms and transfers Mr.
Perfect Knowledge Adversary (Ap): The adversary Ap is stronger than Ag who has access to the target ensemble My. We term Ap as a white-box adversary. The adversary Ap can generate adversarial examples on My knowing the parameters used by all the networks within My.
3 OVERVIEW OF THE PROPOSED METHODOLOGY
In this section, we provide brief description of the proposed methodology used in this paper to enhance classification robustness against adversarial examples using an ensemble of classifiers Mr. The ensemble consists of N neural network classifiers and denoted as MM = U.V. M,. All the M,'s 's are trained simultaneously using the Pairwise Adversarially Robust Loss (PARL) function, which we discuss in detail in Section 4 The final decision for an input image on Mr is decided based on the majority voling among all the classifiers. Formally, let us assume test set of / inputs {z1, {ZI.I2,... .r,...,.I .r. with respective ground truth labels as (81. V/....,. W. The final decision of the ensemble MT for an input æy is defined as
C(MT, majority (13)), Ma(s)), ,MM(a;))
3

---

Under review as a conference paper at ICLR 2022
C(Mr,2)) y, for most 2, in appropriately trained My. The primary argument behind the proposed ensemble method is that all M,'s 's have dissimilar decision boundaries but not significantly different accuracies. Hence, clean example classified as class C₂ in M, will also be classified as Cz in most other Mj's (where j = .... j A i) within the ensemble with a high probability. Consequently, because of the diversity in decision boundaries between M, and Mj (for i 1...., j = 1. and i + j), the adversarial examples generated by zero knowledge adversary (Az) for a surrogate ensemble MMs will have a different impact on each classifiers within the ensemble Mr. i.e., the transferabilityy of adversarial examples will be challenging within the ensemble. A perfect knowledge adversary (Ap) can generate adversarial examples for the ensemble My. However, in this scenario, the input image perturbation will be in different directions because of the diversity in decision boundaries among all M,'s within the ensemble. The collective disparity perturbation directions makes it challenging to craft adversarial examples for the ensemble. We evaluated our proposed methodology considering both the adversaries and presented the results in Section
BUILDING THE ENSEMBLE NETWORKS USING PARL
In this section, we provide a detailed discussion on training an ensemble of neural networks using the proposed Pairwise Adversarially Robust Loss (PARL) function for increasing diversity among the networks. First, we define the basic terminologies used in the construction, followed by a detailed methodology of the training procedure.
4.1 BASIC TERMINOLOGIES USED THE CONSTRUCTION
Let us consider ensemble MT U M,, where M; is the ith network in the ensemble. We assume that each of these networks has the same architecture with Lu number of hidden layers. Let JM, (x. y) be the loss functions evaluating the amount of loss incurred by the network M, for data point x, where y is the ground-truth label for Let J&g (x) be the output of hthh hidden layer on the network M, for the data point x. Let us assume M. (x) has D(Lx) number of output features. Let us consider F. (x) denote the sum of gradients over each output feature of 6th hidden layer with respect to input on the network M, for the data point x. Hence,
D(C+)
(x) [x)
Σ ValFfh (x)),
where V [F 5-),, is the gradient of the 8th output feature of 1hth hidden layer on network M; with respect to input for the data point x. Let X' be the training dataset containing |X' examples.
4.2 PAIRWISE ADVERSARIALLY ROBUST LOSS FUNCTION
The principal idea behind the proposed approach is to train an ensemble of neural networks such that the gradients of loss with respect input in all the networks will be in different directions. The gradients represent the directions which the input needs to be perturbed such that the loss of the network increases, helping transfer adversarial examples. In this paper, we introduce the Pairwise Adversarially Robust Loss (PARL) function, which we will use to train the ensemble. The objective of PARL is to train the ensemble so that the gradients of loss lead to different directions in different networks for the same input example. Hence, the fundamental strategy is to make the gradients as dissimilar as possible while training all the networks. Since the gradient computation depends on all intermediate parameters of network, we force the intermediate layers of all the networks within the ensemble to be dissimilar for producing enhanced diversity at each layer.
The pairwise similarity of gradients of the output of tt hidden layer with respect to input between the classifiers M, and M, for a particular data point x can be represented as
< V (xx, V (x) (x)
(( (x) IV.FM (x)]
where < a, b represents the dot product between two vectors and b. The overall pairwise similarity between the classifiers M, and M, for particular data point x considering Ex hidden layers can be written
4

---

Under review as a conference paper at ICLR 2022
((1d)(x)
Σ
Lz (x)
Next, we define penalty term R(M for all the training examples in X' to pairwise train the models M₁ and Mj as
R(M;,M,) = Σ g(x))) XEX
1
We can observe that computes average pairwise similarity for all the training examples. Now, for network M; and M,, if all the gradients with respect input for each training example are in the same direction, value of R will close to 1, indicating similarity decision boundaries. The value of R will gradually decrease as the relative angle between the pair of gradients increases in higher dimension. Hence, the objective of diversity training using PARL is to reduce the value of R. Thus, add R to the loss function penalty parameter penalize the training for large R value.
In the ensemble My, we compute the values for each distinct pair of M, and M, in order to enforce diversity between each pair of classifiers. We define PARL train the ensemble My
PARL(MT) -
1 1 {
ΣE
JM, (x.) +
E R(M,,
ISiSSN
where "/ and 7/2 are hyperparameters controlling the accuracy-robustnes trade-off. A higher value of "/2 and a lower value of 222 helps to learn the models with good accuracy but is less robust against adversarial attacks. However, a lower value of "/1 and a higher value of %/2 makes the models more robust against adversarial attacks but with a compromise in overall accuracy.
One may note that the inclusion of the penalty values for each distinct pair of classifiers within the ensemble compute PARL has one fundamental advantage. If do not include the pair (Ma. in the PARL computation, the training will continue without any diversity restrictions be- tween Ma and Mr. Consequently, M, and M,, will produce similar decision boundaries, thereby increasing the likelihood of adversarial transferability between Ma and .Ms affecting the robust- ness of the ensemble My. One may also note that the number of gradient computations in an efficient implementation of PARL is linearly proportional to the number of classifiers in the ensem- ble. The gradients for each classifier are computed once and are reused to compute R. values for each pair of classifiers. The reuse of gradients protects the implementation from the exponential computational overhead due to pairwise similarity computation.
5
EXPERIMENTAL EVALUATION
5.1 EVALUATION CONFIGURATIONS
We consider an ensemble of three standard ResNet20 (He al., 2016) architecture for all the ensem- bles used in this paper. We consider two standard image classification datasets for our evaluation, namely CIFAR- 10 (Krizhevsky et al. 2009) and CIFAR-100 (Krizhevsky al., 2009). We consider two scenarios for the evaluation:
Unprotected Ensemble: baseline ensemble (E.SSu of ResNet20 architectures without any countermeasure against adversarial attacks.
Protected Ensemble: An ensemble (ENS2) of ResNet20 architectures, where 2 is the coun- termeasure used to design the ensemble. In our evaluation we have considered three previ- ously proposed countermeasures to compare the performance of PARL. We denote the ensembles SNSADP. E. SGAL, and EN ERGE be the ensembles trained with the methods pro- posed by Pang et al. (2019), Kariyappa & Qureshi (2019), and Yang et al. (2020). respectively. The ensemble trained with our proposed method denoted ENSPARL-
We use the adam optimization (Kingma & Ba 2015) to train all the ensembles with adaptive learning rate starting from 0.001. We dynamically generate augmented dataset using random shifts, flips and crops to train both CIFAR-10 and CIFAR- 100. We use the default hyperparameter settings mentioned in the respective papers for ENSADP. EN and EN ENSPVERGEE We use ?11
We implemented SN following the approach mentioned in the paper. Whereas, we adopted the official GitHub repositories for ENSADP and S. ERGE implementation.
5

---

Under review as a conference paper at ICLR 2022
1.0, 722 0.5, and categorical crossentropy loss for JM, (.) (ref. Equation I for ENSPARL We enforce diversity among all the classifiers in ENSPARL for the first seven convolution layer
We consider four Mate-of-theeart untargeted adversarial attacks Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2013), Basic Iterative Method (BIM) (Kurakin et al. 2017). Momen-
tum Iterative Method (MIM) (Dong et al. 2018), and Projected Gradient Descent (PGD) (Madry
et al. 2018) for crafting adversarial examples. A brief overview on crafting adversarial examples using these attack methodologies are discussed in Appendix A We consider 50 steps for generating adversarial examples using the iterative methods BIM, MIM, and PGD with the step size of e/5, where e is the attack strength. We consider the moment decay factor as 0.01 for MIM. We use 10 different random restarts for PGD to generate multiple instances of adversarial examples for an im- partial evaluation. We use e = 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, and 0.07 for generating adversarial examples of different strengths. We use CleverHans v2. 1 O library (Papernot et al. 2018) to generate all the adversarial examples.
In order to evaluate against stronger adversarial setting for Az, we train black-box surrogate ensemble and generate adversarial examples from the surrogate ensemble instead of a standalone classifier. As also mentioned previously, adversarial examples that mislead multiple models in an ensemble tend to be more transferable (Dong et al. 2018: Liu et al., 2017). All the results in the subsequent discussions are reported by taking average value over three independent runs.
5.2 ANALYSING THE DIVERSITY
The primary objective of PARL is to increase the diversity among all the classifiers within an en- semble. In order to analyze the diversity of different classifiers trained using PARL, we use Linear Central Kernel Alignment (CKA) analysis proposed by Kornblith et al. (019). The CKA metric, which lies between 00, 1]. measures the similarity between decision boundaries represented by a pair of neural networks. A higher CKA value between two neural networks indicates significant similarity in decision boundary representations, which implies good transferability of adversarial examples. We present analysis on layer-wise CKA values for each pair of classifiers within the ensemble EN St and EN ARA trained with CIFAR-10 and CIFAR-100 in Figure to show the effect of PARL on diversity. We can observe that each pair of models in EN Sy show signifi- cant similarity at every layer. However, since EN SPARL is trained by restricting the first seven
We observed that PARL performs better than previously proposed approaches against adversarial attacks with high accuracy on clean examples by enforcing diversity in the first seven convolution layers. We present an Ablation Study by varying the number layers utilized for diversity training using PARL. in Section 3.4
Lincor C&A Analess betwcon Second Enssmbile
Lincor CEA AnAlnsss betwcon
Linor CCA AnAlyss betweon
PEz essamblee
Sooand and
3.0
2.7
EN5,10.8838
EN5,10
(N5,00.8878) SN 8250)
ENS 8233)
EN5 2056)
20
50
10
Layers
Layers
Layers
I Inee CKA AnMlyss tetueen
new CKA Analyce betaren
Lnnor CSA Apalyce setween
38
57
3.5
3.5
2.5
105540.86
SM (3.7506)
322
SM (0.446)
02
BN (0.7317)
50 30
0.1
50.
30
50
30
Layers Reshet20
Layers Reshiet20
Layers Reshvtz0
(d)
Figure 2: Layer-wise linear CKA values between each pair of classifiers in ENSu and ENSPAAL trained with CIFAR-10 ((a), (b), (c)] and CIFAR-100 [(d). (e). (f) datasets showing the similarities each layer. The value inside the braces within the corresponding figure legends represent the overall average Linear CKA values between each pair of classifiers.

---

Under review as a conference paper at ICLR 2022
Table 1:
Ensemble classification accuracy (%) for CIFAR-I0 and CIFAR-I00 on clean examples as well as adversarially perturbed images with attack strength = 0.07 for different adversarial attacks.
Clean Example
FGSM
BIM
MIM
PGD
CIFAR-10
93.11
21.87
7.47
7.27
1.57
EN Su
CIFAR-100
70.88
7.13
7.17
3.93
5.39
CIFAR-I0
92.99
23.53
8.13
7.53
3.08
ENSADP
CIFAR-100
70.01
8.23
10.07
5.83
8.72
CIFAR-10
91.22
22.8
9.3
8,33
7.64
ENSGAL
CIFAR-100
"
CIFAR 10
91.73
28.88
11.78
10.55
9.68
ENRDVERGE
CIFAR-100
CIFAR-10
91.09
28.37
20.8
16.17
15.65
ENSP
CIFAR-100
67.52
12.73
18.97
10.01
21.49
Did not consider CIFAR-100 dataset for evaluation
convolution layers, we can observe a considerable decline in the CKA values at the initial layers. The observation is expected PARL imposes layer-wise diversity in its formulation, as discussed previously in Section 4 The overall average Linear CKA values between each pair of models in Figure 2 are mentioned inside braces within the corresponding figure legends, which signifies that the classifiers within an ensemble trained using PARL shows a higher overall dissimilarity than the unprotected baseline ensemble. In the subsequent discussions, we analyze the effect of the observed diversity on the performance of ENSPPAR against adversarial examples.
5.3 ROBUSTNESS EVALUATION OF PARL
Performance in the presence of Az: We evaluate the robustness of all the ensembles discussed in Section 5. considering both CIFAR-10 and CIFAR-100 where the attackers cannot access the model parameters and rely surrogate models to generate transferable adversarial examples. Un- der such black-box scenario, we use one hold-out ensemble with three ResNet20 models as the surrogate model. We randomly select 1000 test samples and evaluate the performance of black-box transfer attacks for all the ensembles across a wide range of attack strength c. We present the result with clean example accuracies. The methods pre-
for the attack strength (c = 0.07) in TableTalone sented by Kariyappa Qureshi (2019) and Yang et al. (2020) neither evaluated their approach on more complicated CIFAR- 100 dataset nor discussed any optimal hyperparameter settings regard- ing their proposed algorithms for potential evaluation. In our robustness evaluations, we consider all the ensembles discussed in Section [5.]] for CIFAR-10. In contrast, we only consider E.N Su and ENSADA to compare the performance of ENSPARI. for CIFAR-100. We can observe from [Taat the training using PARL does not adversely affect the ensemble accuracy clean exam- plcs compared to other previously proposed methodologies. However, it provides better robustness against black-box transfer attacks for almost every scenario.
A more detailed performance evaluation considering all the attack strengths are presented in Fig- ure For CIFAR-IO evaluation (ref. Figure 3a Figure [d), we can observe that ENSPARL performs at par with EN SNSDVERRS ERGE considering FGSM. However, for stronger iterative attacks like BIM, MIM, and PGD, ENSPARL outperforms other methodologies with higher accuracy for large attack strengths. For CIFAR-100 evaluation (ref. Figure 3el Figure 3h), we can observe that E. SPARI. performs better than both ENN Sy and ENSADD for all scenarios.
Performance in the Presence of Ap: Next, we evaluate the robustness of ensembles when the attacker has complete access to the model parameters. Under such white-box scenario, we craft adversarial examples from the target ensemble itself. We consider the same attack methodologies and settings as discussed in Section 5.1 We randomly select 1000 test samples and evaluate white- box attacks for all the ensembles across wide range of attack strength We present the results for CIFAR- 10 and CIFAR- 100 in Figure A For CIFAR-10 evaluation (ref. Figure 4a Figure 4d), we observe that F.NSPARL performs marginally better than the other ensembles. For CIFAR-100
7

---

Under review as a conference paper at ICLR 2022
310
Biby Bltker
Bibu
BBS
Brb BiS FiS
RiSax
RiSax
20
OWficss
40
201
Attack Strength
Attack Strength
Attack Strength
Attack Strength
(a) FGSM
(b) BIM
(c) MIM
(d) PGD
ENS
ENS
ENS
ENS
10
0
6.01 0.52 0:8 5:1 6:55 586 0.57
E.t 6.52 0:s 3.1 ndss s66 0:37 Strength
5.:: 6.52 0:5
300 0.05 356 0:57
San 6.52 8:8 0n0 sus aie Eor
Strength
Strength
Strength
(c) FGSM
(f) BIM
(g) MIM
(h) PGD
Figure 3: Ensemble classification accuracy (%) v.s. Attack Strength (e) against different black-box
transfer attacks generated from surrogate ensemble for CIFAR-10 [(a), (b), (c), (d)] and CIFAR-100 ((e), (f), (g), (h)] datasets.
230
eits
eftterr
eftioar
efblus
80
effiie 40
20
Eat
6.22 0.03 301 6:35
6.37
EAt
6.32
301
6:37
6.3 n:8
3.0 EAT
Attack
(a) FGSM
(b) BIM
(c) MIM
(d) PGD
185
EN5
IN5
LN5
185rp
2N5rp
2N5rp
2NSrp
5W5ve
632 032 000 304 B35 206 0:37
Ed. 632 000 300 B.3 286 6.37 Stoongth
50. 6.2 0ED 300 EAS 316 6:37
50l 6.2 0E3 300 EAS 336 E.F
Strnngh
Strongth
Strength
(e) FGSM
(i) BIM
(g) MIM
(h) PGD
Figure 4: Ensemble classification accuracy (%) v.s. Attack Strength (c) against different white-box
attacks for CIFAR-10 ((a), (b), (c), (d) and CIFAR-100 [(e), (f. (g), (h)] datasets.
evaluation (ref. Figure 4c Figure 4h). we can observe that ENSPARL performs better than both Sy and ENSADP for all scenarios. Although PARL achieves the highest robustness among all the previous ensemble methods for black-box transfer attacks, its robustmess against white- box attacks is still quite low. The result expected as the objective of PARL is to increase the diversity of an ensemble against adversarial vulnerability rather than entirely eliminate it. In other words, adversarial vulnerability inevitably exists within the ensemble and can be captured by attacks with white-box access. One straightforward way improve the robustness of ensembles against such vulnerability is to augment PARL with adversarial training (Madry et al. 2018), which we look forward to as a future research direction.
5.4 ABLATION STUDY
In all the previous evaluations. consider ENSPARL by enforcing diversity in the first seven convolution layers for all the classifiers during ensemble training using PARL. this section, we provide ablation study by analyzing a varying number of convolution layers considered for the
8

---

Under review as a conference paper at ICLR 2022
Table 2: Ensemble classification accuracy (%) on the test for CIFAR-IO and CIFAR-100. The numbers in the first row after slash denote the number of convolution layers used to enforce diversity
ENSPARL/T
ENSPARL/S
ENSPARL/3
CIFAR-10
91.09
91.18
92.45
CIFAR-100
67.52
67.54
68.81
230
2N6
EN5 Yuuu
20
ENGroa
30
Accurrty
40
20
20
6.57
6:57
0..4 6:47
0.34 (d) PGD
Attack
Attack
Attack
Attack
(a) FGSM
(b) BIM
(c) MIM
320
320
238
200
ENS A
80-
ENB
ENS essus
ENS eusss
accurtry
Accomnry
Accornrry
Attack Strength ld)
Attack Strength to
Attack Strength id
Attack Strength to
(e) FGSM
(f) BIM
(g) MIM
(h) PGD
Figure 5: Ensemble classification accuracy (%) v.s. Attack Strength (e) against different black-box
transfer attacks generated from surrogate ensemble for CIFAR-10 ((a), (b), (c), (d)) and CIFAR- 100 ((e), (f), (g), (h)] datasets. The numbers in the figure legends after slash denote the number of convolution layers used to enforce diversity
diversity training. We consider three ensembles EN PARL/77. ENSPARL/S- and ENSp ARL/3 this study, where ENSPARL/N denotes that the first N convolution layers used for enforcing the diversity. We consider the same evaluation configurations as discussed in Section 5. The accuracies of all the ensembles on clean examples considering both CIFAR-10 and CIFAR- mentioned in T We can observe that as fewer restrictions are imposed, the overall ensemble accuracy increases, which is expected and can be followed from Equation (I). A detailed analysis on the diversity achieved by all these ensembles in terms of Linear CKA metric is provided in Appendix B for interested readers.
EPAL//R for
Next, we evaluate the robustness of these ensembles against black-box transfer attacks for the eval- uation configurations discussed in Section 5.1 and provide the results in Figure 5 We can observe that though the ensemble accuracy on clean examples increases with fewer layer restrictions, the ro- bustness against black-box transfer attacks decreases significantly. The results present an interesting trade-off between accuracy and robustness in terms of number of layers considered while computing PARL (ref. Equation I.).
CONCLUSION
In this paper, we propose an approach to enhance the classification robustness of an ensemble against adversarial attacks by developing diversity in the decision boundaries among all the classifiers within the ensemble. The ensemble network is constructed by the proposed Pairwise Adversarially Robust Loss function utilizing the gradients of each layer with respect to input in all the networks simultane- ously. The experimental results show that the proposed method can significantly improve the overall robustness of the ensemble against state-of-the-an black-box transfer attacks without substantially impacting the clean example accuracy. Combining the technique with adversarial training and ex- ploring different efficient methods to construct networks with diverse decision boundaries adhering to the principle outlined in the paper can be interesting future research directions.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have uploaded the codes as a supplementary material along with a README file for reproducing the results reported in this paper.
REFERENCES
Gcorge-Alexandry Adam, Petr Smirnov, Anna Goldenberg, David Duvenaud, and Benjamin Haibe- Kains. Stochastic combinatorial ensembles for defending against adversarial examples. CoRR, abs/1 abs/1808.0666 808.06645, 2018. htIhttp:: http:://aaxvvoo.o org. abs/ wttp://wwwww////////////
06645
Dario Amodei et al. Deep speech 2 End-to-end speech recognition in english and mandarin. In Proceedings of the 33nd International Conference Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp- 173-182. Jog.. 2016. URL html
.mlr. press/ v48 /s//mmddee
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th Inter- national Conference on Machine Learning, ICML 2018, Stockholmsmässsan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 283.283. PMLR, 2018. URL
mlr. press at lye 18a. html
Osbert Bastani, Yani loannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya V. Nori, and Antonio Criminisi. Measuring neural net robustness with constraints. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2613-2621, 2016.
URL http://papers nips cc/paper/ 6339 measuring neural -eet--oousteess with constraintsstss
Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Dimensionality reduction as a defense against evasion attacks on machine learning classifiers. CoRR, abs/1704.02654, 2017. URL
http::/arxixs http:::/aaxiv.org/eae/ wwwporggg//.//o/eeeee 026544
Mariusz Bojarski al. End to end learning for self-driving cars. CoRR, abs//604.07316, 2016.
URL httth:/tp///xiiie org/ abs/ 0es//6644.03315
Nicholas Carlini and David A. Wagner. Adversarial examples are easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and
Security, AlSec CCS 2017, Dallas, TX, USA, November 2017, pp. 3 14. ACM, 2017a. doi: N0.1145/3128572.334444 URL httpstt::///do org/ 112.11111/1112...111 314044
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pp. 39-57. IEEE Computer Society, 2017b. URL htwp::tp////cm /ttp.//diii//o//////////.... org/ 10. 1109/SP 2017. 49
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 9185- 9193. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR. wttp://cccoooooooooeteeeeeeeeeee
2018,00957. wtt.attppp//oee..ess...eeeeeeeee www.eeeeeessseccc.....eeeeeeeeee www.:ttwwwwwweeeeeeeoooooooeeeee www.o///ccootttteeeeeeeo Dong
2018/html/ CVPR 2018 html
Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. Detecting adversar- ial samples from artifacts. CoRR, abs/1703.00410, 2017. URL
1703.000 00410
Zhitao Gong, Wenlu Wang, and Wei-Shinn Ku. Adversarial and clean data are not twins. CoRR, 7b4/170,,04960, 2017. URL http::/ http://wwwwg/////////ba//// 049600)
10

---

Under review as a conference paper at ICLR 2022
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http::/aaxiv..o
abs/ 141/111222 6572
Edward Grefenstette, Robert Stanforth, Brendan O' O'Donoghue, Jonathan Uesato, Grzegorz Swirszez, and Pushmeet Kohli. Strength in numbers: Trading-off robustness and computation via adversarially--rained ensembles. CoRR, abs/1811.09300, 2018. URL httt:://axxiv..or
abs/1811 09300
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick D. McDaniel. On the (statisticall) detection of adversarial examples. CoRR, abs/1702.06280, 2017. [RRhtt::
7/arxiv.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015. URL http://arxiv...
abs/1412.500 5068
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. TEEE Computer Society, 2016. doi: N0.1109/CVPR.2016.00 90, URL wttptt:ss//.i...o.. htt.://o..oooo.//oo/// /10. wwt.:1111...... /t1.//CVR.........9
Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL
net/
forum? d=B1 dexpDug
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with strong adver- oa31aa..ss111111111131
sary. CoRR,
2015. URL
Jonghoon Jin, Aysegul Dundar, and Eugenio Culurciello. Robust convolutional neural networks under adversarial noise. CoRR, abs/1511.06306, 2015, URL
163160
Sanjay Kariyappa and Moinuddin K. Qureshi. Improving adversarial robustness of ensembles with diversity training. CoRR, abs/11901.099 2019. URL attp://axxiv..org/bbb///
09981
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. |RLhttp: //arxiv. www.ii..rg.aaaa/.eeoa..aaaaaaaaa
6980
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Confer- ence on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California. USA, vol- ume 97 of Proceedings of Machine Learning Research, pp. 3529, PMLR, 2019, URL http:///nn.....p...eeee//////bba http///s//////ooonitttii.i..
httt:://proceedings ww....r...mm.
htm_
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny im- ages. Technical Report, 2009. URL http::/ ist wwwp:eerw...s...s....u.u.. www.pp.....vviie.......
download?doi=100 1. 222. 9200rep-reepsttpeeppff
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL https: //openeeview
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, Oc- tober 22-29, 2017, pp. 5775-5783, IEEE Computer Society, 2017. doi: 10.1109/CC...007...55.
URL htt.lhttps///t //do1 .ww.ooooorrg//ooooo 1109/ TCCV,
www.c.17...... 201...17.1 615.

---

Under review as a conference paper at ICLR 2022
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon. France, April 24-26. 2017. Conference Track Proceedings. OpenReview.net, 2017. htt..htts:///ooo.
net forum?? d GG]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, net/
Conference Truck Proceedings. OpenReview.net, 2018. URL.
Derum?id-rJzlin
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In 5th International Conference on Learning Representations, ICLR 2017, / Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httpt::s: 7/ //openzeview.oo net wtt.:/eetoooomndaaaSSSS333 SwzCssseee. 9xg
Scyed-Mohsen Moosavi-Dezfoolit Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2574-2582. IEEE Computer Society, 2016. URL https: //t:://ddoo org/ 10. 1. 1ww9/1119..... 2016. 282
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pp. 4970-4979, PMLR, 2019. URL URL.httpp://rroeeedino
mlr. wwwppppsssssv...
/ https//////paag1133/ httppaanglll html
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In /EEE Symposium on Se- curity and Privacy, SP 2016, San Jose, CA, USA, May 22-26, 2016, pp. 582-597. IEEE Computer Society, 2016.
doi: 10.1 1091...4....11..11 URL /ttEp::/doii.ore 0ttp::////01..0/01110000000 1109//SP..1...... 2021 2016.
Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, AsiaCCS 2017, Abu Dhabi, United Arab Emirates, April 2-6, 2017, pp. 506 519. ACM, 2017. doi: 3053009
10.1 N0.1145/3052973.3033
/tttpp:///o...or/11..1111 (RLLhttps://dde wwoooogg1110. 111.//1..1115/05521111
Nicolas Papernot, Fartash Faghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Ku- rakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy. Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang. Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, Rujun Long, and Patrick McDaniel. Technical report the cleverhans 1.0 adversarial exam- ples library. CoRR, abs/ 1atss/11000066.. 2018. URL htllhttp://// http://axvvv...///i org/ abs 0017500011111 00768
Andras Rozsa, Ethan M. Rudd, and Terrance E. Boult. Adversarial diversity and hard positive gener- ation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2016, Las Vegas, NV, USA, June 26 July 1, 2016, pp. 410-417, IEEE Computer So- ciety, 2016. doi: N0.100//VVPR.2011.55 URL 0 111p://////1011///P//1 2016 6. 58]
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of neural nets through robust optimization. CoRR, htd//511..04430 2015. URL. http::/aaaa org. abs/ / 05432
Thilo Strauss, Markus Hanselmann, Andrej Junginger, and Holger Ulmer. Ensemble methods as defense to adversarial perturbations against deep neural networks. CoRR, abs/1709.03423, 2017.
URL http://wwwwvvv...aaaa/// httpg//bb///09.// 03423
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In 2nd International Conference
12

---

Under review as a conference paper at ICLR 2022
Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL htt.ttt/://axxx orq/ abs/1312. 6199
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818-2826. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.300 URL https: 7/doi.org 10. 1119/CVPR. 2016.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, lan J. Goodfellow, Dan Bonch, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
Forum?id-rke
R2
Yonghui Wu et al. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.001444 2016. URL
08144
Huanrui Yang, Jingyang Zhang. Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. DVERGE: diversifying vulnerabilities for enhanced robust generation of ensembles. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Process- ing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 62, 2020, virtual, 2020. (RRp:s www.cccbaaaooo hrt.2//0fffcccaaaaaaaa5522222200
paper/2020/1is
html
Stephan Zheng, Yang Song, Thomas Leung and Ian J. Goodfellow. Improving the robustness of deep neural networks via stability training. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 4480 4488. IEEE Computer Society, 2016. doi: 10. (0.1109/CVPR.2016..99
Ut https: /ooi...... org/10.
IIC9/CVPR. 2016 485
A BRIEF OVERVIEW OF ADVERSARIAL EXAMPLE GENERATION
Let us consider benign data point x E Rd classified into class C, by a classifier M. An untargeted adversarial attack tries to add visually imperceptible perturbation 1) E Rd to x and creates new data point Fade = x + 77 such that M misclassifies Fady into another class C, other than C. The imperceptibilityy is enforced by restricting the nnrm of the perturbation r7 to be below threshold e, i.c., [|//|| S (Goodfellow et al. 2015 Madry et al. 2018). We term e as the attack strength.
An adversary can craft adversarial examples by considering the loss function of a model, assuming that the adversary has full access to the model parameters. Let J(H, x, y) denote the loss function of the model, where 6, x, and y represent the model parameters, benign input, and corresponding label, respectively. The goal of the adversary is to generate adversarial example Tadv such that the loss of the model is maximized while ensuring that the magnitude of the perturbation is upper bounded by
Hence, J [(0, Tade: y) > [(, x, y) adhering to the constraint (addv x | < e. Several methods have been proposed in the literature solve this constrained optimization problem. We discuss the methods used in the evaluation of our proposed approach.
Fast Gradient Sign Method (FGSM): This attack proposed by Goodfellow et al. 2015). The adversarial example is crafted using the following equation
Fade = x + sign( (. J x, v))
where (0,
2r 3) denote the gradient of loss with respect to input.
Basic Iterative Method (BIM): This attack is proposed by Kurakin al. (017). The adversarial example is crafted iteratively using the following equation
(k-1) 1) (k) clipe(a sign 2 (k) v)))
13

---

Under review as a conference paper at ICLR 2022
where (0)) is the benign example. If r is the total number of attack iteration, Zadv x()) The parameter a is small step size usually chosen as e/r. The function c.)pe()) is used to generate adversarial examples within e-ball of the original image x=()),
Moment Iterative Method (MIM): This attack is proposed by Dong et al. (0188) which won the NeurIPS 2017 Adversarial Competition. This attack a variant of BIM that crafts adversarial examples iteratively using the following equations
((k) VzJ (1)) y)
g(a)
( g(h) + (kg(k)
((++1) (k) + clipe(a (k1)(
where is termed as the decay factor.
Projected Gradient Sign Method (PGD): This attack is proposed by Madry et al. (2018). This attack is variant of BIM with same adversarial example generation process except for x(()) is randomly perturbed image in the neighborhood of the original image.
B DIVERSITY RESULTS FOR ABLATION STUDY
In this section, we present an analysis on layer-wise CKA values for each pair of classifiers within the ensemble ENS S ARI./T, ENSPARRE and EN PARL// trained with both CIFAR-10 and CIFAR- 100 datasets. The layer-wise CKA values are shown in Figure to demonstrate the effect of PARL on diversity. We can observe a more significant decline the CKA values the initial layers for EN SPAL/// compared to EN SPAL//5 and EN S ARLL/3 which is expected EN SPARL// is trained by restricting more convolution layers. We can also observe that each pair of classifiers show more overall diversity in EN S AAL//7 than in EN APL// and EN SPARL/3 The overall CKA values are mentioned inside braces within the corresponding figure legends.
First and Serend Model n Ensemble
Fire and Toird Model in Essemble
Second and Third Model Ensemblee
1.0
2.9
3.0
0.0
37
37
0
3.
0.
82031
(3.0055)
EMS
(..616)
8215)
EVA 10 8115) (1.
290 of 8828)
20 5805)
30 23
6o
10
20
60
10
20
20
40
66
Layers Reshett2
Layers Reshett0
Layers Reschet20
(a)
(b)
1.0
Fist and Second Motel n Ensemble
Fiis and Third Wozel essemble
Secand and Third Model Ensemble
2.
2.
0.
235
37
37
3.6
EMAS A 8215)
EVA 8117)
EVAS 7854)
02
2905 a18.8594) 8594)
290 nf (0..831) 8551)
10 30
co 70
10
20 30
66 70
10
40 de
Layers Reshiet20
Layers Reshett2
Layers of Reshet22
(d)
(e)
(I)
Figure 6: Layer-wise linear CKA values between each pair of classifiers in ENSPARL/T EN SPAnt/S. and ENSPARL/S trained with CIFAR-10 [(a), (b). (c)] and CIFAR-100 [(d). (c), (f) datasets showing the similarities at each layer. The numbers in the figure legends after slash de- note the number of convolution layers used enforce diversity. The value inside the braces within the corresponding figure legends represent the overall average Linear CKA values between each pair of classifiers.
14


Reviews:
[{"review_id": "cl3r7nCtgcZ", "reviewer": "Reviewer_BhHc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PRO:\n- The paper is overall well written and easy to follow. The review of related work and the motivation of the proposal is well conveyed.\n- The proposed regularizer is simple and easy to implement without too much burden. It is also nice authors attach the codes and even pre-trained models as supplementary materials.\n\nCON:\nHowever, I have to say the idea needs more refinement as the ablation study section does not provide too much information.\n- Although the gradient of each layer of each classifier can be reused for regularization, will this slow down the computation? What will happen if we calculate the regularization more efficiently by sampling pairs of classifier?\n- The gradient of each layer is calculated first in Section 4.1, and then used in the first equation of Section 4.2 (assign numbers for equations in revision). What may happen if the similarity is calculated feature-wised, instead of layer-wised? What if layers or features used for calculating similarity is sampled for computation efficiency?", "summary_of_the_paper": "This paper propose a regularizer of layer-wise gradient to enhance the robustness against adversarial examples of an ensemble of deep neural network classifiers.", "summary_of_the_review": "As explained in the main review, I admit the idea is good but it is at an early stage and needs more refinement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PfpdInlME3g", "reviewer": "Reviewer_Dge8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Using ensembles with diverse members to increase robustness is an established and interesting topic.\n\n- The method is a simple modification of standard training which doesn't incur in major additional cost and achieves some good result in the experimental evaluation.\n\nWeaknesses\n- In the proposed method the gradients, wrt the input, of the feature output of some layers are summed, and then the similarity of this sum with the corresponding one of the other classifiers is minimized: it is not clear to me which is the meaning of using the sum of the features of inner layers as a proxy of the classification loss. For example, if the output layer was used, the decision boundary would be represent by the difference between the logit of the correct class and the maximum of the others. Is this connected to the reason why only early layers are used? Moreover, in PARL the cosine similarity of pairs of models is minimized, but the minimal value -1 can be achieved only by a single pair. Does this create instability during training?\n\n- Defending against black-box transfer attacks is a quite narrow scope. Moreover, the proposed method should, in my opinion, be compared to other techniques for robust (individual) models, first of all adversarial training, which should achieve, for example, around 48% robust accuracy at $\\epsilon=0.031$ (CIFAR-10) against white-box attacks, see e.g. https://robustbench.github.io/, when using a small architecture. This appears to be better than what illustrated in Figure 3 for PARL.\n\n- The strength of transfer attacks mostly depends on how similar the target and surrogate classifiers are: how was the surrogate ensemble used in the experiments trained? Moreover, since PARL uses majority voting to merge the output of the individual classifiers, how are the attacks carried out (see [A] for a discussion of this)?\n\n- The evaluation appears to have some shortcomings: first, the worst-case, for each point, among different attacks should be reported. Second, the attacks used are very similar, that is gradient-based attacks on one loss (which, unless I missed it, is not specified): since the proposed method aims at diversifying the gradients across classifiers, it seems reasonable to test also an attack which doesn't rely on gradient [B, C] to generate the adversarial perturbations to be transferred. Moreover, I think it'd be important to know the success rate of the attacks on the surrogate models.\n\n[A] https://arxiv.org/abs/2011.14031 \\\n[B] https://arxiv.org/abs/1912.00049 \\\n[C] https://openreview.net/forum?id=SygW0TEFwH", "summary_of_the_paper": "The paper proposes a new method to train ensembles of classifiers which are robust to adversarial attacks, in particular black-box transfer-based ones. This is achieved by enforcing the output of early layers of different members of the ensemble to have, on average, gradients with low cosine similarity, which should in turn create different decision boundaries. For this, the authors design a specific loss function, PARL, to be minimized at training time. In the experiments, PARL models achieve higher robustness to transfer attacks that existing methods.", "summary_of_the_review": "Overall, I think that the proposed method is not well-motivated. Also, the experiments do not include important baselines, and it is not clear how strong the evaluation used is (see above).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1lm9b_Ok3J8", "reviewer": "Reviewer_BGda", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of training diverse components makes sense.\nSome details need better explanation or justification:\n* First paragraph of section 4.1: it seems coarse to sum the gradients across a layer, and lots of information can get lost.\n* It seems the equation at top of page 5 should be divided by L_H, because the text says that R(M_i,M_j) is at most 1.\n* Why only compute the loss term on the first seven layers? In particular, Figure 5 suggests that the more the better. And why choose early layers?\n\nThe main weakness is that the results are not strong.\n* According to Figure 3, the defense against transfer attack on CIFAR-10 is weaker than (Madry et al. 2018)'s white-box defense. Although the proposed method has a higher standard accuracy on clean inputs than (Madry et al. 2018), it's not good to have a weaker robustness under transfer attacks than a competitor's white-box robustness.\n* The comparison is against three ensemble methods. There are other non-ensemble methods against transfer attacks that should be included in comparison, for example\nhttps://arxiv.org/pdf/1705.09064\n", "summary_of_the_paper": "This paper addresses the problem of L_inf transfer attacks.\nThe proposed method is an ensemble that is trained with an extra loss term that measures the pairwise and layerwise similarity of components. This loss term is the average cosine similarity between gradients with respect to the input.\nAt inference time, the overall prediction is by voting.\nExperiments show better defense against L_inf transfer attacks than three other ensemble methods.", "summary_of_the_review": "The idea makes sense, but results are weak.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-aVwO4DU_aq", "reviewer": "Reviewer_SqzR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I mainly have the following comments/questions for the authors.\n\n### Comparing with GAL\n1. The high-level idea of the proposed PARL is \"to train an ensemble of neural networks such that the gradients of loss with respect to input in all the networks will be in different directions\". Given that this is the same as the idea of GAL, one of the considered baselines, I think it is necessary to clearly identify the differences between PARL and GAL in Sec. 4.2.\n2. To my understanding, there are two differences between PARL and GAL as to the formulation of the loss function: 1) GAL takes the gradients of the loss while PARL takes the gradients of intermediate layers' outputs, and 2) GAL minimizes the (smoothed) maximum cosine similarity between pairs, while PARL minimizes the sum of the cosine similarity between pairs. An ablation study along these two axes would be very helpful for understanding which part is the key to the performance gain. Or at least, can the authors comment on the intuition/insight on these differences made by PARL?\n\n### Weird trend in the results\n1. The experimental results in general demonstrate the effectiveness of the proposed method. However, there is a weird trend in the results that raises my concern. In Table 1, when the dataset is CIFAR-100, the accuracy of PARL under the multi-step PGD and BIM (21.49% and 18.97%) is noticeably higher than the accuracy under the single-step FGSM (12.73%). To my knowledge the stronger PGD should always lead to lower accuracy than FGSM. I would like to see some clarifications from the authors on this.\n\n### The choice of the layers\n1. The paper presents ablation study as to the number of layers used to compute PARL loss function in Sec. 5.4. However, it remains unclear what is the motivation of choosing the first several layers. What if one uses the last several layers? What if one randomly select some candidates from the whole layers during each step?", "summary_of_the_paper": "This work develops an ensemble-based adversarial defense which diversifies the sub-models to obtain robustness. The idea is to force the gradients of several layers pointing towards different directions across the sub-models such that the adversarial transferability can be reduced. On CIFAR-10/CIFAR-100, the paper demonstrates that PARL outperforms previous ensemble-based approaches.", "summary_of_the_review": "Despite the better empirical results obtained by the proposed method, this work can be improved by identifying key difference to GAL and further investigate/justify the choice of the layers, which will help people understand why PARL is effective (currently it is not that clear). Also, I need some clarifications from the authors on the weird trend in the results. These concerns/issues currently prevent me from recommending acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method to train ensembles of classifiers that are robust to adversarial attacks, in particular black-box transfer-based ones. This is achieved by enforcing the output of early layers of different members of the ensemble to have, on average, gradients with low cosine similarity, which should in turn create different decision boundaries. For this, the authors design a specific loss function, PARL, to be minimized at training time. Two reviewers gave the score of 6 while two reviewers gave the score of 3. The main concerns are: 1) unclear meaning of taking the sum of the gradients of different neurons, and why the similarity of that across models is a proxy for similarity of the decision boundaries; 2) lack of experiments, that is, omitting a simpler baseline like individual robust classifiers. Positive score reviewers also did not champion the paper, thus, the paper should be well addressed these main concerns in the revision and cannot accept to ICLR for now.
Decision: Reject