Random Normalization Aggregation for Adversarial Defense
Authors: Minjing Dong
Xinghao Chen
Yunhe Wang
Chang Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adversarial Robustness
Adversarial Transferability
Normalization

Abstract: The vulnerability of deep neural networks has been widely found in various models as well as tasks where slight perturbations on the inputs could lead to incorrect predictions. These perturbed inputs are known as adversarial examples and one of the intriguing properties of them is Adversarial Transfersability, i.e. the capability of adversarial examples to fool other models. Traditionally, this transferability is always regarded as a critical threat to the defense against adversarial attacks, however, we argue that the network robustness can be significantly boosted by utilizing adversarial transferability from a new perspective. In this work, we first discuss the influence of different popular normalization layers on the adversarial transferability, and then provide both empirical evidence and theoretical analysis to shed light on the relationship between normalization types and transferability. Based on our theoretical analysis, we propose a simple yet effective module named Random Normalization Aggregation (RNA) which replaces the batch normalization layers in the networks and aggregates different selected normalization types to form a huge random space. Specifically, a random path is sampled during each inference procedure so that the network itself can be treated as an ensemble of a wide range of different models. Since the entire random space is designed with low adversarial transferability, it is difficult to perform effective attacks even when the network parameters are accessible. We conduct extensive experiments on various models and datasets, and demonstrate the strong superiority of proposed algorithm. The PyTorch code is available at https://github.com/UniSerj/Random-Norm-Aggregation and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/RNA.

Random Normalization Aggregation for Adversarial Defense
Minjing Dong', Xinghao Chen', Yunhe Wang", Chang Xul School of Computer Science, University of Sydney Huawei Noah's Ark Lab mdon0736@uni. .ttpmii...yydae sydney edu. au, xinghao. chenChuawei.com, yunhe. kang@huawei. com, c. c.xussyd.yy. edu. au
Abstract
The vulnerability of deep neural networks has been widely found in various models as well as tasks where slight perturbations on the inputs could lead to incorrect predictions. These perturbed inputs are known adversarial examples and one of the intriguing properties of them is Adversarial Transfersability, i.e. the capability of adversarial examples fool other models. Traditionally, this transferability is always regarded as a critical threat to the defense against adversaria attacks, however, we argue that the network robustness can be significantly boosted by utilizing ad adversarial transferability from new perspective. In this work, we first discuss the influence of different popular normalization layers on the adversarial transferability, and then provide both empirical evidence and theoretical analysis to shed light on the relationship between normalization types and transferability. Based on theoretical analysis, we propose simple yet effective module named Random Normalization Aggregation (RNA) which replaces the batch normalization layers in the networks and aggregates different selected normalization types form a huge random space. Specifically, a random path is sampled during each inference procedure that the network itself can be treated wide range of different models. Since the entire random space is designed with low adversarial transferability. it is difficult to perform effecttive attacks even when the network parameters are accessible. We conduct extensive experiments on various models and datasets, and demonstrate the strong superiority of proposed algorithm. The Py Torch code available at hhttps: t/gthb.. com/UniSerj/ Nandoom-Noom--gggeegtiiooo and the MindSpore code available at https; /ttp:t/.....nnopppppeee/e////eer
ensemble of a
Introduction
Deep Neural Networks (DNNs) have achieved impressive performance in various tasks [ 2 However, is well known that DNNs are susceptible to maliciously generated adversarial examples the model inputs during inference stage, the model is misled to wrong predictions high rate. Since then, a wide range of attack techniques have been proposed under different settings and show strong attack capability. For example, attackers have full access to the model architecture and parameters, which forms white-box attacks [5) 6]. and attackers have limited query access to the model, which forms black-box attacks 77 Since the high attack success rates of these techniques reveal the high risk of DNNs, defenses against adversarial examples have received increasing attention and adversarial robustness becomes one of the key criteria.
3
4.. Through imperceptible perturbations
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

pM
12.43
01:33
72:65
21:26
64.
BOM
0:44
PM
7264
61-13
21:43
61.78
07.00
HIN
UX
10:16
D4-TE
CN
211
72
40
12.57
0.7
2428
26:44
20/1
GN
BGN
Transerabilitt
Terget
training (c). Traanderabiliity with envertataal traininng
Target
Figure I: (a) denotes the normalized dimensions of LN, GN, and IN, while (b) denotes BGN and BN. (c) and (d) denote the adversarial transferability among different types of normalizations.
To mitigate this risk, adversarial training is proposed to yield robust models through training on generated adversarial examples 6). Besides training procedure, some regularization and image preprocessing techniques are introduced to improve adversarial robustness (9) TO]. Recent work note that the architecture and module designs could play important roles in the robustness [1] [2]. Hence, we pay more attention to the basic modules in the network which are seldom considered for improving robustness, such normalization layers. Existing works have discussed the influence of Batch Normalization (BN) and empirically shown that BN increases adversarial vulnerability and decreases adversarial transferability [13 14). However, the theoretical analysis of this observation is insufficient and how to tackle this pitfall or even utilize this property to defend attacks is unexplored.
In this work, take numerous normalizations into consideration, including Layer Normalization (LN), Group Normalization (GN), Instance Normalization (IN), Batch Normalization (BN), and Batch Group Normalization (BGN) [5 16
[8 191. as shown in Figure [ (a) and (b). To evaluate the influence of different normalizations on the robustness, we first conduct PGD-7 attack (6) to both natural and adversarial trained networks with different normalizations on CIFAR-10. 10, shown in the diagonals of Figure (c) and (d). Not surprisingly. BN obtains the best robustness compared to other variants. However, we have intriguing observation after the transferability evaluations among different normalizations. As illustrated in the heatmaps, the adversarial accuracies in most scenarios are around 70% when fed with transferred adversarial examples, while those under white- box attack are around 50%. This huge gap mainly comes from the adversarial transferability among normalizations. Motivated by this observation, we first explore the relationship between adversarial transferability and normalizations, and show that the gradient similarity and loss smoothness are the key factors of the discrepancy in transferability among different normalizations. Based on the theoretical evidence, we propose to aggregate different types of normalizations to form random space the inference phase where the adversarial transferability can be significantly reduced. With designed random space, the inference phase naturally forms the black-box setting for those attackers who have access to model parameters due to the colossal random space. Together with the proposed black-box adversarial training framework, the adversarial robustness is substantially improved with less reduction of natural accuracy. For example, with the same adversarial training setting, the proposed algorithm improves the natural accuracy by 2.45% and the adversarial accuracy by 8.53% with ResNet-18 18 on CIFAR-10 under 2GD attack. Our contributions can be summarized as:
17
1) We provide both empirical and theoretical evidence that the upper bound of adversarial transferability is influenced by the types and parameters of normalization layers.
2) We propose a novel Random Normalization Aggregation (RNA) module which replaces the normalization layers to create huge random space with weak adversarial transferability. Together with natural black-box adversarial training, RNA boosts the defense ability.
3) We conduct extensive experiments to demonstrate the superiority of RNA on different benchmark datasets and networks. Different variants and components are also studied.
2 Related Work
DNNs are vulnerable to adversarial examples and arouse lots of research interests in the attack and defense techniques 5].. Expectation Over Transformation (EOT) is introduced to generating

---

BN
BGN_33
GN_32
-0.5
0.50
1.0
smoothness
(b). IN smoothness
(c). Smoothness different normalization and
(a) and (b) denote the smoothness of loss w.r.t. input with BN and IN respectively, (c) denotes the smoothness of BN, BGN, LN, GN, and IN with both and 2D plots.
Figure 2:
adversarial examples by computing the gradient over the expected transformation the input [20]. Rice er al. [21] explore the overfitting issue in adversarial training and propose to improve the robustness via early-stopping. Recently, the influence of DNN basic component on adversarial robustness has been paid more attention, such as activation function [22). operation [23], and neural terms of BN, Xie et al. [26] explore the robustness at different network scales and introduce mixture of two BN layers which take care of clean and adversarial examples separately to improve the trade-offs between clean and adversarial accuracy. The mixture BN can also improve the generalization of network with adversarial training (27], Benz el al. IK provide empirical evidence that BN increase the adversarial vulnerability. this paper, we lay emphasis the normalization layers and explore the connections between adversarial robustness and the aggregation of normalization layers to improve defense performance.
architecture [1 24 25).
Adversarial Transferability with Different Normalization
In this section, reveal the connections between adversarial transferability and normalization layers. We first consider network which is identified with an hypothesis h from space H. The network optimized with the loss function [ on input and labels V. The objective is formulated as
h*
argmin E NEH 2.9r (c(((x)
Given a target network and inputs {a,, y) the adversarial examples are defined as perturbed input T + 6, which makes the network l misclassify through maximizing the classification loss as
argmax L(h(t),),
5: 5-
where the perturbation is constrained by its 4o-norm. Adversarial transferability denotes an inherent property of L that these adversarial examples can also boost the classification loss ((h((T), y) of other networks as well, where h' H. We empirically demonstrate that transferability is influenced by the normalization layers in the network h, as shown in Figure (d). For example, taking BN and IN as source models generate adversarial examples, the adversarial accuracies LN are 71.29% and 74.34% respectively. We further provide more theoretical analysis of their relationships.
3.1 Definition of Normalization Layers
Batch Normalization is known as important basic module DNNs. which improves the network performance, and a wide variety of variants are introduced where the activations are normalized with different dimensions as well as sizes. To cover most types of normalization, we divide them into two categories. An illustration is shown in Figure (a) and (b), where LN, GN, and IN compute the mean and variance for each example with different group sizes during inference, while BN and BGN adopt the pre-calculated mini batch statistics which are computed by moving average in the training phase. Note that LN and IN are special cases of GN, which takes the minimum or maximum group number. Likewise, BN is special case of BGN. For simplicity, we use GN and BGN to cover all these normalizations. Considering the activations y E RdxN where N denotes the batch size and denotes the number of features, the normalized outputs after BGN with group number of ((GN) and
3

---

those of GN with group number of S(GN) during inference stage are formulated as
(1(4) (K (u)0): (UBBON))s
(#(BaN)) ((BON) (L / + A(BON), for I
(O(BON)
(U(R),, (MfOm)
(1)
((GN)): = ?(GN)
BKGN)) (5 (+(ON) ) M(GN) (0(GN) + MONN for SSS N, (3)
(d(o))):
c
where (M(GN)); G
E (h))))) Jal (o(o)))
)a (u (((ox))))
where a(G.N) denotes the group size of GN, (M(nON)); and (d(BaN)) denote the tracked
mean and standard deviation of group (
3.2 Variation of Loss Function Smoothness
Existing work on adversarial transferability reveals that the adversarial transferability is mainly influenced by the dimensionality of the space adversarial examples, since the adversarial subspaces of two networks are more likely to intersect with the growth of this dimensionality. [8]] The size of space adversarial examples can be estimated by the maximum number of orthogonal vectors which are aligned with the gradient g V Vxx(h(AX),)) In [28), tight bound is derived as
rT;; <llglla where & denotes the maximum number of ra which implies that the smoothness loss function is inversely proportional to the adversarial transferability. Thus, we analyze influence of different normalization layers on the smoothness of loss function, including GN and BGN.
vi
For simplicity, we dismiss the usage of & in the following equations since the computation both GN and BGN during inference independent of batch size. We denote the loss with GN Eyn and the loss with BGN as Lagn. Since the mean and variance are computed based on current group for both GN and BGN,
compute the partial derivative of loss w.r.t. a group Yj instead of yi where Y, V|GI & G &. +++ Similarly, Z, denotes the activations of group after normalization layers. Based on Eq. the partial derivative of 4.g and form W.r.t Yj can be given as
dign 0Y,
7/gn (G. 0Cgn 1(1, 07g"
a.gn ,,, 2Lgn Y,)),
G. G.og
0zj
07,"
aLngn 0Y,
Bbgn GLrgn byn 3
07,
where (.) denotes the inner product, 03" denotes the standard deviation of Y,, and of hyn denotes ac
the tracked standard deviation of Yy. For simplicity, we denote ač and g az," Taking the Dy,
advantage the fact that the mean of Y, is zero and its norm VG. the squared norm the partial derivative of GN and BGN can be derived as
|1 1|
7gm ( (llagn 11
1 T (1, (l,
C (ggn: p3) 110bg 11
2 Togr pong
19hgm 2|
(5)
Besides the smoothness of the loss, we further consider the smoothness of the gradients of the loss for GN and BGN. Following [29), compute the "effective" 3-smoothness through the quadratic form of Hessian of the loss w.r.f. the group activations in the normalized gradient direction, which measures the change of gradients with perturbations in the gradient direction. For simplicity, denote the hessian w.r.l the layer output as A JY,oy, the hessian w.r.l. the normalization output ac
86
For GN and BGN, we have the normalized gradient as T&ll and g' [M H "Z,0Z,
IT H Hyn 5 Dyn
2/ 10 Iyn Hgnd'n G. (onn }3)} Dhig Hogn
1
Vagn (afamys
Segg Hogn
(6)

---

BNard
BGN
aN
Figure 3: The histograms the gradient cosine similarity of different normalization layers.
3.3 Normalization Layers and Adversarial Transferability
The sufficient conditions and the bounds of adversarial transferabilityy between two networks have been discussed in [30]. We extend this result to the networks with different normalization layers. Since we focus on the influence of different normalization layers, we assume that these networks share the a6
same loss function and weight parameters W, which makes 0L &Lign
ac 87, 82, and a7,07, a7,07,
Meanwhile, Eq. and Eq. 6 can be easily generalized to the input I since a4 assumption, the connections between normalization layers and adversarial transferability can be established via bounded gradient norm and s-smoothness in Eq. and Eq. oas
ac W. With this
Theorem 3.1. Given two networks ha and hy with different normalization layers, the adversarial with attack target label yA and true label yr. Assume
perturbation under white-box attack is 5 ha and hi are "effective" Ba and B6- -m-shhoth respectively, the level of adversarial transferability T between networks ha and hy within the perturbation ball [|ō1|2 I can be upper bounded by
R. + Rs
TS
min(C((, max max 2
1+9
- 36)
where T denotes the attack successful rale, R. and Rs, denotes the empirical risks of network ha and hy, S denotes the upper loss gradient similarity, min(C(a. 4A)) mmnn (I. yA), Ma),Co Cz(x, y and max(| V Cll max (ur----- aA) (La.,, y)ll. VVLo, v)ll). Since the networks share the same loss function and weight parameters, we denote the influence of weight parameters as some constant Cg gradient norm and Cи gradient smoothness. The partial derivative and Hessian of loss w.r.t. the normalization output are the same for different normalization, denoted as g and H respectively. The gradient norm, Ba, and 3o in Eq. 7 Man be bounded
|| S C max
Iigm 3
1 ā (L.
-
(g. 8532,
Togal hug
|1ssl]
(8)
Bub S CH max
1gm (a g HT'
1
(0 (g. 83)
Pagn
g Hg'
G Tgn
Combining Eq. 7 and we observe that the upper bound of adversarial transferability is controlled by the gradient magnitude and gradient smoothness, which is further bounded according to the type and parameters of normalization layers. Specifically, given the same y and a for GN and BGN, GN achieves smaller gradient norm and better gradient smoothness than BGN, which decreases the upper bound of adversarial transferability. Furthermore, the group size G in GN plays important role smoothness. With smaller G, the smoothness of GN increases, and thus the upper bound of adversarial transferability decreases. Similar observations can be found in empirical evidence. As shown in Figure 2 the loss landscapes of different normalization layers w.r.f. input are visualized, which demonstrates that different normalization layers have different smoothness. Furthermore, IN achieves the best performance smoothness, which corresponds to the observation in Eq. g since IN has the minimum group size. The attack success rate is relatively low when the source model is IN, as shown Figure (c)) and (d), which corresponds to the observation in Theorem that the adversarial transfcrability decreases when the network is smoother.
5

---

Activation
Conv
Activation
Input
Network Params and All Pathss
Conv
Attacker
Which Infer?
GN_
BGN
Attack
White-hox
Adv Input
GN
Random
Activation
Conv
Activation
Sample
Transferability
Cony
BGN
Inference
Black-box Adv Train
BGN 2
GN
BGN
Network
Kandum
Path
Act
Sumple
(0). Random Normalization Aggregation
(b). Random Path Inference
(c), Block box Adv Training
Figure 4: An illustration of Random Normalization Aggregation and Black-box Adversarial Training.
4 Random Normalization Aggregation
Since the adversarial transferability is strongly correlated with the type of normalization layers, we ask simple question: Can we utilize the bounded adversarial transferability among normalization layers to defense against white-box attacks? In this work, we propose a Random Normalization Aggregation (RNA) module, which replaces the BN layer in the network. As shown in Figure 4 (a), the normalization layers becomes a combination of different normalization sampled from GNs and BGNs, where the underline denotes the group number. Specifically, the network maintains different normalization layers while only one normalization randomly selected for each layer during forwarding, as shown in Figure layers, the network with RNA module can be treated as a "supernet" with multiple paths. Back to white-box defense setting, we assume that the attackers have access the network parameters. The adversarial examples are generated through backward on randomly sampled path, and then fed to another randomly sampled path due RNA module, which makes the entire white-box attack become "black-bor box" attack, as illustrated in Figure (c). Thus, together with the adversarial transferability study in Section B] it is natural create network with random space in normalization layers where the adversarial transferability is significantly constrained. To achieve a strong defense against adversarial attacks, some concerns still remain: (1). The number of paths are required to be extremely large to reduce the probability of sampling the same path with random sampling strategy; (2). The collaboration with traditional adversarial training; (3). The normalization types need to be carefully selected to enforce low adversarial transferability. We discussion these concerns follows.
(b). (b). Through incorporating randomization normalization
Path Increment in Random Space The adversarial transferability among different normalization has been discussed in Figure (c) and (d). However, the size of random space also matters for effective defense against attacks. the attackers can sample the same path during attack and inference phases with a high probability, the adversarial accuracy will decrease tremendously. To tackle this issue, we introduce layer-wise randomization RNA module, which randomly samples the normalization for each layer in the network. As shown in Figure (b). different normalization types are sampled for different layers, which exponentially increases the number of paths. Given the normalization types in RNA and L layers in the network, the size of random space becomes nz, which reduces the probability of sampling the same path during attack and inference phase to n.. << n
Black-box Adversarial Training It is natural to incorporate RNA module into adversarial training. Consistent with inference phase, we randomly sample path Pa and conduct white-box attack to generate adversarial examples Xp.. Different from traditional adversarial training which optimizes Pa through feeding &p., we feed X,, to another randomly sampled path p, which forms a "black-box" adversarial training, as illustrated in Figure 4 (c). Eq. and Eq. can be reformulated as
h- = argmin E hEW Zpp
[Chha :P).))) where T..
C(h(Ep.:Pa).u)) 3p. 1pp -
argmax
where P denotes the space of paths. The training procedure is shown in Algorithm
Normalization Types Selection In RNA module, multiple normalization types are maintained to form random space. According to Theorem 3.1, the adversarial transferability is bounded by different components, including gradient similarity, empirical risks, gradient magnitude, and gradient smoothness. We first provide empirical evidence that normalization layers from the same category
defined in Section 3.1 tend to have higher gradient similarity. As shown in Figure 3 we visualize

---

Algorithm/ Random Normalization Aggregation with Black-hox Adversarial Training
The training tuple (X, J/; Path set P; Attack step size 17; Attack iterations t; Perturbation size e; Network h with parameters W;
Input:
Replace BN layers with RNA modules, and initialize the network. while not converge do
Sample a batch of data {x. y) from (Ry, Randomly sample path Pe from P: Initialize adversarial perturbation 5; for i to / do
ô = clip. 5 + 1- sign V (2 end for
Randomly sample path p from P: W = W Vw Z(h(x p))).
end while
the histograms over the cosine similarity of two networks with different normalization layers. For example, BN and BGN belong to the same category, and their gradient similarity is much higher than that between BN and IN, comparing Figure (a) and (b). Since the gradient similarity is proportional to the upper bounds of adversarial transferability in Eq. 7 we propose to select normalization from different categories. Thus, RNA module samples the normalization types from both GN and BGN with small group sizes in our experiments, while the evaluation of other combinations is also included.
5 Experiments
In this section, we provide sufficient evaluation of RNA module on various models and datasets.
5.1 Evaluation Setup
CIFAR-10/100 We first conduct experiments on CIFAR-10/100 [3] datasets, which contain 50K training images and 10K testing images with size of 32x32 from 10/100 categories. The networks we use are ResNet-18 18
31 and WideResNet.32 (WRN) (32]. The SGD optimizer with momentum of 0.9 is used. The weight decay is set to 5 x 10-4 The initial learning rate is set to 0.1 with piecewise decay learning rate scheduler. All the baselines are trained with 200 epochs with a batch size of 128. The PGD-10 with = 8/255 and step size of 2/255 is adopted in the adversaris training setting. For the RNA module, we utilize BN and IN form the random space normalization layers. The experiments are performed on one V 100 GPU using Pytorch [33] and Mindspore [34].
The effectiveness of proposed RNA is also evaluated on ImageNet (35), which contains 1.2M training images and 50K testing images with size of 224 X 224 from 1000 categories. The networks we use are Re [3. The SGD optimizer with a momentum of 0.9 is used. The weight decay is set to x 10-4 The initial learning rate is set to 0.02 with a cosine learning rate scheduler. We load a pretrained ResNet-50 and then adversarailly train the network for 60 epochs with a batch size of 512. The PGD-2 with 4/255 is adopted in the adversarial training setting. For the RNA module, we utilize BGNs and GNs with group size of and to form the random space. The experiments are performed on eight V100 GPUs.
ImageNet
Baselines and Attacks Our proposed RNA modules replace the normalization layers in the network. Thus, various normalization layers are involved for comparison, including BN, IN, LN, GN, and BGN [6] [7 18 19. On CIFAR-10/100. we evaluate the robustness of all the baselines under different strong attacks from TorchAttacks [36]. For Fast Gradient Sign Method (FGSM) 4. the perturbation size is set to 8/255. For Projected Gradient Descent (PGD) (6, e is set to 8/255 with step size of 2/255, and the steps are set to 20. For CW attack [37], the steps are set to 1000 with learning rate 0.01. For Momentum variant of Iterative Fast Gradient Sign Method (MIFGSM) (38. is set to 8/ /25 with step size of 2/255, and the steps are set to 5 with decay of 1.0. For DeepFool [39], the steps are set to 50 with overshoot of 0.02. For Auto Attack 40), e is set to 8/255, On ImageNet, we evaluate the robustness of under PGD attacks with of 4/255 and steps of 50.
7

---

Table 1: The adversarial robustness evaluation of adversarial--rained networks CIFAR-10.
Model
Method
Natural
FGSM
PGD
CW
MIFGSM
DeepFool
AutoAttack
BN [18]
81.84
56.70
52.16
78.46
54.96
0.35
47.69
BGN32 [19
77.28
54.60
50.71
73.57
53.05
0.39
46.12
IN 17
77.07
50.07
42.76
72.51
47.63
2.27
38,30
ResNet-18
GN32 16
76.69
52.60
45.66
72.92
50.26
0.72
41.83
LN 13
79.81
53.88
45.44
75.51
50.72
1.13
41,48
RNA(Ours)
84.29
63.10
60.69
84.45
60.70
76.73
65.61
BN [18
85.27
60.65
55.06
82.24
58.47
0.40
52.24
BGN32 [99
83.70
59.66
54.96
80.25
57.85
0.37
51.38
IN 17
84.11
58.14
50.37
80.13
55.37
1.62
46.81
WideResNet32
GN32 16
83.45
58.95
51.94
79.55
56.70
1.37
47.99
LN 13
83.24
57.80
49.74
79.39
54.68
0.95
46.44
RNA(Ours)
86.46
65.73
63.34
85.68
62.84
78.18
67.88
Table 2: The adversarial robustness evaluation of adversarial-traine networks CIFAR-100.
Model
Method
Natural
FGSM
PGD
CW
MIFGSM
DeepFool
AutoAttack
BN [18]
55.81
31.33
28.71
50.94
30.26
0.79
24.48
BGN32 [9]
53.16
30.11
27.75
48.74
29.11
0.54
23.29
IN [17]
52.92
27.56
23.16
47.70
25.91
2.86
19.33
ResNet-18
GN32 [16
51.00
28.32
25.10
45.82
27.10
0.79
21.00
LN 15
48.82
27.05
23.83
44.07
26.93
0.80
19.97
RNA(Ours)
56.79
36.76
35.55
56.86
34.00
51.77
42.12
BN [18]
60.11
35.40
31.69
57.11
34.14
0.23
28.36
BGN32 [19
58.54
34.13
30.97
54.08
33.01
0.46
26.92
IN 17
56.71
31.96
28.09
51.98
30.33
1.83
24.25
WideResNet32
GN32 16
59.08
33.56
29.94
53.89
32.30
1.12
25.78
LN 15
57.09
32.92
29.75
52.19
31.73
0.79
25.71
RNA Ours)
60.57
37.87
36.04
60.21
35.58
55.16
42.43
5.2
Results for Robustness
Main Results We first evaluate the performance of RNA on CIFAR-10 and CIFAR-100 under different types of attacks. The detailed results shown in Table D and 2 layers show similar performance on robustness. However, with random space of different nor- malization layers, the robustness is significantly improved. Comparing RNA with other baselines. RNA consistently achieves the best performance under all attacks, and show strong superiority, For example, RNA with ResNet- 18 achieves 65.61% under Auto Attack CIFAR-I0, which 17.92% higher than BN. Similarly, RNA with WRN achieves 55.16% under DeepFool attacks on CIFAR-100. which is 53.33% higher than IN. The boosted adversarial accuracy shows strong empirical evidence that the constrained adversarial transferability in random space can provide satisfactory defense capability. Furthermore, with proposed black-box adversarial training, RNA achieves better natural accuracy. For example, RNA with WRN achieves 46%46%% on CIFAR-10, which improves BN by 1.19%. We mainly attribute this improvement to the fact that the generated adversarial examples can be treated as "weaker" attack example to other paths during optimization, which naturally achieves better trade-offs between natural and adversarial accuracy.
Popular normalization
Stronger PGD Attacks We further evaluate the defense capability of RNA under stronger PGD attacks. which enhance the number of iterations and enlarge the perturbation size. The comparison with other baselines are shown in Figure 5 (a) and (b). Comparing RNA with other baselines, RNA achieves stable robustness under different attack iterations, such as 60.70% on LO and 59.94% on PGD 100, Meanwhile, the PGD accuracy of RNA is much higher than all other baselines. For example, RNA improves BN baselines by a margin of 7.93%.
terms of larger perturbation size, RNA achieves the best robustness in all the scenarios, and the lowest decrement among all the methods. Specifically, RNA achieves 80.06% with of 2/255 and 24.90% with e of 20/255, whose gap is 56.16%. For comparison, the gap of BN is 64.68 and GN is 58, 48%.

---

331
BON
cura
5013
60.0
0.6
53.0
62.5
50
0.5
40
BN
PGD
07.5
GN+
LN 9-N
42.5
10
60
0.07
Steps
Fath Difference
(4).
Steps
Porturbation
Advorsarial Transfrrbiilit
Space
Figure 5: The evaluation of robustness under PGD attacks settings. (a) denotes larger attack iterations, and (b) denotes larger perturbation sizes. The adversarial transferability in random space evaluated through sampling paths with different levels of diversity in (c).
Table Comparison with defense algorithms.
Table 4: Robustness evaluation of random space built from different normalization com- binations under different attacks.
CIFAR 10
ImageNet PGD
Method
2
AA
RobustWRN [41
59.13
52.48
31.14
Normalization
PGD
DeepFool
AutoAttack
AWP [42
58.14
54.04
BN
52.16
0.35
47.69
RobNet [I]
52.74
37.15
GN+BGN
55,40
70.26
58.90
RPI+RPT [43
53.96
53.30
42.72
GN+LN
46.67
62.85
47.96
SAT [22
56.01
51.83
42.30
LN+BN
55.67
68.14
58.73
RNA(Ours)
63.34
67.88
54.61
IN+BN
60.69
76.73
65.61
Comparison with SOTA Defense Methods To demonstrate the superiority of RNA, we include several SOTA defense algorithms for comparison. RobustWRN [4] explores the importance of network width and depth on robustness. AWP [42 proposes to regularize the flatness of weight loss landscape to achieve robustness. RobNet [1] introduce a NAS framework for robustness. RPI+RPT
43 utilizes randomized precision for adversarial defense. SAT 44 proposes to replace ReLU with its smooth approximations, which exhibits robustness. The results are shown in Table 3 We use WRN on CIFAR- and ResNet-50 on ImageNet. All the baselines are evaluated under the attack 2GD and AutoAttack (AA) CIFAR- 10 and PGD ImageNet. Our proposed RNA module achieves the best performance all the scenarios. On CIFAR-10, 10, RNA achieves 67 88% under AutoAttack, with 13.84% improvement compared with AWP. On ImageNet, RNA achieves 54.61% under 5GD with 12.31% improvement compared with SAT. Note that RNA replaces the normalization layer, which is orthogonal other defense techniques. Similarly, the adversarial training in setting can be replaced by other advanced training strategies to achieve potentially better performance.
Adversarial Transferability in Random Space For better illustration of the adversarial trans- ferability the random space built from RNA, we conduct the adversarial transferability study of ResNet-18 18 applied with RNA on 100. We first define the path difference the number of different normalization layers between attack and inference paths. For example, a path difference of 7 denotes two paths selecting different normalization layers in 7 layers during forwarding. For each path difference, we then randomly sample 10 path pairs for transferability evaluation. We also include different normalization combinations in RNA for comparison. The results shown in Figure 5 (c), which illustrates the PGD accuracy under transferred attacks between path pairs along increasing path differences. The filled areas denote the maximum and minimum PGD accuracy. It is obvious that the combination of BN and IN achieves the best performance, which also corresponds to the analysis Theorem 3.1 With random sampling strategy, the path difference is always around 10 since the number of normalization layers is 20 in this network. For example, the combination of BN and IN achieves average PGD accuracy of 61.09% when the path difference becomes 10. Compared with the baseline BN which achieves 52. 16% PGD accuracy, this lower adversarial transferability in our random space brings a strong defense capability against adversarial attacks.
5.3 Ablation Study
Different Normalization Combinations We first provide more quantitative results of different combinations of normalization layers in RNA module. We conduct comparison with ResNet-18 on

---

Table 5: Ablation studies of RNA with ResNet- 18 on CIFAR-IO, including random space designing. adversarial training, and layer-basse based constraint.
Random Space
ADV Train
Layer-free
Natural
FGSM
PGD)a
CW
MIFGSM
DeepFool
AutoAttack
BN+IN+GNAI.N
WhiteBox
X
10,44
GN]1-64]
WhiteBox
X
75.50
53.62
49.57
74.63
51.81
47.20
50.57
GN[1-64]
BlackBox
77.54
55,06
51.32
76.55
52.44
58.72
$2.59
BGN+GN
BlackBox
73.81
53.98
52.98
73.19
53.07
63.78
56,86
BGN+GN[1-64]
BlackBox
61.72
46.73
46.42
61.89
45.69
56.82
51.26
BGN+GN[1-161
BlackBox
70.99
51.66
49.74
70.48
51.58
63.63
55.98
BGN+GN[1-4)
BlackBox
78.26
58,91
56.29
77.74
56.31
70.83
61,08
BGN+GN[[-1]
BlackBox
84.29
63.10
60.69
84.45
60.70
76.73
65.61
CIFAR-10. As shown in Table we evaluate the performance under PGD DeepFool and Auto Attack, and the combination of IN and BN achieves the best robustness in all the scenarios. Consistent with the observation in Theorem 3.1 the combination of LN and BN has slightly worse performance than that of IN and BN, since IN Is smoother normalization than LN. Similarly, the combination of GN and LN achieves the worst performance, since LN special case of GN so that they have high gradient similarity in our empirical observation, as discussed in Figure 3 Thus, utilizing different normalization layers with smaller group size from GN and BGN, RNA module can form random space with low adversarial transferability to better improve the defense ability.
Effectiveness of Different Components We next demonstrate the effectiveness of each component introduced in RNA module. The detailed results are shown in Table 3 where [.] denotes the range of group size. Besides the normalization combinations, we include more discussion of the random space designing. To form a random space in normalization layers, it is natural to consider combination of BN, IN. GN, and LN, however, it is difficult optimize, as shown in the first row of Table [5] With the normalization layers selected from GNs. the optimization becomes stable, however, the robustness not competitive, as shown the second row. The involvement of black-box adversarial training significantly improves the robustness, shown in the third row. Through expanding the random space with BGNs, the defense capability is slightly improved due to the doubled number of paths. However, the size of random space is still limited. After removing the layer-based constraint, the number of paths exponentially increases, the size of random space for each layer can be reduced for better trade-offs, as shown in the last rows Comparing BGN+GN[1-1] with GN[1-64], better random space designing with an appropriate adversarial training strategy can achieve an improvement of 15.04% under Auto Attack, which demonstrates the necessity of these components.
6 Conclusions
In this paper, we explore the importance of normalization layers in adversarial robustness where the transferability among different normalization layers can be utilized to boost the defense capability. Random Normalization Aggregation (RNA) module is proposed to form random space with low adversarial transfcrability for defense against adversarial attacks. We provide sufficient empirical evidence and theoretical analysis to reveal the connections between adversarial transfcrability and normalization types, which guides the random space designing. With the involvement of black-box adversarial training strategy and the relaxation of layer- based constraint, the robustness provided by RNA module is significantly strengthened. We demonstrate the superiority of RNA module via comprehensive experiments across different network architectures, attack settings, and benchmark datasets. Our work can provide valuable insights into the network module design for robustness.
Acknowledgments
This work supported in part by the Australian Research Council under Project DP210101859 and the University of Sydney Research Accelerator (SOAR) Prize. The authors acknowledge the use of the National Computational Infrastructure (NCI) which is supported by the Australian Government, and accessed through the NCI Adapter Scheme and Sydney Informatics Hub HPC Allocation Scheme. We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
10

---

References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffry E Hinton. Imagenet classification with deep convolutional neural networks. In Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 25, Curran Associates, Inc., 2012.
[2] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-onn; Towards real time object detection with region proposal networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[3] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. CoRR, (bs/1555,,4597, 2015.
[4] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
2014.
[5] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples, 2014.
[6] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[7] Andrew Ilas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-hox adversarial attacks with limited queries and information. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2137-2146. PMLR, 10-15 Jul 2018.
[8] Wieland Brendel ", Jonas Rauber ", and Matthias Bethge. Decision- based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representa- tions, 2018
[9] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. CoRR, abs/1901 008573, 2019.
[10] Cihang Xie, Jianyu Wang, Zhishuai Zhang. Zhou Ren, and Alan L. Yuille. Mitigating adversarial effects through randomization. CoRR. abs/1711.01991, 2017.
[11] Minghao Guo, Yuzhe Yang, Rui Xu, and Ziwei Liu. When NAS meets robustness: In search of robust architectures against adversarial attacks. CoRR, Abs/1911.10699 2019.
[12] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L.. Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 501-509. Computer Vision Foundation / IEEE, 2019.
[13] Philipp Benz, Chaoning Zhang, and In So Kweon. Batch normalization increases adversarial vulnerahility: Disentangling usefulness and robustness of model features. CoRR, abs/2010.03316, 2020.
[14] Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W. Taylor. Batch normalization is a cause of adversarial vulnerability. CoRR, htxs/1905.0216. 2019.
[15] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey F. Hinton. Layer normalization, 2016.
[16] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
[17] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Instance normalization: The missing ingredient for fast stylization. CoRR, abs/1607.08022 2016.
[18] Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.00167 2015.
[19] Xiao- Yun Zhou, Jiacheng Sun, Nanyang Yc, Xu Lan, Qijun Luo, Bo-Lin Lai, Pedro M. Esperança, Guang-Zhong Yang, and Zhenguo Li. Batch group normalization. CoRR, abs/2012.02782,, 2020.
[20] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. PMLR, 2018
In International conference on machine learning, pages
11

---

[21] Leslie Rice, Erie Wong, and Zico Kolter. Overfitinng in adversarially robust deep learning. In International Conference on Machine Learning, pages 8093-8104. PMLR, 2020.
[22] Cihang Xie, Mingxing Tan, Boqing Gong, Alan L. Yuille, and Quoe V. Le. Smooth adversarial training. CoRR, abs/2006. 14536, 2020.
[23] Minjing Dong, Yunhe Wang, Xinghao Chen, and Chang Xu. Towards stable and robust addernets. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 13255-13265. Curran Associates, Inc., 2021.
[24] Minjing Dong, Yanxi Li, Yunhe Wang, and Chang Xu. Adversarially robust neural architectures. CoRR, abs/2009.00902, 2020.
[25] Yanxi Li, Zhaohui Yang. Yunhe Wang, and Chang Xu. Neural architecture dilation for adversarial robustness. CoRR, abs/2108.06885, 2021.
[26] Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. arXiv preprint urXiv:1906.0377 09101.90007772000 2019.
[27] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 319-828.22 2020.
[28] Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples, 2017.
[29] Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization?, 2018.
[30] Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Benjamin I. P. Rubinstein, Ce Zhang, and Bo Li. TRS: transferability reduced ensemble via encouraging gradient diversity and model smoothness. CoRR, fabs/2104000677 2021.
[31] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[32] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/16605.07144 2016.
[33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 2019.
[34] Huawei. Mindspore. httpe://www.iindsprre cn/ 2020.
//wp:///www....
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR '16, pages 770-778. IEEE, June 2016.
[36] Hoki Kim. Torchattacks: A pytorch repository for adversarial attacks. arXiv preprint arXiv:2010.01950. 2020.
[37] Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. CoRR, abs/1608.0466 2016
[38] Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Xiaolin Hu, and Jun Zhu. Discovering adversarial examples with momentum. CoRR, abs/1710.06081, 2017.
[39] Seyed-Mohsen Monsavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method fool deep neural networks. CORR, abs/151 htw//1511.4455 1.04599, 2015.
[40] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. CoRR, 0bs/2003.01699 2020.
[41] Hanxun Huang. Yisen Wang. Sarah Monazam Erfani, Quanquan Gu, James Bailey, and Xingjun Ma. Exploring architectural ingredients of adversarially robust deep neural networks. CoRR, abbs/2110003822 2021.
[42] Dongxian Wu, Shu-Tac Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc.
12

---

[43] Yonggan Fu, Qixuan Yu, Meng Li, Vikas Chandra, and Yingyan Lin. Double-win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Muchine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3492-3504, PMLR, 18 24 Jul 2021.
[44] Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmect Kohli. Adversarial robustness through local lincarization, 2019.
13


Reviews:
[{"review_id": "gtI-3418p5W", "reviewer": "Reviewer_kqKQ", "summary": "This work provides a clear explanation for the relationship between normalizations and adversarial transferability, which further inspires the authors to propose a random normalization aggregation method where the adversarial robustness can be significantly improved.", "questions": "I suggest the authors give an explanation for the difference between RNA (random selection in normalizations) and other random selection methods, i.e., selecting sub-network in parameter or feature space.", "limitations": "Please see [Strengths And Weaknesses]", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Using adversarial transferability to boost adversarial robustness is interesting and inspiring, which is technical and methodology sound.\n\n2. Establishing the relationship between normalizations and adversarial transferability is technically sound, and, in my opinion, it will make some impact on our community and give some insights to more researchers.\n\nWeaknesses:\n\n1. The connection between the mentioned novel viewpoint of using adversarial transferability and improved adversarial robustness should be highlighted, i.e., why transferability can contribute to boosting robustness, similarly why RNA can contribute to boosting robustness, moreover, do these two reasons share the same inspiration?\n\n2. For the introduction to motivation, the authors claim that (line47-line50) the performance gap between BN and other normalizations results from “the adversarial transferability among normalizations,” which is a bit confusing and needs further justification. Specifically, the diagonal results are evaluated using white-box attacks, and the robust accuracy is about 50%, so that the gap may come from the difference between white- and black-box attacks. As this is an essential part of inspiring the authors to investigate the relationship between transferability and normalizations (line50), I suggest the authors provide a detailed justification.\n\n\n3. One concern is that the robustness may come from the designed random selection operation, so it is necessary to employ expectation over transformation, EOT. [1], for the robustness evaluation to defend against the random operation.\n\nTypos:\n\n\\tilde{x} makes Eq.2 confusing.\n\n[1] Synthesizing Robust Adversarial Examples ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dEpSZTfAD-C", "reviewer": "Reviewer_q4wP", "summary": "Inspired by the limited adversarial transferability across different normalizations, the authors proposed to involve randomness into the types of normalization layers and introduce a RNA module that reduces the adversarial transferability in a pre-defined random space, which improves the defense against adversarial examples. To evaluate the effectiveness of their algorithm, the authors provided experimental results on CIFAR-10/100 and ImageNet.\n", "questions": "As mentioned above, several questions are listed below:\n\n1. What is the risk of random sampling strategy? What is the probability of sampling the similar paths?\n\n2. How about the involvement of randomness in weight parameters and architectures? What is the advantage of proposed RNA compared with these baselines?\n\n3. Is there empirical evidence that the smoothness of different normalization layers controls the defense performance?\n", "limitations": "The limitations have been addressed and there is no potential negative societal impact of this work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors studied a simple yet effective randomized mechanism with normalization layers. In general, the paper reads well, and the presentation is clear. The paper is original in that it studies the connections between normalization layers and adversarial defense. The theorical analysis well explains the principle of proposed algorithm. The paper includes a clear experimental setup and a meticulous comparison with other variant baselines as well as state-of-the-art algorithms on popular benchmarks. The results seem convincing.\n\nDespite its contributions, I have several concerns:\n\n1. The authors mentioned that they adopt random sampling strategy to utilize the adversarial transferability in random space. However, there exists a chance that the similar paths are sampled during both attack and inference stage, and there is no discussion of this scenario as well as the probability of it.\n\n2. Besides normalization layers, there exist wide components in the networks which can involve randomness, such as weight parameters and architectures. Although the transferability evaluation in Fig 1 shows that there exists a poor transferability of adversarial examples among different types of normalization layers, it seems to me that this transferability also holds true for weight parameters and architectures. There is no discussion of these variant baselines.\n\n3. The authors claimed that the smoothness of different normalization layers directly controls the adversarial transferability. However, it is hard for me to find corresponding evidence in experiment section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5p_AwfsA59F", "reviewer": "Reviewer_ioBB", "summary": "Researchers explore how different normalization layers affect adversarial transferability. They provide a theoretical upper bound on the adversarial transferability between normalization layers. Then, they proposed a module named Random Normalization Aggregation (RNA). RNA replaces the normalization layers in the network, and samples normalization layers randomly at each forward pass. This can generate an exponential number of possible paths, which makes it harder for attacker to exploit.\n", "questions": "Experiments questions/concerns:\n\t\n1. What if the attacker randomizes a different path (p_a) at each iteration of PGD? Wouldn't it make the attacker more diverse, and therefore generalize better? Can the authors present such experiments?\n\t\n2. Can the authors explain why the results on resNet-18 are better than those on WRN-32? For example, on CIFAR-10, AA results on ResNet-18 are better by almost 4% compared to WRN-32\n\t\n3. Also, for CIFAR-10, FGSM and PGD-20 acc is lower than the Auto-Attack accuracy, which doesn't make sense. Can the authors explain these results?\n\t\n4. WRN-32 with BN and standard AT is essentially the standard AT method by Madry et al. [1]. It was shown that this method reaches AA accuracy of ~52-53% against AA when combined with early stopping. However, in your paper, you claim 46.44% against AA. Did the authors use early stopping for all methods? If not, I think that should present results with early stopping [2] for a fair comparison. Otherwise, the results can be linked to adversarial overfitting [2].\n\n5. It would be interesting to see how the method works when combined with other AT methods.\n\n6. The authors did not refer/compared to related work in the field of normalization layers and robustness [3, 4]. Can the authors compare their method to [3, 4] and present the results?\n\n[1] Towards Deep Learning Models Resistant to Adversarial Attacks https://arxiv.org/pdf/1706.06083.pdf\n\n[2]  Overfitting in adversarially robust deep learning http://proceedings.mlr.press/v119/rice20a\n\n[3] INTRIGUING PROPERTIES OF ADVERSARIAL TRAINING AT SCALE https://arxiv.org/pdf/1906.03787.pdf\n\n[4] Adversarial Examples Improve Image Recognition https://openaccess.thecvf.com/content_CVPR_2020/html/Xie_Adversarial_Examples_Improve_Image_Recognition_CVPR_2020_paper.html\n\n\nmiscellaneous:\n\n1. Figure 4 has twice (b) instead of (c) \n\n2. Line 153 - I suggest rephrasing to- \"… to *defend against* white-box attacks …\"", "limitations": "No.\nAuthors should discuss the limitation of adversarial training in general and their method in particular.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is clear and easy to follow\n\n2. Theoretical part is sound\n\n3. The idea to generate random normalization paths is new to the literature\n\nWeaknesses:\n\n1. Experiment section presents some suspicious results - the results with resNet vs. WRN are counterintuitive, and results on Auto-Attack vs. PGD also do not make sense.  It can be a sign of obfuscated gradients. Additionally, some of the reported results are not in line with known results. See the Questions section.\n\n2. No code/models are provided for evaluation of the results. Can the authors provide the code/models/github link for additional verification?\n\n3. No related work section. Specifically, the authors did not address some related literature that explores the same field (see Questions section).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "37C85Zf-364", "reviewer": "Reviewer_J8qn", "summary": "The paper discusses the relation between normalization layers used in DNNs and adversarial transferability. The author(s) show(s) that the choice of normalization layer highly influences the success rate of transferred adversarial examples. In detail, it is shown that adversarial examples transfer worse (i.e., have a lower success rate of fooling the network) if a different normalization layer is used in the attacked network. This fact is used to motivate a novel technique to robustify existing neural network architectures: the key idea is to randomly select the used normalization layer during inference. Experiments on CIFAR-10, CIFAR-100 and ImageNet show that this approach is effective for the commonly used ResNet-18 and WideResNet32.", "questions": "What is meant with $\\beta$ in Equation (7)?", "limitations": "Limitations and potential negative societal impact have not been addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n- The paper proposes an effective, clear and easy-to-implement method that is well-motivated by experimental and theoretical results.\n- The extensive experimental evaluation on ResNet-18 and WideResNet32 architectures using a plethora of attacks (FGSM, PGD-20, CW, MIFGSM, DeepFool, AutoAttack) is convincing. An ablation study is conducted to show the effectiveness of the different components and the importance of different combinations of normalization layers.\n- The paper presentation is well done and only contains only a few errors (see minor remarks). Used computational resources are mentioned.\n\nWEAKNESSES:\n- The used notation can be improved at some places: \n    1. Equation (1): $x\\sim\\mathcal{X}, y\\sim\\mathcal{Y}$ suggests that $x$ and $y$ are drawn independently form the dataset. Here, $x, y\\sim\\mathcal{X},\\mathcal{Y}$ would be correct. Also in Algorithm 1 $\\{\\mathcal{X},\\mathcal{Y}\\}$ has to be a tuple instead of set.\n    2. In Equation (3): $\\hat{y}(bgn)\\_i^{(k)}$ suggests that $\\hat{y}$ is a function in terms of $(bgn)$, which is not the case. I would suggest to use $\\left(\\hat{y}_{(\\text{BGN})}^{(k)}\\right)_i$ instead.\n    3. Different symbols are used to denote multiplication (Equation (3) vs Equation (8))\n- The stated Theorem 2.1 is hard to understand. Although, I see some value in the theorem, I think the author(s) should spend some time to reformulate it:\n    1. It stated that \"the gradient norm and $\\beta$ in Equation (7) can be bounded as [...]\", however, there is no $\\beta$ in Equation (7) (only  $\\beta_a$ and  $\\beta_b$).\n    2. The variable $T$ is introduced multiple times in a single sentence: \"the level of adversarial transferability\" and \"attack success rate\".\n\nMINOR REMARKS:\n- Typo \"Adversarial Transfersability\" in Abstract, \"hessian\" is sometimes not capitalized\n- subscript and supperscript \"gn\" and \"bgn\" should not be typeset in math mode\n- line 96: extra period in front of references [5, 20]", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept