Robustifying $\ell_\infty$  Adversarial Training to the Union of Perturbation Models
Authors: Ameya Patil
Michael Tuttle
Alex Schwing
Naresh Shanbhag
Venue: NeurIPS 2021 Submitted (2021)
Keywords: adversarial examples
adversarial robustness
efficient adversarial training

Abstract: Classical adversarial training (AT) frameworks are designed to achieve high adversarial accuracy against a single attack type, typically $\ell_\infty$ norm-bounded perturbations. Recent extensions in AT have focused on defending against the union of multiple perturbation models but this benefit is obtained at the expense of a significant (up to $10\times$) increase in training complexity over single-attack $\ell_\infty$ AT. In this work, we expand the capabilities of widely popular $\ell_\infty$ single-attack AT frameworks to providing robustness to the union of ($\ell_\infty, \ell_2, \ell_1$) perturbations while preserving their training efficiency. Our technique, referred to as Shaped Noise Augmented Processing (SNAP), exploits a well-established byproduct of single-attack AT frameworks -- the reduction in the curvature of the decision boundary of networks. SNAP prepends a given deep net with a shaped noise augmentation layer whose distribution is learned along with network parameters using any standard single-attack AT. As a result, SNAP enhances adversarial accuracy of ResNet-18 on CIFAR-10 against the union of ($\ell_\infty, \ell_2, \ell_1$) perturbation models by 14%-to-20% for four state-of-the-art (SOTA) single-attack $\ell_\infty$ AT frameworks, and, for the first time, establishes a benchmark for ResNet-50 and ResNet-101 on ImageNet.

Robustifying to Adversarial Training to the Union of Perturbation Models
Anonymous Author(s)
Affiliation
Address
email
Abstract
Classical adversarial training (AT) frameworks designed to achieve high ad- versarial accuracy against a single attack type, typically &oo norm-bounde per- turbations. Recent extensions in AT have focused defending against the union multiple perturbation models but this benefit is obtained at the expense of a significant (up to 10x) increase in training complexity over single-attack Eoo AT. In this work, we expand the capabilities of widely popular single-attack Eo AT frameworks provide robustness the union of (Lo,, E2, (1) perturbations while preserving their training efficiency. Our technique, referred to as Shaped Noise Augmented Processing (SNAP), exploits a well-established byproduct of single- attack AT frameworks the reduction in the curvature of the decision boundary of networks. SNAP prepends given deep net with shaped noise augmentation layer whose distribution is learned along with network parameters using any standard single-attack AT. As a result, SNAP enhances adversarial accuracy of ResNet- 18 on CIFAR-10 against the union of (foc, l2, f1) perturbations by 14%-to-209 20% for four state-of state-of-the-art (SOTA) single-attack foo AT frameworks, and, for the first time, establishes benchmark for ResNet-50 and BesNet-101 on ImageNet.
2
6
11
12
13
14
15
16
Introduction
17
Today adversarial training (AT) provides state-of-the-art (SOTA) empirical defense against adver- sarial perturbations. For this, adversarial perturbations are used during training to optimize a robust loss function [20, 41, 30, 35]. Early AT frameworks [20, 41] were 7 Tx-to-10x more computationally demanding than vanilla training. More recent works [30, 35, 40] have significantly reduced the computational demands of AT via single-step attacks and supercomvergence..
19
21
23 However, today's 's AT frameworks predominantly focus on single-attack, i.e., they seek robustness
24 to single perturbation, typically lo- bounded [30, 35, 37, 41, 43, 40, 39, 26, 9, 34, 42, 10, 11, 14]. This results in low performance against other perturbations such as E2. &,, or the union of (f Eg & Indeed, as shown in Fig. 1, four state-of-the-art (SOTA) single-attack AT frameworks (black markers)
25
26
employing only bounded perturbations achieve low adversarial accuracy (U) of z 15%-to-20% against the union of (f.c, 22, 21) perturbations. Recent extensions in AT [21, 32, 18] do seek higher A(U) but only the expense of 6xx-to-1xx increase in the total training time (blue markers in Fig. 1). The large training time of these AT frameworks has inhibited their application to large-scale datasets such as ImageNet, e.g., Maini et al. [21], Tramèr & Bonch [32] show results for MNIST and
27
28
30
31
32
CIFAR- 10 only, while Laidlaw et al. [18] only additionally show 64 x 64 ImageNet-100 results.
The high training time for AT frameworks arises from two sources: (i) the need to employ larger 34 networks, e.g., MSD [21] with ResNet-18 achieves higher (U) than PAT [18] with ResNet-50 (see
Fig. 1); and (ii) the need to incorporate multiple perturbations during each attack step and higher
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

36
overall number of attack steps, e.g., 50 in MSD [211, 20 in AVG [32]. Obviously one can always reduce the number of attack steps in MSD/AVG to proportionally reduce training time. Doing so results in training time and All) to rapidly approach the training complexity and A of standard AT frameworks, e,g., a 5-step MSD and 2-step AVG is equivalent in training time and accuracy to PGD training,
37
38
39
40
and TRADES, respectively. Notwithstanding the expensive nature of 50-step
today MSD [21] achieves SOTA A of 47% with ResNet- 18 CIFAR-10.
41
42 This poses a question: can we approach the high robustness of multiple-attack AT such as 50-step MSD against the union of (e:0; &2, {1) perturbations while maintaining the low training time of fast single- attack AT frameworks such as FreeAdy [30] and Fas- tAdv [35]? In our quest to answer this question we find that noise augmentation using adequately shaped noise within standard single-attack AT frameworks employing 6x-- bounded perturbations significantly improves robust- ness against the union of (6 l2, (1) perturbations. The improvement appears to be consequence of a establisd byproduct of AT frameworks - the reduction in the curvature of the decision boundary of networks trained using single-attack AT [6, 23]. We confirm this connection by quantifying the impact of single-attack AT on the geometric orientations of different perturbations.
43
44
46
48
Fubkey Bareeaay
PCp Tocasya
TRsots Jaseeerate
49
50
a
10
51
Total
52
Adversarial accuracy (A against union of (Eoo, l2, 41) vs. measured wall-clock total training time on CIFAR- 10 with different AT frameworks on sin- gle NVIDIA TESLA P100 GPU. 6 (0.031, 0.5, 12) for (foo. l2, &1) perturbations, respectively. SNAP enhances robustness with small increase in training time. All frame- works except PAT employ ResNet-18.
Figure 1:
53
54
56
58
60
Based on this insight, we propose Shaped Noise Augmented Processing (SNAP)- hance robustness against the union of perturbution types by augmenting single-attack AT frameworks. SNAP prepends deep net with a shaped noise (SN) augmentation layer (see Fig. 4) whose dis- tribution parameter Σ is learned with that of the network (0) within any standard single-attack AT framework. SNAP improves the robustness of four SOTA AAT frameworks against the union of Coo, E2, ()) perturbations by 15%-to-20%% on CIFAR- 10 (red markers in Fig. /) with only a modest (~ 10%) increase in training time. This expands the capabilities of widely popular single-attack fo AT frameworks to providing robustness to the union of (L l2, (1) perturbations without sacrificing training efficiency. We validate SNAP's benefits via thorough comparisons with nine SOTA adver- sarial training and randomized smoothing frameworks across different operating regimes on both CIFAR-10 10 and ImageNet.
61
method en-
62
63
64
66
68
69
70
71
72
One tangible outcome of our work - we demonstrate for the first time ResNet-50 (ResNet-101) networks on ImageNet that achieve A A 32% (35%) against the union of (loo (c 2/255), fa(c
73
74
2.0), E (E- = 72.0)) perturbations. Our code and trained models will be shared publicly on GitHub.
Related Work
75
76
We categorize works on adversarial vulnerability of DNNs as follows:
Low-complexity adversarial training: The high computational needs of AT frameworks has spurred significant efforts in reducing their complexity [40, 30, 35, 43). FreeAdv [30] updates weights while accumulating multiple attack iterations. FastAdv [35] employs appropriate use of single-step attacks, 80 while Zheng et al. [43] leverage inter-epoch similarity between adversarial perturbations. However,
78
79
81 these fast AT methods seek robustness against a single perturbation type. e.g., foo norm-bounded perturbations. In contrast, SNAP expands the capabilities of these AT frameworks by enhancing robustness to the union of three perturbation types (fa,, f2, f1), while preserving their efficiency.
82
83
84 Robustness against union of perturbation models: The focus on the robustness against the union of multiple perturbation types is relatively new. Kang et al. [16] studied transferability between different perturbation types, while Jordan et al. [15] considered combination attacks with low perceptual distortion. Stutz et al. [31] proposed a modification in AT to defect images with different models 88 of perturbations via confidence thresholding, but they don't 't attempt to classify perturbed images correctly. For accurate classification in the presence of different perturbation models, Tramèr &
85
86
87

---

90
Boneh [32] studied empirical and theoretical trade-offs involved in including multiple perturbation types simultaneously during training. Maini al. [21] further built upon this work to propose the multi steepest descent (MSD) AT framework which chooses one among the three perturbation models (boa; l2, f1) in each attack iteration during training, achieving SOTA adversarial accuracy on CIFAR- 10 against the union of the (f f2, (1) perturbation models, albeit at high (10x) training time. In contrast, SNAP provides high robustness against the union of (loo; E2; 1) perturbation models using established single-attack so AT frameworks. This enables to showcase the benefits of our approach on large-scale datasets such as ImageNet.
91
92
93
94
95
96
97
98
Recently, Laidlaw et al. [18] developed novel AT framework (PAT) with low perceptual distortion attacks to demonstrate impressive generalization to unseen attacks. In contrast, we focus on extending the capabilities of widely popular AT frameworks to providing robustness against the union of (o E2 (1) perturbations, while preserving their training efficiency.
99
100
101
102
Noise augmentation: Multiple recent works have investigated the role of randomization in enhancing adversarial robustness [12, 24, 8, 25] with theoretical guarantees. Another prominent line of work in this category is randomized smoothing [5, 29, 19, 38], where random noise is used as a tool to compute certification bounds. Rusak al. [28] also explored the role of noise augmentation for improving the robustness against common-corruption [13]. In contrast, in SNAP, noise augmentation -T frameworks to efficiently achieve high robustness against the union of multiple norm- nom-bounded perturbations. As is the characteristic of AT works, our results are primarily empirical in nature. Hence, we follow recent guidelines [33, 21] to evaluate the accuracy against the strongest possible adversaries. We do explicitly compare for ATT+NNAP with randomized smoothing approaches in the Appendix.
103
104
105
106
107
used as a means enable widely popular <
108
109
110
111
112
3
Subspace Analysis of Adversarial Perturbations
In this section, we employ subspace methods to compre- hend the distinction between lx. lg and & perturbations. 115 For each input s; E TR& in dataset X, consider adversar-
115
High curvature in decision boundary (vanillo training)
Low curuature in decision boundary (adversorial training)
114
116 ial perturbations at,, B., and Y/ bounded within Loo, Ez. and E, norms, respectively.
117
118 We begin with hypothesis (see Fig. 2): The perturbations a, B, and y corresponding to input x have directions that
X
Y
119
120
differ significantly if the curvature of the decision bound- ary is high in the neighborhood of x. Conversely, if the curvature of the decision boundary is low, the perturba- tions a, B3 and y tend to point in similar directions.
Perturbation models
121
é norm bounded; (i: fg norm bounded; y: norm bounded Figure 2: Illustration of the role of deci- sion boundary curvature on the distinc- tion between different types of perturba- tions a. 3 and y of the given input æ.
122
123
124 Since prior works [6, 23] have found that single-attack AT reduces the curvature of the decision boundary, we test our hypothesis by studying the following two networks
125
126
127
on CIFAR-I0 data: a non-robust ResNet18 fg trained using vanilla training, and robust ResNet18 rreb trained using the TRADES [41] AT framework employing foo perturbations.
128
129
We compute perturbations a,, 3,, and 7, for each x; E X for both networks, i.e., k E {van, rob]. We compute the singular vector basis pd for the set of &g bounded perturbations A = { ....... 3 x }.
The normalized mean squared projections of the three types of perturbation vectors on the singular vector basis Pr of vanilla trained ResNet-18 (Pvany(Fi. 3(a) and TRADES trained ResNet-18 (Fig ((b) shows a clear contrast.
132
133
134
The perturbations of a vanilla trained network roll-off gradually to occupy a larger subspace as indicated in Fig. 3(a). Specifically, the projections of a and 7 occupy almost all 3000 directions in the basis pvan since their mean squared projections are within 10% of the maximum value Mmax- This shows that the dominant singular vectors of B3 are not well-aligned with a and 7 in a vanilla trained network. With TRADES AT (Fig. 3(b)), however, all three types of perturbations are squeezed w pmb Outside these 250 dimensions, the mean squared projections fall to < 10% of their maximum value.
136
138
into much smaller subspace spanning only the top 250 singular vectors in the perturbation basis
140
142 In summary, the results in Fig. validate the hypothesis that single-attack AT increases the average alignment of different perturbation types due to the reduction in the decision boundary curvature. In
3

---

Vonitee maining
TRADES A7
Pe
137
500
3000
13
yy
jndeej anngular victirs 97
Inder P7
[k)
Figure 3: Normalized mean squared projections of three perturbation types on the singular vector basis ps of perturbations of ResNet18 on CIFAR-I0 after: (a) vanilla training (k van), and (b) TRADES training (k rob). The singular vectors p comprising p = .p......PP) are ordered in descending order of their singular values.
144 Sec. 4, we exploit this behavior of single-attack &x AT to improve its robustness against the union of multiple perturbation models via SNAP.
145
Shaped Noise Augmented Processing (SNAP)
146
least Fiesly
147 We show that single-attack AT can be enhanced to address multiple perturbations by introducing noise appropriately wiggle the -bounded perturbations (Fig. 4(a)). However, to do so, the noise distribution needs be chosen and shaped appropriately to minimize its impact on natural accuracy and robustness to -bounded perturbations.
sapped noise samptes n
148
150
leeey
152
not perturtrron in AT--NAP
153
We experiment with both loc and f2 perturbations in single- attack AT frameworks and find AAT be suitable for our proposed shaped noise augmentation (see Sec. 5.2. 1 for details). Hence, this section, we describe SNAP for single- attack AT frameworks employing Loo perturbations.
154
Perturbation models
at norm bounded; Pt norm bounded; y: norm bounded
155
156
157
SN
Deep Net y (s0r)
y /0z (X)
158
4.1
SNAPnet
A deep net fo(x): : - f0, 1 parametrized by 0 maps the input x E RD to one-hot vector y E 10 over classes
160
Figure 4: SNAP: (a) intuition un- derlying SNAP (not an exact depic- tion), and (b) SNAPnet S con- structed from a given deep net fa(æ) by prepending a shaped noise (SN) augmentation layer which perturbs the primary input æ with noise n whose distribution parameter Σ is learned during AT along with the base network parameter 0.
162
construct SNAP-based deep net (SNAPnet) SaN (a)) by introducing an additive shaped noise (SN) layer (Fig, 4(b)), where the noise distribution parameter Σ is learned during training. Formally,
163
164
185
(I) where ng ~ C(O, IDDD) a zero-mean isotropic Laplace noise vector, Σ Diagio ...... . DD is distribution param- eter denoting its per-dimension standard deviation, LDDD
y = +SN (x) fo + = fx VEno),
186
168
denotes the D x D identity matrix, and V [11 [U1,..., UDD denotes basis in RD. We also studied Gaussian and Uniform distributed ny, but empirically find the Laplace distribution to yield better
170
results (Sec. 5.2.1). We use V I for all our experiments in the main text and study other options for V in the Appendix.
172
The final classification decision d is computed via
arg max
EE y
174 where [a]c denotes the e-th element of vector a. Note, the shaped noise perturbs the input with
175 noise source n = VLng (Fq. (1)). The distribution parameter Σ is learned in the presence of any standard AT method [20, 41, 30] used for learning deep net parameters 0 as described next.

---

Algorithm Training SNAPnett
training X; basis V ....... Vp]: total noise power Pras; minibatch size baseline training method BASE; noise variance update frequency Uy; Total number of epochs T Initialize: noise variances Fo Diag(T1,0,--. p.o)..
Input:
Output: robust network fø.2. noise variances Er Diaglof, ob.
1: for epoch / 1... T do
2: for mini-batch B {x F. do
0 +- BASE fo.., (fx, (12))a1))
BASEO Training
3: end for
if mod U then
> SNAP Distribution Update once every Uy epochs
5: for mini-batch B = f ..... x,} do
6:
M= Ne= + PGD (K)
SN 10
((2)
= x. 2, A i r.
7:
t Pi + Eimi (105. ni))
Vj (1. ......)) D} b Accumulate projections; See Eq. (3)
8:
end for
9:
jj,41 Pinise IR= vTh
TT
Vj E (I,..., D}
Normalize accumulated projections; See Eq. (3)
10: else
11: E141 + Ee
12: end if
13: end for
4.2
Training SNAPnet
Algorithm summarizes the procedure training SNAPnet (ar). each epoch, an arbitrary AT method BASE() (line 2) updates network parameters with input perturbed by noise n. Here BASE() can be any established AT framework [20, 41, 30, 35] employing fx perturbation.
178
179
181
The SNAP parameter
updated once every U, epochs via a SNAP distribution update (lines 4-10). In this update, the per-dimension noise variance of is updated proportional the root mean squared projection of the adversarial perturbations on the basis V given total noise constraint P Proises where Proise denotes the total noise power. Formally,
182
183
j==1 of
184
D Σ f=1
of x Bzex 0,)2)
s.t.
0
P
Phoises
185
where 77 is the & norm- norm-bounded PGD adversarial perturbation for the given input E X (line 6). Note that these l perturbations are employed only for noise shaping and are distinct from the foc perturbations employed by BASEO AT (line 2). Also, fo perturbations cannot be used here since their projections are constant vij when V = Inx D. whereas employing & perturbations leads to poor shaping due to high sparsity.
186
187
188
189
190
Thus, in SNAP, the average squared &a norm of the noise vector n held constant at Paoise while adapting the noise variances the individual dimensions so as align the noise vectors with the adversarial perturbations on average. Intuitively, the decision boundary is pushed aggressively in those directions.
191
192
193
194
4.3 Remarks
185
Note that the SNAP distribution update is distinct from BASE( AT. Hence, SNAP doesn't require any hyperparameter tuning in BASEO. For fairness to baselines we keep all hyperparameters identical
196
197
when introducing SNAP in all our experiments. However, SNAP introduces a new hyperparameter
(U) P Proise. which permits to trade adversarial robustness Adv for natural accuracy Ana. This trade-off is
199
explored in Sec. 5.2.2.
The computational overhead of SNAP is small (~ 10%) since the SNAP Distribution Update occurs once 10 epochs using just 20% of the training data to update noise standard deviations aj. We provide more details about the SNAP Distribution Update in the Appendix.
201
5

---

Method
A
0.03
All
aav
Method
Ams
0.03
0.5
1
(2)
PGD AT with ooc perturbations
High Complexity AT with E.o. perturbations
FGD
84.6
48.8
62.3
15.0
15.0
PGD
84.6
48.8
62.3
15.0
15.0
+NNAI[G]
80.7
45.7
66.9
34.6
31.9
+SNAP
83.0
44.8
68.6
40.1
35.6
+SNAP[U]
85.1
42.7
66.7
28.6
26.6
+SNAP[L]
83.0
44.8
68.6
40.1
35.6
TRADES
82.1
50.2
59.6
19.8
19.7
+SNAP
80.9
45.2
66.9
46.6
41.2
PGD AT with 1/ perturbations
FGD
89.3
28.8
67.3
31.8
25.1
Low Complexity AT with & perturbations
+SNAP[G]
83.0
35.0
65.8
39.9
30.2
FreeAdv
81.7
46.1
59
15.0
15.0
+SNAPIU]
86.4
32.3
66.7
30.2
25.0
+SNAP
83.5
39.7
66.2
34.3
29.6
+SNAP[L]
84,8
33.4
66.1
42.5
30,8
FastAdv
85.7
46.2
60.0
13.2
13.2
Table I: ResNet-18 CIFAR-10 results showing
+SNAP
84.2
40.4
67.9
36.6
30.8
the impact of SNAP augmentation of PGD [20] AT framework with foc (top) and &2 (bottom) per- turbations where [G], [U], and [I.], denote shaped
Table 2: R8sNet-18 CIFAR-10 results showing
the impact of SNAP augmentation of established -AT frameworks. The computational overhead of SNAP is limited to ~ 10%.
Gaussian, Uniform, and Laplace noise.
Experimental Results
203
5.1
Setup
205
Following experimental settings of prior work [41. 30, 21], we employ a ResNet- 18 network for 10 experiments and both ResNet-50 and 100 networks for ImageNet experiments. Accuracy on clean test data is referred to with Anst and accuracy on adversarially perturbed test data is (faz) refemed to via A adv Alle)
207
(f1) for loo. E2, and & norm bounded perturbations, respectively. Accuracy against the union of all three perturbations is denoted by (C)
(fa) and adv
209
210
For fair robustness comparison, our evaluation setup closely follows the setup of Maini et al. [21] for CIFAR- 10 data: (1) choose norm bounds e = (0.031, 0.5, 12.0) for (foe, Ez, {1) perturbations, respectively; (2) scale norm bounds for images to lie between [0. 1]: (3) choose the PGD attack configuration to be 100 iterations with 10 random restarts for all perturbation types'; and (4) estimate
211
212
213
(U) ady as the fraction of test data that is simultaneously resistant to all three perturbation models.
214
Following the guidelines of Tramer et al. [33], carefully design adaptive PGD attacks that target the full defense - SN layer - since SNA Pnet is end-to-end differentiable. Specifically, we backpropagate to primary input x: through the SN layer (see Fig. 4). Thus, the final shaped noise distribution is exposed to the adversary. We also account for the expectation E,[--] in Eq. (2) by explicitly averaging deep net logits over No 8) noise samples before computing the gradient, which eliminates any gradient obfuscation, and is known to be the strongest attack against noise augmented models [29]. In the Appendix we also show robustness stress tests and evaluate more attacks.
216
218
219
220
221
222
On CIFAR-10 data, compare with the following seven key SOTA AT frameworks: PGD [20], TRADES [41], FreeAdv [30], FastAdy [35], AVG [32], MSD [21], PAT [18]. We also compare with two randomized smoothing frameworks [5. 29] in the Appendix. Thanks to their GitHub code releases, we first successfully reproduce their results with a ResNet- 18 network in our environment. In the case of PAT [18), we evaluate and compare with their pretrained ResNet-50 model CIFAR-10. We compare all training times on a single NVIDIA P100 GPU. On ImageNet data, we primarily compare Fde [30]. We train ResNet-50 and its SNAPnet version with FreeAdv on Google Cloud server with four NVIDIA P100 GPUs to compare their accuracy and training times. We will
224
226
228
229
230
231
release our pretrained models and code on GitHub.
232
5.2
Ablation Studies
233
5.2.1
Impact of Noise Distribution and Model of BASEO AT Perturbations
In this subsection, we first study the impact of employing foo vs. l2 perturbations in BASE AT( (see line 2 in Alg. 1) on (U)) For each choice, we further experiment with three distributions for the 236 SN layer in Fig. 4(b) viz. Gaussian, Uniform, and Laplace. We don't consider é perturbations in
234
Following Maini et al. [21], we also run all attacks on subset of the first 1000 test examples with 10 random restarts for CIFAR- 10 data.

---

237
BASE ATQ since Maini et al. [21] showed that employing & single-attack AT achieves very low robustness all attacks. We choose PGD [20] AT as BASE AT() for this ablation study. For fair comparison across the noise distributions, we fix mosee = 160, enforcing all noise vectors to have the same average l norm. For each distribution, the noise shaped per the procedure summarized in Alg. 1.
238
240
As observed in Table 1, 2e -PG AT achieves much lower A than 22-PGD AT, observation also reported by Maini et al. [21]. With SNAP, however, we find that there is an interaction between the rerturbation model in PGD AT and the noise distribution in SNAP. For instance, SNAP[U] enhances
243
(C)
245
by 11% with PPG AT while not achieving any improvement with 22-PGD AT. In fact, adv
SNAP appears to be particularly suitable for AA, since it always improves A by 1 11%-to-2006%% irrespective of the noise distribution.
247
Finally, of the three noise distributions, we find the Laplace distribution to be distinctly superior, achieving the highest (! (35.6% and 30.8%) due significant improvement in A for both So and &g PGD AT, respectively. The superiority of the Laplace distribution in achieving high (f1)
249
250
A Ady stems from its heavier tail compared to the Gaussian and Uniform distributions with the same variance. Shaped Laplace noise generates the highest fraction of extreme values in a given noise sample. Hence, it is more effective in improving accuracy against & bounddd attacks, which are
252
253
254
the strongest when perturbing few pixels by large magnitude [21, 32]. We discuss this further in the Appendix. Henceforth, unless otherwise mentioned, we choose Laplace noise for SNAP and foo
255
perturbations for BASEO AT as the default setting since it achieves the highest A
258
5.2.2
Impact of Pooise
258
Next, we explore the impact of the SNAP hyperpa- rameter Pnoise, which constrains the average squared l2 norm of the noise vector n. enables to trade between adversarial and natural accuracy.
99.0
259
47.5
280
Adversshall Micursry (M)
A (fa)
262
Fig. 5 shows that, as Puoise increases, Affa) improves from 31 % to 47% accompanied by graceful (5%)
Anx
Remura
(fx)
254
drop in Ana and a small drop of 2% in A that stabilizes to = 45%. These results show: (1) SNAP preserves the impact of foo perturbations which
265
266
150
250
267
not surprising since PGD AT [20] explicitly includes those, and (2) Pnoise provides an explicit knob to control the Anal vs. Andv trade-off. Henceforth, we choose Paoise values that incur < 1.5% drop in Anal for all SNAP+AT experiments.
Penise
Figure 5: ResNet-18 CIFAR-10 results: ad- versarial accuracy (6)) ((x)) and natural accuracy Ana vs. total noise power Pooise for PGD+SNAP.
268
270
5.2.3
SNAP augmented SOTA AT Frameworks
273 Table 2 shows the effectiveness of SNAP for four SOTA AT frameworks: high complexity frame-
274 works, such as PGD [20], TRADES [41], and low complexity frameworks such as FreeAdv [30], FastAdy [35]. All are trained against foo attacks with 0.031. As expected, while they achieve
high A (foz) their A (fa) adv and (6) are lower.
276
adv
For high-complexity AT, SNAP enhances (E2) and (66) by ~ 6% and ~ 25%, respectively, while incurring only a drop of 5% in (fxx), Thus overall, SNAP improves robustness (A((T) by 20% against the union of the three perturbation models. Note that this robustness improvement comes at only a 1% drop in Ana (see Table 2). For low-complexity ATs, SNAP improvements in union robustness (A(4)) are also significant (~ 15%). Again, presence of SNAP improves (E2) and (E))
277
278
280
281
This time the drop A(rx) ~ 7%. We believe this is due to the fact that these frameworks employ weaker single-step attacks during training. Note that in the case of SNePAdv+SNAP, we actually observe ~ 2% increase Ana, trend we also observe in the ImageNet experiments described later.
282
284
7

---

Method
LR schedule
Epochs
Aua
Total time
(minutes)
Set A: Total Time
12 Hrs
AVG 50 Siep [32]
cyclic
50
84.8
40.4
4217
AVG Step [32]
eyclic
50
85.6
40.4
1834
AVG Step [32]
eyclic
50
86.7
38.9
956
PAT [18]
step
100
82.4
36.6
1364
MSD Step [21]
cyclic
50
81.7
47.0
1693
MSD Step [21]
eyclic
50
82.4
44.9
978
Set B: Hrs
Total Time Hrs
AVG 5 Step [32]
cyclic
50
87.8
33.7
489
MSD 20 Step [21]
cyclic
50
83.0
37.3
690
TRADES [41]
step
100
82.0
19.7
516
TRADES
step
100
80.9
41.2
566
Set C: Hrs
Total
Time < Hir
MSD Step [21]
cyclic
50
342
PGD [20]
step
100
84.6
354
PGD+SNAP
stop
403
Set D: Hrs
Total
Time Hrs
AVG 2 Step [32]
cyclic
50
88.4
22.0
232
MSD Step [21]
cyclic
50
84.0
12.6
185
PGD [20]
cyclic
50
82.8
15.7
177
TRADES [41]
cyclic
50
80.0
21.4
258
PGD+SNAP
cyclic
50
82.3
33.5
199
TRADES
cyclic
50
78.8
40.8
280
Set Total Time
Hrs
FreeAdy [30]
step
200
81.7
15.0
66
FastAdv [35]
eyclic
50
85.7
13.2
47
FreeAdv4SNAP
step 200
83.5
29.6
88
FastAdy+SNAP
cyclic
50
84,2
30.8
69
(U))
Table 3: CIFAR-10 results for comparing adversarial accuracy udv vs. training time (on single NVIDIA P100 GPU) for different AT frameworks and the improvements by introducing proposed SNAP technique. All frameworks except PAT [18] (which employs ResNet-50) employ ResNet-
Training
Ama (%)
A(
((5))
(77) adv
Total time (minutes)
c 2/255
= 2.0
=
72.0
ResNet-50
Are (30)
61.7
47.8
19.9
14.8
12.6
3590
PreeAdvSNNAP
66.8
46.1
37.8
37,4
32,4
3756
ResNet-101
FrecAdv (30]
65.4
51.8
22.8
18.8
16.1
5678
FreeAdv+SNAP
69.7
50.3
41.1
40.2
35.4
5904
Table ImageNet results: Iso-hyperparameter introduction of SNAP yields 20% improvement in adversarial accuracy (A())) with modest impact on training time for ResNet-50 and ResNet-101.
286
5.3
Robustness vs. Training Complexity
287
Next we quantify adversarial robustness vs. training time trade-offs. Table 3 shows that SNAP augmentation of single-attack AT frameworks achieves the highest A constrained to 12 hours (sets B, C, D. and E).
(t) (t) adv when training time is
289
For instance, TRADES+SNAP achieves a 4% higher (13) (= 41%) than MSD-20 with 2 hours lower training time (Set B in Table 3). Similarly, PGD+SNAP achieves 2% higher (U)) than MSD-10 while having a similar training time (Set C). Note that both PGD and TRADES here use 100 training epochs with standard step learning rate (LR) schedule, while MSD frameworks employ a cyclic learning rate schedule to achieve superconvergence in 50 epochs.
290
282
293
294
In Set following Maini al. [21]. employ cyclic learning rate schedule for PGD, TRADES, as well as for PGD+SNAP and TRADES+SNAP to achieve convergence in 50 epochs. Improvements
296
in (U) for PGD+SNAP and TRADES+SNAP similar to those in Sets B and C. Most notably,

---

PGD+SNAP with cyclic learning rate achieves 20% and 11.5% higher A than MSD-5 and AVG-2, respectively, while having a similar training time (~ 3 hours). Set E augments the data from
298
Table 2 with training times. FastAdv+SNAP and FreeAAv+SNAP achieve a high A ~ 30% while preserving the training efficiency of both FastAdv and FreeAdv. Notably, FastAdv+SNAP achieves
300
301
18% higher A adv than MSD-5, while being ~ 2.7x more efficient to train.
302
303
5.4
ImageNet Results
304
Thanks to SNAP's low computational overhead combined with FreeAdv's fast training time, we for the first time able to report adversarial accuracy of ResNet-50 and ResNet- 101 against the union of (loo, l2. { attacks on ImageNet.
305
308
307
We closely follow the evaluation setup of Shafahi et al. [30]. Specifically, we use 100 step PGD attack, one of the strongest adversaries considered by Shafahi et al. [30], and evaluate on the entire test set. We first reproduce FreeAdv [30] results using the same hyperparameters and then introduce SNAP. All hyperparameter details are specified in the Appendix.
308
309
311
In order to clearly demonstrate the contrast between robustness to different perturbation models, we evaluate with (2/255, 2.0, 72.0) for (foe, f2, (1) attacks, respectively. 2 As shown in Table 4, FreeAdv achieves a high A(fx) 47.8% with ResNet-50, but a lower A((e) = 20% and A(
312
313
15,, and consequently, low A (t) of 12.6% against the union of the perturbations. In contrast, SSeAAdv+SNAP improves A ((x)) and A())) by 17% and 22%, respectively, accompanied by a 5% adv
315
adv
improvement in Ana and small 2% loss in A(fx)) This results in an overall robustness improvement of 20% against the union of the perturbation models, setting a first benchmark for ResNet-50 on ImageNet. Upon increasing the network to ResNet-101, 101, both natural and adversarial accuracies improve by R 4% for FreeAdv, a trend also observed by Shafahi et al. [30]. SNAP further improves
316
317
318
319
FreeAdv's 's results for Anat and A(U)) by 4.3% and 19.3%.
320
Discussion
321
322
Given the wide popularity of AA, in this paper, we propose SNAP as an augmentation that generalizes the effectiveness of AAT to the union of (fo,, f2, fi) perturbations. SNAP's strength is its simplicity and efficiency. Consequently, this work sets first benchmark for ResNet-50 and ResNet- 101 networks which are resilient to the union of (foo l2, E1) perturbations on ImageNet. Note that bounded perturbations include large class of attacks, e.g., gradient-based [20, 27, 32, 21, 4, 22], decision-based [3] and black-box [1] attacks.
323
324
325
328
327
More work is needed to extend the proposed SNAP technique to attacks beyond norm- corn-bounded additive perturbations, e.g., functional [17, 36], rotation [7], texture [2], etc. We provide preliminary evaluations in this direction in the Appendix. It is important to note that SNAP is meant to be an efficient technique for improving AAT, and not a new defense. Indeed defending against a large variety of attacks simultaneously remains open problem, with encouraging results from recent efforts [21, 18).
329
331
332
333
Another limitation of our approach is that its benefits are demonstrated empirically. It is an inevitable consequence of a lack of any thcoretical guarantees for underlying AT frameworks. An interesting direction of future work is to explore whether any theoretical guarantees can be derived for anisotropic shaped noise distributions in SNAP by building upon the recent developments in randomized smooth- ing [29, 38). This could be potential avenue for bridging the gap between certification bounds and empirical adversarial accuracy.
335
337
339
Finally, we believe that any effort on improving adversarial robustness of deep nets has net positive societal impact. However, recent past in this field has shown that any improvements in defense techniques also lead to more effective threat models. While such a cat-and-mouse game is of great intellectual value in the academic setting, it does have an unintentional negative socictal consequence of equipping malicious outside actors with a broad set of tools. This further underscores the well- recognized need for provable defenses.
341
343
345
Note that Ea and é norms of PGD perturbation with Exo norm of 2/255 can be as large as ~ 3.0 and 1100 for images of size 224 x 224 x 3.

---

References
348
[1] Andriushchenko, M., Croce, F.. Flammarion, N., and Hein, M. Square attack: a query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer. 2020.
348
349
350
[2] Bhattad, A., Chong, M. J. Liang, K., Li, B., and Forsyth, D. A. Unrestricted adversarial examples via semantic manipulation. arXiv preprint arXiv:1904.00::: 2019.
351
352
[3] Brendel, W., Rauber. J., and Bethge, M. Decision- adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations, 2018.
[4]
Chen, P.-Y., Sharma, Y., Zhang. H., Yi, and Hsich, C.-J. Ead: clastic-net attacks deep neural networks via adversarial examples. In Thirty-second AAAI conference on artificial intelligence, 2018.
355
356
[5] Cohen, J., Rosenfeld, E., and Kolter, Z. Certified adversarial robustness via randomized smoothing, In International Conference on Machine Learning (ICML), 2019.
357
358
[6] Dezfooli, S. M. M., Fawzi, A., Fawzi, O., Frossard, P., and Soatto, S. Robustness of classificrs to universal pertur-bations. A geometric perspective. In International Conference on Learning Representations (ICLR), 2018.
359
360
[7] Engstrom, L.,, Tran, B., Tsipras, D., Schmidt, I.., and Madry, A. Exploring the landscape of spatial robustness. In International Conference on Machine Learning, pp. 1802-1811. PMLR, 2019.
362
383
Gilmer, J., Ford, N., Carlini, N., and Cubuk, E. Adversarial examples are a natural consequence of test error in noise. In International Conference on Machine Learning, pp. 2280-2289, 2019.
364
365
[9] Gowal, S., Qin, C., Uesato, J., Mann, T., and Kohli, P. Uncovering the limits of adversarial training against norn-bounded adversarial examples, arXiv preprint arXiv:0010.03593, 2020.
366
387
[10] Gui, S., Wang, H., Yu, C., Yang, H., Wang, Z., and Liu, J. Model compression with adversarial robustness:
368
A unified optimization framework. arXiv preprint 19X..0990200333 2019.
369
[11] Guo, M., Yang, Y., Xu, R., Liu, Z., Lin, D. When nas meets robustness: In search of robust architectures against adversarial attacks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition. pp. 631-640, 2020.
370
371
372
[12] He, Z., Rakin, A. S., and Fan, D. Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
373
374
375
[13] Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference Learning Representations, 2018
[14] Hu, T.-K.,, Chen, T., Wang, H., and Wang, Z. Triple wins: Boosting accuracy, robustness and efficiency together by enabling ipputaadative inference. arXiv preprint 2XXi:20022.1025 2020.
378
379
[15] Jordan, M., Manoj, N., Goel. S.. and Dimakis, A. G. Quantifying perceptual distortion adversarial examples. arXiv preprint arXiv:1902.0880 08265, 2019.
380
381
[16] Kang, D.. Sun, Y. Brown, T., Hendrycks, D., and Steinhardt, J. Transfer of adversarial robustness between perturbation types. arXiv preprint arXiv: 0001199050004,. 2019.
382
383
[17] Laidlaw, C. and Feizi, S. Functional adversarial attacks. Advances in Neural Information Processing Systems, 2019.
[18] Laidlaw, C., Singla, S., and Feizi, S. Perceptual adversarial robustness: Defense against unseen threat models. International Conference on Learning Representations (ICLR), 2018
386
387
[19] Li. B., Chen, C., Wang, W., and Duke, L. C. Certified adversarial robustness with addition gaussian noise. Neural Information Processing Systems (NeurIPS),201 2019.
388
389
[20] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations (ICLR), 2018.
380
381
[21] Maini, P., Wong, E., and Kolter. J. Z. Adversarial robustness against the union of multiple perturbation models. In International Conference Machine Learning (ICML), 2020.
10

---

393
[22]
Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2016.
395
396
[23]
S.-M., Fawzi, A., Uesato, J., and Frossard, P. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
387
398
[24] Pinot, R., Meunier, L.. Araujo, A., Kashima, H., Yger, F., Gouy-Pailler, C., and Atif. J. Theoretical evidence for adversarial robustness through randomization: the case of the exponential family. Advances in Neural Information Processing Systems, 2019.
400
401
402
[25] Pinot, R., Ettedgui, R., Rizk, G., Chevaloyre, Y., and Atif, J. Randomization matters. how to defend against strong adversarial attacks. In International Conference on Machine Learning (ICML), 2020.
403
[26]
Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg, F., Wiles, O., and Mann, T. Fixing data augmentation improve adversarial robustness, arXiv preprint arXiv:2103.01944 2021.
405
406
[27] Rony, J., Hafemann, L. G. Oliveira, L. S., Ayed, I. B., Sabourin, R., and Granger, E. Decoupling direction and norm for efficient gradient- gradient-based 12 adversarial attacks and defenses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4322-4330, 2019.
407
[28] Rusak, E., Schout, L.,, Zimmermann, R. S., Bitterwolf, J., Bringmann, O., Bethge, M., and Brendel, W. A simple way to make neural networks robust against diverse image corruptions. In European Conference on Computer Vision, pp. 53-69. Springer, 2020.
410
411
412
[29] Salman, H., Li, J., Razenshteyn, I, Zhang, P., Zhang, H., Bubeck, S., and Yang, G. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems, pp. 11289-1 1300, 2019.
415
[30] Shafahi, A., Najibi, M., Ghiasi, A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T. Adversarial training for free! Advances in Neural Information Processing Systems (NeurIPS), 2019.
416
417
[31] Stutz, D., Hein, M., and Schicle, B. Confidence-calibraeed adversarial training: Generalizing to unseen attacks. In International Conference on Machine Learning, pp. 9155-9166. PMLR, 2020.
[32] Tramèr, F. and Boneh, D. Adversarial training and robustness for multiple perturbations. In Advances in Neural Information Processing Systems, pp. 5858-5868, 2019.
420
421
[33] Tramer, F, Carlini, N., Brendel, W., and Madry, A. On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.088:
422
2020.
[34] Vivek, B. and Babu, R. V. Single-step adversarial training with dropout scheduling. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 947-956. IEEE, 2020.
425
[35] Wong, E.,, Rice, L., and Kolter, J. Z. Fast better than free: Revisiting adversarial training. International Conference on Machine Learning (ICLR), 2020.
426
427
(36] Xiao, C., Zhu, J.-Y., Li, B., Hc, W., Liu, M., and Song, D. Spatially transformed adversarial examples. In International Conference on Learning Representations, 2018
[37] Xie, C. and Yuille, A. Intriguing properties of adversarial training at scale. In International Conference on Learning Reprexentations, 2020.
430
431
[38] Yang, G., Duan, T., Hu, E., Salman, H., Razenshteyn, I., and Li, J. Randomized smoothing of all shapes and sizes. International Conference on Machine Learning (ICML), 2020.
432
433
[39] Yang, Y-Y.,, Rashtchian, C., Zhang, H. Salakhutdinov, R., and Chaudhuri. K. A closer look at accuracy vs. robustness. Advances in Neural Information Processing Systems, 33, 2020.
434
435
[40] Zhang, D... Zhang, T., Lu, Y., Zhu, Z., and Dong, B. You only propagate once: Accelerating adversarial training via maximal principle. arXiv preprint (rXiv:190500087 2019.
436
437
[41] Zhang, H., Yu, Y., Jiao, J., Xing, E., Bl Ghaoui, L., and Jordan, M. Theorctically principled tradc-off between robustness and accuracy. In International Conference Machine Learning (ICML), 2019.
[42] Zhang. J., Xu, X., Han, B., Niu, G., Cui, L., Sugiyama, M., and Kankanhalli, M. Attacks which do not
440
kill training make adversarial learning stronger. In International Conference on Machine Learning, pp. 11278-1122 PMLR, 2020.
441
442
[43] Zheng, H., Zhang, Z., Gu, J. Lec, H., and Prakash, A. Efficient adversarial training with transfcrable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition, pp. 1181-1190, 2020.
11

---

Checklist
445
448
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
448
contributions and scope? [Yes]
449
(b) Did you descrihe the limitations of your work? [Yes] See Section 6.
450
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
451
452
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
454
2. If you are including theoretical results...
455
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
456
(b) Did you include complete proofs of all theoretical results? [N/A]
457
3. If you ran experiments..
458
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] An URL to our code as well as the pretrained models is provided in the Appendix.
460
461
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All training hyperparameters are mentioned in the Appendix. Our technique does introduce new hyperparameter, whose impact is discussed in Sec. 5.2.2.
462
463
464
465
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We do run a subset of experiments multiple times to obtain error bars (see Appendix). In doing so we confirm that our technique is effective across random initializations. However, some of the training runs in our work are too expensive to run multiple times.
466
487
469
470
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Yes, we explicitly mention the type of GPUs used and the training times in Section 5.
472
473
4. If you are using existing assets (e.g., code, data, models) or curating/releusing new assets...
474
(a) If your work uses existing assets, did you cite the creators? [Yes] We appropriately cite the relevant papers while using their code to reproduce/extend their results.
(b) Did you mention the license of the assets? [Yes] Our own codes & models, as well as, all the other codes that we use are available freely in public domain. We do mention so explicitly in Section 5.1
477
479
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We do share our own code and pretrained model as a part of the supplemental material
480
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
482
483
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
484
5. If you used crowdsourcing conducted research with human subjects...
486
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
487
488
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
490
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
481
12


Reviews:
[{"review_id": "pLDsGZFBxrH", "reviewer": "Reviewer_ymM2", "summary": "This paper produces SNAP -- that augments a noise layer to networks during standard adversarial training. On 9 different adv training methods for l_inf perturbations, the authors find that adding a noise layer helps improve robustness against the union of l1, l2, l_inf perturbations. The key hypothesis that the work is built on is that since AT reduced the curvature of the decision boundary, various attacks belonging to different perturbation types get closely aligned. Using this property, the authors attempt to modify the l_inf model boundary by adding a specifically chosen noise.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### After rebuttal: Rating changed to 5\n\n### Merits:\n1. The method is a simple addition to any AT method to increases robustness \n2. The training time increases is only ~10% as opposed to past works that use 3x the time.\n3. Reasonable accuracy improvements over standard training methods.\n\n\n### Questions and Negative points\n1. *Removal of Noise Layer at test time*: The authors say that using N = 8 restarts ensures that there is no gradient masking because they use EOT. My review heavily depends on the following result -- What happens when one removes the noise layer of SNAP+AT model at test time. According to the hypothesis, SNAP should only help in training, and not inference. In that case, we should not need the noise layer at inference. If the results hold, then it empirically justifies no gradient masking, however, if they don't -- it suggests that gradients are being masked and the evaluation was not sufficient.\n2. *Hypothesis Graph*: Figure 3 appears problematic to me. Why should p_j^2 not be divided while graphing the expectation? It can mislead the reader into believing in higher alignment. I would like to view the graph after this change to make any inferences because it seems rather surprising that the adversarial attacks should align so well even in non-AT setting.\n3. *Do adversarial vectors really align?*: It has been shown in past work that adversaries in different perturbation regions do not subsume each other: see table 3 in https://arxiv.org/pdf/1905.11213.pdf. The hypothesis that they should align seems surprising.\n4. I find it difficult to understand that even if the Hypothesis in Section 3 holds, how does it motivate adding a random noise layer to \"wiggle\" the l_inf boundary? It seems that the two observations are disjoint. Can you help understand this link better?\n5. *Evaluation*: Since Maini et. al. was first published in 2019, the standards of adversarial training evaluation have significantly changed. I request the authors to use attacks from AutoAttack library as standardized by RobustBench for l2, linf. Additionally please use, Fast Adaptive Boundary Attack (FAB) for all of l1,l2,l_inf to perform a thorough evaluation. These attacks are faster to run, and often help identify situations where gradients are masked.\n6. *What about MNIST?*: It has been found in past work that hyperparameter tuning was especially tricky in case of the MNIST dataset. Does SNAP also successfully augment noise in case of AT methods on MNIST?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "jQDHbiDNr2", "reviewer": "Reviewer_tGS8", "summary": "This paper presents a new adversarial training procedure that enhances existing $L_\\infty$ AT methods by adding specific noise to the images, with the goal of improving robustness to multiple types of perturbation ($L_1, L_2, L_\\infty$).\n\nThe proposed method improves upon existing AT methods in two ways: \n1. For the underlying $L_\\infty$ procedure, the addition of noise adds considerable robustness to other norms, at the expense of a small reduction in robustness for $L_\\infty$. \n2. In comparison of other approaches targeting robustness against union of perturbation, the proposed method reduces the runtime by employing a single attack AT, with a relatively cheap noise augmentation. While competing approaches have to compute multiple attacks, the proposed method only computes one attack and only occasionally updates the noise distribution.\n\nThe improvement in performance allows the method to be applied to large datasets (ImageNet), which is prohibitive under competing approaches.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the presentation of the results is well done. The figures and tables are easy to read and well-elaborated with useful visual aids, which I found very helpful in some instances. The experimental setup is well described and the choice and discussion of experiments contribute to a cohesive narrative. While there is a lack of theoretical results and the originality of the method is somewhat reduced, this is compensated by a solid experimental evaluation.\n\nThe efficacy of the proposed method is clearly demonstrated with the empirical results.There are relevant comparison with SOTA methods that tackle both reducing complexity of AT and/or improving robustness for multiple attacks. In particular, Table 3 makes a compelling argument for the trade-off between the robustness and the training time. It does a good job of highlighting the benefits of SNAP when the training time is restricted to less than 12 hours. \n\nThe flexibility of choosing any base AT procedure and the infrequent updates of the noise distribution made it possible to obtain good robustness results on ImageNet, which is exceedingly costly with other methods.\n\n**Weaknesses:**\n\nIn essence, SNAP is a combination of randomized smoothing and adversarial training, i.e. adversarial training with noise-augmented image, where the noise distribution is occasionally updated throughout the training. However, it took me a while to understand this. The clarity of the paper could be improved by describing the similarities (and differences) to a method such as SmoothAdv earlier in the paper. Also, due to the strong parallels between AT+SNAP and SmoothAdv, I expected to see it included in the results of Tables 1 and 2. While there is some discussion in the supplementary material, having this comparison in the mentioned tables could contribute to set SNAP apart from the other randomized smoothing + AT approach.\n\nThe spectral subspace analysis, while interesting, was a bit hard to interpret. For example, why the dimensionality of the projection implies that the noise vectors are aligned? Couldn't they still be pointing in different directions even though they lie in a small subspace? Also, in Figure 3a, the dimension seems to be very different for each type of noise. This section brings an interesting discussion about how AT affects the geometry of the perturbations, but it is not easy to interpret and understand the implications of the analysis.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and societal impact in the final discussion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "U6krhl7HCiJ", "reviewer": "Reviewer_d8RZ", "summary": "This paper expands the capabilities of widely popular single-attack $\\ell_{\\infty}$ adversarial training (AT) frameworks to provide robustness to the union of $(\\ell_1, \\ell_2, \\ell_{\\infty})$ perturbations while preserving the training efficiency. The proposed method, named SNAP, could reduce the curvature of the decision boundary of networks. SNAP prepends a given deep net with a shaped noise augmentation layer whose distribution is learned along with network parameters using any standard single-attack AT. SNAP enhances adversarial accuracy of ResNet-18 on CIFAR-10 against the union of $(\\ell_1, \\ell+2, \\ell_{\\infty})$ perturbations for four single-attack $\\ell_{\\infty}$ AT frameworks, and establishes a benchmark for ResNet-50 and ResNet-101 on ImageNet.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. $V=[v_1,\\dots,v_D]$ denotes a group of basis in $\\mathbb{R}^D$. Is V the orthonormal basis? \n\n2. The authors claim that \"the average squared `$\\ell_2$ norm of the noise vector $n$ is held constant at $P_noise$ while adapting the noise variances in the individual dimensions to align the noise vectors with the adversarial perturbations on average.\" Why can such an alignment help to defend the union attack?\n\n3. The proposed method sacrifices the performance of the clean data to improve robustness against special attacks. However, Eq. (1) can be treated as mitigating through randomization. Unfortunately, this mechanism has been broken by [1]. Therefore, the vulnerability of SNAP should be fully considered under various kinds of adversarial methods. \n\n\n[1] Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "GFmG2G_4ptS", "reviewer": "Reviewer_dv2i", "summary": "This paper aims to improve the robustness of networks trained using ell_infty norm-based adversarial training to the union of perturbations like ell_1, ell_2, and ell_infty. And the basic idea is to add a shaped augmentation layer that can inject noise during adversarial training.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper looks like a combination of adversarial training and randomized smoothing to me so I tends not to think it is novel enough.\n\nQuality: The main claim of this paper is that by injecting noise during AT, the model can be robust to a union of perturbations without costing much extra time. However, from the results, the improvement looks mostly come from the ell_1 perturbation. It then makes sense why the authors found that Laplacian noise is found to be the best choice because ell_1 certified robsutness is achieved by randomized smoothing with Laplacian noise.\n\nClarity: The writing is clear.\n\nSignificance: Given the improvement is mostly from ell_1 robustness, the reviewer kind of doubts whether it is meaningful to consider the problem in the paper, i.e., being robust to the union of perturbations efficiently.\n\n---\nUpdate after rebuttal:\nThe results authors pointed in the paper together with the new results do address my concern on the source of improvement. Thus I would like to increase my score by 1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is indeed a borderline paper. Based on the majority of opinions from reviewers and the suggested accept rate of program committee, a current rejection decision is made. 
Decision: Reject