Composite Adversarial Training for Multiple Adversarial Perturbations and Beyond
Authors: Xinyang Zhang
Zheng Zhang
Ting Wang
Venue:  (2021)
Keywords: adversarial examples
deep learning
robustness

Abstract: One intriguing property of deep neural networks (DNNs) is their vulnerability to adversarial perturbations. Despite the plethora of work on defending against individual perturbation models, improving DNN robustness against the combinations of multiple perturbations is still fairly under-studied. In this paper, we propose \underline{c}omposite \underline{a}dversarial \underline{t}raining (CAT), a novel training method that flexibly integrates and optimizes multiple adversarial losses, leading to significant robustness improvement with respect to individual perturbations as well as their ``compositions''. Through empirical evaluation on benchmark datasets and models, we show that CAT outperforms existing adversarial training methods by large margins in defending against the compositions of pixel perturbations and spatial transformations, two major classes of adversarial perturbation models, while incurring limited impact on clean inputs.

Under review as a conference paper at ICLR 2021
COMPOSITE ADVERSARIAL TRAINING FOR MULTIPLE ADVERSARIAL PERTURBATIONS AND BEYOND
Anonymous authors
Paper under double-blind review
ABSTRACT
One intriguing property of deep neural networks (DNNs) is their vulnerability to adversarial perturbations. Despite the plethora of work on defending against indi- vidual perturbation models, improving DNN robustness against the combinations of multiple perturbations is still fairly under-studied. composite adversarial training (CAT), a novel training method that flexibly inte- grates and optimizes multiple adversarial losses, leading to significant robustness improvement with respect to individual perturbations as well as their "compo- sitions". Through empirical evaluation on benchmark datasets and models, we show that CAT outperforms existing adversarial training methods by large margins in defending against the compositions of pixel perturbations and spatial transfor- mations, two major classes of adversarial perturbation models, while incurring limited impact on clean inputs.
this paper, we propose
INTRODUCTION
Despite their state-of-the-art performance in tasks ranging from computer vision (Szegedy et al,, 2016) to natural language processing (Seo et 201, deep neural networks (DNNs) are inherently susceptible to adversarial examples (Szegedy et al. 2014). which are maliciously crafted samples deceive target DNNs. A flurry of adversarial attacks have been proposed, which craft adversarial examples via either pixel perturbation (Goodfellow et al. 2015b Moosavi-Dezfool et al., 2016
Carlini & Wagner (2)77a) or spatial transformation (Engstrom al. 2017] et al.) [2018] Alaifari et al. 2019). To defend against such attacks, a line of work attempts to improve DNN robustness
by developing new training and inference strategies (Kurakin et al. 2017 Guo et al. 2018 Liao
et al. 2018 Tramer et al. 2018). Yet, the existing defenses are often circumvented or penetrated by adaptive attacks (Athalye et al. 2018), while adversarial training (Madry et al. 2018 Shafahi et al. 2019) proves to be one state-of-the the-art defense that still stands against adaptive attacks.
While most adversarial training methods are primarily designed for individual attacks which are either fixed (Madry et al. 2018) selected from pre-defined pool (Tramèr & Boneh 2019 Maini ct al. 2020). in realistic settings, the adversary is not constrained to individual perturbation models but free to "compose" multiple perturbation models construct more powerful attacks. Despite their robustness against individual attacks, the DNNs trained using existing methods often fail to defend against such composite attacks (details in S Moreover, the existing adversarial training methods focus on pixel perturbation-base attacks (c.g.. bounded by fy-norm balls), while the research on training DNNs robust against spatial transformation-basee attacks is still limited.
To bridge this striking gap, in this paper, we present CAT, a novel adversarial training method able to flexibly integrate and optimize multiple adversarial robustness losses, which leads to DNNs robust with respect to multiple individual perturbation models as well as their "compositions". Specifically, CAT assumes an attack model that composes multiple perturbations and, while bounded by the over- all perturbation budget, optimally allocates the budget to each iteration. To solve the computational challenges of this formulation, we extend the recent advances on fast projection to E,,1 mixed-norm ball (Liu & Ye, 2010 Sra 2012 Béjar et al. 2019) to our setting and significantly improve the optimization efficiency. We validate the efficacy of CAT on benchmark datasets and models. For instance, MNIST, CAT outperforms alternative adversarial training methods (Tramèr & Boneh, 2019) by over 44% in terms of adversarial accuracy against attacks that combine pixel perturbation and spatial transformation (details in $4), with comparable clean accuracy and training efficiency.

---

Under review as a conference paper at ICLR 2021
Our contributions be summarized as follows. First, demonstrate that class of adver- sarial attacks, which "compose" multiple perturbations, render most existing adversarial training methods ineffective; then, propose CAT, the first adversarial training method designed for mul- tiple perturbation models as well as their compositions; further, we validate the efficacy of CAT by comparing it against alternative methods benchmark datasets and DNNs; finally, explore the optimization space of composite perturbations, leading to several promising research directions.
2
FUNDAMENTALS
2.1 ADVERSARIAL TRAINING
Adversarial training is class techniques to train robust DNNs by minimizing the worst-case loss with respect to a given adversarial perturbation model. Formally, let fo be a DNN parameterized by 8, the loss function, and Dirain {æ,, Vi 1 the training set. Then the adversarial training with respect to an lp adversary with perturbation magnitude is defined as:
0" = arg min Σ
max ((fo(1; + 81).
(1)
6EBp(e)
where (c) = {5 : | S } is the fy-norm ball of radius e. Here, the inner maximization problem essentially defines the target adversarial attack. For instance, instantiating it as the foc projected gradient descent (PGD) attack leads to the well-known PGD adversarial training.
Despite their effectiveness against considered perturbation models (e.g.. foo perturbation), the exist- ing adversarial training methods often fail to defend against perturbations that they are not designed for (Tramèr & Boneh 2019). Motivated by this, some recent work explores conducting adversarial training with respect to multiple perturbation models simultaneously.
AVG and MAX Tramer & Boneh (2019) propose two methods, AvG and MAX, aggregate multiple perturbations. Specifically, AvG formulates the robustness optimization as:
0- arg min ΣΣ
max ((fe(z; 6p). EBpl))
where = (1,2,00), Compared with Eq. 1 Eq. 2 aggregates multiple adversarial perturbations in the inner loop. Similarly, instead of averaging multiple perturbations, MAX selects the perturbation resulting in the largest loss:
0 arg min
max (CEB,()))EAA)
{(Solz; + 6), us)
If A contains only one adversarial perturbation, Eq. Eq and Eq I are all equivalent.
MSD While AVG and MAX achieve varying degrees of robustness the considered perturbations, it is practically difficult to minimize the worst-case loss with respect to the union of perturbations. To this end, Maini al. (2020) propose multiple steepest descent (MsD) which improves MAX (and AvG) along two aspects. First, selects the largest (or average) perturbation at each inner itera- tion; Second, applies the steepest descent instead of the projected gradient descent in generating adversarial inputs. Formally, MSD formulates the optimization at the (-th iteration as:
g(t+1) Projg,(e) (80) + (1((())))
for p E A
(4)
g((+1) = arg max (foo 5% 5((+1)), ),
(5)
g(D
where (.ojec() is the projection operator onto the convex set C, and (p((()) is the steepest descent direction for lp perturbation, (6) = arg max | 8 Vé(fo(x + 5), 8), where is the step size.
2.2 COMPOSITE ADVERSARIAL ATTACK
While the existing adversarial training methods seem effective against individual perturbation mod- els which are either fixed or selected from a pre-defined pool (i.e., the union of perturbations). in
2

---

Under review as a conference paper at ICLR 2021
truth label
sirplane
bird
deer
dog
Frog
horse
ship
truck
natural image
pteel
perturbed
image
predicted
label
frog
truck
deer
dog
horse
bird
dog
dog
airplane
Figure I: Samples produced by composite attacks on CIFARIO (ep = 0.015, cy = 0.175, untargeted).
realistic setting. the adversary able to combine multiple perturbation models to construct more destructive attacks, which we exemplify with a new class of composite adversarial attacks.
Intuitively, the attack constructs an adversarial example by applying sequence of perturbation models (A::) each A; bounded by an independent perturbation budget c, Here, we treat A; as an abstract operator A,(-,G) (e.g.. pixel perturbation or spatial transformation), which applies the corresponding perturbation (bounded by e;) over the output of its previous perturbation:
(1 x (1-1) 84-1, 5, (z ()) for ===:..... 1,...,m
(6)
We can further generalize the attack to a more flexible setting which the adversary, while bounded by the overall perturbation budget, is able to optimally allocate the budget to each optimization iteration. The details of this generalization are discussed Appendix B. Figure[Tillustrato samples generated by the composite adversarial attack which combines pixel perturbation (with budget cp) and one spatial transformation (with budget cy).
Given attacks {A,, with A, bounded by ct, in the composite attack, re-scale the perturbation budget by a factor of 1/m (i.e., f/ m for A.). to make the composition of (AA,) ) comparable with the union attack. Intuitively, with proper setting of { the composition of { s=1 is strictly
stronger than each individual attack as well as their union (detailed proofs in Appendix A). Figure
compares the robust accuracy of AVG, MAX, and MSD on the unions and compositions multiple
perturbations. Observe that while effective on the union attacks, the existing methods fail to defend
against the composite attacks, with robust accuracy drop as large as 40%.
Accuracy
8.4
MNIST
CIFARIO
0.
0.0
MAX
M5D
MSD
Union
Composition
Figure 2: Adversarial accuracy of AvG, MAX, and MSD w.r.t. unions and compositions of = (12, fack.
CAT: COMPOSITE ADVERSARIAL TRAINING
We now present CAT, a new adversarial training method to defend against multiple perturbations as well as their compositions.
3.1 FORMULATION
At high level, CAT adopts the composite adversarial attack as the inner loop of Eq which generates adversarial examples through sequence of perturbations:
x²" = max E(fol +
Σ 8.). v)
16
(7)
1
s.t.
| v for
1
3

---

Under review as a conference paper at ICLR 2021
As concrete instances, in the of composing pixel perturbations (i.e., (1. l2. for perturbations) with budget 61, C,, and 6go. respectively, the straightforward composite adversarial examples is produced by x²
ò1 62 d,c, where omit the clipping operator.
Next we consider composing pixel perturbation and spatial transformation (Xiao et al. 2018 Alai- fari et al. 2019) with budget Cp and of respectively. We first give brief introduction of spatial transtormation. Instead of directly perturbing the values of pixels, spatial transformation displaces their coordinates. Formally, the input x represented
of tuples æ {(us, Us J=) where
(us, Us) are the coordinates of x's i-th pixel and b; is its value. We set f = {(Au;, An))E (with
abuse of notations here), where (Au, Avi) are the displacement of z's i-th pixel. When we apply f to to generate the adversarial input x', (ug 14) = (uf + Au,, Do + Aer). Typi- cally, bi-linear interpolation applied to handle fractional pixel positions (Jaderberg et al. 2015):
D/ ΣyeN(uu...) (g(1 (n4- Ug () le: -yg),, where .N (u,,, 02) is the neighboring points of
(us, Uz). Following Alaifari et al. (2019), measure the perturbation magnitude s's -nrmm
|| max max;
Aut. max; Auip.
Attack A Attack
Attack A Atlack
Attack A Atlack
Attack A Attack
1-th Iteration
i-th Iteration
H-t Iteration
Max
idh Iteration
Avg
Figure 3: Comparison of different adversarial training frameworks.
3.2 DISCUSSION
Figure compares different adversarial training methods (AvG, MAX, MSD, and CAT). The design of CAT enjoys two major benefits. First, by definition, the composite adversarial attack naturally covers the strongest individual attack and the union of these attacks, as demonstrated in the two instantiations above. Second, CAT generalizes adversarial robustness from individual attacks which are either fixed or selected from a fixed pool their compositions.
As composite adversarial attacks are by nature stronger than individual attacks, setting the pertur- bation budget overly large in CAT may cause accuracy degradation with respect to clean inputs. We propose variant a-CAT to mitigate this issue. With 0 < < a hyper-parametes we re-scale the perturbation budget of each component attack A, to ac, during the adversarial training.
CAT-r
One issue with the above a-CAT formulation is the trade-off between attack strength of individual component attack and clean accuracy. With smaller a, we expect high clean accuracy. However, the component attack might not strong enough to cover the original perturbation size. For instance, 3. the trained model may have low robust accuracy for each individual attack and
if we take hence their union. On the contrary, a very large a (close to 1) causes significant drop in the clean accuracy, make the robust model is not useful. Our work-round solution for this use smaller to ensure clean accuracy, while we sample component attacks with replacement during adversarial training enhance the robustness against individual attacks. In fact, under this implementation. we may sample multiple stack of the same type attack, which corresponds to that attacker with larger perturbation size.
EMPIRICAL EVALUATION
We empirically evaluate the efficacy CAT in various settings. All the experiments are performed on MNIST and CIFAR10 dataset. Specifically, results convey two key messages. First, CAT trains DNNs robust against both composition perturbations and union perturbations the 4, space (6,, la and (,)), suggesting that composite adversarial robustness generalization of the adversar-
4

---

Under review as a conference paper at ICLR 2021
ial robustness with respect to the union of multiple perturbations. Second, CAT is able to train DNNs robust against both pixel perturbation (e.g., foo perturbation) and free-form spatial transformation.
Models and Hy per-pparaeeeers On MNIST, we the network architecture used by both Mainn ct al. (2220) and Madry ct al. (2018), which is a DNN consisting of 4 convolutional layers, followed by 2 fully connected layers. On CIFARIO, we use pre-activation version of ResNet32 (He et al., 2016). which is build up with 16 residual blocks, followed by global averaging pool layer, and fully connected layer.
Setting of ax- As discussed in 3.2, to minimize the performance degradation on clean inputs, we re-scale the allowed perturbation magnitude uniformly by a. To set a properly, we conduct grid search within {1.0, 0.9, 0.8, 0.7, 0.6, 3.5) and select the optimal a value by optimizing the trained model's robust accuracy (under the union threat model) and clean accuracy (with less than 10.0% drop from all baselines under the union threat model).
We implement all the algorithms with Py and run all the experiments on single Nvidia RTX 6000. The detailed setting of models and (hyper- (hyper-)parameee )parameters is summarized in Appendix B.
4.1 ROBUSTNESS IN lp SPACE
In the first part, we evaluate CAT and baselines on the regular pixel perturbation- attacks.
Baselines We compare CAT with MAX- worst-case perturbation, AvG- average-case perturba- tion (Tamer & Boneh, 2019), and MSD (Maini et al. 2020). Besides, we also include robust DNNs trained with PGD attacks with respect to individual perturbations.
Component Attacks for CAT- We consider 3 commonly used lp perturbations, namely foo. lz, and & in CAT. For fo attack, we use foo PGD attack (Madry et al.] 2018). For lz attack, we implement an &g PGD adversary. For & attack, we use the enhanced E attack proposed in Maini et al. (2020).
Attacks Used for Evaluation - To evaluate the robustness of DNNs trained by CAT and baselines, we consider collection of representative adversarial attacks.
Individual perturbations For Eo attack, we consider both Eo PGD attack and Fast Gradient Sign attack (Goodfellow et al.) [015a);; for & attack, we consider Eg PGD, DeepFool (Moosavi-Dezfooli cet al., 2016), C& attack (Carlini & Wagner 2017b), and Salt&Pepper attack (Rauber et al. 2017); for E attack, we use E PGD attack.
Combined perturbutions Besides, we consider both unions and compositions of multiple perturba- tions. Following Maini et al. (2020), in the union threat model, the adversary applies all the attacks on the given input and is considered successful if one of the attacks succeeds. In the composite threat model, we consider
of composite attacks with A -PGD, A f--PGD, and As = f-PGD. For the composite attacks, we measure the robust accuracy under different settings of re-scale factor a, which defined similarly in CAT-a.
Results- - Table and Table 2 summarize the results with respect to pixel perturbation- -based attacks on CIFAR1O and MNIST. The perturbation budget for each type of f, attack is shown in the tables. To measure models' performance on clean inputs, we use their accuracy on the test set. To evaluate the robustness of each model, we measure the robust accuracy of all the models on random sample of 1,000 test inputs. The robust accuracy is defined as the fraction of test inputs that are misclassified initially or are predicted to wrong classes after an attack. In reporting robust accuracy, we aggregate results of all the attacks from each target norm. In other word, an input is correctly predicted after attacks for f, norm if and only if all the attacks of f, norms from above fail for the input. Similarly, for the union setting, all the attacks except combined perturbations are considered.
On CIFAR1O, with a slight degradation of clean accuracy, CAT achieves the same robustness aceu- racy as MSD under the union of three lp perturbations and outperforms all other baselines. Further- more, CAT is much more robust against composite adversarial attacks when a = 0.5, outperforming MsD and other baselines by over 10%.
On MNIST, PGD-Ex (Px) achieves the best performance across all the settings, which however is attributed to the holding gradient masking effects (Trmer & Boneh 2019). In Appendix C, we show the robust accuracy of models in Table D on two decision-based attacks, and we confirm the higher attack success rate with black-box models of Poo. Thus, exclude Pao in the following
5

---

Under review as a conference paper at ICLR 2021
Pbo
P2
P
MAX
AVG
MSD
CAT
CAT-F
clean accuracy
83.3%
90.2%
73.3%
81.0%
84.6%
81.1%
72.6%
81.6%
Loo attacks (e = 0.03)
49.7%
21.4%
0.0%
45.9%
42.8%
47.5%
46.9%
43.6%
82 attacks (c 0.5)
59.0%
65.5%
0.0%
62.7%
67.6%
65.8%
60.3%
66.7%
E attacks 12)
16.6%
25.8%
10.2%
39.3%
57.3%
54.8%
59.6%
63.1%
Union
16.6%
18.1%
0.0%
34.5%
42.1%
46.7%
46.7%
43,5%
Composite (0.5)
11.2%
11.6%
0.3%
31.0%
33.5%
37.7%
47.0%
43.4%
Table 1. Performances CAT and baselines on CIFARIO with e, perturbations (p
1, 2, oc). Rows represent
attacks, and columns denote robust trained models. Po... P2, and P1 are models adversarially trained with PGD attacks (with corresponding norms).
P's
F2
M4
MAX
AVG
MSD
CAT
CAT-r
clean accuracy
99.1% %
99.2%
99.3% %
98.6%
99.1%
98.3%
91.7%
98.7%
Loo attacks (e 0.3)
92.1%
1.5%
0.0%
57.5%
71.0%
68.1%
42.5%
67.1%
E2 attacks (c = 2)
63.0%
75.9%
45.8%
68.1%
71.0%
74.7%
67.5%
77.6%
( attacks (E = 10)
71.5%
69.3%
77.2%
57.1%
65.1%
70.9%
74.3%
75.2%
Union
59.6%
1.5%
0.0%
47.7%
55.8%
63.4%
42.4%
63.9%
Composite (0.5)
49.0%
3.3%
0.0%
7.8%
12.0%
18.5%
38.8%
49,4%
Table 2. Performances of CAT and baseline methods on MNIST with &p perturbations (p = 2, zo). Rows
represent attacks, and columns denote robust trained models. Poo, P2, and Pi are models adversarially trained with PGD attacks (with corresponding norms).
discussion. Across all the other methods, CAT only performs slightly worse than MAX, AVG, and MSD under the union threat model. Under the composite threat model (ax = 0.5), CAT outperforms baselines in terms of robust accuracy by margins over 20%.
The above results indicate that CAT assumes strong adversarial attack model during the adversarial training process and produces DNNs with robustness not only against individual perturbations (and their unions) but also against their compositions.
4.2 ROBUSTNESS AGAINST PIXEL PERTURBATIONS AND SPATIAL TRANSFORMATIONS
Next evaluate CAT and baselines against compositions of pixel perturbations and spatial trans- formations.
Component Attacks For pixel perturbation, we consider to PGD attack. For spatial transforma- tion, we consider a projected gradient descent approach. The key differences between this formu- lation and ADef (Alaifari et al. 2019) are that 1) PGD is much faster than the DeepFool procedure proposed in ADet and 2) it uses free-form flows instead of the smoothed flows.
Attacks Used for Evaluation We evaluate CAT and baselines against the above individual pertur- bations as well as their unions and compositions.
Baselines Since MSD only applicable to pixel perturbation-basse attacks, we consider AvG and MAX as the baselines. Plus, we use two DNNs that are adversarially trained using the above piexl perturbation and spatial transformation respectively, which we refer as Ppixel and Phow.
Results Table 3 and summarize the results. We have the following observations. First, CAT achieves similar (on MNIST) or even better (on CIFARIO) robust accuracy than the baselines under the union threat model. Second, CAT outperforms all the baselines under the composite threat model by large margins. For instance, on MNIST, the robust accuracy of all the baselines drops to close to O even with 0.5 (i.e., half of the specified perturbation budget); in contrast, CAT attains 46% robust accuracy.
P Ppixel
Piow
MAX
AVG
CAT
CAT-r
clean accuracy
99.1%% %
98.8%
98.0%
98.2%
95.6%
96.7%
pixel attack 0.3)
92.1 %
0.0%
77.3%
85.2%
91.8%
88.9%
flow attack (E = 0.75)
3.0%
52.4%
44.8%
43.3%
41.3%
53.5%
Union
3.0%
0.0%
42.5%
42.4%
41.2%
53.5%
Composite (0.5)
12.5%
0.0%
1.0%
2.0%
46.0%
72.4%
Table 3. Performance of CAT and baselines on MNIST with respect to pixel and spatial perturbations. Rows represent attacks, and columns denote robust trained models.

---

Under review as a conference paper at ICLR 2021
P pixeel
fhow
MAX
AVG
CAT
CAT-r
clean accuracy
83.3%
82.5%
77.2%
79.8%
71.3%
74.7%
pixel attack (e = 0.03)
50.0% %
0.0%
45.5%
45.3%
47.4%
39.8%
flow attack (c = 0.35)
21.3%
48.4% %
40.8%
40.6%
44.1%
39.0%
Union
18.7%
0.0%
35.4%
32.8%
38.6%
32.3%
Composite (0.5)
26.5%
0.2%
38.2%
36.7%
43.0%
38.6%
Table 4. Performance of CAT and baselines CIFARIO with respect pixel and spatial perturbations. Rows represent attacks, and columns denote robust trained models.
Rooudt Accurnny
Accurazy
0-3-
0.04
0.11
0.0
AVG
MAXX
Composite Typa
Compos 09
Composita (0:89
Composito (1:0)
Composht Type
Composita (0.5)
Composioo (1:00
Composike (1.01
(a) CIFARIO: by space
(b) CIFARIO: pixel and spatial perturbations
Figure 4: Robust accuracy of CAT and basclines on CIFARIO dataset under composite adversarial attacks with re- re-scaling factor a=0.5, 0.8, and 1.0.
4.3 ROBUSTNESS AGAINST COMPOSITE ATTACKS WITH VARYING
Thus far, we have assumed the defender and attacker use the same setting of re-scaling factor (a 0.5) in attacking the DNNs. Next, evaluate the impact of varying by the attacker on the robust accuracy. Figure 4 summarizes the results on CIFARIO under a = 0.5, 0.8, and 1.0 (which correspond to stronger attacks). Observe that as expected, both CAT and baselines expe- rience performance degradation under large a and yet, CAT still consistently outperforms all the baselines by large margins across all the settings.
5 EXPLORING THE SPACE OF COMPOSITE PERTURBATIONS
In this section, we empirically study the critical properties of composite adversarial attacks and CAT. One particular aspect the impact of the ordering of component attacks on the adversarial training. Besides, we consider an even stronger composite attack which, bounded by overall perturbation budget, is able to optimally allocate the budget to each round.
5.1 ORDERING COMPONENT ATTACKS
We evaluate the impact of the ordering of component attacks under the setting of ép perturbations only well compositions of pixel perturbations and spatial transformations.
this set of experiments, we consider the compositions of -PG and &g- PGD attacks. Table 5 and 6 summarize the performance of CAT under the compositions of foo and l perturbations. We have the following observations. First, flipping the two attacks in CAT has little impact both clean accuracy and robust accuracy of CAT. Second, observable from the last two columns of each table, the effectiveness of composite adversarial attacks seems also independent of the ordering of component attacks.
6p Perturbations
Clean Accuracy
toc
Union too Eg (0.5)
8 loo (0.5)
to = 0.3 = 2.0
99.1%
92.5%
77.7%
77.4%
84.2%
84.2%
f2 2.0 f~ 0.3
99.1%
92.8%
79.5%
79.1%
85.7%
84.7%
foc f.x = -0.15
2.0
99.4%
96.9%
58.2%
58.2%
86.0%
85.7%
l 2.0-
99.4%
96.8%
57.7%
57.7%
85.3%
85.3%
E.c 0.3 1.0
99.1%
92.4%
95.9%
92.1% 1%
93.3%
93.4%
& 1.0 & 0.3
99.0%
92.2%
95.8%
92.0%
93.4%
93.8%
Table 5. Impact of the ordering of component attacks CAT on MNIST under compositions of Eoc and E
perturbations (Ai is the ordering used in CAT and composite attacks).

---

Under review as a conference paper at ICLR 2021
Clean Accuracy
t0
L2
Union
L. L2 (0.5)
82 to (0.5)
<cr = (0.3-4,, 0.5
82.7%
48.8%
64.4%
48.8%
57.0%
57.0%
8 = 0. = 0.03
82.8%
50.3%
64.9%
50.3%
58.5%
58.6%
Lo 0.015
0.5
88.8%
69.1%
68.2%
67.1%
68.3%
68.4%
& 0.5- = 0.015
89.1 1%
69.8%
67.2%
66.7%
68.7%
68.7%
6 0.03 -2 122 0.25
83.1%
50.0%
74.5%
50.0%
62.7%
62.5%
f 0.25 foc 0.03
82.7%
51.0%
73.4%
51.0%
62.7%
62.7%
Table Impact the ordering component attacks CAT on CIFARIO under compositions of too and &
perturbations (Ai AA is the ordering used in CAT and composite attacks).
Clean Accuracy
Pixel
Flow
Union
Pso foo (0.5)
foo Poc (0.5)
Poe foo Jo0 Poo
95.6%
91.8%
41.3%
41.2%
46.0%
52.6%
MNIST
96.7%
92.8%
34.4%
34.4%
40.6%
42.7%
Poo foo
71.3%
47.4%
44.1%
38.3%
43.0%
42.3%
CIFARIO
foo Poo
70.3%
48.0%
43.2%
38.8%
42.8%
42.6%
Table 7. Impact of the ordering of component attacks (pixel and spatial perturbations) in CAT on MNIST and CIFAR1O (A1 AAg denotes the ordering of component attacks used in CAT and composite attacks, while the perturbation budget is the same as and
Pixel and Spatial Perturbations Similarly, we empirically evaluate the impact of the ordering of pixel and spatial perturbations. Here we follow the same setup as $14 (the same attacks and a for each dataset). The results are summarized in Table 7 We observe that CIFAR10, the ordering has fairly limited impact as in the fp case; however, we find that first applying pixel perturbation and then spatial transformation results in more robust model on MNIST. We consider the study of the root cause of this interesting phenomenon as our ongoing work.
5.2 ROBUSTNESS AGAINST FINE-GRAINED COMPOSITE ADVERSARIES
Finally, we consider an extension of the basic composite adversarial attack (each perturbation is applied only once) to multiple round setting. Under this setting, the adversary, while bounded by the overall perturbation budget, is able optimally allocate the budget to each iteration, leading to even stronger attacks.
Formulation and Solution We sketch the differences between CAT and K-round CAT here, with full details deferred to the appendix B.
Perturbation Accounting - For each component perturbation A,, instead of seeking a single 8, we divide it into K parts oi,k for k 1,..., K. We measure the overall perturbation cost as the sum of ( where p Ep norm. In other word, the new constraint is Σ ||6 S cq.
Optimization Solution - This new constraints essentially specifies Lip mixed-norm ball constraint. Therefore, we can still solve this new optimization problem with projected gradient descent. We extend the recent advances (Liu & Ye 2010 Béjar et al.) 2019) to solve this new problem for the cases p = and p = oo respectively (details in the appendix B).
0.46
0.44
Robust Accuracy
0.12
0:40
0.38
0.36
K Rounds
MNIST.
CIFARID
MN ST-sieel-spafia
CIFAR1O-picel-spallaa
Figure 5: Robust accuracy versus the number of rounds K on MNIST and CIFARIO under composite adver- sarial attacks.
Results Figure displays how K-round composite attack impacts the robust accuracy of CAT under the settings of E, perturbations only as well as pixel plus spatial perturbations. As increases, the robust accuracy decreases by 4% and 7% respectively MNIST and CIFAR10 10 under the setting
8

---

Under review as a conference paper at ICLR 2021
of pixel plus spatial perturbations, while the decrease is much less evident under the setting of lp perturbations only, indicating that CAT is fairly robust to K K-round composite attacks. Moreover, by incorporating K-round composite attacks in the adversarial training of CAT, we expect to see further robustness improvement.
CONCLUSION
While effective against individual perturbation models, existing adversarial defenses often fail to defend against combinations of multiple perturbations. In this paper, we first present a new class of composite attacks that combine multiple perturbations and penetrate the state-of-the-an defenses. We then propose composite adversarial training (CAT), a novel training method that improves DNNs robustness not only against individual perturbations but also against their compositions. Empirical evaluation on benchmark datasets and models shows promising performance.
REFERENCES
Rima Alaifari, Giovanni S. Alberti, and Tandri Gauksson. ADef: an iterative algorithm to construct adversarial deformations. In Proceedings of International Conference Learning Representations (ICLR), 2019.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give False Sense of Security: Circumventing Defenses to Adversarial Examples. In Proceedings IEEE Conference on Machine Learning (ICML), 2018.
Benjamín Béjar, Ivan Dokmanić, and René Vidal. The fastest ,, prox in the west, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 201
Nicholas Carlini and David A. Wagner. Towards Evaluating the Robustness of Neural Networks. In Proceedings of IEEE Symposium on Security and Privacy (S&P), 201
Gustavo. Chau, Brendt. Wohlberg, and Paul. Rodriguez. Efficient projection onto the &oo.1 mixed-norm ball using newton root search method. SIAM Journal on Imaging Sciences, 12(1):604-623, 2019.
L. Engstrom, B. Tran, D. Tsipras, I. Schmidt, and A. Madry. Rotation and Translation Suffice: Fooling CNNs with Simple Transformations. ArXiv e-prints, 2017.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Hameessing Adversarial Examples. In Proceedings of International Conference on Learning Representations (ICLR), 2015a.
Ian Goodfellow, Jonathon Shlens, Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Reprexentations, 2015b. TRRL:tttpp//axxxv.orrg///// wwwpal/wwwwrr./b.ooa/aoo...bssss 1412.
65
Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens van der Maaten. Countering Adversarial Images Using Input Transformations. In Proceedings of International Conference Learning Representations (ICLR), 2018.
C. Gustavo, B. Wohlberg, and P. Rodriguez. Fast projection onto the mixed norm ball using steffensen rool search. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). pp. 4694-4698, April 2018
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, pp. 2017-2023 Curran Associates, Inc., 2015. URL htttt:tt/ppppeeet httpppppprs.is....ppppppppppppee cc/
haper/58544spatee a trrssrrannnneeeeee wwweerrreeewttee.e pdt
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of Interna- tional Conference Learning Representations (ICLR), 2015.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial Machine Learning Scale. In Proceedings of International Conference on Learning Representations (ICLR), 2017.

---

Under review as a conference paper at ICLR 2021
Fangzhou Liao, Ming Liang. Yinpeng Dong. Tianyu Pang. Xiaolin Hu, and Jun Zhu. Defense against Adver- sarial Attacks Using High-Level Representation Guided Denoiser, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Jun Liu and Jieping Ye. Efficient 11/lq norm regularization. ArXiv e-prints, 000/1009.47666 2010.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant Adversarial Attacks. In Proceedings International Conference Learning Representations (ICLR), 2018.
Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial robustness against the union of multiple perturbation models. In Proceedings of IEEE Conference on Machine Learning (ICML), 2020.
Mohsen Moosavi Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method fool deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Moosavi-Dezfooli,
Ariadna Quattoni, Xavier Carreras, Michael Collins, and Trevor Damrlll, An efficient projection for L1 regularization. In Proceedings of the International Conference on Machine Learning, pp. 857-864. ACM, 2009.
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox:
python toolbox to benchmark robust- ness of machine learning models. In Reliable Machine Learning in the Wild Workshop. 34th International Conference on Machine Learning, 2017. URL http: /wwpivaaaiivv.vooio org/ abs/1707. 1707.04131
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In Proceedings of International Conference on Learning Representations (ICLR), 2019.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannanch Hajishirzi. Bidirectional attention flow for machine comprehension. In International Conference on Learning Representations, 2017. 2017.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor. and Tom Goldstein. Adversarial training for free! In Advances in Neural Information Processing Systems, pp. 3353-3364, 2019.
Suvrit Sra. Fast projections onto mixed-norm balls with applications. Data Mining and Knowledge Discovery, 25(2):358-377, 2012.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representutions, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna, Rethinking the incep- tion architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818 2826, 2016.
F. Tramer, A. Kurakin, N. Papernot, I. Goodfellow, D. Bonch, and McDaniel. Ensemble Adversarial Train- ing: Attacks Defenses, Proceedings of International Conference on Learning Representations (ICLR), 2018.
Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. In Proceedings of Advances Neural Information Processing Systems (NeurIPS), 2019.
C. Xiao, J.Y. Zhu, B. Li, W. He, M. Liu, and D. Song. Spatially Transformed Adversarial Examples. In Proceedings of International Conference on Learning Representations (ICLR), 2018.
A PROOFS
We assume set of component models (A.) each A, bounded by budget e;.
Lemma 1. The union of (A is stronger than each individual attack.
10

---

Under review as a conference paper at ICLR 2021
Proof. (Lemma 1) The adversarial loss of given imput (z. y) with respect to A, is defined as:
Cx( max (cB)(e)
((f(x + 5).9)
where (e) is the feasible set for A, with bound Meanwhile, the adversarial loss with respect to union of {Ar] is defined as:
C. max max {(((+ 2 (x)x) (EB,(e)
(9)
Given that A, is arbitrarily chosen, the union attack is stronger than each component attack.
Therefore, we only need to show that the composite attack is stronger than the union attack. We prove this by showing that the feasible set for the composite attack is larger than the union attack. To simplify the discussion, we consider two é attacks A1 and A with budget e and 6 respectively, and d is the data dimensionality.
Lemma 2. If d(Ei + ) / /2d 2(E1 ExO > e4/2" + de the composition of Ai and Aso is strictly stronger than their union.
61
Composition Figure 6: Comparison of union and composition for d = 2.
Union
Proof. (Lemma 2) First, it can be verified that with the constraint, 6o0 de thus the d- dimensional hypercubes represented by e and 6 intersect but none of them completely contains the other. figurelossows the case for d = 2.
The feasible set B., for the union attack is given by:
B, = Bi(E)) U (xx()
(10)
where Bp,) denotes the feasible set for Ap (p 1, 20). The volume of B,, Vol(B,) is given by:
Vol(B.) 2dd 2d+1 (e1 coo
(11)
In comparison, the volume of the feasible set B. for the composite attack (with budget c1/2 and /x for A₁ and Aco) given by:
Vol(B.) (c1 (s)) -d/d
(12)
Given the constraint, it is trivial to see Vol(B.) Vol(B.., indicating that the composite attack entails larger perturbation space than the union attack.
This result can be generalized the cases of other fp attacks and more than component attacks.
B EXPERIMENT SETTINGS
We present the detailed setting of CAT, baseline methods, well the attacks used in
11

---

Under review as a conference paper at ICLR 2021
Case
M
CIFARIO-
MNIIT-pixel and spatial
CIFARIO-pixcl and spatial
0.8
0.8
1.0
0.7
Table 8. The rescale factor a for CAT models presented in the paper.
B.
MODEL TRAINING
We reuse few pre-trained models from repository provided by previous work. For the rest of models, their training use the following setups for each dataset.
The models are optimized with an Adam optimizer (Kingma & Ba 2015). The learning rate linearly increases from 0.001 in the first epochs, and then linearly decreases to O in the last 9 epochs.
MNIST.
CIFARIO. The models are optimized by a SGD optimizer with momentum of 0.9. The learning rate linearly increases from 0 to 0.1 in the first 20 epochs, then linearly decrease to 0.005 in the next 20 epochs, and it linearly decay to 0 in the last 10 epochs. Besides, we regularize models with a & weight decay of x 10-4.
B.2 CAT
lp cases. We run the composite attack during CAT model training with step sizes of 0.1 c for each p E {1, 21 x). The number of iterations for attacks is 50.
pixel and spatial perturbations. We run the composite attack during CAT model training with step sizes of 0.1 6p and 0.1 ef for pixel and spatial perturbation. The number of iterations for attacks is 40.
We also present the re-scaling factor for the models presented in the main text in Table 8
B.3 ADVERSARIAL ATTACKS
In evaluation, we run all the attacks except C&W with 5 random restarts. Below we summarize all the attacks we used in the evaluation and their hyper-parameters.
Attacks from Foolbox. We use the following attacks from Foolbox 3.1. H foo PGD, & PGD, PGD, Fast Gradient Sign Method, DeepFool, C&W, Palp& We take their default settings from Foolbox in the evaluation.
Our Implementation of Ep PGD Attacks. Plus, we also implement PGD attacks for three lp norms ourselves, which achieves higher attack succces rate than the version from the Foolbox. For foo PGD, we run 200 iterations with a step size of 0.1 X e. For & PGD, we run 500 iterations with a step size of 0.05 c. Our implementation of PGD attack is based upon Section A. of (Maini et al. 2020). where we set the the range of number of pixels modify in each iteration [k k2 as: k
and hy 20. The number iterations
of this attack is 200, and the step size is set to 0.05 x e.
The PGD Spatial Perturbation Attack. We run this attack with 200 iterations and step size of 0.1 é--
Composite Adversarial Attacks. We run this attack with 200 iterations and step sizes of 0.1 x e for all the component attacks.
B.4 BASELINES
Perturbations. We use pre-trained robust models provided by MSD Maini et al. 2020 for all the baseline methods, including Px.. P2, PI. AVG, MAX, and MSD.
Pixel and Spatial Perturbations. The adversarial trained models with foo PGD is the same as in the Ep perturbations case. For Pfoow. the number of iterations for spatial attack is 50, and the step size set to 0.1 cg. The same rule applies to AVG and MAX on the both two attacks.
12

---

Under review as a conference paper at ICLR 2021
C K-ROUND COMPOSITE ADVERSARIAL ATTACK
We describe the detailed formulation for the multiple round composite adversarial attack and tech- nical tools to solve this new attack.
C.I. FORMULATION
We denote component attacks A1:..., Am and whose perturbation sizes are C,.... Cm spectively. Unlike the definition in 8/2 here we represent each attack A; as (2 6;), which denotes applying perturbation d; onto with the mechanism of A,, The K-round attack generalizes compos- ite adversarial attack in the following sense, the attack runs in K-round, At k-th round, it performs composite attack with perturbations ...1....: 5g. The constraint for adversary is the overall magnitude of perturbations he spent for each attack. Formally,
(......; 8m arg min ((KK
8,
Fg = x
s.t.
2x Am (... A (Ai (I 82,1). 8x,2) 8p.m) k = 1..... K
(13)
Σ Σ |6 S C i -.......
K
where 25 is the perturbed sample after the k-th round and 6, (ōt,1, _(,k) is the concatenation of the perturbation at each iteration. Observe that by specifying K, Eq instantiates a spectrum of attacks. As K approaches infinity, eqssentssy considers all finite combinations of ways of allocating the total budgets {- using the m perturbation mechanisms.
C.2 DERIVATION
Now we present an iterative projected gradient descent algorithm to find a solution to Eqn (33 We use the superscript (L) to denote the value of related variables at (-th iteration. We randomly initialize ((0) At the t-th iteration, the update rule is defined as:
(t) 2llx
g
(14)
a8!")
((++1) II (8, (lse, (6
og;"" (2)
(15)
where a: is the learning rate and (.) is the projection operator of a convex set S. The computation of Eqn (14 is straightforward. We focus our discussion on the projection operator in Eqn (15).
Without loss of generality, we omit the subscript i for simplicity. Let V be a matrix with its k-th row 6g (for k 1....,,KK K)) Then the summation Σk= ||& S can be rewritten Σk || i-th component attack is measured with lp norm for
Additional, We suppose the perturbation some p 21, )o), which hold for all the experiments in the paper. Thus, reach to the fp.1 mixed-norm of V, denoted as [V which is a special case of fp.g mixed-norm ball: [V = |// Hence, we cast Eqn (15) as the projection operator onto an lp, mixed-norm ball. For p 1, the calculation straightforward since it reduces to the & projection operator for the concatenated vectors. We work with p 2 and oo in the next.
m
Li=1
C.3 IMPLEMENTATION FOR
We leverage Algorithm in Sra 2012) to compute the 62,00 mixed-norm of an input æ. Interested readers could find more details (Sra 2012
C.4 IMPLEMENTATION FOR = 00
We leverage the method proposed in (Béjar al. 2019). which solves the proximal operator of & mixed- norm using an active set approach and attanss better efficiency than previous methods (Quat-)
toni al. 2009 Gusstvv et al. 2018 Chau et al. 2019). However, this is primitive procedure
13

---

Under review as a conference paper at ICLR 2021
in CAT, which is executed for hundreds of iterations, the basic implementation in (Béjar et al. 2019) is not scalable for our setting.
We now improve the scalability of (Béjar et al. 2019) for CAT, with their original algorithm sketched in Algorithm| [ The method is based on computing the proximal operator of the dual form of 2 mixed-norm ball, fac.1 mixed-norm. We define U abs(V). Given initial radius of & ball for its row u, denoted by r, it iteratively improves r by first projecting each row with & norr exceeding to the E, ball of radius (line 4 8) and then computing a larger based on the nonzero elements of the projected row (line 9). In particular, Algorithm uses sorting to solve the {n-nrmm projection problem, which is detailed in Algorithm We state our more efficient implementation based on two key observations here.
The projection radius (line 6) Algorithm | for related rows increases every iteration. Thus, for each related row v, we face set of queries with increasing radii r1 S .... S rr, where T the number of queries. Following the notations of Algorithm 2 let denote the sorted v in nonddecrasinng order, and we define h2 ΣØ= increasing:
Observation 1
kug. Note that monotonically
h h& uk+1 (k- + 1)ux+1 + kuy k(u uk+1) 2
Leveraging this observation, we optimize Algorithm 2 as follows. At line 2, K is the largest item among 1, ,......V NN such that hx; we only need scan hy pass to find the optimal K₂ for all a, due to that both h and r are increasing. At line 3, we only need to pre-process the partial sums once for each row before the main loop.
Observation - To update the radius r of the 4-norm ball Algorithm we need to access the statistics of line 7 ~ 9. A critical observation is that it is not necessary to explicitly compute the projected vector at every iteration to compute the statistics. Specifically, line 7 computes the number of non-zero elements J. for the projected i-th row; and line updates using the of the projected rows Σ == Fi,y and Jr. Combining the previous observation, if K2, is the optimal K (line 2 of Algorithm for the t-th radius r, and the i-th row, it holds that
T Ke,i and Σ
Ui.j
Σ
(Ia. Tr,)) re Ki,
JEJ.
(fm1)>0)}
where Ti, calculated based on line 3 of Algorithm for i-th row with radius Ty. Thus, we can avoid computing the intermediate l projections.
Algorithm 1: Proximal operator of mixed 61,00 norm: pro (-) Béjar et al. 2019
Input: matrix V
Output: X
U e abs (V);
+ initial radius computed via Lemma 2 and Lemma 3 in Béjar et al. 2019
do
M +- {ilr [u ||1} foreach i in M do
æ; - IT 11 (u:)) T. - ( pii > 0}
ΣIEM DoT ESES, N4.
Diem पि7ा
while or {J.).mm change;
10 for I....,m do pa +- max
Ejer, "6.--" A1Jell
0):
X sgn(V) O max (U Au1"" .0:
D ADDITIONAL RESULTS
D.1 DECISION-BASED ATTACKS ON MNIST
Table summarizes the results of the models in Table 2 on two decision-based attacks: & Pointwise attack and 4 Pointiise attack Schoot al. 2019). One may notice that the inferior performance of Po. model on these black-box attacks compared to its performance in Table 2 for white-box attacks. In summary, find strong gradient masking effects within this model.
14

---

Under review as a conference paper at ICLR 2021
Algorithm 2: Algorithm for projection
R
onto simplex Σn= Un 7, and
2 0 for
n 1,.., N.
Input: y E R
Output: æ
u f sort e in non non- increasing order: U1 2 ... 2 uN
K < m {k] [k[[_;=1
r//h < ux)
EΣ k= ,, N do for
K
2n + max 7, 0
Poc
P2
Pi
MAX
AVG
MSD
CAT
Pointwise (62)
68.9% 9%
97.3%
98. 98.4%
94.0%
92.9%
95.3%
87.5%
Pointwise (61)
22.5%
92.7%
93.9%
83.4%
73.0%
86.6%
82.0%
Table 9. Robust Accuracy to decision-basc based attacks for models trained with CAT and baseline methods on
MNIST dataset.
D.2
VISUALIZATION OF COMPOSITE ADVERSARIAL EXAMPLES ON MNIST
Figure dataset.
complements Figure D with adversarial examples against natural model for MNIST
truth label
natural image
0
2
3
/
5
6
7
8
9
Aylun!
djew
perlurbed image
3
S
?
8
4
predicted Tobel
2
9
3
Figure 7: Samples produced by composite attacks MNIST. (ep 0.09, ef 0.225, untargeted).
D.3 PRELIMINARY RESULTS FOR CIFAR 100 DATASETS
Table TO and Table 11 present performances of baseline models and CAT for CIFAR100 dataset. Similar to previous results, with slightly reduction in clean accuracy, CAT outperforms MAX and AVG as well as simple PGD adversarial trained models. The MSD is excluded here temporally due to it takes extreme long time to train. We will fill MSD's result when it is available.
D.4 COMPUTATIONAL EFFICIENCY OF CAT
describe the computational cost MAX, AVG, and CAT here. MSD is excluded here due to it requires way more time, as the official implementation. We consider filling its result with our implementation later. All the experiments are with CIFARIO dataset and of Ep settings. The batch size of all the experiments is 50, and the total number of epochs is 50. The result is in Table 12
15

---

Under review as a conference paper at ICLR 2021
Poo
Pg
Pı
MAX
AVG
CAT
clean accuracy
49.0%
%
64.5%
56.7%
62.8%
53.3%
for attacks 0.03)
25.9%
2.8%
0.0%
23.6%
17.8%
22.6%
f2 attacks (e = 0.5)
34.5%
25.6%
0.0%
36.8%
38.7%
38.6%
8 attacks (e 12)
9.6%
5.0%
0.0%
29.6%
36.7%
37.6% %
Union
9.2%
2.4%
0.0%
21.8%
17.7%
22.3%
Composite (0.5)
8,3%
1.6%
0.0%
19.3%
15.8%
22.7%
Table 10. Performances of CAT and baselines CIFAR100 with 6 perturbations (p
1, oo). Rows
represent attacks, and columns denote robust trained models. Poo, P2, Pi are models adversarially trained with PGD attacks (with corresponding norms).
Ppisel
PHow
MAX
AVG
CAT
clean accuracy
57.7%
61.4%
55.1%
57.0%
54.0%
pixel attack (e 0.03)
26.1%
5.0%
23,6%
22.0%
23.7%
flow attack 0.35)
23.2%
34.8%
29.1%
29.9%
32.0%
Union
19.5%
5.0%
22.3%
20.9%
22.9%
Composite (0.5)
24.1%
3.4%
25.2%
25.1%
27.2%
Table Performance CAT baselines on CIFAR 100 with respect to pixel and spatial perturbations. Rows represent attacks, and columns denote robust trained models. Phow and Ppixel are models trained with PGD attacks (with corresponding perturbation spaces).
Method
Time per epoch (minute)
MAX
63.2
AVG
63.5
MSD
33.3
CAT
24.7
Table 12. Computational Cost of CAT and baseline methods.
16


Reviews:
[{"review_id": "xWSQ9IJy5M1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper proposes adversarial training with a novel threat model. Specifically, the authors propose to compose multiple adversaries, such as the ones based on l_p norm and spatial transform, in a predefined order to create a strong adversary in the adversarial training. The paper empirically demonstrated that the composite adversary is effective against previous adversarial defense mechanisms. It also demonstrated that the proposed adversarial training can lead to the classifier robust against the composite attack as well as the individual or union of multiple adversaries.\n\nPros\n+ The composite adversary seems to be novel and effective in terms of both adversarial attack and adversarial training.\n+ The paper is generally well-written and easy to read.\n+ The experiment results convey comprehensive evaluation and analysis. I especially enjoyed that it covers various attack scenarios, such as the ones with unseen attacks and composite attack with a random order, etc.\n   \nConcerns & Suggestions\n- It is not clear why the composited thread model can be stronger than individual or union attacks as claimed by the authors. If there are some theoretical justifications/proofs, it would be interesting to see such discussions (e.g., the composited attack consistently leads to higher classification loss (inner maximization of adversarial training objective)).\n- Although I appreciate authors for their comprehensive experiments, the current results are based on fairly small and easy datasets and it would be still interesting to see the results on more complex datasets such as Cifar-100 or mini-ImageNet.\n- It is unclear how exactly the l_p attacks are implemented. In Section 4.1, the authors mentioned various methods for l_p attacks, such as PGD, FGSN, C&W, DeepFool, Salt&Pepper, etc., but it is unclear how they are actually used in the experiments, for instance in Table 1 and 2. \n- It would be clear if authors add constraints on total attack budget on Eq.(8)\n- Table 5 & 6: Please clarify that the rows are the adversarially-trained models and columns are threats. It is confusing since rows and columns are different from the previous tables.\n\n--- post rebuttal update ----\n\nThe authors successfully addressed my initial concerns regarding more analysis and experiments on a larger dataset. Therefore, I keep my rating weak accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "wA3w_pMvSTs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary ####\nThis paper tackles the problem of adversarial training for the image classification task. It proposed a novel adversarial training method called composite adversarial training (CAT) against combined attacks constructed by multiple perturbations. First, CAT is based on the composite adversarial attacks, in which the attackers explore different sources of perturbations. Second, CAT leverages the composite adversarial attacks as the inner loop for optimization during the training. The experimental evaluations have been focused on comparing the proposed CAT with existing robust training methods including adversarial training with PGD attacks, AVG, MAX (Tramer and Boneh, 2019), and MSD (Maini et al. 2020) on MNIST and CIFAR-10 classification benchmarks.\n\n#### Comments ####\nThis paper studies an important problem in adversarial machine learning. The paper is well-motivated with novel technical contributions (Section 3.1) supported by reasonably designed experiments. However, reviewer feels the submission in the current form is a borderline case mainly due to mixed or inconclusive experimental results.\n\nW1: The clean accuracy of CAT  (Table 1 - 4, first row, last column) is significantly worse than methods such as AVG & MAX and MSD, especially on CIFAR-10 where the accuracy drops 20+% (I assume the state-of-the-art model has 90+% accuracy for the 10-way classification on CIFAR-10). This seems to be a major weakness of the proposed method. Reviewer understands the tradeoff between clean accuracy and accuracy under attack, but not sure how much value it is given the proposed defense method sacrifices too much on the clean accuracy. What makes it worse, this is just the performance drop of 10-way classification on CIFAR dataset. Reviewer is worried if this gap is even more significant on CIFAR-100 or ImageNet (w/ 1000 classes). It would be good to have some ablation studies.\n\nW2: Besides the drop on clean accuracy, reviewer fails to see a clear winner between MSD and CAT (see the last two columns in Table 1 and Table 2). CAT seems to be more robust to composite attacks but not as robust as MSD on other attacks. Such comparisons are missing in Section 4.2 (pixel perturbation and spatial transformations). It would be good to comment on this.\n\nW3: It would be good to report the computational cost (e.g., number of iterations in optimization, training time) of the proposed composite training method and explain how it is compared to the existing methods.\n\nMinor1 (applied for all the tables): it would be good to mention each row is a different attack method and each column is a different defense (robust training) method. It is not crystal clear at the first glance. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "eR_h1NmfbV8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposed an interesting new form of adverserial attack (composite adversarial attack) as well as an algorithm to defend this form of attack (CAT). The new form of attack is constructed as a composition of different individual perturbation models including pixel perturbation and spatial transformations. The CAT is proposed to defend both individual attack and composite attack by penalising the maximum accuracy loss during the sequential generation process of a composite attack perturbation. Empirical experiments comparing the proposed algorithm under both individual and composite attacks are conducted on benchmark datasets against baseline methods. The proposed CAT outperformed the baselines under composite attacks. Further analysis and discussion on different variations of composite attack as well as CAT are also presented with possible future exploration directions. \n\nPros:\n- The paper is well structured in general and easy to understand.\n- The idea of composite attack is interesting and meaningful to the neural network adversarial attacking area. \n- The proposed method improves the network robustness under composite attack.\n- The detailed analysis on composite attacks is valuable. \n\nCons:\nMy main concern with the paper is the general performance of the proposed algorithm and the fairness of the comparison. \n- While the paper claims outstanding performance on individual perturbation model attacks, it is not always true across the two dataset. And the proposed algorithm always presents a lower clean accuracy in most of the experiment settings by a relatively large margin. There seems to be a clear tradeoff between the clean accuracy and the robustness towards a more aggressive attack (composite attack). The result limits the strength of the algorithm. \n- I am concerned about the fairness of the comparison against baseline methods like MAX/ AVG. Since the paper used pretrained models from previous work,  MAX/AVG baseline models are trained based on Eq(2)(3) and evaluated under the composite attack. In this case, the underlying perturbation space considered in Eq(2)(3) is different from (smaller than) the one in composite attack. (E.g. true maximum perturbation will not never be considered when training these models)\n- Another question is:  what does alpha mean for baseline methods during training? Is alpha used to rescale the perturbation during baseline training or not? If not, then Figure 4 presents very limited information since alpha is an unfair information available to the proposed model. If yes, then isn’t the whole experiment a scaling version of the main results?\n\n\nOther comments:\n- I would move the introduction of spatial transformation perturbation to section 2 as it is part of the fundamentals. \n- Some details of baselines in Appendix A should be moved to the main text to provide a more self-contained experiment section. E.g. how the baseline models are trained.\n- It would be nice to bold the best performance number in the tables. \n\n\n---------------------------------------\npost-rebuttal\n\nI would like to thank the authors for their efforts to improve the methods and the draft. Part of my concerns was resolved. \nFor clean accuracy, CAT-r did provide a better trade-off. However, it is improved after the submission deadline, it can't be counted into the original contribution in theory. \nFor the concern that the comparison to the baseline presents unfairness as the proposed method was designed for the composite attack with a larger perturbation space, I think the author agrees with my point to some extend. \nI decided to keep my original score deal to the remaining weakness in the paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AVCMFSvCEIr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method for dealing with *composite* adversarial attacks, which are defined as a sequence of perturbation operators each applying some constrained perturbation to the output of the previous operator. Their method models the composed adversarial examples $x^*$ as the sum of the unperturbed example with a series of perturbations $\\delta_i$ which maximize the estimator's loss. They compare their results to other existing adversarial training methods against multiple types of adversarial attacks.\n\nPros:\n- Interesting idea, seems like a very natural continuation of existing work\n- Good experimental design, results are reasonably thorough\n- Some results are encouraging\n\nCons:\n- Explanation of method (CAT) is somewhat lacking. It's not clear to me exactly what their method does differently than the baselines explained in the background.\n- Results are mixed with discussion focusing almost entirely on the positive parts. For example, CAT consistently performs significantly worse than baselines on \"clean accuracy\" and worse than one or more baselines on other singular attacks (see Tables 1,2,3,4).\n- Results in section 5.2 lack explanation (i.e. what do the table columns/rows actually mean)\n- Minor formatting issues\n\nOverall, I think the central problem that the authors are trying to solve is important and their work makes a reasonable contribution towards the solution. Despite the apparent mixed results, this paper should be a candidate for acceptance.\n\nAdditional comments for the authors:\n- It would be helpful to provide references for the definitions of \"robust accuracy\" and \"clean accuracy\"; I'm sure these are metrics that have been defined and used in prior work but this can sometimes make it difficult for outside readers to find where they are rigorously defined.\n- As mentioned in the Cons, you should make it more clear what the reader should be looking for in the tables. Reading just by the accuracy scores, it seems like CAT often performs worse or about the same as baselines in multiple experiments.\n- Table captions should be above, not below, the table. This particularly problematic with Table 4/Figure 4 where the Table caption looks like the title of Figure 4.\n- As mentioned before, equation 8 does not (for me) satisfactorily explain what CAT actually does.\n- In equation 8, $\\delta_i$ appears in the constraint but not in the expression; perhaps you meant to write:\n$$\nx' = \\underset{x^{(m)}}{\\arg\\max} \\ell (f_{\\theta}(x^{(m)} + \\delta_i,y)\n$$\n- The distinction between the different indexing notations $x_i$ and $x^{(i)}$ is not always clear\n- It's not clear what the notation means in Tables 5, 6, and 7 and how it relates to \"ordering\" of perturbations.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank authors and reviewers for discussions. Reviewers found the paper (specially the CAT-r method proposed in the rebuttal period) interesting but there are some remaining concerns about the significance of the results and experiments. Given all, I think the paper still needs a bit of more work before being accepted. I encourage authors to address comments raised by the reviewers to improve their paper.

- AC
Decision: Reject