Towards A Unified Min-Max Framework for Adversarial Exploration and Robustness
Authors: Jingkang Wang
Tianyun Zhang
Sijia Liu
Pin-Yu Chen
Jiacen Xu
Makan Fardad
Bo Li
Venue:  (2020)
Keywords: Ensemble attack
adversarial training
diversity promotion

Abstract: The worst-case training principle that minimizes the maximal adversarial loss, also known as adversarial training (AT), has shown to be a state-of-the-art approach for enhancing adversarial robustness against norm-ball bounded input perturbations. Nonetheless, min-max optimization beyond the purpose of AT has not been rigorously explored in the research of adversarial attack and defense. In particular, given a set of risk sources (domains), minimizing the maximal loss induced from the domain set can be reformulated as a general min-max problem that is different from AT. Examples of this general formulation include attacking model ensembles, devising universal perturbation under multiple inputs or data transformations, and generalized AT over different types of attack models. We show that these problems can be solved under a unified and theoretically principled min-max optimization framework.  We also show that the self-adjusted domain weights learned from our method provides a means to explain the difficulty level of attack and defense over multiple domains. Extensive experiments show that our approach leads to substantial performance improvement over the conventional averaging strategy.

Under review as a conference paper at ICLR 2020
TOWARDS A UNIFIED MIN-MAX FRAMEWORK FOR ADVERSARIAL EXPLORATION AND ROBUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
The worst-case training principle that minimizes the maximal adversarial loss, also known as adversarial training (AT), has shown to state-of-the-art approach for enhancing adversarial robustness against norm-ball bounded input perturbations. Nonetheless, min-max optimization beyond the purpose of AT has not been rig- orously explored in the rescarch of adversarial attack and defense. In particular, given a set of risk sources (domains), minimizing the maximal loss induced from the domain mentally different from AT since the maximization is taken over the probability simplex of the domain Examples of this general formulation include attacking model ensembles, devising universal perturbation under multiple inputs or data transformations, and generalized AT over different types of attack models. We show that these problems can be solved under a unified and theoretically principled min-max optimization framework. We also show that the self-adjusted domain weights learnt from our method provide holistic tool to explain the difficulty level attack and defense over multiple domains. Extensive experiments show that our approach leads to substantial performance improvement over the conventional heuristic strategies
be reformulated
general min-max problem that is funda-
INTRODUCTION
Training a machine learning model that is capable of assuring its worst-case performance against all possible adversaries given a specified threat model is a fundamental yet challenging problem. especially for deep neural networks (DNNs) (Szegedy et al. 2013 Goodfellow et al. 2015 Carlini
Wagnne 2017). A common practice train an adversarially robust model is based on a specific form of min-max training, known as adversarial training (AT) (Goodfellow et al. 2015 Madry et al.,
2017), where the minimization step learns model weights under the adversarial loss constructed at the maximization step in alternative training fashion. On datasets such MNIST and CIFAR- 10, AT has achieved the state-of-the-ee defense performance against lp fp-norm-bal input perturbations (Athalye al. 2018b).
Motivated by the success of AT, one follow-up question that naturally arises is: Beyond AT, can other types of min-max formulation and optimization techniques advance the research in adversarial robustness? In this paper, we give an affirmative answer corroborated by the substantial performance gain and the ability of self-learned risk interpretation using our proposed min-max framework on several tasks for adversarial attack and defense.
We demonstrate the utility of general formulation for min-max optimization minimizing the maximal loss induced from a set of risk sources (domains). Our considered min-max formulation fundamentally different from AT, as our maximization step is taken over the probability simplex of the set of domains. Moreover, we show that many problem setups in adversarial attacks and defenses can in fact be reformulated under this general min-max framework, including attacking 2018: Liu et al. 2018), devising universal perturbation to input samples (Moosavi-Dezfooli et al. 2017) or data transformations (Athalye & Sutskever
model ensembles (Tramer et al.
2018 Brown
et al. 2017), and generalized AT over multiple types of threat models (Tramèr & Boneh 2019 Araujo
ct al. 2.19). However, current methods for solving these tasks often rely on simple heuristics (c.g..
For reproducibility, the code and trained models will be released accompanying this paper.

---

Under review as a conference paper at ICLR 2020
uniform averaging). resulting in significant performance drops when comparing to our proposed min-max optimization framework.
Specifically, based on the general min-max framework, show that these problems can be solved under the same optimization procedure and prove the rate of its algorithmic convergence. As a byproduct and exclusive feature, by tracking the weighting factor associated with the probability simplex during training, our method can provide tools for self-adjusted risk assessment and obtain novel insights on the set of domains for the associated tasks.
Contributions (I) We indicate the utility of min-mas max optimization beyond AT by proposing a general and theoretically grounded framework on adversarial attack and defense. (2) We demonstrate the effectiveness of our min-max framework by evaluating the proposed APGD attack on MNIST and CIFAR-10. 10. In theory, we show that APGD has an ( convergence rate, where T is the number of iterations. In practice, we show that APGD obtains 17.48%, 35.21% % and 9.39% improvement on average compared with PGD attack on CIFAR-IO. (3) We propose a generalized AT scheme under mixed types of adversarial attacks and demonstrate that the diversified attack ensemble helps adversarial robustness. Compared with vanilla AT, our new training scheme leads to better worst-case robustness even if the defender lacks prior knowledge of the strengths of attacks. (4) We show how the weighting factors of the probability simplex help to obtain novel insights for associated tasks and interpreting the importance of candidates in domains.
Related Work
Recent studies have identified that DNNs are highly vulnerable to adversarial
manipulations in various applications (Szegedy al. 2013
Carlini al.
[016 Tia & Liang 2017
Lin et al. 2017: Huang et al. 20117; Carlini & Wagner 2018
Zhao et al. 2018 Eykholt et al. 2018
Chen et al. 20188| Lei al. 2019). thus leading an arms race between versarial attacks (Carlini
2017 et al. 2018b; Goodfellow et al. 2014 Papernot et al. 201 Moosavi Dezfooli
ct 2016 et al. 22] 8b;[Xu et al. al.] 2019) and defenses (Madry et 2017 Papernot et al.
201
Meng & (0eem]
et al. 2017; Xu al. 2018). One intriguing property of adversarial
examples is the transferability across multiple domains (Liu et al. 2017 Tramer et al. 2017; Papernot
et al. 2017 Su et al., 2018). which indicates more challenging yet promising research direction
devising universal adversarial perturbations over model ensembles (Tramèr et al.) [2018] Liu et al.)
2018), input samples MMoosai-Dezzoooli et al. 2017: Metzen et | [017]] Shafahi al.) 2018)
and data transformations (Athalye et al. 2018b Athalye & Sutskever] 2018 Brown et al. 2017).
However, current approaches suffer from significant performance loss for resting the uniform
averaging strategy. We will compare these works with our min-max method in Sec. 4 As a natural extension following min-max attack, we study the generalized AT under multiple perturbations
(Tramer & Boneh 2019; Araujo et al. 2219;; Kang et al.)
[019 Croce & Hein 2019). Finally, our
min-max framework is adapted and inspired by previous literature on robust learning over multiple
domains (Qian et al. 2018 Rafique et al. 2018 Lu et al. 2018 2019a).
2 MIN-MAX POWER IN ADVERSARIAL EXPLORATION AND ROBUSTNESS
We begin by introducing the principle of robust learning over multiple domains and its connection to a specialized form of min-max optimization. We then show that the resulting min-max formulation fits into various attack settings for adversarial exploration: a) ensemble adversarial attack, b) universal adversarial perturbation and c) robust perturbation over data transformations. Finally, we propose generalized adversarial training (AT) framework under mixed types of ad vvversaiaal attacks to improve model robustness.
2.1 GENERAL IDEA: ROBUST LEARNING OVER MULTIPLE DOMAINS
Consider K loss functions [Fi(v) (each of which is defined learning domain), the problem of robust learning over K domains can be formulated as (Qian et al. 2018 Rafique et al. 2018 Lu ct al. 2018)
minimize maximize vev
ΣK F.V).
WEP
where v and w are optimization variables, V is a constraint set, and P denotes the probability simplex p = [w] 1' 1, E [0, 1), vij. Since the inner maximization problem in (1) is linear function
2

---

Under review as a conference paper at ICLR 2020
of w over the probabilistic simplex, problem 1 is thus equivalent to
minimize maximize F((v):
vEV sE[K]
where [K] denotes the integer set {1, (I, KK,
Benefit and computation challenge of ( Compared to multi-task learning in a finite-sum formu- lation which minimizes K losses on average, problem (I) provides consistently robust worst-case
performance across all domains. This can be explained from the epigraph form of (2).
minimize t, subject to Fi(v) S t, E [K], veV,
(3)
where l is an epigraph variable (Boyd & Vandenberghe 2004) that provides the t-level robustness at each domain.
Although the min-max problem ① offers great robustness interpretation in solving it becomes more challenging than solving the finite-sum problem. It is clear from (2) that the inner maximization problem of ④ always returns the one-hot value of w, namely, ith standard basis vector, and = arg max; (Fi(v)) The one-hot coding reduces the generalizability other domains and induces instability of the learning procedure in practice. Such an issue is often mitigated by introducing a strongly concave regularizer in the inner maximization step (Lu et al. 2018
e;, where e; is the
Qian al. 2018).
Regularized problem formulation Spurred by (Qian et al. 2018), we penalize the distance be- tween the worst-case loss and the average loss over K domains. This yields
minimize maximize WEP
Σ wvV.(v)
31 J|ww-/KK[3
(4)
vEV
where 7 >> 0 is regularization parameter. As O, problem ④ is equivalent ( By contrast, it becomes the finite-sum problem when - co since w - 1/K. In this sense, the trainable w provides an essential indicator on the importance level of each domain. The larger the weight is, the more important the domain is. We call w domain weights in this paper. We next show how the principle of robust learning over multiple domains can fit into various settings of adversarial attack and defense problems.
2.2 ROBUST ADVERSARIAL ATTACKS
The general goal of adversarial attack to craft an adversarial example x' Xo + E IRd to mislead the prediction of machine learning (ML) deep learning (DL) systems, where Xu denotes the natural example with the true label fo, and 5 is known as adversarial perturbation, commonly subject to f,-norm (p {0, 1, 2, (0)) constraint X' == {8 ||0|| e, Xo 5 [0, 1]d} for given small number e. Here the Ep norm enforces the similarity between x' and Xo, and the input space of ML/DL systems normalized to [0, 1]4.
Ensemble attack over multiple models Consider K ML/DL models (M:) the goal is find robust adversarial examples that can fool all K models simultaneously. In this case, the notion of "domain' in (4) is specified as 'model', and the objective function F, in (4) signifies the attack loss f Xo, No. M,) given the natural input (Xn, yo) and the model M,. Thus, problem (4) becomes
minimize maximize WEP
Σ w.f(6: Xo. yo M.) || - 1/K
(5)
where w encodes the difficulty level of attacking each model.
Universal perturbation over multiple examples Consider natural examples {(x;, K and single model M, our goal is to find the universal perturbation ō so that all the corrupted K examples can fool M. In this case, the notion of 'domain' (4) specified as 'example', and problem becomes
minimize maximize SEX
ΣK w.f(f:;x, /// 1/K
(6)
WEP
where different from 5) encodes the difficulty level of attacking each example.
3

---

Under review as a conference paper at ICLR 2020
Ad Adversarial attack over data transformations Consider K categories of data transformation {pr), ). e.g., rotation, lightening, and translation (Athalye et al. 2018a), our goal is to find the adversarial attack that is robust to data transformations. In this case, the notion of 'domain' in (4) is specified as
'aaa transformer', and problem (4) becomes
minimize maximize WEP
Σ (F + 6): yo. M)] 2 |w - 1/K
where Byoup, ([((xx++)))))))))))))) yo; denotes the attack loss under the distribution of data transformation Ps, and w encodes the difficulty level of attacking each type of transformed example Xo.
2.3 ADVERSARIAL TRAINING (AT) UNDER MIXED TYPES OF ADVERSARIAL ATTACKS
Conventional AT is restricted to a single type of norm-ball constrained adversarial attack (Madry et al. 2017). For example, AT under &o attack yields
minimize Exxy)ED maximize fe= 8; x,
(8)
0
[|6 Se
where 0 E IR" denotes model parameters, 6 denotes e-tolerant loo attack, and [O,(O, 6; x, y) the training loss under perturbed examples {(x 8, y)}. However, there possibly exist blind attacking spots across multiple types of adversarial attacks so that AT under one attack would not be strong enough against another attack (Araujo al. 2019). Thus, interesting question is how generalize AT under multiple types of adversarial attacks. One possible way is to use the finite-sum formulation
minimize Exxy)eD maximize 0
Σ (v, 8; x),
(6,Ex))
K 1=1
where 6, E Xr is the ith type of adversarial perturbation defined on X, e.g., different lp attacks.
Moreover, one can map 'attack type' to 'domain' considered in (T). We then perform AT against the strongest adversarial attack across K attack types in order to avoid blind attacking spots. That is, upon defining F; (0) == maximizes,ex, Str(0, d,; y), we solve the problem of the form 2
minimize IE (x,))D maximize (.))..
(10)
KE(K)
In fact, problem (T0) is in the min min--aa-maan max form, however, Lemma shows that problem 100 can be further simplified to the min-max form.
Lemma 1. Problem (TO) is equivalent
minimize (x)))) maximize
(11)
(E..(
where w E RK represent domain weights, and P has been defined in
Proof: see Appendix
Similar a strongly concave regularizer -7/21 1/K can be added into the inner maximization problem of (IT), which can boost the stability of the learning procedure and strike balance between the max and the average attack performance. However, solving problem (TT) and its regularized version is more complicated than (8) since the inner maximization involves both domain weights and adversarial perturbations {
We finally remark that there was an independent work (Tramer & Bonch 2019) which also proposed the formulation (TOp for AT under multiple perturbations. However, what we propose here is the regularized formulation of (II). As will be evident later, the domain weights w in our formulation have strong interpretability, which learns the importance level of different attacks. Most significantly, work has different motivation from Tramerr & Boneh 2019), and our idea applies to not only AT but also attack generation in Sec. 2.2
PROPOSED ALGORITHM AND THEORY
In this section, we delve into technical details how efficiently solve problems of robust adversarial attacks given by the generic form and problem (IT) for generalized AT under mixed types of adversarial attacks.
4

---

Under review as a conference paper at ICLR 2020
3.1 ALTERNATING ONE-STER STEP PGD FOR ROBUST ADVERSARIAL ATTACK GENERATION
We propose the alternating one-step projected gradient descent (APGD) method (Algorithm[]] Th to solve problem 4 For clarity, we repeat proh- lem 4 under the adversarial perturbation ó and its constraint set X' defined in Sec.
Algorithm 1 APGD to solve problem
1: Input: given w and 5())
2: for t = 1,2,.... T do
3:
outer min.: fixing w = w (((-1), call PGD (3) to update g(t)
minimize maximize WEP
ΣE MX (6).
(12)
BE
4: inner max.: fixing δ = - update w(()
We show that each iteration, APGD takes only one-step PGI) for outer minimization and one-step projected gradient ascent for inner max- imization (namely, PGD for its negative objective fiunciion. We also show that each alternating step has closed-form expression, and the main computational complexity stems from computing the gradient of the attack loss w.r.t. the input. Therefore, APGD is computationally efficient like PGD, which is commonly used for design of conventional single fn-norm based adversarial attacks (Madry) et al. 2017).
via (14)
5: end for
Outer minimization Considering w and F(Ø)= Ete w. P((8) in we per- form one-step PGD to update ó at iteration t, (t) projx (8(t-1) - V5
(13)
where proj(-) denotes the Euclidean projection operator, i.e., ( arg min ||x all2 at the point a, a > is given learning rate, and V denotes the first order gradient w.r.t. 5.
In 13 the projection operation becomes the key to obtain the closed-form of the updating rule (3) Recall from Sec. 2.2 that
{8||8||p _ S S é), where p E 1, x)), and é Xo co, then the projection function becomes the clip function. However, when E {0, 2,). the closed-fore of projection operation becomes non- non-trivial. In Proposition | we derive the solution of proj (a) under different lp norms.
and è = 1 - xo (implying è < 0 < é). If
Proposition 1. Given a point Rd and constraint set X' {o|||||,, S c, S 5 s c), the Euclidean projection 8" =proix(aa) has a flosed-form solution when E 1,2
Proof: See Appendix B
Inner maximization
By fixing 5(9) and letting v'(w) == ΣK w; wi((1))))) - 3ll
1/K
in problem (4) we then perform one-step PGD (w.rtt -1/) to update w,
((t)
p (-(t-1) + uV (w)(t-1)) = =(b- 1 1114 (b--1144
(14)
where 3 > is a given learning rate, (w)(w) = (tt)) 7(w - 1/K). and ((t) (8))) ).. Fx ( (Z (14), the second equality holds due to the closed-form of projection op- eration onto the probabilistic simplex P (Parikh et al. 2014), where (-)+ denotes the elementwise non- negative operator, i.e.. (I)+ = max 2), and μ is the root of the equation 1 1 ,1)+ = 1. Since
1' (b min, [b,) + 1/K)+ 1/K 1, and 1"(b max, {b.}1 + 1/K)+ < 1 = 1,
the root exists within the interval [min: [b.} - 1/K, m,x, (b.) - 1/K] and can be found via the
bisection method (Boyd & Vandenberghe, 2004).
Convergence analysis We remark that APGD follows the gradient primal-dual optimization frame- work (Lu et al. 2019a), and thus enjoys the same optimization guarantees. In Theorem ! we demonstrate the convergence rate of Algorithm[[for solving problem (4).
Theorem 1. (inherited from primal-dual min-max optimization) Suppose that in problem F F) has I. L-Lipschitz continuous gradients, and X is convex compact set. Given learning rates S I and B 1, then the sequence {8(0), w(() Jt= generated by Algorithm 7 converges a first-order stationary poin of problem under the convergence rate )
"The stationarity measured by the la norm of gradient of the objective in w.r.t. (6, w).
5

---

Under review as a conference paper at ICLR 2020
Proof: Note that the objective function of problem is strongly concave w.r.t w with parameter 7 and has y 7y-Lisschitz continuous gradients. Moreover, we have ||ll22 S 1 due to w E P. Using these facts and (Lu et al. 2019a Theorem 1) (Lu et al. 2019b Theorem 1) completes the proof.
3.2 ALTERNATING MULTI-STEP PGD FOR GENERALIZED AT
We next propose the alternating multi-step projected gradient descent (AMPGD) method to solve the regularized version of problem which is repeated follows
1/K (15)
minimize ((D))
maximize wEP (o
(0 w, (6.))
Problem 15 is in more general non-convex
Algorithm 2 AMPGD to solve problem (5
non-concave min-max setting, where the inner maximization involves both domain weights w
1: Input: given ((0) (())), 5(0)
and K 0.
and adversarial perturbations [ō:j. It was shown
2: for = 1, 1,2,... 2....." T do
in (Nouiehed al.] 2019) that the multi-step
3: given w- and ((t-1), perform SGD to
PGD is required for inner maximization in or- der to approximate the near-optimal solution. This is also in the similar spirit of AT (Madry et al. 2017), which executed multi-step PGD attack during inner maximization. We summa- rize AMPGD in Algorithm At 4tep of Algorithm each PGD step to update w and å can be decomposed
update ( (Madry al. 2017)
4: given O(t), perform R-step PGD to update
w(1) and 5(()
5: end for
wh projp
w + V
(1)
w (0(1)
- (0 (-
(0)
Vr [R]. Vr E [R] Vi E [K]
(t)) 1r
projx;
((() i-- 8V ( w
10 (
where let w = (t-1) and 8.(0 1.1 = o( Here the subscript / represents the iteration index of AMPGD, and the subscript r denotes the iteration index of R-step PGD. Clearly, the above projection operations can be derived for closed-fone form expressions through (14) and Lemma[] To the best of our knowledge, is still an open question build theoretical convergence guarantees for solving the general non-convex non-concave min-max problem like (15). except the work (Nouiehed et al. 2019) which proposed (1)/T) convergence rate the objective function satisfies Polyak- Lojasiewicz conditions (Karimi et al. 2016).
Improved robustness via diversified ép attacks. was recently shown (Kriiyapa & Qureshi 2019; Pang et al. 2019) that the diversity of individual neural networks improves adversarial robust- ness of an ensemble model. Spurred by that, one may wonder the promotion of diversity among &p attacks is beneficial adversarial robustness? We measure the diversity between ad adversarial attacks through the similarity between perturbation directions, namely, input gradients (V fir 6,; x, y) in [15 We find that there exists strong correlation between input gradients for different lp attacks. Thus, we propose to enhance their diversity through the encouraging diversified prediction of ensemble models in (Pang al. 2019)
cerpoonally--rrmmotin regularizer used for
h(0, {8:) x, 3) := logdet (edeetG"GGGG G),
(16)
where G
Bd is d K matrix, each column of which corresponds to a normalized input gradient Vs, (0, 5;; y) for i E [K], and h(O, {ō,} }:x. y) reaches the maximum value 0 as input gradients become orthogonal. With the aid of (16). we modify problem (5) to
minimize B(xyyJD maximize (8, w, (6.)) Ah(0, (6.)x..)). WEP.(B.EX.))
(17)
The rationale behind (17) is that the adversary aims to enhance the effectiveness of attacks from diversified perturbation directions (inner maximization), while the defender robustifies the model 0, which makes diversified attacks less effective (outer minimization).
4 EXPERIMENTS
In this section, we first evaluate the proposed min- max optimization strategy on three attack tasks. We show that our approach leads substantial improvement compared with state-of-the-art attack

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison of average and min-max (APGD) ensemble attack over four models MNIST and CIFAR-10. Ace (%) represents the test accuracy of classifiers on adversarial examples. Here we set the iterations of APGD as 50 for attack generation. The learning rates a, 3 and regularization factor 7 are provided in
Appendix
(a) MNIST
(b) CIFAR-10
Box constraint
Opt.
Acca
Acca
Accc
Actp
ASR,,
Lit
Box construint
Acca
Aceg
Acce
Accp
ASRai
Lif
301
ung-
7.03
2.36
1.51
11.27
4.99
2.48
84.03
9.45%
50)
27.86
18.74
8.66
3.15
9.64
5.16
9.70
6.17
65. 15.16
71.44
9.64%
20)
nuin maix
orgi
6.12
0.15
2.53
21-18
8.43
5.11
28.64%
300
12.46
5.62
5.86
78.65
31.68%
(x - 3.0)
muin manx
agg-
6.88
1.51
0.03
0.89
26.28
3.50
14.50
2,06
95.31
69.12
37.89%
lc 2.0)
7.17
24.3
1.51
3.03
4.65
4.59
4.20
5.14
69.55
83,95
20.70%
C. (e - 0.2)
ongg min max
2.47
1.05
0.07
0.37
41.10
7.30
35.03
5.81
90.16
48.17
N7.17%
Fe 0.05)
19.00
7.21
2,68
1.55
5.61
4.74
4.50
4.26
73.29
84.36
15.10%
methods such ensemble PGD (Liu et 2018) and expectation over transformation (BOT) (Athalye) et al.] 2018b] Brown et al. 2017 Athalye et al., (0118)). We next demonstrate the effectiveness of the generalized AT for multiple types of adversarial perturbations. We show that the use of trainable domain weights in problem (15) can automatically adjust the risk level of different attacks during the training process even if the defender lacks prior knowledge on the strength of these attacks. We also show that the promotion of diversity of Ep attacks help improve adversarial robustness further.
We thoroughly evaluate our APGD/AMPGD algorithm on MNIST and CIFAR-10. 10. A set of di- verse image classifiers (denoted from Model A to Model H) are trained, including multi-layer perceptrons (MLP), All-CNNs (Springenberg et al., 2015), LeNet (Lecun et al.) 1998). L2 VGGI6 (Simonyan & Zisserman [2015), ResNer50 (He et al., 2016). Wide-ResNet (Madry et al., 2017 Zagoruyko & Komodakis 2016) and GoogLeNet (Szegedy et al. 2015). More details about model architectures and training process are provided in Appendix C.I
4.1 ROBUST ADVERSARIAL ATTACKS
Most current works play a min-max game from defender's perspective, i.e., adversarial training. However, we show the great strength of min-max optimization also lies at the side of attack generation. Note that problem formulations (5)(7) are applicable to both untargeted and targeted attack. Here we focus on the former setting and use C& W loss function (Carlini & Wagner 2017: Madry et al.. 2017 The details of crafting adversarial examples are available in Appendix 2.2
Ensemble attack over multiple models We craft adversarial examples against an ensemble of known classifiers. The work (Liu et al. 2018, 5th place at CAAD-18) proposed an ensemble PGD attack, which assumed equal importance among different models, namely, w; = I/K K in problem ①.. Throughout this task, we measure the attack performance via ASRau the attack success rate (ASR) of fooling model ensembles simultaneously. Compared to the ensemble PGD attack (Liu et al. 2018). our approach results in 40.79% % and 17.48% ASRall improvement averaged over different -n-normm constraints on MNIST and CIFAR- R-00, respectively. In what follows, we provide more detailed experiment results and analysis.
In T we show that our min-max APGD significantly outperforms ensemble PGD in ASRall- Taking &x -attakk on MNIST as an example, our min-max attack leads to a 90.16% ASRah which largely outperforms 48. 17% (ensemble PGD). The reason is that Model C, D are more difficult to attack, which can be observed from their higher test accuracy on adversarial examples. As a result, although the adversarial examples crafted by assigning equal weights over multiple models are able to attack (A, B) well, they achieve a much lower ASR (i.e., 1 Acc) in [C, D). By contrast, APGD automatically handles the worst case {C. D) by slightly sacrificing the performance on (A, B): 31, 1..4% averaged ASR improvement on (C, D) versus 0.86% degradation on (A, B). More results on CIFAR-10 and more complicated DNNs (e.g., GoogLeNet) are provided in Appendix D
Figure depicts the ASR of four models under average/min-man max attacks as well as the distribution of domain weights during attack generation. For ensemble PGD (Figure Ia), Model C and D are attacked insufficiently, leading to relatively low ASR and thus weak ensemble performance. By contrast, APGD (Figure lb) will encode the difficulty level to attack different models based on the current attack loss. It dynamically adjusts the weight w; as shown in Figure [c] For instance, the weight for Model D is first raised to 0.45 because D is difficult attack initially. Then it decreases to
7

---

Under review as a conference paper at ICLR 2020
1.0
0.8
A: MLP
B: AI--NNS
Attack
0.4
0.4
LeNet
0.4
0.2
0.2
D: LeNetV2
Success (all)
0.0
10
20
30
0.0
10
20
30
0.0
Number of iterations
Number of iterationn
Number iterations
(a) average case
(b) min max
(c) weight {w,
Figure 1: Ensemble attack against four DNN models on MNIST. (a) (b): Attack success rate of adversarial
examples generated by average (ensemble PGD) or min-max (APGD) attack method. (c): Boxplot of weight w
in APGD adversarial loss for four models. Here adopt the same to attack Table
av3. minnax
avg
0.8
0.0
0.5
8...
Aeteck surcess
0.6
0.5
0.6
0.
0.4
0.4
0.2
avg
0.2
0.4
AVD
0.0
0.0
9.0
6.3
0.3
9.55
0.30
0.15
0.0
0.2
0.3
0.01
3.06
0.15
(a) MNIST (A, B, C)
(b) CIFAR (A, B, CJ
(c) MNIST (A, B, C.
(d) CIFAR (A, B, C. DJ
Figure 2: ASR average and min-max loo ensemble attack versus maximum perturbation magnitude e.
0.3 once Model D encounters the sufficient attack power and the corresponding attack performance is longer improved. It is worth noticing that APGD is highly efficient because wy converges after small number of iterations. To perform a boarder evaluation, we repeat the above experiments (fo norm) under different in Figure 2 The ASR of min-max strategy is consistently better or on part with the average strategy. Morcover, APGD achieves more significant improvement when moderate e chosen: MNIST [0.15, 0.25)) and CIFAR-IO (E E [0.03, 0.05)).
Lastly, we highlight that tracking domain weights w provides us novel insights for model robustness and understanding attack procedure. From our theory. model with higher robustness always corresponds to larger w because its loss is hard to attack and becomes the "worst" term. This hypothesis can verified empirically. According to Figure Ic. we have w. > wa > Wa > wh indicating a decrease in model robustness for C, D, A and B, which is exactly verified by Accc Accp Acc > Aceg in Table T (norm))rm).
Universal perturbation over multiple examples We evaluate APGD in universal perturbation on MNIST and CIFAR-10, 10, where 10,000 test images randomly divided into equal-size groups (containing K images per group) for universal perturbation. We measure two types of ASR (%), ASRavy and 9p Here the former represents the ASR averaged over all images in all groups, and the latter signifies the ASR averaged over all groups but a successful attack is counted under a more restricted condition: images within each group must be successfully attacked simultaneously by universal perturbation. When K- 5, our approach achieves 42.63 % and 35.21 % improvement over the averaging strategy under MNIST and CIFAR-10, respectively.
In Ta we compare the proposed min-max strategy (APGD) with the averaging strategy on the attack performance of generated universal perturbations. As we can see, our method always achieves higher ASR gp for different values of K. The universal perturbation generated from APGD can successfully attack 'hard' images (on which the average-based PGD attack fails) by self-adjusting domain weights, and thus leads to a higher gr Besides, the min-mat universal perturbation also offers interpretability
"image robustness" by associating domain weights with image visualization: see Figure A9 and A10 (Appendix F. for an example in which the large domain weight corresponds to the MNIST letter with clear appearance (c.g., bold letter).
Robust adversarial attack over data transformations EOT (Athalye et al. 2018a achieves state- of-the-an art performance in producing adversarial examples robust to data transformations. From (7). we could derive EOT as a special case when the weights satisfy wi = 1/K (average case). For each input sample (ori), we transform the image under series of functions, e.g., flipping horizontally
8

---

Under review as a conference paper at ICLR 2020
Table 2: examples. The adversarial examples are generated by 20-step -PPDD with a
Comparison of average and minmax optimization universal perturbation over multiple input
50 and y 4.
Setting
K
=
K
Dataset
Model
Opt.
|ASRay
AS
Lift (1)
ASRayy ASRgp
Lift (1)
ASSRag ASRgp
Lift (1)
ASRang
ASRgp
Lift (1)
MI.P
ang-
97.19
94.48
85.13
56.64
79.11
38.05
60.53
3.50
min
98.15
96.96
2.62%
83.76
72.32
27.68%
72.28
53.70
41.13%
30.10
6.70
91.43%
All-CNNs
avg..
97.76
95.52
85.19
51.92
80.02
31.25
65.79
2.10
MNIST
mii muax
99.69
99.38
4.04%
75.64
45.69%
53.50
71.20% %
43.54
4.30
104.8%
LoNet
min max
ang-
94.78
96.60
94.58
89.96
5.14%
62.12
55.50
28.77
36.72
27.86 %
51.84
42.79
25.80
19.15
34.73%
22.48
30.29
7.20
4.30
67.44%
LeNetV2
dnig.
94.72
90.04
61.59
26.60
50.42
17.05
26.49
4.80
mni max
97.33
95.68
6.26%
55.38
35.52
33.53 %
40.22
21.05
23.46 %
19.73
7.10
47.92%
AlI-CNNs
ang-
91.09
83.08
85.66
54.72
82.76
40.20
71.22
4.50
min max
92.22
85.98
3.49%
87.63
65.80
20.25% %
85.02
55.74
38.66 %
65.64
11.80
162.2% %
LeNetV2
ang.
93.26
86.90
90.04
66.
28
72.02
8.90
CIFAR-10
min maxe
93.34
87.08
0.21% %
91.91
71.64
8.35 %
91.21
63.55
15.55 %
82.85
25.10
182.0 %
VGG16
drig-
90.76
82.56
89.36
63.92
88.74 74
55,20
85.86
22.40
mii
92,40
85.92
4.07%
90.04
70.40
10.14% %
88.97
63,30
14.67 %
79.07
30,80
37.50%
mnin Goe
axgg-
85.02
72.48
75.20
32.68
71.82
19.60
59.01
0.40
87.08
77.82
7.37%
77.05
46.20
41.37% %
71.20
33.70
71.94%
45.46
2.40
600.0%
Table 3: Comparison of average and min-max optimization on robust attack over multiple data transformations on CIFAR-10. 10. Acc (%) represents the test accuracy of classifiers on adversarial examples (20-step APGD
0.03) with 2: 3 and 7 = 10) under different transformations.
Model
Opt.
Accari
Acc/1A
Accyie
AcCars
AcCgam
ACCerop
ASR,
ASRgp
Lift (t)
A
aug.
10.80
21.93
14.75
11.52
10.66
20.03
85.05
55.88
min max
12.14
18.05
13.61
13.52
11.99
16.78
85.65
60.03
7.43%
avg.
5.49
11.56
9.51
5.43
5.75
15.89
91.06
72.21
min max
6.22
8.61
9.74
6.35
6.42
11.99
91.78
77.43
7.23%
avg.
7.66
21.88
15.50
8.15
7.87
15,36
87.26
56.51
min max
8.51
14.75
13.88
9.16
8.58
13.35
88.63
63.58
12.51
D
ang.
8.00
20.47
13.46
7.73
8.52
15,90
87.65
61.13
min max
9.19
13.18
12.72
8.79
9.18
13.11
88.97
67.49
10.40%
(fth) or vertically (flv), adjusting brightness (bri), performing gamma correction (gam) and cropping (crop), and group each image with its transformed variants. Similar to universal perturbation, ASR and ASR are reported to measure the ASR over all transformed images and groups of transformed images (each group is successfully attacked signifies successfully attacking an example under all transformers). In Table 3 compared EOT, our approach leads to 9.39% averaged lift in ASR gp over given models on CIFAR-10 by optimizing the weights for various transformations. Due to limited space, we leave the details of transformers in Append C33 and the results under randomness (c.g., flipping images randomly w.p. 0.8; randomly clipping the images at specific range) in Appendix
4.2 ADVERSARIAL TRAINING FOR MULTIPLE ADVERSARIAL PERTURBATIONS
Compared to vanilla AT, we show the generalized AT scheme produces models robust to multiple types of perturbation, thus leads to stronger "overall robustness". We measure the training performance using two types of Acc (%): Acemax examples with the strongest perturbation (Exo or E2), and Acc Rdy ady denotes the averaged test accuracy over examples with all types of perturbations (l>o and f2). Moreover, we measure the overall worst-case robustness S. in terms of the area under the curve 'Acelmax vs. 6' (see Figure 33b..
adv and Acc where Ace denotes the test accuracy over
In Table we present the test accuracy of MLP in different training schemes: a) natural training. b) single-norm: vanilla AT (Eso or E2), c) multi-norm: generalized AT (avg and min max), and d) generalized AT with the adversary only performs single-type attack, training and testing on the same attack type leads to the best performance (diagonal of -ly block). However, when facing fo and lg attacks simultancously, multi nooli-noormaee generalized AT achieves better Acemax and Acc ady than single-norm AT. In particular, the min-max strategy slightly outperforms the averaging strategy under multiple perturbation norms.
attack regularization (DPAR, X = 0.1 in problem (16)). If
avg

---

Under review as a conference paper at ICLR 2020
Table 4: Adversarial training of MNIST models single attacks (f.x and E2) and multiple attacks (aeg. and min max). The perturbation magnitude e for loo and & attacks are and 2.0, respectively. Top test accuracy on each metric are highlighted. Complete table for varied c is given in Table A8 (Appendix
(a) MLP
(b) LeNet
Opt.
Acc.
Acc-foc
Acc-fz
Acc
Acc
Opt.
Acc.
Acc-foc
Acc-fa
Acc
Acc
natural
98.30
2.70
13.86
0.85
8.28
natural
99.25
17.93
39.32
17.57
28.63
Exc 98.08
77.70
69.17
66.34
73.43
Exe
99.18
93.80
78.97
78.80
86.39
98.72
70.03
81.74
69.14
75.88
Eg
99.22
85.84
87.31
84,06
86.58
aug.
98.62
75.09
79.00
72.23
77.05
avg.
99.22
88.96
85,59
84.29
87
DPAR
98.50
76.75
79.67
74.14
78.21
DPAR
99.25
89.96
86.49
85.44
88.23
min max
98.59
75.96
79.15
73.43
77.55
min max
99.32
89.21
85,
84,82
87.60
DPAR
98.58
76.92
79.74
74.29
78.35
DPAR
99.22
90.19
86.47
85.47
88.33
1.00
0.8
0.75
12
0.6
0.50
0.4
natural
0.4
adv (t.)
0.25
0.2
0.2
adv (minmax)
0.0
2.5 3.0 3.5 <(Pa)
4.0
1.5
0.5
2.0 elfy)
g(f2)
(a) weight (w:)
(b) Acc
(c) Acc
Figure 3: (a): Violin plot of weight w in APGD versus perturbation magnitude e of fz-attack in AT; (b) & (c):
Robustness of MLP under different AT schemes. Supplementary result for LeNet is provided in Figure A2 (Appendix E
DPAR further boosts the adversarial test accuracy, which implies that the promotion of diversified lp attacks is beneficial supplement to adversarial training.
In Figure 3 we offer deeper insights on the performance of generalized AT. During the training procedure we fix cx.,. for foo attack during training) as 0.2, and change 6tz from 0.2 to 5.6 (eg., x x so that the &oo and Eg balls are not completely overlapped (Araujo et al. 2019). In Figure 3al et, increases, &2tattcck becomes stronger so the corresponding w also increases, which consistent with min-max spirit defending the strongest attack. We remark that min max ang training does not always lead to the best performance on Acc and Acc avg ady> especially when the strengths of two attacks diverge greatly (see Table A8). This can be explained by the large overlapping between loo and le balls (see Figure A3). However, Figure 3b and show that AMPGG is able to achieve rather robust model matter how changes (red lines), which empirically verifies the effectiveness of our proposed training scheme. In terms of the area-under-the tedder-the-curee theveurve measure S., AMPGD achieves the highest worst-case robustness: 6.27% and 17.64% improvement compared the vanilla AT with foo and & attacks. Furthermore, we show in Figure A4a that our min-max scheme leads to faster convergence than the averaging scheme due to the benefit of self-adjusted domain weights.
CONCLUSION
In this paper, we propose a general min-max framework applicable to both adversarial attack and defense settings. We show that many problem setups can be re-formulated under this general framework. Extensive experiments show that proposed algorithms lead to significant improvement on multiple attack and defense tasks compared with previous state-of-the-art approaches. particular, we obtain 17.48%, 35.21% and 9.39% improvement on attacking model ensembles. devising universal perturbation to input samples, and data transformations under CIFAR-10, 10, respectively. Our min- max scheme also generalizes adversarial training (AT) for multiple types of adversarial attacks, attaining faster convergence and better robustness compared to the vanilla AT and the average strategy. Moreover, approach provides holistic tool for self-risk assessment by learning domain weights.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexandre Araujo, Rafael Pinot, Benjamin Negrevergne, Laurent Meunier, Yann Chevaleyre, Florian Yger, and Jamal Atif. Robust neural networks using randomized adversarial training. arXiv preprint arXiv: 1tti::/933.01999 2019.
A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok. Synthesizing robust adversarial examples. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 284 293, 10-15 Jul 2018a.
Anish Athalye and Ilya Sutskever. Synthesizing robust adversarial examples. ICML, 2018.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv: mee::880220444 2018b.
S. Boyd and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. CoRR, abs/1712.09665, 2017.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pp. 39-57. IEEE, 2017.
Nicholas Carlini and David A. Wagner. Audio adversarial examples: Targeted attacks on speech-to- text. In JEEE Symposium on Security and Privacy Workshops, pp. 1-7. IEEE Computer Society, 2018.
Nicholas Carlini, Pratyush Mishra, Tavish Vaidya, Yuankai Zhang, Micah Sher, Clay Shields, David Wagner, and Wenchao Zhou. Hidden voice commands. In USENIX Security Symposium, pp. 513-530, 2016.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. CoRR, abs/ (bs//992.00675, 2019.
Hongge Chen, Huan Zhang, Pin- Yu Chen, Jinfeng Yi, and Cho-Jui Hsieh. Attacking visual language grounding with adversarial examples: A case study on neural image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume pp. 2587-2597, 2018a.
Pin- Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. EAD: elastic-net attacks to deep neural networks via adversarial examples. AAAI, 2018b.
Francesco Croce and Matthias Hein. Provable robustness against all adversarial 4p-perturbations for 2 1. arXiv preprint arXiv: /213, 2019.
K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song. Robust physical-worid attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. 2015 ICLR, arXiv preprint
2015.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv: 2014.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Matthias Hein and Maksym Andriushchenko. Formal guarantees the robustness of classifier against adversarial manipulation. In Advances in Neural Information Processing Systems, pp. 2266-2276, 2017.
11

---

Under review as a conference paper at ICLR 2020
Sandy Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. In ICLR (Workshop). OpenReview.net, 2017.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In EMNLP, pp. 2021-2031. Association for Computational Linguistics, 2017.
Daniel Kang, Yi Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardt. Testing robustness against unforeseen adversaries. arXiv preprint arXiv:/908) (90.. 08016, 2019.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- condition. In Joint European Conference on
gradient methods under the Machine Learning and Knowledge Discovery in Databases, pp. 795-811. Springer, 2016.
Sanjay Kariyappa and Moinuddin Qureshi. Improving adversarial robustness of ensembles with diversity training, arXiv preprint 190998,,,,, 2019.
Y. Lecun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (((1)):277-2324 Nov 1998. ISSN 0018-9219. doi: 10.179995.72671.
Qi Lei, Lingfei Wu, Pin-Yu Chen, Alexandros G Dimakis, Inderjit Dhillon, and Michael Witbrock. Discrete adversarial attacks and submodular optimization with applications text classification. SysMI, 2019.
Yen-Chen Lin, Zhang-Wer Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming- Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In ISCAI, pp. 7766-3762. ijcai.org. 2017.
Jiayang Liu, Weiming Zhang, and Nenghai Yu. CAAD 2018: Iterative ensemble adversarial attack. CoRR, abs/181 1.03456, 2018
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In ICLR. OpenReview.net, 2017.
Lu, I. Tsaknakis, and M. Hong. Block alternating optimization for non-convex min- max problems: Algorithms and applications in signal processing and communications. 2018. URL http:
/pepple.cce umn. wwt / Max Min--iinnll ndff
Lu, R. Singh, X. Chen, Y. Chen, and M. Hong. Understand the dynamics of GANs via primal-dual optimization, 2019a.
//t.tp/ss//ppeeeeeee..... net. forummmiioo... 1id=rydrryy 1I33R9K7
Songtao Lu, Joannis Tsaknakis, and Mingyi Hong. Block alternating optimization for non-convex min-max problems: algorithms and applications in signal processing and communications. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 135 147. ACM, 2017.
Jan Hendrik Metzen, Mummadi Chaithanya Kumar, Thomas Brox, and Volker Fischer. Universal adversarial perturbations against semantic image segmentation. In ICCV, pp. 2774-2783, IEEE Computer Society, 2017.
Seyed Mohsen Moosavi Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), number EPHL-CONF-211800 2016.
Seyed-Mohsen Deosavi-Dezfooli,, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 86-94, 2017.
12

---

Under review as a conference paper at ICLR 2020
Maher Nouiched, Maziar Sanjabi, Jason D Lee, and Meisam Razaviyayn. Solving class of non- convex min-max games using iterative first order methods. arXiv preprint FrXXv:1902.08299) 2019.
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. arXiv preprint wrXiv:1901.088440 2019.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In Security and Privacy (EuroS&P), 2016 IEEE European Symposium pp. 372-387. IEEE, 2016a.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks, In Security and Privacy (SP), 2016 IEEE Symposium on, pp. 582-597. IEEE, 2016b.
Nicolas Papernot, Patrick McDaniel, lan Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, pp. 506-519, ACM, 2017.
Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and Trends in Optimization, (3):127-239, 2014.
Q. Qian, S. Zhu, Tang, R. Jin, B. Sun, and H. Li. Robust optimizaaiion over multiple domains. CoRR, abs/ 000.0/885,07588, 2018. /RA/tttt///arri...rr///////////
07588
H. Rafique, M. Liu, Q. Lin, and T. Yang. Non-convex min-max optimization: Provable algorithms and applications in machine learning. arXiv preprint 1rXiv:1810.02000 2018.
Ali Shafahi, Mahyar Najibi, Zheng Xu, John P. Dickerson, Larry S. Davis, and Tom Goldstein. Universal adversarial training. CoRR, abs/1811.11304, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Jost Tobias Springenberg. Alexey Dosovitskiy. Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (Workshop), 2015.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study the robustness of 18 deep image classification models. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 63 631-648, 2018.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv: 1312.6799 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pp. 1-9. IEEE Computer Society, 2015.
F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, Ensemble adversarial training: Attacks and defenses. 2018 ICLR, arXiv preprint arXiv:1705.07200. 2018.
Florian Tramèr and Dan Bonch. Adversarial training and robustness for multiple perturbations. arXiv preprint arXiv: 19044 13000, 2019.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv:1704.03455, 2017.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. arXiv preprint 1711 1719999911111111 2017.
Kaidi Xu, Sijia Liu, Pu Zhao, Pin- Yu Chen, Huan Zhang, Quanfu Fan, Deniz Erdogmus, Yanzhi
Wang, and Xue Lin. Structured adversarial attack: Towards general implementation and better interpretability. In International Conference on Learning Representations, 2019. URL BRLhttps //openreviee net. wttp://wwww.eeeeooro//o/o-o--eo.
13

---

Under review as a conference paper at ICLR 2020
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. NDSS. The Internet Society, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC. BMVA Press, 2016.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. In ICLR. OpenReview.net, 2018.
14

---

Under review as a conference paper at ICLR 2020
CONTENTS
Proof of Lemmal)
17
Proof of Proposition
17
Experiment Setup
19
Model Architectures and Training Details
19
Crafting adversarial examples
20
Details of conducted data transformations
20
D
Supplementary Results Robust adversarial attacks
21
D.1
Ensemble attack over multiple models)
21
D.2
Comparison with heuristic weighting schemes
21
D.3
Robust adversarial attack over data transformations
22
D.4
Sensitivity analysis of regularizer on probability simplex
22
E
Supplementary Results Adversarial training against multiple types of adversarial at-
tacks
23
Robustness evaluation
23
Comparison with universal adversarial training (UAT)
23
Overlap of norm balls
24
Learning curve under different traming schemes
25
Interpretability of domain weight w on universal perturbation to multiple images
25
LIST OF FIGURES
A1
Sensitivity analysis of the regularizer on the probability simplex (Sec.
23
A2 Violin plot of weights w in AMPGD & overall model robustness under different AT schemes (LeNet. Sec. [4.2]
23
A3
Overlapping of adversarial examples under different t, balls (Sec. 4.2
24
Learning curves of MLP under different AT schemes (faster convergence, Sec.
25
LIST OF TABLES
A1
Neural network architectures used in the paper (Sec.
20
A2
Clean test accuracy of the DNN models (Sec.
20
A3
Ensemble attack over multiple models (A, E, F, H) (Sec.
21
A4
Comparison with heuristic weight schemes on ensemble attack (Sec. 4.1)
22
A6
Robust attack over multiple transformations (Sec. [.2]
22
A7
Comparison with universal adversarial training (Sec. [.2]
24
A8
Generalized adversarial training under mixed types of perturbations (Full table, Sec.
4.2
26
15

---

Under review as a conference paper at ICLR 2020
A9 Interpretability of domain weight for universal perturbation (digits 0-4, Sec. (4.1)
27
A10 Interpretability of domain weight w for universal perturbation (digits 5-9, Sec.)4.1)
28
16

---

Under review as a conference paper at ICLR 2020
A
PROOF OF LEMMA I
Lemma 1. Problem (O) is equivalent to
minimize (x,))ED maximize weP (6..
Σ ue (0, x,y),
where w E RK represent domain weights, and P has been defined in T
Proof of Lemma 7
Similar to ( problem 10 is equivalent to
minimize (x,y))D maximize WEP
wy F()).
(18)
Recall that F.(0): maximizes, fir x, v), problem can then be written
minimize (xY))D maximize WEP
n0 maximize (e, 6,; y)]- 8,EA,
(19)
According to proof by contradiction, is clear that problem
19 is equivalent to
K
minimize E(xy)eD maximize (6.EX.
Σ
We (0, x,;
(20)
B PROOF OF PROPOSITION
Proposition Given a point a E Rd and a constraint set X {8|||81|p é, < s é}, the
Euclidean projection 8 proix(() (a) has the closed-form solution when {0,1,2)
1) 1fp 1, then 8* is given by
8; = =
(a)a;(aa)) Peree (sign(a;) max {lail - \1, 01)
Ei (at Se
(21)
otherwise,
where x; denotes the ith element of vector x; (-) denotes the clip function over the in- terval [2,, &); sign(x) = if 0, otherwise O; d1 max; |asl e/cl is the root of Dd
max {lay1 (1,,),,
2) If 2, then 8* is given by
8 = {
(ai)))a;) Plane (a:/(A2 + 1))
Et (P c²
(22)
otherwise,
where X2 (0, ||al|//c 1] is the root of Ei-i (Pr (as/ (2e + 1)))² = ?
3) 1fp = 0 and e E N4, then 8* is given by
{ 8 n 2 [n]
20z0: 22
04 < c
8
0 otherwise,
T
20,0;
22
0; c
(23)
aail
otherwise.
where (n] denotes the e-th largest element of n, and 8, (ae)
Proof of Proposition 7
17

---

Under review as a conference paper at ICLR 2020
1 norm When we find the Euclidean projection of onto the set X, we solve
minimize
1 | a + (,,a((0)
(24)
subject
||ō||
where (.) the indicator function of the set (é, 6). The Langragian of this problem is
L = 6 al2 + (c. + (1 - c)
(25)
Σ 2 (b: aa)² A11o01 ((,,)(00)) λiE. 1=1
(26)
The minimizer 8 minimizes the Lagrangian, is obtained by elementwise woft-thresholdng
8 Pe.. (sign( (a;) max {|as] \1. 0+).
where x; is the ith element of a vector x, P(.,e (-) is the clip function over the interval [č,, 2.].
The primal, dual feasibility and complementary slackness
\ 0. ||
Σ
16
P (aoaa
(27)
f=1
d
or \ > 0,11811 =
16
((egaa)sign(()) (sign(a;) max {|ail - -1, =e.
(28)
1=1
If Ei | (a:)) S 6, = a (a:). Otherwise 8
(sign( max {lasl - 'o 0}),
where A is given by the root of the equation Σ Peer,a (sign(a;) max [aall - \1, 0}) c.
Bisection method can be used to solve the above equation for Å1, starting with the initial interval
(0, max, |o,1 e/d]. Since Σ (Pgg,)))sig((,,) e in this case, and >n ( (a;) max {las] max; |as| + t/d,
(sign(a,) max {|a 0,0))] Σ aP
Edh (PPaaa((sgn((a(((/))) s Ed (c/d) c
& norm
When we find the Euclidean projection of onto the set X, we solve
minimize
||6 a a12 + (()
(29)
subject to || 2,
where (.) the indicator function the set [é, c). The Langragian of this problem is
L 15 all2 (() + >2(110113
2)
(30)
d
Σ (6 - a;)² 2 + x + (o,,)(())) - A2
(31)
The minimizer 8* minimizes the Lagrangian, it is
5r = Plaee
Xo + a.).
The primal, dual feasibility and complementary slackness are
A2 = 0,11,112 =
Σ ((Pe.a)(a:))2 5
(32)
1=1 1=1
or b2 > 0,118112
8 (Per
A2 +1 az))² =e?
(33)
18

---

Under review as a conference paper at ICLR 2020
If (E=I(PEE,)(a)) & =,, 6 = (a))))(oi). Otherwise of = Peraal] ( where X2 is given by the root of the equation Eiz( (P....)) X22- az))² e. Bisection method can be used to solve the above equation for 22, starting with the initial interval (0, v Ei=n(a)))/ 1). Since
E EP-i(Ple, (04) = (P)).))(a))) > e² in this case, and Σ (Pla,e ( lx+1
E EP=I(PA E. (a))))))) S (a)
tp norm For Ey norm in X, it is independent to the box constraint. So we can clip a to the box constraint first, which is 8/ Pes. (a:), and then project it onto Eo norm.
We find the additional Euclidean distance of every element in a and zero after they are clipped to the box constraint, which is
Va (a; 62)
a, à a à otherwise.
7
Vag (a; las
6;)2
(34)
It can be equivalently written as
v 2a,02 d2
< c 0;
7/
ai > & otherwise.
(35)
|ad
To derive the Euclidean projection onto &p norm, we find the e-th largest element in n and call it [n].. We keep the elements whose corresponding 7); is above or equals to e-th, and set rest zeros. The closed-form solution is given by
{
0 2 [n]e
8 =
(36)
otherwise.
Difference to Hein & Andrüushchenko 2017 Proposition 4.1). We remark that Hein & An-
driushchenko 2017) discussed a relevant problem of generating f--norm based adversarial examples under box and linearized classification constraints. It was shown in (Hein & Andriushchenko [017] Proposition 4) that the problem is convex and the solution can be derived using KKT conditions. However, Proposition[]in in our paper is different from (Hein & Andriushchenko 2017 Proposition 4.1). First, we place fp norm as a hard constraint rather than minimizing it in the objective function. This difference will make our Lagrangian function more involved with newly introduced non- negative Lagrangian multiplier. Second, the problem of our interest is projection onto the intersection of box and Ep constraints. Such projection step can then be combined with attack loss (no need of linearization) for generating adversarial examples. Third, we cover the case of to norm.
C EXPERIMENT SETUP
C.I MODEL ARCHITECTRRES AND TRAINING DETAILS
For comprehensive evaluation of proposed algorithms, adopt set of diverse DNN models (Mode] A to H), including multi-layer perceptrons (MLP), AlI-CNNs Springenberg et al. (2015). LeNet Lecun al. 19988, _ VGG16 VGGi6\Simonyan Zisserman (2015). ResNet50 He et al. (2016), Wide-Rest models, we use the exact same architecture as original papers and evaluate them only on CIFAR-10 10 dataset. The details for model architectures are provided in Table AT For compatibility with our framework, we implement and train these models based on the strategies adopted in pytorch-cifaf" and achieve comparable performance clean images; see Table A2 To foster reproducibility. all the trained models are publicly accessible in the anonymous link. Specifically, we trained MNIST
Madry et al. (217)) and GoogLeNet (Zegeddy et al. (2015). For the last four
An enhanced version of original LeNet with more layers and units (see Table A1 Model D).
https: / //gi thwb. http://uuanggiugytttccchh--e far
19

---

Under review as a conference paper at ICLR 2020
classifiers for 50 epochs with Adam and constant learning rate of 0.001. For CIFAR-10 classifers, the models are trained for 250 epochs with SGD (using 0.8 nesterov momentum, weight decay 5e 4). The learning rate is reduced epoch 100 and 175 with a decay rate of 0.1. The initial learning rate set as 0.01 for models [A, B, C, D, H) and 0.1 for (E, F, G). Note that no data augmentation is employed in the training.
Table AI: Neural network architectures used on the MNIST and CIFAR-10 dataset. Conv: convolutional layer,
FC: fully connected layer, Globalpool: global average pooling layer.
A (MLP)
B (AII-CNNs, 2015
C (LeNet. 1998
D (LeNetV2)
FC(128) Relu
Conv([32, 64), 3. 3) Relu
Conv(6, 5, 5) + Relu
Conv(32. 3, 3) Relu
FC(128) Relu
Conv(128, 3, 3) (ropout(0.5)
Maxpool(2, 2)
Maxpuol(2, 2)
FC(64) + Relu
Conv([128, 128), 3, 3) + Relu
Conv(16, 5, 5) + Relu
Conv(64, 3. 3) Relu
FC(10)
Conv(128, 3, 3) (ro)
Maxpool(2. 2)
Maxpuol(2, 2) FC(128) Relu
Softmax
Conv(128, 3) 3)+ Relu
FC(120) Relu
(1nv(128, 1, + Relu
PC(84) + Relu
Dropout(0.25) FC(10)
Conv(10, 1, 1) Globalpool
FC(10)
Softmax
Softmax
Softmax
E (VGG16, 2015
(ResNet50, 2016
G (Wide-ResNet, 2017
II (GoogLeNet, 2015
Table A2: Clean test accuracy of DNN models on MNIST and CIFAR-10 10.
We roughly derive the model
robustness by attacking models separately using FGSM FGSM[Goodielool Gooddellow et al. 2014).
The adversarial examples are
generated by FGSM Exx -atackk (c- = 0.2).
MNIST
CIFAR-I 10
Model
Acc.
FGSM
Model
Acc.
FGSM
Model
Acc.
FGSM
A: MLP
98.20%
18.92%
A: MLP
55.36%
11.25%
E: VGG16
87.57%
10.83%
B: All-CNNs
99.49%
50.95%
B: All-CNNs
84.18%
9.89%
F: ResNet50
88.11%
10.73%
C: LeNet
99.25%
63.23%
C: LeNet
64.95%
14.45%
G: Wide-ResNet
91.67%
15.78%
LeNetV2
99.33%
56.36%
D: LeNetV2
74.89%
9.77%
H: GoogLeNet
90.92%
9.91%
C.2 CRAFTING ADVERSARIAL EXAMPLES
We adopt variant C&W loss in APGD/PGD as suggested in Madry et al. (217); Carlini & Wag ner (207)) with confidence parameter implementation. The adversarial examples are generated by 20-step PGD/APGD unless other- wise stated (e.g., 50 steps for ensemble attacks). Note that proposed algorithms will not be affected largely by the choices of hyperparameters (a, B, 7). In consequence, we do not finely tune the parameters on the validation set. Specifically, The learning rates a, 3 1
50. Cross-entropy loss is also supported in
robust and
and regularization factor for Table | are set as (a) MNIST: to : a 1,3 = 1000 7 = 7,
41 : 3 100: 7 5, l2
10 100 7 3; lo = 1: 1
1
50.7 3; (b) 1
CIFAR- 10: to : = 1,3 1 1,6 1.8 100 7= 5, 82 a 1.3 100 7 3;
lx : a 5 7 7 =6
Due to varying model robustness on different datasets, the perturbation magnitudes are set sepa- rately Carlini et al. (2019). For universal perturbation experiments, the are set as 0.2 (A, B), 0.3 (C) and 0.25 (D) on MNIST; 0.02 (B, H), 0.35 (E) and 0.05 (D) on CIFAR- 10. For generalized AT, the models on MNIST are trained following the same rules in last section, except that training epochs are prolonged to 350 and adversarial examples are crafted for assisting the training with a ratio of 0.5. Our experiment setup is based on CleverHans packag and Carlini and Wagner's framework
C.3 DETAILS OF CONDUCTED DATA TRANSFORMATIONS
To demonstrate the effectiveness of APGD in generating robust adversarial examples against multiple transformations, we adopt series of common transformations, including a&b) flipping images
20

---

Under review as a conference paper at ICLR 2020
horizontally (fth) and vertically (ffv); c) adjusting image brightness (bri); d) performing gamma correction (gam), e) cropping and re-sizing images (crop); f) rotating images (rot).
Moreover, both deterministic and stochastic transformations particular, Table and Table|A6jan deterministic settings - rot: rotating images 30 degrec clockwise; crop: cropping images in the center (0.8 x 0.8) and resizing them 32 x 32; bri: adjusting the brightness of images with a scale of 0.1; gam: performing gamma correction with a value of 1.3. Differently, in Table A5 we introduce randomness for drawing samples from the distribution rot: rotating images randomly from -10 to 10 degree; crop: cropping images in the center randomly (from 0.6 1.0); other transformations are done with probability of 0.8. In experiments, we adopt tf. image [ processing the images.
considered in our experiments. In
D SUPPLEMENTARY RESULTS ROBUST ADVERSARIAL ATTACKS
D.1 ENSEMBLE ATTACK OVER MULTIPLE MODELS
Table A3 shows the performance of average (ensemble Liu ct al. (2018) and min-max (APGD) strategies for attacking model ensembles. Our min-max approach results in 15.69% averaged improvement on ASRait models (A, E, F, H) CIFAR-10.
Table A3: Comparison of average and min- (APGD) ensemble attack over four models CIFAR- 10. Acc (%) represents the test accuracy of classifiers on adversarial examples. The learning rates a, and regularization
factor y are set as fo : a 1 = 1 ? = 1, f a = 4.13 105 7 5, lg : a . 105+ 7 = 3;
for 5 B 7 6. The attack iteration for APGD set 50.
Box constraint
Opt.
Acc
Accæ
Acc
Accи
ASRal
Lift (t)
to (c = 70)
avg.
27.38
6.33
7.18
6.99
66.56
min max
19.38
8.72
9.48
8.94
73.83
10.92%
E (E 30)
avg.
30.90
2.06
1.85
1.84
66.23
min max
12.56
3.21
2.70
2.72
83.13
25.52%
f (E = 1.5)
ang.
20.87
1.75
1.21
1.54
76.41
min max
10.26
3.15
2.24
2.37
84.99
11.23%
Lx 0.03)
ang.
25.75
2.59
1.66
2.27
70.54
min max
13.47
3.79
3.15
3.48
81.17
15.07%
D.2 COMPARISON WITH HEURISTIC WEIGHTING SCHEMES
To further demonstrate the effectiveness of self-adjusted weighting factors in proposed min-max framework, we compare with heuristic weighting schemes in Table Specifically, with the prior knowledge of robustness of given models (C> > A > B), we devised several heuristic baselines including: (a) We+d' ensemble PGD models C and D only; (b) Wo+c+di ensemble PGD on models A, and D only; (c) Welipi clipped version of C&W loss (threshold / = 40) to balance model weights in optimization
suggested in Shafahi et al. (2018); (d) Wprior: larger weights the more
robust models, Uprior = (. WB, WC, WD = 00.2, 0.1, 0.4, 0.3); (e) W the converged mean weights of min-max (APGD) ensemble attack. For &a (e 3.0) and foo (E 0.2) attacks, Watatic [WA, WB, we, WD] are [0.209 0.046, 0.495, 0.250] and [0.080, 0.076, 0.541, 0.303], respectively.
Table A4 shows that our min-max approach outperforms all static heuristic weighting schemes by a large margin. Specifically, our min-max APGD also achieves significant improvement compared to Wsstotic setting, where the converged optimal weights are statically (i.c.. invariant w.r.t different images and attack procedure) adopted. It again verifies the benefits of proposed min-max approach by automatically learning the weights for different examples during the process of ensemble attack generation (see Figure Ic).
21

---

Under review as a conference paper at ICLR 2020
Table A4: Comparison of average. min-max (APGD) ensemble attack and some heuristic weighting schemes over four models on MNIST. (%) represents test accuracy of classifiers
adversarial examples. Wo4d and waeo4d: average ensemble attack over models {C. D) and [A, C. DJ; Welep clipped version of
C&W loss (Shafahi ct al. 2018 Wprior: larger weights on the models that are more difficult to attack
[WA, up, Wo, Wp] (0.2, 0.1, 0.4, 0.3]; Mistatcc converged mean weights (over all data) in min max ensem-
ble attack. The experimental setting the same Table
Box construint
Opt.
Acc,
Accp
Accc
Accp
ASR
ASR
Lift (T)
avg.
6.88
0.03
26.28
14.50
88.08
69.12
Woted
69.03
14.58
5.11
0.34
77.74
28.65
-58.56%
Watord
1.34
24.53
11.69
2.79
89.91
67.45
-2.42%
3.0)
Walp
2.70
12.69
4.13
95.12
85.33
23.45%
Mprior
6.28
0.05
6.78
2.65
96.06
88.25
27.68%
Watatic
4.52
0.27
3.35
4.15
96.93
90.53
30,98%
min max
1,51
0.89
3.50
2.06
98.01
95.31
37.89 %
avg-
1.05
0.07
41.10
35.03
80.69
48.17
Wo+dd
60.37
19.55
15.10
1.87
75.78
29.32
339.13%
Waanta
0.46
21.57
25.36
13.84
84.69
53.39
10.84%
800 (e 0.2)
Walip
0.66
0.03
23.43
13.23
90.66
71.54
48.52%
Wprior
1.57
0.24
17.67
13.74
91.70
74.34
54,33%
Mstaticc
10.58
0.39
9.28
10.05
92.43
77.84
61.59%
min max
2.47
0.37
7.39
5.81
95.99
90.16
87.17%
D.3 ROBUST ADVERSARIAL ATTACK OVER DATA TRANSFORMATIONS
Table A5 and A6 compare the performance of average (EOT Athalye et al. (0188)) and min- max (APGD) strategies. Our approach results in 4.31 % and 8.22% averaged lift over four models fA, B, C. DJ CIFAR-10 under given stochastic and deterministic transformation sets.
Table AS: Comparison of average and min min-max optimization on robust attack over multiple data transformations on CIFAR-IO. Note that all data transformations are conducted stochastically with a probability of 0.8, except for crop which randomly crops central area from original image and re-size it into 32 x 32. The adversarial examples generated by 20- step fo APGD 0.03) with 2.
and 7 10.
Model
Opt.
Accort
Accyis
Aue fte
Accari
AcCerop
ASRavg
ASR
Lift (t)
11.55
21.60
13.64
12.30
22.37
83.71
55.97
min max
13.06
18.90
13.43
13.90
20.27
84.09
59.17
5.72%
B
avg.
6.74
11.55
10.33
6.59
18.21
89.32
69.52
min max
8.19
11.13
10.31
8.31
16.29
89.15
71.18
2.39%
8.23
17.47
13.93
8.54
18.83
86.60
58.85
min max
9.68
13.45
13.41
9.95
18.23
87.06
61.63
4.72%
D
ang-
8.67
19.75
11.60
8.46
19.35
86.43
60.96
min MAX
10.43
16.41
12.14
10.15
17.64
86.65
63.64
4.40
Table A6: Comparison of average and min-max optimization on robust attack over multiple data transformations CIFAR-10. Here new rotation (rot) transformation is introduced, where images are rotated 30 degrees clockwise. Note that all data transformations conducted with probability of 1.0. The adversarial examples and Y
are generated by 20-step l APGD (c 0.03) a 1
T06
Model
Opt.
Acc.rt
Acc
Acc
Acc
Acc gemn
ACC
Acc
ASR
ASR
Lift (1)
A
avg.
11.06
22.37
14.81
12.32
10.92
20.40
15.89
84.60
49.24
min max
13.51
18.84
14.03
15.20
13.00
14.79
84.66
52.31
6.23%
B
aug.
5.55
11.96
9.97
5.94
16.42
11.47
90.44
65.18
min max
6.75
9.13
10.56
7.11
10.80
90.96
70.38
7.98%
C
avg.
7.65
22.30
15.82
8.17
8.07
15.44
15.09
86.78
49.67
min
9.05
15.10
14.57
9.57
9.31
14.11
14.23
87.72
55.37
11.48%
D
ang-
8.22
20.88
13.49
8.71
16.33
14.98
87.07
$3.52
min
10.17
14.65
13.62
10.03
10.35
14.36
13.82
87.57
$7.36
7.17%
D.4 SENSITIVITY ANALYSIS OF REGULARIZER ON PROBABILITY SIMPLEX
22

---

To further explore the utility of quadratic regularizer on the probability simplex in proposed min-max frame- work, conducted sensitivity analysis y and show how the proposed regularization affects the eventual performance (Fig taking ensemble attack as example. The experimental setting is the same as Ta- ble D except for altering the value of y from 0 10. Figure A1 shows that too small or too large leads to relative weak performance due the unstable conver- gence and penalizing too much for average case. When y is around 4, APGD will achieve the best performance so we adopted this value in the experiments (Table Under review as a conference paper at ICLR 2020
0.880
0.875
Attack
0.865
0.860
2
10
Figure A1: Sensitivity analysis of the regular-
Moreover, when y + co, the regularizer term dom- inates the optimization objective and it becomes the average case.
||w 1/K 1|ž on the probability simplex.
E SUPPLEMENTARY RESULTS ADVERSARIAL TRAINING AGAINST MULTIPLE TYPES OF ADVERSARIAL ATTACKS
E.1 ROBUSTNESS EVALUATION
Figure A2 presents "overall robustness" comparison of our min-max generalized AT scheme and vanilla AT with single type of attacks (fo and E2) MNIST (LeNet). Similarly, min-max training scheme leads to higher "overall robustness" measured by Sc- In practice, due to the lacking knowledge of the strengths/types of the attacks used by adversaries, it is meaningful enhance "overall robustness" of models under the worst perturbation (Acenax). Specifically, our min-max S, compared to single-type AT with
generalized AT leads 6.27% and 17.63% improvement eo and & attacks. Furthermore, weighting factor w of the probability simplex helps understand the behavior of AT under mixed types of attacks. Our AMPGD algorithm will adjust automatically according to the min-max principle defending the strongest attack. In Figure A2a 68, increases,
f2-attack becomes stronger so its corresponding w increases as well. When 6t2 2 2.5, f2-attack dominates the adversarial training process. That is to say, our AMPGD algorithm will put more weights on stronger attacks even if the strengths of attacks are unknown, which meritorious feature in practice.
1.00
0.75
0.8
0.8
$ 0.50
AcC
0.6
0.6
natural
adv [f.)
0.2
adv (minmax)
1.0 1.5
2.0
2.5
3.0
3.5
4.0
0.5
2.5
1.0
2.5
alt))
((ts)
alta)
weight {w.)
max
(c) Ace
Figure A2: (a): Violin plot of weight w in APGD as a function of perturbation magnitude e of & attack in adversarial training; (b) & (c): Robustness of LeNet (Model C) under different adversarial training schemes.
Table A8 shows complete results on the test accuracy of models in different training schemes. In general, the min-max generalized AT obtains better performance than averaging strategy. AMPGD always leads to Top-2 Ace and "vg adv
E.2 COMPARISON WITH UNIVERSAL ADVERSARIAL TRAINING (UAT)
Shafahi al. (0018) also propose variant of adversarial training to defend universal perturbations over multiple images. To produce universal perturbations, they propose uSGD to conduct gradient descent the averaged loss of one-batch images. In consequence, their approach can be regarded as
23

---

Under review as a conference paper at ICLR 2020
variant of our generalized AT in average case. The difference is that they do AT across multiple adversarial images under universal perturbation rather than mixed by fy-norm perturbations.
We added UAT [1] generated by uSGD (fx norm, e = 0.3) with a batch size of 128 following (Shfahii ct al. (2018). We find that our proposed approach outperforms UAT under per-image E attacks. Taking A as an example, our avg and min max generalized AT (with DPAR) result in average 17.85% % and 17.97% improvement in adversarial lest accuracy (ATA), b) our approach has just 3.72% degradation in ATA when encountering universal attacks, and <) both methods yield very similar normal test accuracy. It not surprising that our average and min- max training schemes can achieve better overall robustness while maintaining competitive performance on defending universal perturbation. This is because the defensed model is trained under more general (fp norm) and diversity promoted perturbations. As a result, proposed generalized AT' is expected to obtain better overall robustness and higher transferability as shown Table and A7
of our defense baselines in Table A7 The universal perturbation is
Table A7: Adversarial training of MNIST models on single attacks (foc and ε2), multiple attacks (avg. and min max) and universal perturbation (wni). The perturbation magnitude for & and Eg attacks are 0.2 and 2.0, respectively. uSGD indicates universal adversarial training following Shafahi al. (2018) (foo norm, = 0.3). Top 2 test accuracy on each metric are highlighted.
(a) MLP
(b) LeNet
Opt.
Acc.
Acc- &oo
Acc-Fa
Acc-uni
Acc
Dvg
Opt.
Acc.
Acc- foo
Acc-f2
Acc-umi
Acc
natural
98.30
2.70
13.86
21.61
12.72
natural
99.25
17.93
39.32
58.93
38.73
Ex
98.08
77.70
69.17
90.90
79.26
99.18
93.80
78.97
98.70
90.49
82
98.72
70.03
81.74
82.49
78.09
Fy
99.22
85.84
87.31
96.63
89.93
uSGD (Csc)
98.73
56.21
64.51
91.27
70.66
uSGD (foc)
99.44
72.81
66.39
98.37
79.19
any- DPAR
98.62
79.00
86.69
80.26
ang-
99.22
88.96
85.59
97.41
90.65
98.50
76.75
79.67
87,88
81,44
DPAR
99.25
89.96
86.49
97.36
91.27
min max
98.59
75.96
79.15
83
80.41
min max
99.32
89.21
85.98
98.22
91.14
DPAR
98.58
76.92
79.74
87.55
81.40
DPAAR
99.22
90.19
86.47
97.77
91.48
E.3 OVERLAP OF 2--NORM BALLS
As reported in Sec. our min-max generalized AT does not always result in the best performance on the success rate of defending the worst/strongest perturbation (Ace ) for given (Ct,., <t2) pair, especially when the strengths of two attacks diverge greatly (e.g., for loo and &g attacks are 0.2 and 0.5). In what follows, we provide explanation and analysis about this finding inspired by recent work Araujo al. (2019).
2.1
example
0.8
0.8
2.0
0.8
0.6
0.4
0.4
adv. example U3)
0.4
0.2
0.2
0.0
0.5
2.0
0.55
2.0
0.5
2.0
cl6)1
clt,)
((h)
(a) 20 E 82
(b) 82 E 80o
(c) average Eg- fg-norm
Figure A3: (a) & (b): Comparison of the percentage of adversarial examples inside Eao ball (left, blue area) and inside &a ball (right, red area). particular, the (blue) area (a) (or (b)) represents the percentage of adversarial examples crafted by (f2) attack that also belong to & (f ball. We generate adversarial examples on 10,000 test images for each attack. (c): Average lp norm of adversarial examples as a function of perturhation magnitude ct.- The top (bottom) side represents the f2-norm (foo) of the adversarial examples generated by f. (Ez) attack et, for generalized AT increases. Note that the same as the AT procedure is used while attacking trained robust models.
Figure A3 shows the real overlap of foo and lg norm balls in adversarial attacks for MLP model on MNIST. Ideally, if ee, satisfies et_, < 622 cew Vd, loo and eg balls will not cover each other
24

---

Under review as a conference paper at ICLR 2020
completely Araujo al. (2019). In other words, AT with loo and l attacks cannot interchange with each other. However, the real range of et, for keeping lo and & balls intersected is not (Gew. ftx x vd), because crafted adversarial examples are not uniformly distributed in f--norm balls. In Figure A3bl 99.98 99.98% adversarial examples devising using & attack are also inside 6oo ball, even if 0.2 < Eg2 0.5 < 5.6. In consequence, AT with &oo attack is enough to handle fy-attack in overwhelming majorty cases, which results in better performance than min-max optimization (Table A8a)
Figure A3c presents the average lp distance of adversarial examples with Et, increasing. The average 22-norm (green líne) of adversarial examples generated by &~2 attack remains around 2.0 with a slight rising trend. This consistent to our setting fixing çt, as 0.2. It also indicates model robustness et, increases, robustly trained MLP model becomes more
may effect the behavior of attacks robust against E2 examples, so the &o attacker implicitly increases lg norm to attack the model more effectively. On the other hand, the average &o -nom increases substantially to 2.5. When Et, arriving at 0.85, the average &oo norm gets close to 0.2, so around half adversarial examples generated by 22-attack are also inside t. balls, which is consistent with Table A3b
et, increases from 0.5
E.4 LEARNING CURVE UNDER DIFFERENT TRAINING SCHEMES
Figure A4 shows the learning curves of model A under different AT schemes, where two setting are plotted: (a) (ft.. ff2) (0.2, 0.5); (b) (ft..: = (0.2, 2.0). Apart from better worst-case robustness shown in Table A8 our min-max generalized AT leads to faster convergence compared to average-based AT, especially when the strengths of two attacks diverge greatly. For instance, when Et, 0.5 (Figure A4a), the robust model trained with AMPGD reaches 70% test accuracy on the worst perturbation (I-Rmex) within 210 epochs versus 280 epochs in average setting. When cE2 2.0 (Figure ( the learning curves for min-max and average strategy are very close because the strengths of two attacks are similar, which is verified by approximately equal weights in Figure 3a
1.0
1.0
0.8
0.8
Acc
0.6
Accita
0.4
(L.)
0.4
(t,) adv
0.2
(minmax)
0.2
adv (minmax) adv (avg)
adv (avg)
0.0
0.0
100
200
300
0
100
200
300
epochs
epochs
(a) (er (12) = (0.2, 0.5)
(b) (er 422) = (0.2, 2.0)
Figure A4: Learning curves MLP model under different adversarial training schemes on MNIST, Note that each experiment is repeated ten times with different random seeds.
F INTERPRETABILITY OF DOMAIN WEIGHT w ON UNIVERSAL PERTURBATION TO MULTIPLE IMAGES
Tracking domain weight w of the probability simplex from our algorithms is an exclusive feature of solving problem I In Sec. 4 we show the strength of w in understanding the procedure of optimization and interpreting the adversarial robustness. Here we would like to show the usage of in measuring "image robustness" on devising universal perturbation to multiple input samples. Table A9 and A10 show the image groups on MNIST with weight w in APGD and two metrics (distortion of & minimum for PPGD of measuring the difficulty of attacking single images. The binary search is utilized to searching for the minimum perturbation.
Although adversaries need consider a trade-off between multiple images while devising universal perturbation, we find that weighting factor w in APGD is highly correlated under different é, norms. Furthermore, is also highly related to minimum distortion required for attacking single image
25

---

Under review as a conference paper at ICLR 2020
Table A8: Adversarial training of MNIST models with single attacks (foe and 62) and multiple attacks (aog. and min max). During the training process, the perturbation magnitude ét.,, is fixed as 0.2, and 6t, are changed crafted using 20-step
from 0.5 to 3.0 with step size of 0.5. For min-max scheme, the adversarial examples looD with g. 8 50 and y The ratio of adversarial and benign examples in adversarial training set as 1.0. For divvrsity-promoting attack regularizer (DPAR) in generalized AT, the hyperparameter À 0.1.
(a) (E E22) (0.2, 0.5)
(b) (ELo 622
(0.2, 1.0)
Model
Opt.
Acc.
Acc-f
Acc-fq
Acc
Acc
Acc.
Acc-E30
Acc-fz
Acc
Acc
natural
98.28
2.78
93.75
1.80
48.27
98.30
3.65
72.39
1.17
39.01
foo
98.22
77.82
97.11
77.23
87.46
98.29
78.15
93.28
77.95
85.71
82
98.71
12.04
97.10
11.73
54.57
98.98
36.02
94.39
34.68
65.20
MLP
aeg--
98.83
74.07
97.70
73.67
85.88
98.77 72
73.97
94.63
73.70
84,30
+ DPAR
98.56
77.32
97.74
76.98
87.53
98.66 60
76.57
94.41
76.39
85.49
min max
98.73
75.88
97.43
75.56
86.66
98.77 72
75.18
94.29
74.92
84.74
DPAR
98.75
77.04
97.81
76.72
87.43
98. 68
76.59
95.11
76.49
85,85
natural
99.17
18.16
97.56
15.23
57.86
9.16
18.24
89.97
15.36
54.10
foc
99.27
93.60
98.74
93.26
96.17
99.28
93.51
96.49
93.13
95.00
82
99.43
34.30
98.49
26.89
66.39
99.50
63.48
96.62
57.94
80.05
LeNet
aung-
99.29
90.69
98.89
90.34
94.79
99.40
89.39
96.94
89.02
93.16
+ DPAR
99.28
91.81
98.87
91.52
95.34
99.38
99.09
97.13
89.99
93.61
min max
99.35
90.81
98. 74
90.21
94.78
99.31
90.82
97.20
90.56
94.01
+ DPAR
99.34
91.82
98.77
91.60
95.30
99.35
90.88
97.07
90.80
93.98
(c) (Er 622)
(0.2, 1.5)
(d) (E 622
(0.2, 2.0)
Model
Opt.
Acc.
Acc-foo
Acc-f2
Acetmax
Accuis
Acc.
Acc-f30
Acc-fz
Aocaay
Acc
natural
98.39
2.77
35.70
2.32
19.23
98.30
2.70
13.86
0.85
8.28
foo
98.34
78.96
85.94
77.42
82.45
98 08
77.70
69.17
66.34
73.43
f2
99.00
60.37
89.96
59.82
75.16
98.72
70.03
81.74
69.14
75.88
MLP
aung-
98.61
75.01
88.85
74.76
81,93
98.62
75.09
79.00
72.23
77.05
+ DPAR
98.68
76.55
88.52
76.18
82.53
98.50
76.75
79.67
74.14
78.21
min max
98.76
75.66
88.78
75.33
82.22
98.59
75.96
79.
73.43
77.55
+ DPAR
98.77
77.54
89.57
77.24
83.55
98.58
76.92
79.74
74.29
78.35
natural
99.22
14.31
67.69
12.34
41.00
99.25
17.93
39.32
17.57
28.63
too
99.22
93.76
91.11
90.26
92.43
99.18
93.80
78.97
78.80
86.39
f2
99.35
79.92
93.27
77.39
86.60
99.22
85,84
87.31
84.06
86.58
LeNet
azig.
99.31
89.26
93.29
88.77
91.28
99.22
88.96
85.59
84.29
87.28
DPAR
99.27
90.75
93.44
89.96
92.11
89.96
86.49
85.44
88,23
min max
99.40
89.83
92.96
89.00
91.39
99.32
89.21
85.98
84.82
87.60
+ DPAR
99.35
90.64
93.27
89.80
91.96
99.22
90.19
86.47
85.47
88.33
(e) (e 622)
(0.2, 2.5)
(1) (cr.o (F2) = (0.2, 3.0)
Model
Opt.
Acc.
Acc-
Acc-fz
Acc
Acc
Acc.
Acc-
Acc-tz
Aoc
Acc
natural
98.31
3.37
6.02
2.27
4.70
98.24
2.92
2.42
1.54
2.67
foo
98.25
77.91
51.28
49.40
64.59
98.35
79.15
32,58
31.23
55.86
f2
98.10
73.94
70.01
67.66
71.97
97.55
73.86
58.24
57.83
66.05
MLP
ang-
98.47
75.35
64.39
63.37
69.86
98.17
75.07
49.75
49.49
62.41
+ DPAR
98.18
76.33
66.49
65.54
71.41
97.85
74.61
51.16
51.04
62.89
min
98.44
75.48
66.12
64.99
70.80
74.71
$0.45
50.54
62.58
+ DPAR
98.20
76.98
66.42
65.55
71.70
97.97
76.13
$1.12
51.00
63.63
natural
99.23
15.25
16.08
11.16
15.67
99. 24
13.76
4.74
2.57
9.25
fxx
99.18
94.09
60.18
58.47
77.13
99.30
93.14
39.48
32,93
65.81
Eg
98.94
87.57
78.45
78.42
83.01
98.55
87.87
68.69
68.34
78.28
LeNet
aetg.
99.10
89.88
74.68
74.39
82.28
99.10
89.19
59.87
60.01
74.53
+ DPAR
99.14
90.17
75.16
75.09
82.67
98.95
89.80
62.21
61.19
75.50
min max
99.21
88.88
74.97
74.42
81.93
99.01
88.93
61.15
60.76
75.04
+ DPAR
99.09
89.34
75.55
75.45
82.45
98.98
89.53
63.22
63.18
76.37
26

---

Under review as a conference paper at ICLR 2020
successfully. It means the inherent "image robustness" exists and effects the behavior of generating universal perturbation. Larger weight w usually indicates an image with higher robustness (e.g., fifth 'zero' in the first row of A9), which usually corresponds to the MNIST letter with clear appearance (e.g., bold letter).
Table A9: Interpretabillity of domain weight wo for universal perturbation to multiple inputs on MNIST (Digit O to 4). Domain weight w for different images under -orro (p 0, 1, 2, oc) and two metrics measuring the difficulty of attacking single image recorded, where dist. (f2) denotes the the minimum distortion of successfully attacking images using C&W (l2) attack; emin (foc) denotes the minimum perturbation magnitude for 2e PPG attack.
O
Image
Eg
0.
0.
0.
1.000
0.248
0.655
0.097
0.
0.
0.
0.
1,000
0.07
0.922
0.
0.
Weight
E2
0.
0.
0.
0.
1.000
0.441
0.248
0.156
0.155
0.
0.
0.
0.
0.
1.000
0.479
0.208
0.145
0.168
dist. W E2)
1.839
1.954
1.347
1.698
3.041
1.545
1.982
2.178
2.349
1.050
Metric
Emin (Lxo)
0.113
0.167
0.073
0.121
0.199
0.167
0.157
0.113
0.114
0.093
1
1
Image
Eo
0.
0.
0.613
0.180
0.206
0.223
0.440
0.337
81
0.
0.
0.298
0.376
0.327
0.
0.
0.397
0.433
0.169
Weight
82
0.
0.
0.387
0.367
0.246
0.
0.242
0.310
0.195
0.253
0.087
0.142
0.277
0.247
0.246
0.
0.342
0.001
0.144
0.514
dist. f2)
1.090
1.182
1.327
1.458
0.943
0.113
1.113
1.357
1.474
1.197
Metric
Emin (fo0)
0.075
0.068
0.091
0.105
0.096
0.015
0.090
0.076
0.095
0.106
Image
2
2
2
2
2
2
2
2
2
2.
Eo
0.
1,000
0.
0.909
0.091
81
0.
1,000
0.
0.
0.
0.
0.
0.843
0.157
Weight
82
0.
0,892
0.
0.
1.108
0.
0.
0.788
0.
0.112
0.
0.938
0.
0.
0.062
0.
0.
0.850
0.150
dist. Eg)
1.335
2.552
2.282
1,229
1.884
1.928
1.439
2.312
1.521
2.356
Metric
Canin (foo)
0.050
0.165
0.110
0.083
0.162
0.082
0.106
0.176
0.072
0.171
3
3
3
3
3
3
3
3
3
3
Image
Eo
0.481
0.
0.378
0.352
0.648
f
0.690
0.
0.310
0.
0.
0.
0.093
0.205
0.701
Weight
E2
0.589
0.069
0.208
0.
0.134
0.064
0.260
0.077
0.
0,600
Eoc
0.864
0.
0.084
0.
0.052
0.079
0.251
0.156
0.514
dC& Ex)
2,267
1.656
2.053
1.359
0.861
1.733
1.967
1.741
1.031
2,413
Metric
Emin (Lxo)
0.171
0.088
0.143
0.117
0.086
0.100
0.097
0.096
0.038
0.132
4
4
4
4
4
4
4
4
el
4
Image
Eo
0.
0.
0.753
0.
0.247
0.
1.000
f
0.018
0.
0.567
0.
0.416
0.347
0.
0,589
0.063
Weight
E2
0.
0.
0.595
0.
0.405
0,346
0.
0.
0.654
0.
Exe:
0.
0.651 0.
0.
0.349
0.239
0.
0.
0.761
0.
(C& Ez)
1,558
1.229
1.939
0.297
1.303
0,940
1.836
1.384
1.079
2.027
Metric
Emin (loo)
0.084
0.088
0.122
0.060
0.094
0.115
0.103
0.047
0.125
0.100
27

---

Under review as a conference paper at ICLR 2020
Table A10: Interpretability of domain weight w for universal perturbation to multiple inputs MNIST (Digit 5 Domain weight for different images under ly-norm (p 0, 2, oc) and metrics measuring the difficulty attacking single image are recorded, where dist. (E2) denotes the minimum distortion of successfully attacking images using C&W (f2) attack; emin (fes) denotes the minimum perturbation magnitude for tx PPDD attack.
5
S
5
5
5
5
S
5
S
5
Image
Eg
0.062
0,254
0.
0.684
0.457
0.
0.542
0.
8
0.131
0.250
0.
0.
0.619
0.033
0.157
0.005
0.647
0.158
Weight
0.012
0.164
0.121
0.
0.703
0.161
0.194
0.508
0.136
0.158
0.008
0.258
0.
0,576
0,229
0.179
0.401
0.191
dist, (lz)
1.024
1.532
1.511
1.351
1,584
1.319
1.908
1.020
1.402
1.372
Metric
Emin (fxz)
0.090
0.106
0.085
0.069
0.144
0.106
0.099
0.0748
0.131
0.071
6
6
6
6
6
G
6
6
Image
E 0.215
0.
0.
0.194
0.590
0.805
0.
0.
0.195
0.
0.013
0.
0.441
0.546
0.775
0.
0.225
0.
Weight
8
0.031
0.
0.410
0,560
0.767
0.
0.233
0.
Exc
0.
0.
0.
0.459
0.541
0.854
0.
0.
0.146
0.
dist. (62)
1.199
0.653
1.654
1.156
1.612
2.158
0.
1.063
1.545
0.147
Metric
Emin (fxc)
0.090
0.017
0.053
0.112
0.158
0.159
0.020
0.069
0.145
0.134
Image
7
7
7
7
7
7
7
7
7
2
ty
0.489
0.
0.212
0.298
0.007
0.258
0.117
0.482
0.136
0.525
0.190
0.215
0.070
0.470
0.050
0.100
0.343
0.038
Weight
&2
0.488
0.165
0.
0.175
0.172
0.200
0.175
0.233
0.378
0.014
loo
0.178
0.263
0.354
0.205
0.258
0.207
0.109
0.426
dist. (E2)
1.508
1.731
1.291
1.874
1.536
1.719
2.038
1.417
2.169
0.848
Metric
Emin (fxz)
0.110
0.125
0.089
0.126
0.095
0.087
0.097
0.084
0.135
0.077
8
8
8
8
8
}
8
8
8
8
Image
Eg
0.
0.
1.000
0.
0.
0.246
0.
0.
0.754
8
0.
08100
0.442
0.378
0.
0.171
0.
0.
0.
0.829
Weight
82
0.
0.298
0.593
0.109
0.
0.330
0.
0.
0.670
Exc
0.
0.377
0.595
0.028
0.
0.407
0.
0.
0.
0.593
dist. (Ez)
1.626
1.497
1,501
1.824
0.728
1.928
1.014
1,500
1.991
1.400
Metric
Emin (fxc)
0.070
0.153
0.156
0.156
0.055
0.171
0.035
0.090
0.170
0.161
Image
9
9
q
9
9
G
9
9
9
9
to
1.
0.
O.
0.
O.
0.
0.665
0.331
0.
0.004
8 E 0,911
0.918
0.012
0.
0.070
0.510
0.490
0.
Weight
0.
0.089
0.
0.
0.
0.510
0.490
0.
0.
exc
0.935
0.
0.065
0.
0.
0.
0.665
0.331
0.
0.004
dist. (&x)
1.961
1.113
1.132
1.802
0.939
1.132
1.508
1.335
1.033
1.110
Metric
6min (Lx))
0.144
0.108
0.083
0.103
0.079
0.041
0.090
0.103
0.083
0.044
28


Reviews:
[{"review_id": "rkxIOeMg9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a unified framework for adversarial training & robustness. The problem is important and interesting. The proposed framework has solid theory and is well conceived. A generic method is proposed with O(1/T) convergence rate, which is also empirically demonstrated with good performance on often-used MNIST and CIFAR-10 benchmarks. An alternating multi-step PGD is also proposed. Empirical experiments are thorough and well organized. Overall I feel it is a well written paper with sufficient contributions and is of interest to a range of ICLR audience.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyg4PfI4qB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Note: I applied a higher standard to this paper given that it significantly exceeds the recommended page limit. Furthermore, important details are left out in the appendices, which make it difficult to read the main body of the paper in a self-contained fashion. Given that the main body was already over the recommended page limit, I did not read the appendices.\n\nThis paper generalizes the min max formulation of adversarial training, and proposes a formulation that encompasses adversarial training of an ensemble, robustness to universal adversarial examples, and robustness to non-adversarial transformations. This formulation is used to derive an adversarial training procedure that trains against the worst-case adversarial example among adversarial examples generated by a set of attacks. Experiments seek to demonstrate applicability of this framework to both attacks and defenses.\n\nAs far as experiments are concerned, Section 4.1 presents results on MNIST, which is known to be a poor dataset to study adversarial examples on [https://arxiv.org/abs/1902.06705]. If models C and D are more difficult to attack, could better baselines be employed than attacking the ensemble A+B+C+D? For instance, would an adversary evading models C+D only perform better? It is difficult to draw insights that are generally applicable from a single ensemble. How was the ensemble chosen? Why would a defender add models which are known to be significantly less robust to the ensemble?\n\nWhen discussing universal perturbations, how are they generated? Given that the performance of the proposed approach significantly degrades average evasion across all images from all groups, what is the threat model for an adversary being interested in group-level success rather than average evasion across all images from all groups? How were the values of K chosen? This comment also \tapplies to experiments over data transformations. For these experiments, what was the value of K?\n\nAs far as the defensive perspective is concerned, it is not clear whether the improvements observed are statistically significant. Were multiple runs averaged to produce Table 4? Given that without DPAR, the improvement is negligible, this is important to interpret results. It appears that most of the robustness gains in both the average and max settings stem from DPAR. This should be clearly surfaced in the introduction and presentation of contributions if DPAR is required for the proposed generalized min max formulation to improve robustness. In particular, it is not clear whether DPAR is “a beneficial supplement to adversarial training” or a required supplement to adversarial training - per the formulation in this paper.\n\n\nThere are issues with grammar throughout the document, which make it difficult to read. Some specific issues:\n\n1 - Adversarial attack is a tautology (an attack is always adversarial)\n\n7 - What does “robust” adversarial attack mean?\n\n7 - What is CAAD-18?\n\n7 - Define ASR_all: what does evade mean here? Is the attack targeted or untargeted?\n\n7 - What is an “advanced” DNN?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkexHiNTqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies how a min-max framework can incorporate different tasks related to adversarial robustness. Specifically, the authors study adversarial attacks against model ensembles, universal perturbations, and attacks constrained by the union of Lp norms. They propose optimizing a probability distribution over \"domains\" (models in an ensemble, inputs, Lp balls; respectively per task) and regularizing it to be close to uniform. They perform experiments to evaluate their method.\n\nFrom a conceptual point of view, I did not find the contribution of the paper significant. All of the tasks discussed are direct application of the min-max framework and have been studied to a certain extent in prior work (https://arxiv.org/abs/1811.11304, https://arxiv.org/abs/1706.04701, https://arxiv.org/abs/1904.13000). The novel tools introduced are the regularizer on simplex probability and the attack diversity regularizer. However, the theoretical justification for these tools is rather weak and their utility would need to be demonstrated empirically.\n\nFrom an experimental point of view, the baselines considered are very weak. At a high level, the authors compare their version of min-max optimization to a very simple average-case optimization. In order to demonstrate the utility of the tools introduced the authors would need to at least compare to a reasonable min-max baseline. For instance, a very simple heuristic capping the loss of each domain in the finite-sum formulation (https://arxiv.org/abs/1811.11304) would be the bare minimum. In their current state, the experiments only demonstrate that a min-max approach outperforms an average case approach, which is fully expected. At the same time, the diversity regularizer does seem to offer some empirical gains and I would encourage the authors to investigate further.\n\nOverall, the conceptual and experimental contributions of the paper are rather weak and I thus recommend rejection.\n\n=========================\nUPDATE: I appreciate the authors' response and additional experimental results. \n\nI am still quite concerned about the universal perturbation baseline. I suspect that the clipping factor used might be too large since clipping barely has any impact (the attack is still focusing too much on B ignoring C). Conceptually, clipping should be quite similar to a min-max formulation. \n\nI do see that the proposed method outperforms the one proposed in Shafahi et al in terms of universal adversarial training. I feel like this is a more reasonable baseline and I am increasing my score to a 3.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission studies an interesting problem. However, as some of the reviewers point out, the novelty of the proposed contributions is fairly limited.
Decision: Reject