Sensible adversarial learning
Authors: Jungeum Kim
Xiao Wang
Venue:  (2020)
Keywords: adversarial learning
deep neural networks
trade-off
margins
sensible reversion
sensible robustness

Abstract: The trade-off between robustness and standard accuracy has been consistently reported in the machine learning literature. Although the problem has been widely studied to understand and explain this trade-off, no studies have shown the possibility of a no trade-off solution. In this paper, motivated by the fact that the high dimensional distribution is poorly represented by limited data samples, we introduce sensible adversarial learning and demonstrate the synergistic effect between pursuits of natural accuracy and robustness. Specifically, we define a sensible adversary which is useful for learning a defense model and keeping a high natural accuracy simultaneously. We theoretically establish that the Bayes rule is the most robust multi-class classifier with the 0-1 loss under sensible adversarial learning. We propose a novel and efficient algorithm that trains a robust model with sensible adversarial examples, without a significant drop in natural accuracy. Our model on CIFAR10 yields state-of-the-art results against various attacks with perturbations restricted to l∞ with ε = 8/255, e.g., the robust accuracy 65.17% against PGD attacks as well as the natural accuracy 91.51%.


Under review as a conference paper at ICLR 2020
SENSIBLE ADVERSARIAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The trade-off between robustness and standard accuracy has been consistently re- ported in the machine learning literature. Although the problem has been widely studied understand and explain this trade-off, the problem seems to remain as open problem. In this paper, motivated by the fact that the high dimensional distribution is poorly represented by limited data samples, we introduce sensi- ble adversarial learning and demonstrate the synergistic effect between pursuits of natural accuracy and robustness. Specifically, we define a sensible adversary which is useful for learning defense model and keeping high natural accuracy simultaneousll. We theoretically establish that the Bayes rule is the most robust multi-class classifier with the 0-1 loss under sensible adversarial learning. We propose a novel and efficient algorithm that trains a robust model with sensible adversarial examples, without a significant drop in natural accuracy. Our model on CIFARI0 10 yields state-of-the-art results against various attacks with perturba- tions restricted for with 8/255, e.g., the robust accuracy 57. 57.23% against PGD attacks as well as the natural accuracy 91.51%.
INTRODUCTION
With many impressive successes of deep learning, there are a multitude of applications of deep neural networks (DNNs) that permeate in everyday life. As DNNs applied in security- critical systems such as malware detection, face identification, and autonomous driving, robustness of DNNs against adversarial attacks, i.e., the intently perturbed inputs to fool the system, has become an important research topic (Szegedy et al., 2013 Papernot et al. 2016 Biggio et al.) 2013).
One of the most widely studied classes of adversarial perturbations is lp-norm constrained adver- sarial perturbations (Szegedy et al.] (2013). Madry et al. (2017) formalize the adversarial learning against this class of perturbations
minimization problem of adversarial risk defined in follow- ing way. Let (X, y) E X' x y/ be from some unknown distribution Px,y. Given a loss function IR and constraint constant e O, the adversarial robust risk is
E: x
Rfo Ex,y max {(f(X + 5)] S
Many adversarial learning methods be interpreted empirical minimization of (1) (Goodfellow let al. 2014) Kurakin et al.) 2016b] Ruitong Huang & Szepesvari 2013] Madry et al.) (017). For this optimization problem, Madry et al. (2017) propose to train robust model with the augmented data generated by the projected gradient descent method (PGD). On this adversarial training, they make two important observations. First, it costs natural accuracy. A network trained with adversarial examples tends to have a lower natural accuracy than a naturally trained network. This trade-off is observed even with a small training c. Second, the adversarial training requires larger model ca- pacity than the natural training does. If the model capacity is only sufficient for the natural learning, the adversarial training can converge to constant function.
For large e, the optimization problem of (I) itself may pose the trade-off. For instance, Tsipras jet al. (2018) show an example of an inherent tension between pursuits of accuracy and robustness when is large enough to change the true class. For smaller e, however, the formulation in does not explicitly pose any conflict between the pursuit of robustness and accuracy. Note that Ryos is upper bound of the standard risk of f. A perfectly robust model f with Ryo 0 is also perfeetly accurate for natural learning. If the perfect classifier exists in a given model class, the trade-off may be caused by the large sample complexity of adversarially robust generalization

---

Under review as a conference paper at ICLR 2020
(Schmidt et al.) 2018 Yin al., 2018: Stutz et al. 2019). Without sufficiently large amount of data, the empirical minimization of may result in a large standard risk by converging to a model of a poor robust risk. On the other hand, if robust learning converges to a constant function, it cannot achieve natural accuracy. In this sense, to resolve the trade-off problem, we may need to deal with the increased requirement on the model capacity.
In this paper, we propose a novel framework, called sensible adversary. in order to overcome the trade-off between natural accuracy and robustness. In particular, we restrict adversarial perturbations not to cross the Bayes decision boundary besides the e-ball constraint, so that the perturbation ball adaptively modified for every single data point. Our main contributions are:
Under the framework of sensible adversary, the pursuit of robustness and accuracy given an enough model capacity can align with each other, i.e., there is no trade-off. We theoretically establish the Bayes rule is most robust against the sensible adversary. If the Bayes decision boundary can be far from data manifolds at least by our pursuit of sensible robustness does not cost any adversarial robust risk.
We propose efficient algorithm for sensible adversarial training. which utilizes sensible adversaries the absence of the true Bayes rule. This sensible adversarial training enjoys robustness without significant drop of natural accuracy. Furthermore, the algorithm is not sensitive to the model capacity. When insufficient model capacity is given, our algorithm does not collapse constant function. Instead, it trains model robust as possible.
We experimentally demonstrate that sensible adversarial training enables to stably learn robust and accurate model. In particular, CIFARIO, achieve 91.51% natural test accuracy and 57.23% robust test accuracy against &oo PGD attacks constrained to = 8/255. To the best of our knowledge, there is no approaches known to achieve natural accuracy more than 90%. while achieving more than 55% of robust accuracy against PGD attacks of e = 8/255. Moreover, no previous approaches pursuing robustness against this attack achieved the natural accuracy more than
1.1 RELATED WORK
Madry et al. (2017) formalize adversarial learning as a mini-max problem given perturbation restric- tion, and theoretically and empirically established the feasibility of the optimization. Our sensible adversary redefines the set of perturbation in the inner maximization problem on which their the- oretical result is directly applicable. Tsipras al.] (2018) investigate the possible source of robust trade-off. The key idea is that when there are features that are useful for natural classification but vulnerable to adversarial perturbations, a robust model would abandon these features because other- wise all of these features can adversarially move to promote incorrect prediction. Our work explores the possibility of learning a robust model while not allowing such collective adversarial migration. While a class change by adversarial examples typically has been prevented by using small Sug- gala et al. (2018) explicitly ignore an adversarial perturbation that crosses the decision boundary of Bayes rule. Zhang et al. (2019) also investigate the Bayes decision boundary to resolve the trade-off problem. They search for model f having a small weighted sum of the natural risk and probability that adversarial example can cross the decision boundary of f. Gilmer et al. (2018) show that in high dimensional setting, even small test error can imply the existence of adversarial examples for most of data points. Our effort to prioritize natural accuracy to find a robust model consistent to the view in Gilmer et al. 22018) More related work will be presented in Appendix A
1.2 NOTATION
Let F denote the class of functions represented by DNNs with a fixed architecture. An opti- mal robust model w.r.t Рx, is denoted by frot arg minsez (f). Denote the stan- dard risk w.r.t Px, by (fta(f) = Ex.y((((()),
and its optimal natural model by fat = arg min (ote(f). Let Px,Y = Px,ylexy denote restricted distribution of Px,y on subset & x y C X x V. Likewise, denote the standard risk w.r.t TP xx, by Rota(f) and the robust risk w.n. Px, by RFa For a set A, the e-neighborhood in fn-norm is defined as B(A,e) (ullly x æ1Ip I 6, A}, and the interior is denoted by (A). Denote the
2

---

Under review as a conference paper at ICLR 2020
neighborhood of the decision boundary of by Df( {a| Br' B(x, s.t. f(x) + f(z))) Let Pf. (z) denote the predicted probability of the label of being y by model f.
ADVERSARIAL LEARNING MAY HELP STANDARD LEARNING
In this section, we use toy example to investigate the synergistic effect between the pursuit of robustness and natural accuracy.
Example: We can easily find an example of trade-off wherever an e-ball attack can cross the manifold boundary between classes. For example, consider a two-dimensional random vector X = =X1, X2) on X' = (0, 1) x (0, 1) with binary class Y Ber(p), where p > 0.5. Let the conditional distribution be (X1, X2) 0 ~~ (ni 2) x (0, (0.1)) and (X1, (2),
Unif((2 1) (0, 1)). Then the Bayes Rule is (x) sign(x1 0.5). and it perfect classifier, in that Redd (
with as the 0-1 loss. If > 0.5, the robust classifier against e-ball attacks is frot (z) sign [I1 (0.5-e)). Its decision boundary is deviated by -e from that of the Bayes rule, and this deviation costs natural accuracy by Rrea(froo) (f) (1 p)e. However, when the data points reside in high-dimensional space, the samples are too sparse to represent the underlying true distribution. To take this phenomenon into account, consider a distribution Fx,y = Px,yl.s x
on subset X x y C X x y. Assume that we only observe data generated from Px,y, v. and the training and test sets do not provide any information on X x V. Now by applying this support re- striction to the example above, demonstrate how adversarial learning dramatically changes from harming to benefiting natural accuracy.
The Cheese hole distribution: For the above example, we restrict the support on E = U.
Hij. where H, (( 2 ) + 1)) x ((2, $a 2 + 1)), a 6. and = 1, 2, 3. Therefore the sampling support X' comprises of nine small squares equally spaced by a. This is illustrated in Figure (a). Among all classifiers which predict the same as f on X, the worst one, denoted by f predicts exact opposite on z X as illustrated in Figure (b). For this worst case, the true standard risk w.r.t y is 3 although the standard risk w.r.t. F Fx,y is zero.
We argue that pursuing robustness can mitigate this discrepancy. Among minimizers of Ryoss let freb be the worst classifier in that it predicts incorrectly outside X' as depicted in Figure (c). On B(X,e), c), frve should correctly classify, except on A, = {(=1, In this situation, the inaccuracy of figh is compensated by its increased accuracy B(X,e) c) \ A.. Moreover, the composition of Rros (f (f is particularly interesting, and it easy to show that
e S xι S 0.5) r ((X,e).
Roob P( (())B(())))))) x, E DB(jBr.)) 3/4,
where 3/4 is from the standard inaccuracy on X², The major part of Rros(f comes from the inaccuracy of jB+ outside of the sampling support X. Therefore by simply reducing this in- accuracy on Xo, we can lower the robust risk by 3/4. Another interesting observation is that F1 < 0.5 + + is the only area where the Bayes rule fB has its ro- bust risk. Note that (f) robust risk. The reason why f is not most robust is simply because is accurate on B.. This prevents f from being robust function under the current adversarial robustness framework.
B. = [(x11
IP E Be, [X)
Y) 2e which is greater than the optimal
These observations have two implications. First, the robust pursuit can help to increase natural accuracy outside of X where the samples are poorly representative. Second, by accurately correcting the model outside of X in a standard sense, the robust risk can be significantly decreased. This is where the pursuit of robustness and accuracy coincides.
Toward respecting the original data structure: What if we regard the adversarial robust risk near the class border reasonable gullibiliry? This corresponds to pursuit of robustness only against adversarial examples which do not cross between the class manifolds, i.e., as long as this does not harm the natural accuracy on X. We call this robustness as sensible robustness, which respects the structure represented by data. In this example, the sensible robustness can increase both robustness and natural accuracy. Let fee denote the worst case sensibly robust classifier w.r.t P xxy- The
3

---

Under review as a conference paper at ICLR 2020
X2
X2
X2
X1
X1
(a)
(b)
(c)
(d)
Figure Cheese holes distribution. (a) The outer square is the support of the underlying true distribution Px,,, but the sampling is restricted on the small squares (b) The worst case naturally optimal model w.r.t. P Px,.. (c) The worst case robustly optimal model w.r.t. TP x,y- (d) The worst case sensibly robust model w.r.t. Px,y-
with distribution Px,y.
decision boundary of f should not deviate to the left ( matter how large is, as depicted in Figure (d). Note that when 2,2,, fros is the unique minimizer of Rfob(f) as B B(X,e) e) covers X.
SENSIBLE ADVERSARIAL ROBUSTNESS
In this section, introduce sensible adversary framework. Consider general multi-class with the 0-1 loss, where y [K]. Assume the model capacity is enough so that the Bayes rule I F. We consider f--norm constrained adversarial attacks, where p {0, 1, ..,, oo).
Definition 1. (sensible adversarial example) For a classifier f. let (f) = {a X1I| æ|11 I
6, f()) = y}. Then the sensible adversarial example of (x, y) w.r.t f is defined as
if f + otherwise.
T, arg max {()(=),), =fB)
Definition 2. (sensible robustness) Let the sensible adversarial loss be (S,
2, v) = (f(#),)) v)
where à is a sensible adversarial example as defined above. Then the sensible robust risk of a model is defined by
(f) = Exx.Y (Fos, (f. (f,X,Y)) = Exxy (()((),))
(3)
We call its minimizer as a sensibly robust model w.n.t IXx,y and denote by fram i.e.
f arg min JEF (f).
Remark 1. Intuitively, sensible adversarial example is an adversarial example restricted not to cross the decision boundary of the Bayes rule. In addition, a sensible adversary does not perturb a data point that the Bayes rule incorrectly classifies. Therefore, it is natural to expect that pursuing sensible robustness would not cost natural accuracy, and the following theorem confirms it.
Theorem Let Rid denote the minimum standard risk which is Rf) Then we have (f Rudd- Furthermore, f is the unique minimizer of among f E
Theorem 2. Let Sx,e be an e-ball centered at X. Then for any f F and for any set A C X' (B()
PP PP3 E Sx s.1. H. (())) <P(3'' E Sx, s.1. [(x') ((x)
Remark 2. According the Theorem I, a sensibly robust model w.r.t PX, is the Bayes rule,
i.., frob B. This sensible robustness costs adversarial robustness since f8 may have larger
4

---

Under review as a conference paper at ICLR 2020
Algorithm Sensible adversarial training for Eoo norm restriction
Input: Initialized f = Jo, e. E (0, 1), step number K, step sizes y, 1/2 data x X
2: repeat
3: iori=l,,.. m, s.t. f(x ((av))) = Yt
4: for k L..,,K,,
Liade t (I) sign( V af (o vc)) + x ((-T))) II: the projection operator
(k)
5:
6:
if (E, Wi) > log !
7:
(sensible reversion) F. ii,do = It i,ado
(K)
(k-1)
8:
break
9: 0 - 1/2 Di=1 xo 263/m
10: until training converged
adversarial robust risk compared with frob. a direct minimizer of (T. However, Theorem 2 shows f is equally or even more robust than frub except on certain area. In particular, Theorem shows that f is the most robust model except on DB <). Therefore, f is most robust almost everywhere if the decision boundary of f can lie outside of ( e), that X' DB(f" <) X. For example, this happens when each class has its own support apart from each other by at least 2e.
The next theorem shows that even when only have data from Px,y restricted on X y, we can find frob, the optimal function w.r.t. Px,.-
Theorem 3. Let A.
E ((f(z) fx),, Ve E Sx.((B)) = 1} and R Epx, o (f, X, f)). Then, for any e > O, (f) is only minimized by any f E Ac. Further- more, if ((X, e) X, f is the unique minimizer (f).
Remark 3. is interesting to compare work with that of Suggala et al. 2018). We notice there are some critical diffcrences between these two works. Given y E { 1-1,1 1,, Suggala et al. (2018) define adversarial risk
max Radv (f) = E ((z)= ()
Unlike our definition of sensible adversary, if is a Bayes rule f. their adversary tries to increase the loss w.r.t not y but deterministic function of z. Therefore, for making f being an optimal robust model, their objective should always have additional term, e.g., Rnat(f) + XRtav (f) for < oo because an s.t. f(x) A g(x) w.p. can minimize (f)v Unlike their approach, sensible adversarial risk 33 alone be optimized making the Bayes rule as the optimal model.
ALGORITHM
A transition from theory to algorithm poses two main challenges. First, the 0)1 loss function in the theory hard to optimize. Therefore, as common practice, we use the cross-entropy loss. Second, fB on the entire space is practically unavailable. We note that a model that performs well on natural data can be nice approximation of f on the restricted support X V. Therefore, generalize sensible adversary in (2) to utilize general loss function and reference model that substitutes f
Definition 3. (generalized sensible adversarial example) Consider loss function é. For classi- fier and E (0, 1], let (2.) {2 E X'|||= - xlIpp S {(f(2), y) < log 2 U {z). Then given reference model fr, the sensible adversarial example for (z, y) is defined
x, arg max {(f(2).)) =f-),.(f)
if (x) + otherwise.
(4)
If is the cross-entropy loss, the condition ((fr(x),)) v) log e is equivalent to (z) 2 In binary case with c = 0.5, this requires the perturbed examples not to cross the decision boundary of fr. and for general not reach vicinity of the boundary.
5

---

Under review as a conference paper at ICLR 2020
The test result on natural examples and -atackk for CIFARIO (E = 8/255). attacks are generated with 20 random restarts and counted the worst case only.
Table 1:
The PGD
NAT
CW40
DeepFool
FGSM
LBFGS
MIFGSM
PGD100
SENSE
91.51
67.01
78,89
72.72
85.94
68.87
57.23
TRADE
84.92
62.19
61.38
61.06
81.58
57.95
54.72
Table 2: The transfer attack results between the TRADE and SENSE model on CIFAR10. The to PGD40 and MIFGSM attacks are generated. The subscripts of the column names denote the generating model. The denominator and numerator in each cell are the number of adversarial attacks and correct predictions respectively (e 8/255).
Defence model
PGD PGDTRADE
MIFGSMSENSS
MIFGSMTRADE 2916/4112
TRADE
7831/10000
5655/10000
1584/3113
SENSE
6499/100000
6961/10000
6887/10000
In our algorithm we set f, as a naturally trained model or a current model. Given a fr. the implemen- tation of sensible adversarial attacks is straightforward. For a correctly classified natural example by fr. we add perturbations in the similar way to the PGD method (Madry et al. (017). The difference is that during the K-step of PGD iterations, once the loss of a currently generated example exceeds log et we reverse it back to the previous step and break the iteration. This requires no additional forward- or backward- proppaaatoogttiionn, compared with the PGD method. The proposed algorithm is in Algorithm [ for the & norm and in Algorithm
22 Appendix [ for the other l, en-orrs..
The definition of the sensible adversary in 4 divides the data into three subsets: AG, (x)f(())
y or Sr,c(fr) = {=)) By, = {aIS2.. 7 Sr.e. (fr) * {a) and Cj, = X' (Ay B Therefore, our algorithms generate sensible adversarial example i= 5=- in three different ways: (1) E x for x E Ay. (2) i + F² for x E By, and (3) 8 5" for I E Cy, where T" is full K-step PGD attack. Therefore, sensible adversarial training inherently involves a set selection mechanism. The sensible adversarial loss (f x, y) can be written as
((S, ((,,,,))leA, E(S,"" 5", ly)n, + ((J,,",u)lzEC; =((f, x, v) [(t. l(f,f" y))=s> (ff...))igg) + E(f, 2", v)) wiSlog?
(5) When c < 0.5, this optimization problem has a strong motivation to increase |Cy ,, i.e., the number of sensible adversarial attacks that are identical to full PGD attacks. Because A, = (x)(((((x).)) > log } for e 2 0.5, the loss on the natural stage A, is always greater than the loss of the sensibly reversed stage By and the full PGD stage Cy. Furthermore, the loss for E By is always ap- proximately log _ by adaptive sensible perturbations. Therefore, paradoxically, the smallest loss is only achievable by full PGD examples. In other words, sensible adversarial training penalizes when & Cy. We note that the optimization problem has a smooth landscape; Although the data points may jump around between A,, By. and C'y. there is no obvious discontinuity in the loss. Therefore, during the training. (* x. y) smoothly slides down, making both {(f, x. y) and ((f, SP, y) smaller than log ! This enables to learn naturally and adversarially accurate model. More discussion on the stability of our algorithms are presented in Appendix D
EXPERIMENTS
5.1 EXPERIMENT 1: ROBUSTNESS AGAINST e-BALL ATTACKS
CIFAR10 We train WideResNet-34-10 10 (He et al. 2016) with sensible adversarial examples of e 8/255 and c = 0.7 with the step number K 10 and size 7iI = e/5. We attack our model with various white-box attacks with perturbations restricted to foo with the performance with TRADE using the same architecture (Zhang et al. 2019). which is know to be robust and accurate. The result is in Table I Our model achieves 91.51% 1% natural accuracy. This is 3.7% drop in natural accuracy from 95.2%, which is an accuracy that a naturally trained model can achieve (Madry et al. 2017). With this architecture, Madry et al. (207)) achieve 47.04% robust
8/255. We compare

---

Under review as a conference paper at ICLR 2020
Table MNIST: test results of our models on natural examples and foo based attacks (e = 0.3).
Defence model
Natural
PGD500
C&W40
SENSE
99.51
91.74
96.02
TRADE
99.48
93.30
96.90
adveraria accuracy
natural accuracy
capacity
capacity
capacity 3
capacity
capacity 5
1.00
Accuracy
0.75
0.50
0.25-
0
4
8
2
8
8 Hyperparameter C
2
2
.8
.
8
Figure 2: The natural and robust accuracies of our models for the varying parameter c.
accuracy against PGD20 attacks and 87.3% natural accuracy. As a black-box attack, we try transfer attacks between the TRADE and SENSE model by the foo based PGD and MIFGSM, which known to be effective for transfer attack (e = 8/ 8/255). We obtain adversarial examples by applying generating model, and then
PGD and MIFGSM The result is in Table Overall we observe that our model outperforms both the TRADE and Madry model. In particular, Sensible adversary achieves more than 55% of robust accuracy against PGD) attacks of = 8/ 8/255. This performance is consistent to the test margin distribution in Figure 8 and this is discussed in detail Appendix [E, For PGDI00, we conduct random 20 restarts and count only the worst case. The step number 100 and step size 2/255 of the PGD attacks are justified by the plot Figure in Appendix H
the examples attack a defense model.
We consider a CNN model with three convolutional layers followed by a fully connected linear layer, which is the same architecture in (Zhang et al.) 2019). We train MNIST model with sensible adversarial examples of e = 0.3 and c = 0.5 with the step number K = 10 and size Mi 0.05. The robust test result in Table 3 shows the comparable performance of our model. The step size of the PGD500 attack is 0,01, and the serenity check on the PGD attack is in Appendix H We conduct 100 random restarts and count only the worst case for each example.
MNIST
5.2 EXPERIMENT 2: SENSITIVITY ANALYSIS
We perform the sensibility analysis understand the effect of and the model capacity for two reasons. First, the sensible training prevents full PGD perturbations on an example until the loss on it becomes less than log 1, which could hardly happen for model with a small capacity. Second, as Madry et al. (2117) point out, when the model capacity is insufficient for adversarial learning, the model collapses to a constant function. We are interested a range of that keeps sensible learning from collapsing. Therefore, we consider a sequence of CNNs of the increasing number of kernels similar to Madry al. (2017), where we denote the capacity by q E (1, 2, 4, 5). The details of the model architectures are presented in Appendix H When we train them with natural examples, the networks of capacity and 2 achieve about 95% and 97% accuracy, whereas the networks of the other capacities achieve more than 99%. When trained with regular PGD examples, the networks with capacity 1,2, and collapse. Therefore, capacity 3 is enough only for natural learning, and capacities and are possibly insufficient for natural learning.
MNIST:

---

Under review as a conference paper at ICLR 2020
capacity 5 Natural margin
capacity 5
Adversarial margin
0.03
0.03
c=0.5
density
C=0.9
0.02
0.02
NAT
PGD
0.01
0.01
0.00
0.00
100
200
300
-100
100
260
300
margin
Figure 3: The prediction margins at convergence of capacity 5 on the test set. The natural margin of a model f at (x. y) is log ff (a) maxy'y log Pf (z). The adversarial margin is calculated by log (x²) maxxa Ay log Pra for (i"), where i is a full PGD attack.
For each capacity q. we train the models with sensible adversarial examples on MNIST with the hyperparameters
00,0.1,... 0.9, 1}. Note that sensible adversarial training is identical to natural training when c = 1, and to adversarial learning without any perturbations for incorrectly classified natural examples when
O. Figure 2 shows the natural and robust accuracy against PGD-40 attacks with varying c for each capacity. general, the natural accuracy tends to increase increases while robustness decreases. For capacities
almost 4 and 5, the accuracies insensitive to varying c. On the other hand, the tendency is most distinct in capacity 1. In this case, the network obtains best robustness when 0.5, which the least loss bound among c 0.5, the range of с with stable learning property. Even for capacity 1, however, the sensible learning does not collapse except when
We also see that c 0.5 is least sensitive to the model capacity.
Note that capacities 2 and 3 do not collapse even with c= O. When the sensible adversarial learning from the regular PGD training that the sensible learning requires robustness only for the correctly classified natural examples. When = 0, at the convergence of the models of capacities and 2, about 5% of the data points are allowed to free from the perturbation. By paying only the 5% of robust training accuracy, the sensible learning avoids collapsing and obtains about 80% robust accuracy and 90% of natural accuracy.
0. the only difference of
Intuitively, an insufficient model capacity locally close class manifolds can make a virtual deci- sion boundary that is inevitable to keep a nice natural performance. the algorithm, the sensible reversion prevents adversarial examples from crossing this boundary. This effectively reduces the requirement of the model capacity posed by the regular adversarial learning. The sensible reversion also allows robust model have larger margins than the PGD trained model for the majority of the dataset as Figure 3 The PGD trained model has majority adversarial margins as positive. Instead, has much smaller natural margins than the naturally trained model. The SENSE model with e 0.5 has comparably large adversarial and natural margins. Instead, the number of data points with negative adversarial margins is larger than that of the PGD trained model. As increases to 0.9, this model has much smaller and more negative adversarial margins. On the other hand, its two types of margins are generally even larger. This phenomenon is consistent the decreasing robustness in Figure 2 for capacity 5. In general, for a fixed capacity, increasing increases the natural and adversarial margins of the majority the data, while it also increases the portion of data of negative adversarial margins.
The cost for this highly confident prediction is the robustness near the decision boundary at con- vergence, i.e., the portion of the data points in the natural and sensibly reversed stage. When consider the margin on the training set, we find that there is linear relationship between the portion of the data points having negative adversarial margins and the test accuracy. In practice, such por- tion can be an indicator about the robustness the model or the sufficiency of the model capacity, without directly testing the model.
8

---

Under review as a conference paper at ICLR 2020
Table 4:
The sensitivity of CIFAR models of WideResNet
CIFAR WideResNet
c=0.0
c=0.3
c=0.5
c=0.6
c=0.7
c=0.8
Natural data
82.88
86.76
90.42
90.87
91.51
92.35
PGDT00
43.70
46.90
50.95
55.90
57.80
55.60
Table 5: The sensitivity of c on CIFAR models with CNNs
CIFAR CNNs
c=0.1
c=0.5
c=0,9
natural training
NAT
66.26
75.70
82.02
80.85
PGD40
26.67
20.26
3.95
0.00
We train train WideResNet-34-100 10 with several different e values. Except that for c = 0.8 we stopped learning at 120 epoch, other models are trained for 300 epochs. We report the adversarial accuracy against and PGD100 attacks with step size 7/1 2/255 for the first 2000 test examples, with random 20 restarts. The results are in Table
CIFAR:
Interestingly, while the natural accuracy positively correlated to the value, the robustness does not show clear negative correlation to c. Rather, as e become closer to 0.7, more robust result the model shows. As see that 0.8 has better robustness than e S 0.5, in CIFAR, the main reason of the observed trade-off could attribute to the influential adversarial perturbations.
For the CIFAR dataset, also try to train robust model with small capacity. We intentionally choose the CNNs model that used in Experiment 1 for MNIST. This model can be not enough for adversarial training CIFAR. We report the results in Table Given the serious lack of model capacity, the decrease in natural accuracy of SENSE models compared with the naturally trained model is not serious, while achieving better robust accuracy numbers. Note that none of our models collapse, and therefore the possibility of sensible adversarial training is not sensitive to the choice of even in the lack of model capacity.
CONCLUDING REMARKS
In this paper, we proposed a sensible adversary which is useful for learning defense model, keep- ing high natural accuracy simultaneously. We theoretically establish that the Bayes rule is most robust under the framework of sensible adversarial learning. Our learning algorithm efficient and stable, and not sensitive to the choice of the main hyperparameter c. Also, has a clear meaning as the lowest prediction- -rrttiiiitpo bound. Our empirical experiments yield state-of-the-ar results of adversarial learning on the CIFAR 10 and MNIST datasets. In addition, approach can effectively deal with the lack of model capacity. This is because by paying a robust accuracy certain area, the algorithm protects the model from being collapsed by influential adversarial examples. Furthermore, the sensible adversarial learning trains a model to have high prediction margins on both natural and adversarial examples.
showed that the sensible
We now mention several future directions for research on sensible adversary. One remaining the- oretical problem is develop generalization error bounds for sensible adversary learning, so that we can theoretically justify our empirical performance. In fact, in this work, we did not tackle the lack of sample size. In particular, as our algorithm tends produce model with large natural and adversarial prediction margins, in the lack of sample size, it is not clear if this large margins are always beneficial. Therefore, there is much remaining work be done to theoretically understand the high margin tendency of the models trained with sensible adversarial examples with relation to generalization.
9

---

Under review as a conference paper at ICLR 2020
REFERENCES
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Sindić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Sébastien Bubeck. Eric Price, and llya Razenshteyn. Adversarial examples from computational constraints. arXiv preprint arXiv: /805. 10204, 2018.
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang. On the sensitivity of adversarial robustness input data distributions. arXiv preprini arXiv:1902.08336, 2019a.
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: An adversarial robustness toolbox based on pytorch. arXiv preprint Xot::::900.006600 2019b.
Yinpeng Dong, Fangzhou Liao, Tianuu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting versarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow, Adversarial spheres. arXiv preprint arXiv:1801.02774 2018.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and hamessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: (007:00660..25533 201 6a.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arxiv:1611.01236. 2016b.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [Online]. Avail- able: http://yann. lecun. com/exdb/nmist, 2:18, 2010.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1xXi:1706606088 2017.
Seyed- Mohsen Moosavi Meosavi-Dezfooli, Alhussein Fawzi, and Paseal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
Maria-Irina Nicolac, Mathicu Sinn, Minh Ngoe Tran, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalic Baracaldo, Bryant Chen, Heiko Ludwig, Ian Molloy, and Ben Edwards. Adversarial robustness toolbox v0.8.0. CORR, 1807.01069 2018.
000000000
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE Fumpean Symposiun on Security and Privacy (FuroS&P), pp. 372-387. IEFF, 2016.
Jonas Rauber, Wieland Brendel, and Mathiis Bethge. Foolbox v0.8.0: A python toolbox to benchmark the robustness of machine learning models. CoRR, abs/1107.04411 044131, 2017. /tLp::////a/: www.woorggaaaaa abss/
1707.04131
Dale Schuurmans Ruitong Huang. Bing Xu and Csaba Szepesvari. Learning with a strong adversary. arXiv preprint arXiv:1511.030:: 2015.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. In Advances in Neural Information Processing Systems, pp. 5014- 5026, 2018
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the IEFF Conference Computer Vision and Pattern Recognition, pp. 6976-6987, 2019.
10

---

Under review as a conference paper at ICLR 2020
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi. Pin-Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
Arun Sai Suggala, Adarsh Prasad, Vaishnavh Nagarajan. Pradeep Ravikumar. Revisiting adversarial risk. arXiv preprint arXiv:/8000 0tt.//06..022.2 02924, 2018.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruma, Dumitru Erhan, Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Pedro Tabacof Eduardo Valle. Exploring the space of adversarial images. 2016 International Joint Conference on Neural Networks (IJCNN) pp. 426-434 226-433. IEEE, 2016.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Tumner and Aleksander Madry. Robustness may be at odds with accuracy. stat, 1050:11, 2018.
Dong Yin, Kannan Ramchandran, Peter Bartleu, Rademacher complexity for adversarially robust general- ization. arXiv preprint 1810. 2018.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoreti- cally principled trade-off between robustness and accuracy. arXiv preprint arXiv:1901.08577 2019.

---

Under review as a conference paper at ICLR 2020
Appendices
A ADDITIONAL RELATED WORK
Bubeck et al. 2018)) conjecture that learning robust model is information theoretically possible but computa- tionally intractable. They introduce example which not robustly learnable in polynomial time. Our view consistent with Bubcck al. (2018) in that robust model exists, but we search for efficient algorithm to estimate a robust model in reasonable time. [Su et al. (018)) compared various naturally trained models on ImageNet, and found that the trade-off varies among different model architectures. Also, they empirically discovered that more accurate models tend to be less robust when the models are trained with natural examples. Stutz et al. (2019) demonstrate that given large training set, adversarial training can produce robust model that is as accurate a naturally trained model. Our work maximize and adversarial accuracy is consistent to their demonstration. Also, they show that most of PGD attacks are off- manifold of the original data, and by on on-manifold adversarial training, the natural accuracy can be improved. we see the restricted space as data manifold, and X as off-manifold, our sensible framework aligns with their view, given the data manifolds of different classes are separated by at least 2e. Kurakin et al. (2016b) suggest adversarial learning that trains with data randomly divided into two parts, natural and adversarial set. We divide data in data adaptive way into three parts including sensibly reversed adversarial The relationship between their approach and our algorithm is discussed more Appendix F
synergistic effect between natural
B PROOFS OF THEOREMS
To save the space, we use v)(k(z)) denote P(Y k|x =
Theorem 1. (Restated) Let Ritd denote the minimum standard risk which is R.aa(F)). Then have Rfoo(f") = Ryss- Furthermore, f" is the unique minimizer of Roos(f) among f E F.
Proof.
R1) = =P(F((x)+)
-P(S # Y) +X) and 3x' {x,((f))s. s.t. (8))
-P(S +0
by the definition of (f
= R)) Reed.
It is obvious that fd is minimizer of (f) because Rita is lower bound of (ff for any f F.
Now we show f is the unique minimizer of Ro (f)-
(F) P(f ( +,1x)
Y, Sx,(ff) s.t. f(x') +
Σ
3r' Sx 8.t. f(x')(())))))))))) ) # k. =
Y=
P(F -
PP (x) k. 3r' S2. st. f(x') # Y = = x)dP(x)
K
/ L h (e att. (())))
Σ
K
= R Σ 1 1y0t sl81 ((()))))
(6)
k=1
The last equality is by (() = (1 (L-)
The first term Rate(f) is uniquely minimized by the Bayes rule 88 The second term is always non-negative, and is zero when f Therefore, Ri uniquely minimized by S8
12

---

Under review as a conference paper at ICLR 2020
Theorem 2. (Restated) Let Sx,e be e-ball centered X. Then for any f E F and for any set X DB()"
p(3d' E S... s.t. (x) A A) Ep(ad' E Sx.e s.t. [(1)+ + v.N E A).
Prouf. For any set X .)) E ex s.l. A) # Note that any subset B X, the Bayes rule has the least error probability. Therefore,
A) < P(f(X) & Y,X X A) s Pa E Sx.e st. f(x') # Y, A). The last inequality is trivial because ((X( # Y Dr E Sx,, sd. ((x) +Y.
Theorem (Restated) A. {s e F[P([(z) = x Ve Sx.c(f))) 1 and (f) Exx,y L (f, Y)). Then, any e O, Roa is only minimized by any A.. Furthermore, B(&,)) 5 X, f is the unique minimizer of (J).
Proof. The sensible risk of w.r.t. the restricted distribution corresponding to i
S 1/0(0) st. flrt 1()))))(((())))())))
Rf Rsta(s) Σ/
(7)
= R f Σ
K
[1pr at ( 1())) (0 (x)l))))
8 k=1
= =R.ue(]) (132/ES2, (z))))"())) 1 ( )n( (as)))daa (x))x)d(((1)
(8)
because P(Y = k|X 2) P(Y = x|X 2) for &. The minimum of R2.on (f) is achieved by f with the minimum value R R Therefore, any function f that Roaas) R* cannot achieve the minimum of (f) because the term (s,,' (18) ) a.t ((()))))()) B) in (8) is always non-negative. Therefore. only functions in A { E (IX( l 1} need to be considered possible minimizers of (f).
Note that Ar A. By the definition, we know that A Ac if and only if
(F( =1
(f)x = (x), Vo Sx.c(f")) <1
Therefore, for / E AA P(x A) and 3r' B) for Ve Ast. f(a) # ( For this f. the equation in 8 can be written Ri (f) = R2 for some O. Now show that 0.
Note that by the definition of the Bayes rule, P(y fd (x)]X x) 2 2 Otherwise, 1 = Ex= Ply k: X = x) S. KPP( = f (x)[X = x) < 1, which is contradict. Then, for the f E A As and c X that are described above,
f (IavESE.((9)) (z)))8(0) 110 (n) x)))(1)
2 1
K 1 (Lares en (()))"(>)) (())
1
K / (Lar'8n.. a (o1))) dP((x) by f E A
1
(PA) K 0.
2 K (IawrEs,, a. 12)4 dP(()
Therefore, > Note that for any f Ac, the second term in X zero by the definition of Ac. Therefore, first result of the theorem is proved. Furthermore, for such that B(² 5 X. = 8) Therefore, when B <) 5 X. fd is the unique minimizer of (f).
Theorem Let A {s FI²(((X) = (x)) = 13. and take reference function fr E A. Consider extended sensible adversarial examples, with e < and as the 0-1 loss. Then, for any e > 0, Ri f+) E 1(((()), Y)] is uniquely minimized by fr-
13

---

Under review as a conference paper at ICLR 2020
Theorem 4 says that for fa E A.. which behaves the Bayes rule f the restricted support, the correspond- ing sensible adversarial risk E Fxxy [X(f(X)s..) Y)] is minimized only by fr. This implics that if we do not have any information about the Bayes rule X the sensibly optimal model w.r.t. Px,y can be arbitrary on & although this optimal model is the Bayes rulc on X. Our algorithm deals with this arbitrariness by searching for better reference function in each iteration. As current model is used estimation
reference function, ie., the defense model and the reference model is identical, the algorithm essentially pursues sensi- bleness on X' and robustness Xe of the trained model. Note that on X, sensibleness sufficient condition for natural accuracy.
Proef. By using the same way to derive (6 and 8 and noting that f-(x) = f on X, get
K
(11o( Raaas) +
Σ (
a4. f(x))* ((o))
km
= Rota(f) + / (150'E8 sa. (z)))) - -1 (t))ss (x)
(9)
Because f+(x) (x) X, Ret(lF)) R.) This implies fr minimizes (f((( because P) is the minimum of Raaas) for all F and the second term in is non-negative. This implies any function f s.t. Ras(() > Ram(s²) cannot achieve the minimum of Reo Therefore, as a minimizer Rioh we only need to consider f E A. Note that {())))(())) for A X. Therefore, by letting Rind be (e) (), equation in ( can be written as
f lar' st. "orrs"" (z)())dd
Riof( = RRied"
-Ried + f lavesudd a. 10 (x)
(10)
Note that (n("((x))2)) (x)]z) is positive on X. Therefore, for f E A minimize (0 must satisfy that P(3r' (x. s.l. f(x') f-(x)) This essentially says f should be fr-
C ALGORITHM
Algorithm Sensible adversarial training for E norm restriction
Input: Initialized fe. c (0, 1), step number and sizes K, m. U2 data x0 adv X
2: repeat
for = ..., . s.t. f(x y/ 4: for k = 1,..,,K K
T Fi [Bpp(re))
(N)
V V. (A-1) kad V=((s(a! (aade m.)
+ x (((-))))
II: the projection operator
6:
if e(f ",ador Wi) > log !
(sensible reversion) (K) (1-
8:
break
0 - 0 172 Em Vgl(f, (K) 10: until training converged
i,ddu 91mmm
Algorithm 2 is a straightforward extension of Algorithm from foo norm to Ep norm. In addition, we present another way to realize the sensible reversion. After stepping back when the loss exceeds the threshold, we can add a random noise as the next step instead of just breaking the iteration as in Algorithm [ [nddd Considering the nature of (mini-) batch leaning, this random noise does not add much computational cost because until every loss exceeds the threshold, the for loop would keep calculating the forward and backward loop with fixed perturbation for the reversed examples. This algorithm loo norm presented in Algorithm 3, of which objective function is also (5). Algorithm also reverses the adversarial example when the loss is greater than log J. The only difference is that instead of breaking the iteration after the reversion, it considers a random noise the next step. This can lead more effective search for the local loss maximizer in the ball z.e(f) in definition 4 The extension of Algorithm to & norm straight forward, and thus omitted.
14

---

Under review as a conference paper at ICLR 2020
Algorithm 3 Sensible adversarial training for Eoo norm restriction
Input: Initialized f = Jo. e. E (0, 1), step number K, step sizes m, 1/2 data x X repeat
3: for 1..,, m, s.t. fx = Y/ for k = L...,K K
i; s- (R(()))((sep))) (k)
+ + ((ade) II: the projection operator
6:
if ((f, Tz,y1) V1) S log
I i,ado = Zz
step; (k+1) = yn signi (osggn/Ve([ffeer (al U:))
9:
else
(A)
I s,ade
x aaae
(k+1) = random noise
step;
12: 0 0- 12 > V vel(f, aadd 2:3/ until training converged
D THE LANDSCAPE FOR SENSIBLE ROBUST OPTIMIZATION
Note that when the original examples are incorrectly classified, full PGD examples can be very influential in regular adversarial training. If we do not add any perturbations on these potentially influential examples and add full PGD perturbations on the other examples, the resultant examples are equivalent to the sensible adversarial examples when O. Experiment shows this small change keeps the models from collapsing, demonstrating how influential PGD perturbations on the incorrectly classified natural examples. However, an adversarial training with the sensible adversarial examples with c = O. the empirical loss can largely fluctuate during the training: Once incorrectly classified natural example becomes correctly classified, its full PGD attack can pose sudden large gradient for the model update. Then, when it is incorrectly classified again, the loss it suddenly reduces the natural loss that distinctly smaller than the large loss value on its full PGD attack. Therefore, whenever an example changes its state between a correctly and incorrectly classified example, i.c., the full PGD and natural stage, the corresponding loss can fluctuate making the learning unstable.
However, if we add sensible reversion step. particularly if e 2 0.5, this fluctuation does not occur. Between the two stages, the sensibly reversed stage provides kind of cushion between the natural and full PGD stage, preventing the sudden change the loss value as described. Until Sf,x Sz,c. the sensibly adversarial perturbation for x is adapted to make the loss of the current function approximately equal to log ?. As the abstraction in Figure 4 it is like to have a virtually extended area at P5., = c on the loss function that set of j that are in the sensibly reversed stage. However,
Sz.c]. i.e., in spite of the existence of such flat loss area, the model still can learn with j. in sensibly reversed stage. This obvious because the cross-entropy loss has its non-zero gradient when is log ! Interestingly, when the model is updated in a way to decrease the loss of the previous 8", the new sensible perturbation is again adapted to have the loss approximately equal to log ! Therefore. the course of training directs the model to have sensible adversarial examples in the full PGD stage. Furthermore, as long sensibly reversed stage, still has perturbation an adapted e-ball. This helps obtaín a robustness although it is not on a full e ball.
is locally flat on [i (x.().)())
0, even when S is in
Note that for e 2 0.5, the learning is very stable because only natural examples can overpower the training; The large loss values are only achievable by sensible attack in the natural stage. Any single full PGD attack cannot dominate the next update of the training because the loss function has relatively small gradients full PGD stage. Therefore, our algorithms not only effectively ignore any influential full PGD attacks that may overpower the next update but also train a model that allows as many sensible attacks to be full PGD attacks. In other words, our algorithms can stably learn a robust model.
the
E HIGH MARGIN PROPERTY OF SENSIBLE ADVERSARIAL TRAINING
In sensible adversarial learning, the natural accuracy clearly takes priority over the adversarial accuracy. The perturbed example is not allowed to cross the decision boundary of the Bayes rule in 2 or to reach the vicinity of the decision boundary of reference function in 4. As mentioned, when the cross-entropy loss is used, (x) is prerequisite for adding any adversarial perturbation
We note that this condition provides
15

---

Under review as a conference paper at ICLR 2020
An abstraction of the sensible adversarial loss
Natural
Sensibly revesed
Full PGD
lfss(f, ,)
(x(
Figure 4: An abstraction of the sensible adversarial loss when c 2 0.5. When x in the sensibly reversed stage for a current model f. the loss of 3 is approximately log 1 Although the loss approximately the same while x stays in this stage, the model updates in a way to pushes the sensible adversarial example become a full PGD attack as the arrow.
a lower bound of the following natural margin as
M(f x, v) (og(r.((x)
max logir,, (x) log zy
(r.((()) (2)
c 2 log 1-c
Therefore, the priority of natural accuracy hints that the learning will prevent the natural margin from being sacrificed for the sake of adversarial robustness. We that the natural margin of x is an upper bound of its adversarial margin. Therefore, a model cannot confidently predict a natural example x, neither can the model confidently predict any adversarial examples of x.
We investigate the margins of the models trained in Experiment understand the effect of c. Figure 5 we draw density plots the margins on the test set for varying e for the fixed capacities. Overall, we see that larger c results in a larger adversarial margins and natural margins. In Figure 6 we also draw the margins but for varying model capacity. In general, for each c, smaller capacity has more data points of negative margins. However, for naturally trained models, i.c., the models with 1.0, larger model has smaller adversarial margins. This is consistent the observation of Su et al. (2018) that accurate models tend be less robust when the models are trained with natural examples. On other hand, although not displayed, the plots corresponding to = 0.9 are essentially similar to the plots in the second low. This implies even for large c, our method is not like natural learning; The models trained with 0.9 still have larger natural and adversarial margins.
Experiment 1
Note that in Figure the regular PGD model of capacity 5 has only few negative adversarial margins. Instead, their natural margins significantly smaller than those of the natural models. In contrast, our model has negative adversarial margins for a few more data points. Note that the majority of hoth natural and adversarial margins of the models significantly larger than that of the regular PGD models. For capacity 1,2 and 3, the regular PGD models collapse having small "mean" of adversarial losses. On the contrary, the lack of model capacity by letting more portion of examples to have negative margins. As demonstrated by Figure 5 in our methods the mean adversarial loss can be arbitrarily large. Instead, it maintains a large portion of points having relatively large adversarial margins, i.c., being far from decision boundarics.
models deal with
The flip side of the advantage of high margin is the possibility of over fitting. For e 2 0.5 in Figure 7 the best robust accuracies not achieved when capacity is 5 but when capacity is 3. On the contrary, the PGD method achieves better robustness as the capacity increases. Therefore, there is a possibility over-fitting problem that arises form the high margin property of sensible adversarial learning.
16

---

Under review as a conference paper at ICLR 2020
The margin distributions for the varying the loss parameter
capacity
capacity
Natural margin
Adversarial margin
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
20
40
-40
-20
20
capacity
capacity
Natural margin
Adversarial margin
0.06
0.06
density
0.04
0.04
0.02
0.02
0.00
0.00
100
200
300
400
-200
200
400
capacity
capacity
Natural margin
Adversarial margin
0.025
0.020
0.02
0.015
0.010
0.01
0.005
0.000
0.00
100
200
300
-100
100
200
300
margin
0.1
0.5
0.9
Figure 5: The prediction margins at convergence of the models in Experiment the test set.
Experiment In Figure g we draw the plot of natural and adversarial margins on the testset of CIFARIO. Compared with the distribution of the TRADE model, the natural margin of the SENSE model is large. For the adversarial margins, the SENSE model has two clearly separated clusters; one is of negative margins and the other of positive margins. Instead, the positive margins are distributed on the larger values. We remark that this phenomenon consistent the sensible idea allow to fooled near the decision boundary of Bayes rule. When the capacity is not enough, the concept of the decision boundary is not of the Bayes rule. Its is projected to inevitable boundary of model with nice natural performance, caused by the lack of model capacity. The portion of negative adversarial margin obtain robustness much as possible given model capacity. We see portion adversarial margins of SENSE model in the negative area in Figure iis not small. This may imply the current model capacity and the sample size for SENSE are not enough.
cost for sensibility and with this cost, the model
F COMPARISON WITH OTHER METHODS
The sensible selection and reversion in approach distinguish sensible learning from other approaches that balance between the natural and adversarial accuracy. The objective function of our algorithm be rewritten as follows. For f E
C())
T Σ (f.
| Rase [|
|Br] Rea (f))) By)++ Cyl (f
(11)
where (f, y) is defined in
17

---

Under review as a conference paper at ICLR 2020
The margin distributions for the varying capacity
0.1
0.1
Adversarial margin
Natural margin
0.06
0.06
0.04
0.04
0.02
0.00
0.00
50
50
75
0.5
0.5
Adversarial margin
Natural margin
density
0.06
0.06
0.04
0.04
0.02
0.02
0.00
0.00
-100
0
100
200
100
200
==
1.0
Adversarial margin
Natural margin
0.06
0.06
0.04
0.04
0.02
0.00
0.00
-200
-100
100
50
100
150
margin
capacity
3
4
5
Figure 6: The prediction margins at convergence of the models in Experiment the test set.
Kurakin 2016b) suggest adversarial training that randomly divides into two parts, natural and adversarial set. The objective function can written as the following. For random index sets and C' s.t. |A| + +CI = n and Anc Φ
CN = A + AIO
1
Σ
[((((s). Mb) + Σ ((S(E.).
sEC
|A] |A| + X|C|
|C|
~ X|C| R (((C)
(12)
where F. is an adversarial example of Ir.
Our approach is similar in that also divide the data for different usage. However, divide the data random but an adaptive way, so that f" (f, 3i, yi) 2 (f, J.j Ms) 3; A, and æj Cy. Also, we have an additional set other than a natural and fully adversarial set. This additional set, sensibly reversed adversarial set, plays an important role in allowing a data point smoothly changes its identity between a natural example and a full adversarial example. Note that we do not have any weight controllers like À in (12 The hyperparameter itself controls the importance of the natural accuracy in comparison the adversarial accuracy.
Zhang al. 2019) investigate the Bayes decision boundary to resolve the trade-off problem. They propose TRADE, which the objective function for 3 > 0,
L() - ()
max X'EB(X.0)
E(F(X'),
(13)
This formulation shows several key differences between TRADE and SENSE, First, (13) uniformly restricts the perturbation for all data points, whereas [T] selects the sets for A,, B, and Cy and restricts
18

---

Under review as a conference paper at ICLR 2020
adveraria accuracy
natural accuracy
-Inf
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.00
0.75-
acc
0.50
0.25-
1
3
1
5
3
5
1
5
3
5
3
5
1
3
5
1
3
5
3
5
1
3
capacity
Figure 7: Another visualization of Figure The name of each panel denotes the hyperparameter c. -Inf denotes the model trained with the PGD method.
The margins the test set of CIFAR of the SENSE and TRADE model
0.15
0.2
density
method
density
method
SENSE TRADE
SENSE
TRADE
0.1
0.05
0.0
0.00
-10
10
-10
10
nat_maa mat_margin
adv Lav_margin
Figure 8: Adversarial and natural prediction margins on CIFARIO of the SENSE and TRADE model. The margins are calculated by M(J,2,) v) log (x) max,'yy log Py (x) 8y(x)
max 2y Sy (x), where s denote score function of f. i.c., the output of the neural network.
19

---

Under review as a conference paper at ICLR 2020
Table 6: MNIST: test results of the our models on natural examples and Loo based attacks.
Defence model
0.3
0.33
0.36
6= 0.39
SENSE
96.46
92.89
83.15
63.51
TRADE 96.72
90.56
46.94
11.66
Table 7: CIFAR: test results of the our models on natural examples and Lo based attacks.
Defence model
10/255
12/255
c= 14/255
16/255
SENSE
62.63
60.39
58.05
55.09
TRADE
47.61
39.05
31.57
25.15
the perturbation in different ways. Second, the main parameter 3 [3] controls the importance of the natural accuracy in comparison to the smoothness of the model. However, the main parameter in (IT) directly controls the lower bound of the natural and adversarial loss the individual data, This is the lower bound on the prediction probability с if é is the cross entropy loss. Third, the term ",
in 13 lcads the model be smooth to all directions in the input space, but e" (f, 3) in (11) leads be close Therefore, intuitively, TRADE achieves robustness by obtaining smoothness achieves robustness by reformulating (T) a way to promote high confidency
the model, whereas SENSE robust prediction. This may provide an intuition on the plot of the margin density in Figure X By this intuition, we apply PGD attacks with larger perturbation of the training We apply the attacks on our trained models and the TRADE model Table Table
by Zhang al. 2019) for MNIST and CIFARIO. The results
G ADDITIONAL INFORMATION ABOUT CHEESE HOLE DISTRIBUTION
Figure (e) compares worst-case standard ReaaSSrah and Reea( (Bn). Although Rase(()iss) does consistently decrease increases, is always much smaller than Rr Therefore, pursuit of robustness leads to more naturally accurate classifier. Figure 9 (e) shows that, as increases from c a/2 to 1, the standard risk the robustly optimal model gradually increases whereas the sensibly robust model keeps zero risk. Figure 9
demonstrates the robustness against e-ball attacks of ffl (black), fres (bluc), and fro (red). Although sensibly robust models have large adversarial robustness increasing to as increases, this because more and more adversarial examples can cross the border. while the model keeps its decision boundary consistent to the class border line. On the other hand, adversarially robust functions have constant robust risk for 1/4. This is because the robust functions predict as for every X.
Standard Risk
Adversarial Risk
X2
Risk
Risk
0.4
X1
X1
0.0
0.2
0.4
0.0
0.2
0.4
(e)
(f)
(g)
(h)
Figure 9: Cheese holes distribution. (c) and (f) The natural and robust risk when = 0.55. The black, blue, and red colors are the worst cases naturally, adversarially, and sensibly robust func- tions. (g) The robustly optimal model when e > 0.25. (h) The sensibly robust model when > 1/12.
The sketch of the proof on the standard and adversarial robust risks in (e) and (f) in Figure first calculate the three classes of functions which minimize natural, adversarial robust, and sensibly robust risk respectively, wart Px,.. Then for each class, consider the worst case function from each class, in that the function maximizes the standard risk Px... The corresponding stondard risks in Figure (c).
20

---

Under review as a conference paper at ICLR 2020
Likewise, for each class, consider the worst function from each class, in that the function maximizes the adversarial robust risk w.r.t Px,y. The corresponding adversarial robust risks are in Figure (f).
First, the minimizers of cach risk w.r.t Px,y are following.
1) Let FB be a set of naturally optimal functions w.r.t Px,y:
sign(x1 - 0.5) for (x1, E x
Let Frob be set of sensibly optimal functions w.r.t Px.y:
Fiu (f E Flf(2) sign(21 0.5) for (I1, B(),
3) Let Frob be set of robustly optimal functions w.r.t
Fras (S E g(x) for (x1, E B(λ,e))
where
g(x) = sign(1i - 0.5 +f a/2 2 3a/4 ((112005--)) - (31330/24.) if a/222/22222 S < 3n/4
(14)
Second, we consider the worst case standard risk w.r.t. Px, for each class above.
1) max JEfg (f) Although p * 0.5, due the symmetry the shape X, max Rataff) is the area on X' \2, the area outside the small nine squares.
2) max JEF: Retalf): For the same reason above, max JEFi, Reta(s is area on X' B( c). the area outside small nine squares extended by
3) mex Rai(f): When < a/2 or 2 3a/4, consider the deviated line on B(X, c). and regard the model outside B(X,e) incorrect. The risk is calculated easily by using the fact that the risk of any I E Frab on is ((², c) is 3 x min(a 2c, 2a) x (1 p) x min(2e, 0.5) 00.5 When a/2 & < < 30/4, since B(X,e) c) covers X. the worst case functions are in form of f(x = sign(æ1 - 0.5 + ce) for some e on the entire X. For each s.t. 3a/2 + æ1 < 0.5 it easy to find the corresponding ce.
Last, worst case adversarial robust risk w.r.t. Px,y, we can calculate the risks in similar way to above.
H ADDITIONAL INFORMATION ABOUT EXPERIMENTS
H. EXPERIMENT
We consider the MNIST and CIFAR-10 dataset (LeCun al. 2010 Krizhevsky & Hinton 2009).
Training For each dataset, initialize model a naturally trained model. Then, train the initialized model with sensible adversarial examples with the specifications in Table
Table 8: The learning specifications for the SENSE models in experiment 1
Dataset
K
Initial m/2
Epoch
MNIST
0.3
0.05
10
0.5
0.01
500
CIfARI0
X
10
0.7
0.1
300
255
255
10
Testing with white-box attacks For white-box attacks, we consider PGD (Madry et al. 2017 C&W
(Ding et al. 2019a). DeepFool (Moosvvi-Dezoooli et al. 2016). FGSM (Kurakin al. (2016a). LBFGS (Taba- cot & Valle 2016), and MIFGSM (Dong et al. 2018). In Experiment with E. norm less than 6, where 0.3 the MNIST dataset and = 8/255 the CIFARI dataset.
consider adversarial perturbations
We attack our models with the white box attacks. We use the attacks implemented in Foolbox (Rauber et all 2017). Advertorch (Ding et al.) 20192). and Adversarial Robustness 360 Toolbox (ART) (Nicolae et al. 2018) The attack specifications are in Table 9 The options that not listed in the table are kept default of the attack generating functions.
21

---

Under review as a conference paper at ICLR 2020
Table 9: The white-box attack specifications. We denote the step size and step number by on and K.
Dataset
Attack
7/1
K
Python package
Function
MNIST MNIST
PGD500 C& W40
0.01
500
Advertorch
LinfPGDAttack
0.01
40
ART
CIFARIO
PGDI00
2 255
100
Advertorch
LinfPGDAttack
CIfARIO
C& W40
40
ART
CarliniLInfMetiinn
255
20
CIfARIO
DeepFool
default
default
Foolbox
DeepFoolLinniiig Attack
CIfARIO
FGSM
8 255 default 255
Advertorch
GradientSign Attack
CIfARIO
LBFGS
default
Foolbox
LBFGSA
CIfARIO
MIFGSM
40
Foolbox
NomeentumleeratiicAttaaee
40.001
(distance=Linfnnity
return return.early- early=False)
We set the step size for MIFGSM as slightly smaller than 265 from having the perturbation norm greater than 8/255. For the performance of TRADE C& W40, apply C& W40 attack with the same specifications Table 9 For other results TRADE in Table [] we refer to Zhang et al. (2019). For PGD attacks, we draw accuracy and loss plots for increasing step numbers in Figure[T0jndd Figure II They show that for the chosen step sizes, the chosen step numbers are enough to generate proper local maximizer of the PGD objective function. Particularly, for CIFAR, 2/255 rather than x 2/255 as it more efficient.
30 in order keep the adversarial example
the step size
PGD attack serenity check 500 seem enough by Figure
We test with 100 random restarts, and for the step size 0.01, the step number
MNIST with PGD for the first 100 examples PGD loss
PGD accuracy
1.00
0.7
0.6
0.98
0.5
0.97
0.4
0.3
0.95
0.2
0.1
0.94
250
500
750
1000
250
500
750
1000
Figure 10: The convergence check for the PGD attacks on the MNIST model. We used step size 0.01. We can see that K 500 is enough to achieve the lowest point by counting the worst case of random restarts.
Testing with black-box attacks attack our models PGD40 and MIFGSM swith specifications Table [ As the Foolbox implementation for MIGSM only returns the successful attacks on the generating model, we only apply these attacks on the defense model. We note that the argument return sourr_early of MIFGSM is set to False as in Table g For TRADE, we use the models by Zhang et al. (2019) for both MNIST and CIFAR10.
22

---

Under review as a conference paper at ICLR 2020
CIFAR10 with PGD for the first 100 examples
PGD loss
PGD accuracy
0.85
0.75
0.70
0.65
0.60
50
100
150
200
50
100
150
200
Figure 11: The convergence check for the PGD attacks on the CIFAR model. We used step size 0.01 and for CIFAR. 2/255. We can see that K 100 is enough to achieve the lowest point by counting the worst case of random restarts.
H.2 EXPERIMENT 2
Model Architecture We conduct Experiment 2 on the MNIST dataset (LeCun et al. 2010 We consider sequence of CNNs with the increasing number of kernels. A network capacity q convolutional layers with 2(d-1) and 2 filters respectively, followed by fully connected lincar layer of 2(d+4) units. Each layer is activated by ReLU. Each convolutional layer followed by x max-pooling layer. The size of all convolutional filters is 5.
With similar sequence of CNNs, Madry al. (2017 investigate the model behavior when the capacity creases. They have capacity scale 1,2,4.8 and 16. In their experiment, capacity scale and 2 collapse. our capacitics 2 and 3 are comparable to the capacity scale and 2 Madyy et al. (2017). Likewise, our capacities 4 and 5 comparable to their capacity scale 4 and $ capacity 1,2 and collapse, is consistent to the result by Madry al. (2017).
Therefore, our result, which shows the PGD models of
Training We train the sequence MNIST models with sensibly adversarial example with 0.3, γ). 0.05 and K = for varying c E І0.0, 0. 0.9}. The initial learning rate 722 is 0.01, and we train for 500 epochs. When training the PGD models, we use the same hyperparameters except c.
Testing The MNIST models are tested with foo PGD attacks of e 0.3 with the step number K = 40 and step size = 0.01. We generate the attacks by using Python package Advertorch by Ding et al. (201b).
H.3 ADDITIONAL EXPERIMENT FOR TABLE 6 AND TABLE
We conduct our additional experiment on the MNIST and CIFAR- 10 dataset (LeCun et al. 2010 Krizhevsky & Hinton, 2009). For each dataset, consider the SENSE model trained in Experiment and the TRADE model by Zhang al. (2019). We note that for each dataset, the TRADE and SENSE model share the same architecture. On theses models, we apply the PGD attacks with perturbations larger than the training e. generate the PGD attacks, using a Python package Advertorch by Ding et al. (201b) with the following attack specifications.
MNIST
Let = 0.3, which is the training for each model. For å E [1.1, 1.2, 1.3), we generate the foc PGD attacks of e deg with the step number K 40 8 and step size 7 0.01.
Our models could he slightly smaller than the counterparts by Madry et layers do apply any padding.
2017
because our max-pooling
23

---

Under review as a conference paper at ICLR 2020
CIFAR-10 Let cu 8/255, which is the training for each model. For 8 [10 12
16 }. generate
the &oo PGD attacks of deu with the step number K 40 x step size ny. Eu/ 20.
24


Reviews:
[{"review_id": "Bylzgg27qB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes the notion of a \"sensible\" adversary that does not perturb data points on which the Bayes-optimal classifier is incorrect. The authors then provide theory showing that minimizing robust risk against such a sensible adversary yields the Bayes-optimal classifier, which addresses the question about standard vs. robust risk posed in prior work. On the experimental side, the authors then introduce a simple yet effective variation of adversarial training / robust optimization. Instead of maximizing the loss over the perturbation set, the proposed variant stops as soon as the loss exceeds a certain threshold. This can be seen as a variant of gradient clipping that reduces the influence of examples with a very high loss. The authors show that their modification yields an 8 - 9% improvement in robust accuracy on CIFAR-10, which gives state-of-the-art performance.\n\nI find the empirical improvements achieved with their modification of PGD-style adversarial training very interesting and recommend accepting the paper. However, it is not clear to me how well the theory is connected to the empirical findings. Moreover, there are additional experiments the authors can conduct to investigate the performance of their method more thoroughly. Concretely:\n\n- The theory relies on the \"reference model\" f_r being the Bayes-optimal classifier, while the experiments use the current model as reference model. Especially early in training, the current model performs significantly worse than a Bayes-optimal classifier. Moreover, it is unclear if the proposed training modification is effective if the reference model f_r is a separate classifier. It would be interesting to use a separately trained CNN (standard training without any robustness interventions) as a reference model to see if the training modification still yields improvements.\n\n- If the improvements of the proposed method come from the loss of a few adversarial examples dominating the overall loss in a batch, it would be interesting to measure and plot the loss distribution over examples in a batch with experiments.\n\n- As a baseline, comparing the proposed approach to clipping the loss or gradients of each example would be interesting.\n\n- In the robustness evaluation, have you experimented with randomly-restarted PGD?\n\n- To ensure that PGD works as intended, it would be helpful to see a plot of PGD iteration vs. adversarial accuracy.\n\n- It would be interesting to see accuracy numbers (standard and robust) for models trained with different values for the parameter c. This would also provide information about the sensitivity of this hyperparameter.\n\n\nFurther comments:\n\n- I encourage the authors to release their code and pre-trained models in a format that is easy for other researchers to build on (e.g., PyTorch model checkpoints).\n\n- Page 3, \"Note that R_rob(f^B) = P(X_1 [...]\" - should this be X instead of X_1?\n\n- Line 9 of Algorithm 1: should the sum go from 1 to m?\n\n- Equation 5: is \"x <= log 1/c\" in the subscript a typo?\n\n- Page 8: \"Our model achieves 91.51natural accuracy.\": percent symbol and space missing", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygQh5fMoH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper studies the phenomenon of trade-off between robust and standard accuracies that is usually observed in adversarial training. Many existing studies try to understand this trade-off and show that it is unavoidable. In contrast, this work shows that under a sensible definition of adversarial risk, there is no trade-off between standard accuracy and sensible adversarial accuracy. It is shown that Bayes optimal classifier has optimal standard and sensible adversarial accuracies. The authors then go on to propose a new adversarial training algorithm which tries to minimize the sensible adversarial risk. Experimental results show that models learned through the proposed technique have high adversarial and standard accuracies. \n\nComments:\n1) Sensible Adversarial Risk: The first contribution of the work is to define a new notion of adversarial risk which the authors call ''sensible adversarial risk'' and study its properties.  There is a recent work[1] which also proposes a new definition of adversarial risk and which is similar in spirit to what the current work tries to achieve.  In [1], the authors define a perturbation as adversarial only if it doesn't change the label of the Bayes optimal classifier, which is similar to what the current paper does. Owing to this similarity, the properties of sensible adversarial risk obtained in Theorems 1,2 in the current work look similar to [1]. So the authors should discuss/compare their results with [1].\n\n2) Sensible Adversarial Training: I believe the major contribution of the paper is to propose an algorithm for minimizing sensible adversarial risk. However, I have some concerns with the proposed algorithm. The authors say that since the Bayes optimal classifier is unknown, they use a reference model f_r (which can be naturally trained). Consider the following scenario. Suppose the true data is given by (x, f^B(x)), for some unknown f^B; that is, the Bayes optimal classifier has perfect standard accuracy. Suppose f_r has perfect standard accuracy, but very bad adversarial accuracy on train and test sets. Suppose f_r is substituted for f^B in the sensible adversarial risk (with 0-1 loss). Then it is easy to see that f_r is a minimizer of the resulting objective. So the proposed algorithm will just output f_r in this scenario. This is clearly not desirable. Given this, I believe a more thorough understanding of the proposed algorithm is needed. When will the algorithm converge to non-robust classifiers? How should one initialize the algorithm to avoid such undesirable behavior?\n\nWhile the notion of sensible adversarial risk is sensible, it is not clear why it should result in such high adversarial accuracies as reported in Tables 1,5. The adversarial perturbation of epsilon=8/255 on cifar10 is considered so small that sensible adversarial risk (with reference model f^B) at any point will almost always be equal to the existing notion of adversarial risk at that point. So, minimizing sensible adversarial risk (assuming you are given f^B) is exactly equivalent to minimizing adversarial risk. But it is known that minimizing adversarial risk results in models with low standard accuracies. So I feel sensible adversarial risk is not the reason behind such high accuracies. Could the authors explain what is the reason for such good adversarial accuracies reported in the paper?\n\n3) Experiments:  While the experimental results on cifar10 look impressive, I have some concerns about the way the PGD attacks are run. I downloaded the model provided by the authors and ran PGD attack on it.  I ran L_infty attacks with epsilon=8/255, step size=2/255. The results I obtained seem to differ from the results presented in the paper:\nPGD Steps | Adversarial accuracy of SENSE\n20            62.09\n50            60.34\n100           59.99\n\nI believe PGD attack with multiple random restarts will reduce the adversarial accuracy even further. Given this, I'd appreciate if the authors perform more careful attacks (with appropriate hyper-parameters) on their model. It'd also be great if the authors report the performance of PGD trained model using the same attacks used to report the performance on their model. \n\n4) Other comments:  I'm not sure if the toy example (cheese hole distribution) in Section 2 is helpful. What the authors seem to conclude from it is that adversarial training can improve standard accuracy. But I do not agree with these conclusions. What if Figure 1b is the true distribution? Will the same conclusions hold? In general, these toy examples need not be illustrative of the behavior on real datasets. So instead of having these toy examples, I'd suggest the authors have a thorough discussion on theoretical and experimental results.   \n\n[1] Suggala, A. S., Prasad, A., Nagarajan, V., & Ravikumar, P. (2018). Revisiting Adversarial Risk. arXiv preprint arXiv:1806.02924.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkx94fgQqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated from the so-called trade-off between robustness and standard accuracy in the existing adversarial learning, this paper has proposed a \"sensible\" adversarial example framework without losing  significantly  performance in natural accuracy. Some toy examples have been presented, showing its reasonableness of the model. The proposed algorithm looks very simple, but it appears that it could be effective through some experiments on two data sets.\n\nThough the reviewer did not fully understand all the details, the proposed idea  seems reasonable. In general, the paper chose those adversarial examples that won't change the predication class. Such adversarial examples, called sensible adversarial examples, mean the perturbation which may not mislead the decision boundary.\n\nThere are two concerns with the paper from the reviewer.\n\n(1) The paper states that there is always a trade-off between between robustness and standard accuracy in the adversarial learning work, this seems arguable to the reviewer. In some adversarial example learning literatures, the adversarial learning appears able to improve the performance of both natural examples and adversarial examples.  For example, the VAT approach published in PAMI. The reviewer would like to see more comments about this.\n\n(2) Though there are some theoretical analysis in the paper, the empirical validation may not be very convincing. MNIST and CIFAR 10 are relatively easier datasets, I am not sure if a same observation can be attained when the algorithm is applied on some more complicated and challenging dataset. It remains unclear to me if the \"sensible\" way could indeed be consistently useful in practical and more challenging cases.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your detailed feedback to the reviewers, which clarified us a lot in many respects.
However, there is still room for improvement; for example, convergence to a good solution needs to be further investigated.
Given the  high competition at ICLR2020, this paper is unfortunately below the bar.
We hope that the reviewers' comments are useful for improving the paper for potential future publication.
Decision: Reject