Title: SENSIBLE ADVERSARIAL LEARNING

Abstract: The trade-off between robustness and standard accuracy has been consistently reported in the machine learning literature. Although the problem has been widely studied to understand and explain this trade-off, the problem seems to remain as an open problem. In this paper, motivated by the fact that the high dimensional distribution is poorly represented by limited data samples, we introduce sensible adversarial learning and demonstrate the synergistic effect between pursuits of natural accuracy and robustness. Specifically, we define a sensible adversary which is useful for learning a defense model and keeping a high natural accuracy simultaneously. We theoretically establish that the Bayes rule is the most robust multi-class classifier with the 0-1 loss under sensible adversarial learning. We propose a novel and efficient algorithm that trains a robust model with sensible adversarial examples, without a significant drop in natural accuracy. Our model on CIFAR10 yields state-of-the-art results against various attacks with perturbations restricted to ∞ with = 8/255, e.g., the robust accuracy 57.23% against PGD attacks as well as the natural accuracy 91.51%.

1. INTRODUCTION:

With many impressive successes of deep learning, there are a multitude of applications of deep neural networks (DNNs) that permeate in our everyday life. As DNNs are applied in securitycritical systems such as malware detection, face identification, and autonomous driving, robustness of DNNs against adversarial attacks, i.e., the intently perturbed inputs to fool the system, has become an important research topic (Szegedy et al., 2013; Papernot et al., 2016; Biggio et al., 2013) .

One of the most widely studied classes of adversarial perturbations is p -norm constrained adversarial perturbations (Szegedy et al., 2013) . Madry et al. (2017) formalize the adversarial learning against this class of perturbations as a minimization problem of adversarial risk defined in a following way. Let (X, y) ∈ X × Y be from some unknown distribution P X,Y . Given a loss function : Y × Y → R and a constraint constant > 0, the adversarial robust risk is

EQUATION

Many adversarial learning methods can be interpreted as empirical minimization of (1) (Goodfellow et al., 2014; Kurakin et al., 2016b; Ruitong Huang & Szepesvari, 2015; Madry et al., 2017) . For this optimization problem, Madry et al. (2017) propose to train a robust model with the augmented data generated by the projected gradient descent method (PGD). On this adversarial training, they make two important observations. First, it costs natural accuracy. A network trained with adversarial examples tends to have a lower natural accuracy than a naturally trained network. This trade-off is observed even with a small training . Second, the adversarial training requires a larger model capacity than the natural training does. If the model capacity is only sufficient for the natural learning, the adversarial training can converge to a constant function.

For a large , the optimization problem of (1) itself may pose the trade-off. For instance, Tsipras et al. (2018) show an example of an inherent tension between pursuits of accuracy and robustness when is large enough to change the true class. For a smaller , however, the formulation in (1) does not explicitly pose any conflict between the pursuit of robustness and accuracy. Note that R rob (f ) is an upper bound of the standard risk of f . A perfectly robust model f with R rob (f ) = 0 is also perfectly accurate for natural learning. If the perfect classifier exists in a given model class, the trade-off may be caused by the large sample complexity of adversarially robust generalization (Schmidt et al., 2018; Yin et al., 2018; Stutz et al., 2019) . Without sufficiently large amount of data, the empirical minimization of (1) may result in a large standard risk by converging to a model of a poor robust risk. On the other hand, if robust learning converges to a constant function, it cannot achieve natural accuracy. In this sense, to resolve the trade-off problem, we may need to deal with the increased requirement on the model capacity.

In this paper, we propose a novel framework, called sensible adversary, in order to overcome the trade-off between natural accuracy and robustness. In particular, we restrict adversarial perturbations not to cross the Bayes decision boundary besides the -ball constraint, so that the perturbation ball is adaptively modified for every single data point. Our main contributions are:

• Under the framework of sensible adversary, the pursuit of robustness and accuracy given an enough model capacity can align with each other, i.e., there is no trade-off. We theoretically establish the Bayes rule is most robust against the sensible adversary. If the Bayes decision boundary can be far from data manifolds at least by , our pursuit of sensible robustness does not cost any adversarial robust risk.

• We propose an efficient algorithm for sensible adversarial training , which utilizes sensible adversaries in the absence of the true Bayes rule. This sensible adversarial training enjoys robustness without a significant drop of natural accuracy. Furthermore, the algorithm is not sensitive to the model capacity. When insufficient model capacity is given, our algorithm does not collapse to a constant function. Instead, it trains a model as robust as possible.

• We experimentally demonstrate that sensible adversarial training enables to stably learn a robust and accurate model. In particular, on CIFAR10, we achieve 91.51% natural test accuracy and 57.23% robust test accuracy against ∞ PGD attacks constrained to = 8/255. To the best of our knowledge, there is no approaches known to achieve natural accuracy more than 90%, while achieving more than 55% of robust accuracy against PGD attacks of = 8/255. Moreover, no previous approaches pursuing robustness against this attack achieved the natural accuracy more than .

1.1 RELATED WORK Madry et al. (2017) formalize adversarial learning as a mini-max problem given perturbation restriction, and theoretically and empirically established the feasibility of the optimization. Our sensible adversary redefines the set of perturbation in the inner maximization problem on which their theoretical result is directly applicable. Tsipras et al. (2018) investigate the possible source of robust trade-off. The key idea is that when there are features that are useful for natural classification but vulnerable to adversarial perturbations, a robust model would abandon these features because otherwise all of these features can adversarially move to promote incorrect prediction. Our work explores the possibility of learning a robust model while not allowing such collective adversarial migration. While a class change by adversarial examples typically has been prevented by using a small , Suggala et al. ( 2018) explicitly ignore an adversarial perturbation that crosses the decision boundary of a Bayes rule. Zhang et al. (2019) also investigate the Bayes decision boundary to resolve the trade-off problem. They search for a model f having a small weighted sum of the natural risk and a probability that an adversarial example can cross the decision boundary of f . Gilmer et al. (2018) show that in high dimensional setting, even small test error can imply the existence of adversarial examples for most of data points. Our effort to prioritize natural accuracy to find a robust model is consistent to the view in Gilmer et al. (2018) . More related work will be presented in Appendix A.

1.2. NOTATION:

Let F denote the class of functions represented by DNNs with a fixed architecture. An optimal robust model w.r.t. P X,Y is denoted by

f rob = arg min f ∈F R rob (f ). Denote the stan- dard risk w.r.t P X,Y by R std (f ) = E X,Y [ (f (X), y)]

and its optimal natural model by 

f std = arg min f ∈F R std (f ). Let PX,Y = P X,Y | X ×Y denote a restricted distribution of P X,Y on a subset X × Y ⊂ X × Y.

(f, ) = {x ∃x ∈ B(x, ) s.t. f (x) = f (x )}.

Let pf,y (x) denote the predicted probability of the label of x being y by a model f .

2. ADVERSARIAL LEARNING MAY HELP STANDARD LEARNING:

In this section, we use a toy example to investigate the synergistic effect between the pursuit of robustness and natural accuracy.

Example: We can easily find an example of trade-off wherever an -ball attack can cross the manifold boundary between classes. For example, consider a two-dimensional random vector X = (X 1 , X 2 ) on X = (0, 1) × (0, 1) with a binary class Y ∼ Ber(p), where p > 0.5. Let the conditional distribution be

(X 1 , X 2 )|Y = 0 ∼ U nif ((0, 1 2 ) × (0, 1)) and (X 1 , X 2 )|Y = 1 ∼ U nif (( 1 2 , 1)×(0, 1)).

Then the Bayes Rule is f B (x) = sign(x 1 -0.5), and it is a perfect classifier, in that R std (f B ) = 0 with as the 0-1 loss. If p > 0.5, the robust classifier against -ball attacks is f rob (x) = sign(x 1 -(0.5 -)). Its decision boundary is deviated byfrom that of the Bayes rule, and this deviation costs natural accuracy by R std (f rob ) -R std (f ) = (1 -p) . However, when the data points reside in a high-dimensional space, the samples are too sparse to represent the underlying true distribution. To take this phenomenon into account, consider a distribution PX,Y = P X,Y | X ×Y on a subset X × Y ⊂ X × Y. Assume that we only observe data generated from PX,Y , and the training and test sets do not provide any information on X c × Y. Now by applying this support restriction to the example above, we demonstrate how adversarial learning dramatically changes from harming to benefiting natural accuracy.

The Cheese hole distribution: For the above example, we restrict the support on

X = ∪ 3 i=1 ∪ 3 j=1 H ij , where H ij = (( α 2 , 3α 2 )+2α(i-1))×(( α 2 , 3α

2 )+2α(j -1)), α = 1 6 , and j = 1, 2, 3. Therefore the sampling support X comprises of nine small squares equally spaced by α. This is illustrated in Figure 1 (a). Among all classifiers which predict the same as f B on X , the worst one, denoted by f B * , predicts exact opposite on x ∈ X c as illustrated in Figure 1 (b) . For this worst case, the true standard risk w.r.t P X,Y is 3 4 , although the standard risk w.r.t. PX,Y is zero. We argue that pursuing robustness can mitigate this discrepancy. Among minimizers of Rrob , let f * rob be the worst classifier in that it predicts incorrectly outside X as depicted in Figure 1 (c). On B( X , ), f * rob should correctly classify, except on A = {(x 1 , x 2 )|0.5 -≤ x 1 ≤ 0.5} ∩ B( X , ). In this situation, the inaccuracy of f * rob is compensated by its increased accuracy on B( X , ) \ A . Moreover, the composition of R rob ( f B * ) is particularly interesting, and it is easy to show that

R rob ( f B * ) = P f B * (Y ) = X, X ∈ DB( f B * , ) + 3/4,

where 3/4 is from the standard inaccuracy on X c . The major part of R rob ( f B * ) comes from the inaccuracy of f B * outside of the sampling support X . Therefore by simply reducing this inaccuracy on X c , we can lower the robust risk by 3/4. Another interesting observation is that B = {(x 1 , x 2 )|0.5 -< x 1 < 0.5 + } is the only area where the Bayes rule f B has its robust risk. Note that R rob (f B ) = P(X ∈ B , f B (X) = Y ) = 2 which is greater than the optimal robust risk. The reason why f B is not most robust is simply because it is accurate on B . This prevents f B from being a robust function under the current adversarial robustness framework.

These observations have two implications. First, the robust pursuit can help to increase natural accuracy outside of X where the samples are poorly representative. Second, by accurately correcting the model outside of X in a standard sense, the robust risk can be significantly decreased. This is where the pursuit of robustness and accuracy coincides.

Toward respecting the original data structure: What if we regard the adversarial robust risk near the class border as reasonable gullibility? This corresponds to pursuit of robustness only against adversarial examples which do not cross between the class manifolds, i.e., as long as this does not harm the natural accuracy on X . We call this robustness as sensible robustness, which respects the structure represented by data. In this example, the sensible robustness can increase both robustness and natural accuracy. Let f s * rob denote the worst case sensibly robust classifier w.r.t PX,Y . The decision boundary of f s * should not deviate to the left on B( X , ) no matter how large is, as depicted in Figure 1 (d) . Note that when ≥ α/2, f * rob is the unique minimizer of Rrob (f ) as B( X , ) covers X .

3. SENSIBLE ADVERSARIAL ROBUSTNESS:

In this section, we introduce a sensible adversary framework. Consider a general multi-class case with the 0-1 loss, where Y = [K]. Assume the model capacity is enough so that the Bayes rule f B ∈ F. We consider p -norm constrained adversarial attacks, where p ∈ {0, 1, ..., ∞}. Definition 1. (sensible adversarial example) For a classifier f , let S x, (f ) = {z ∈ X | z -x p ≤ , f (z) = y}. Then the sensible adversarial example of (x, y) w.r.t f is defined as

x =    x, if f B (x) = y arg max z∈Sx, (f B ) (f (z), y),

otherwise.

(2)

Definition 2. (sensible robustness) Let the sensible adversarial loss be s rob, (f, x, y) = (f (x), y) where x is a sensible adversarial example as defined above. Then the sensible robust risk of a model f is defined by

EQUATION

We call its minimizer as a sensibly robust model w.r.t P X,Y and denote by f s rob , i.e., f s rob = arg min

f ∈F R s rob (f ).

Remark 1. Intuitively, a sensible adversarial example is an adversarial example restricted not to cross the decision boundary of the Bayes rule. In addition, a sensible adversary does not perturb a data point that the Bayes rule incorrectly classifies. Therefore, it is natural to expect that pursuing sensible robustness would not cost natural accuracy, and the following theorem confirms it. Theorem 1. Let R * std denote the minimum standard risk which is R std (f B ). Then we have R s rob (f B ) = R * std . Furthermore, f B is the unique minimizer of R s rob (f ) among f ∈ F. Theorem 2. Let S X, be an -ball centered at X. Then for any f ∈ F and for any set A ⊂ X \ DB(f B , ),

P ∃ x ∈ S X, s.t. f B (x ) = Y, X ∈ A ≤ P ∃ x ∈ S X, s.t. f (x ) = Y, X ∈ A .

Remark 2. According to the Theorem 1, a sensibly robust model w.r.t P X,Y is the Bayes rule, i.e., f s rob = f B . This sensible robustness costs adversarial robustness since f B may have a larger

1: Input: Initialized f = f θ , c ∈ (0, 1), step number K, step sizes η 1 , η 2 , data X (0) adv = X 2: repeat 3: for i = 1, ..., m, s.t. f (x (0) i,adv ) = y i 4: for k = 1, ..., K 5: x (k) i,adv ← Π B(xi, ) (η 1 sign(∇ x (f (x (k-1) i,adv ), y i )) + x (k-1) i,adv ), Π: the projection operator 6: if (f, x (k) i,adv , y i ) > log 1 c 7: (sensible reversion) x (K) i,adv = x (k-1) i,adv 8: break 9: θ ← θ -η 2 m i=1 ∇ θ (f, x (K)

i,adv , y i )/m 10: until training converged adversarial robust risk compared with f rob , a direct minimizer of (1). However, Theorem 2 shows f B is equally or even more robust than f rob except on a certain area. In particular, Theorem 2 shows that f B is the most robust model except on DB(f B , ). Therefore, f B is most robust almost everywhere if the decision boundary of f B can lie outside of B(X , ), so that X \ DB(f B , ) = X . For example, this happens when each class has its own support apart from each other by at least 2 .

The next theorem shows that even when we only have data from PX,Y restricted on X × Y, we can find f s rob , the optimal function w.r.t. P X,Y . Theorem 3.

Let A = f ∈ F P f (x) = f B (x), ∀x ∈ S X, (f B ) = 1 and Rs rob (f ) = E PX,Y [ s rob, (f, X, Y )].

Then, for any > 0, Rs rob (f ) is only minimized by any f ∈ A . Furthermore, if B( X , ) ⊃ X , f B is the unique minimizer of Rs rob (f ). Remark 3. It is interesting to compare our work with that of Suggala et al. (2018) . We notice there are some critical differences between these two works. Given Y ∈ {-1, 1}, Suggala et al. ( 2018) define an adversarial risk as

R adv (f ) = E[ max g(x)=g(x+δ), δ ≤ (f (x + δ x ), g(x)) -(f (x), g(x))].

Unlike our definition of sensible adversary, if g is a Bayes rule f B , their adversary tries to increase the loss w.r.t. not y but a deterministic function of x. Therefore, for making f B being an optimal robust model, their objective should always have an additional term, e.g.,

R nat (f ) + λR adv (f ) for 0 < λ < ∞ because an f s.t. f (x) = g(x) w.p. 1 can minimize R adv (f ).

Unlike their approach, our sensible adversarial risk (3) can alone be optimized making the Bayes rule as the optimal model.

4. ALGORITHM:

A transition from theory to algorithm poses two main challenges. First, the 0-1 loss function in the theory is hard to optimize. Therefore, as a common practice, we use the cross-entropy loss. Second, f B on the entire space is practically unavailable. We note that a model that performs well on natural data can be a nice approximation of f B on the restricted support X × Y. Therefore, we generalize sensible adversary in (2) to utilize a general loss function and a reference model that substitutes f B . Definition 3. (generalized sensible adversarial example) Consider a loss function . For a classifier f and c

∈ (0, 1], let S x, (f ) = {z ∈ X | z -x p ≤ , (f (z), y) ≤ log 1 c } ∪ {x}.

Then given a reference model f r , the sensible adversarial example of f for (x, y) is defined as

EQUATION

If is the cross-entropy loss, the condition (f r (x), y) ≤ log 1 c is equivalent to pfr,y (x) ≥ c. In binary case with c = 0.5, this requires the perturbed examples not to cross the decision boundary of f r , and for general c, not to reach to a vicinity of the boundary. In our algorithm we set f r as a naturally trained model or a current model. Given a f r , the implementation of sensible adversarial attacks is straightforward. For a correctly classified natural example by f r , we add perturbations in the similar way to the PGD method (Madry et al., 2017) . The difference is that during the K-step of PGD iterations, once the loss of a currently generated example exceeds log 1 c , we reverse it back to the previous step and break the iteration. This requires no additional forward-or backward-propagation, compared with the PGD method. The proposed algorithm is in Algorithm 1 for the ∞ -norm and in Algorithm 2 in Appendix C for the other p -norms.

The definition of the sensible adversary in (4) divides the data into three subsets: 

A fr = {x|f r (x) = y or S x, (f r ) = {x}}, B fr = {x|S x, (f r ) = S x, , S x, (f r ) = {x}}, and C fr = X \ (A f ∪ B f ).

(f, x, y)1 x∈A f + (f, xs , y)1 x∈B f + (f, xp , y)1 x∈C f = (f, x, y)1 (f,x,y)>log 1 c + (f, xs , y)1 xp =x s , (f,x,,y)≤log 1 c + (f, xp , y)1 (f,x p ,y)≤log 1 c .

(5) When c ≥ 0.5, this optimization problem has a strong motivation to increase |C f |, i.e., the number of sensible adversarial attacks that are identical to full PGD attacks. Because A f = {x| (f (x), y) > log 1 c } for c ≥ 0.5, the loss on the natural stage A f is always greater than the loss of the sensibly reversed stage B f and the full PGD stage C f . Furthermore, the loss for x ∈ B f is always approximately log 1 c by adaptive sensible perturbations. Therefore, paradoxically, the smallest loss is only achievable by full PGD examples. In other words, sensible adversarial training penalizes when x ∈ C f . We note that the optimization problem has a smooth landscape; Although the data points may jump around between A f , B f , and C f , there is no obvious discontinuity in the loss. Therefore, during the training, s (f, x, y) smoothly slides down, making both (f, x, y) and (f, xp , y) smaller than log 1 c . This enables to learn a naturally and adversarially accurate model. More discussion on the stability of our algorithms are presented in Appendix D.

5. EXPERIMENTS:



5.1. EXPERIMENT 1: ROBUSTNESS AGAINST -BALL ATTACKS:



CIFAR10:

We train WideResNet-34-10 (He et al., 2016) with sensible adversarial examples of = 8/255 and c = 0.7 with the step number K = 10 and size η 1 = /5. We attack our model with various white-box attacks with perturbations restricted to ∞ with = 8/255. We compare the performance with TRADE using the same architecture (Zhang et al., 2019) , which is know to be robust and accurate. The result is in Table 1 . Our model achieves 91.51% natural accuracy. This is 3.7% drop in natural accuracy from 95.2%, which is an accuracy that a naturally trained model can achieve (Madry et al., 2017) . With this architecture, Madry et al. (2017) achieve 47.04% robust MNIST We consider a CNN model with three convolutional layers followed by a fully connected linear layer, which is the same architecture in (Zhang et al., 2019) . We train an MNIST model with sensible adversarial examples of = 0.3 and c = 0.5 with the step number K = 10 and size η 1 = 0.05. The robust test result in Table 3 shows the comparable performance of our model. The step size of the PGD500 attack is 0.01, and the serenity check on the PGD attack is in Appendix H. We conduct 100 random restarts and count only the worst case for each test example.

5.2. EXPERIMENT 2: SENSITIVITY ANALYSIS:



MNIST::

We perform the sensibility analysis to understand the effect of c and the model capacity for two reasons. First, the sensible training prevents full PGD perturbations on an example until the loss on it becomes less than log 1 c , which could hardly happen for a model with a small capacity. Second, as Madry et al. (2017) point out, when the model capacity is insufficient for adversarial learning, the model collapses to a constant function. We are interested in a range of c that keeps sensible learning from collapsing. Therefore, we consider a sequence of CNNs of the increasing number of kernels similar to Madry et al. (2017) , where we denote the capacity by q ∈ {1, 2, 3, 4, 5}. The details of the model architectures are presented in Appendix H. When we train them with natural examples, the networks of capacity 1 and 2 achieve about 95% and 97% accuracy, whereas the networks of the other capacities achieve more than 99%. When trained with regular PGD examples, the networks with capacity 1,2, and 3 collapse. Therefore, capacity 3 is enough only for natural learning, and capacities 1 and 2 are possibly insufficient even for natural learning. For each capacity q, we train the models with sensible adversarial examples on MNIST with the hyperparameters c ∈ {0, 0.1, • • • , 0.9, 1}. Note that sensible adversarial training is identical to natural training when c = 1, and to adversarial learning without any perturbations for incorrectly classified natural examples when c = 0. Figure 2 shows the natural and robust accuracy against PGD-40 attacks with varying c for each capacity. In general, the natural accuracy tends to increase as c increases while robustness decreases. For capacities 3, 4 and 5, the accuracies are almost insensitive to varying c. On the other hand, the tendency is most distinct in capacity 1. In this case, the network obtains best robustness when c = 0.5, which is the least loss bound among c ≥ 0.5, the range of c with stable learning property. Even for capacity 1, however, the sensible learning does not collapse except when c = 0. We also see that c = 0.5 is least sensitive to the model capacity.

Note that capacities 2 and 3 do not collapse even with c = 0. When c = 0, the only difference of the sensible adversarial learning from the regular PGD training is that the sensible learning requires robustness only for the correctly classified natural examples. When c = 0, at the convergence of the models of capacities 1 and 2, about 5% of the data points are allowed to be free from the perturbation. By paying only the 5% of robust training accuracy, the sensible learning avoids collapsing and obtains about 80% of robust accuracy and 90% of natural accuracy.

Intuitively, an insufficient model capacity or locally close class manifolds can make a virtual decision boundary that is inevitable to keep a nice natural performance. In the algorithm, the sensible reversion prevents adversarial examples from crossing this boundary. This effectively reduces the requirement of the model capacity posed by the regular adversarial learning. The sensible reversion also allows a robust model to have larger margins than the PGD trained model for the majority of the dataset as in Figure 3 . The PGD trained model has majority adversarial margins as positive. Instead, it has much smaller natural margins than the naturally trained model. The SENSE model with c = 0.5 has comparably large adversarial and natural margins. Instead, the number of data points with negative adversarial margins is larger than that of the PGD trained model. As c increases to 0.9, this model has much smaller and more negative adversarial margins. On the other hand, its two types of margins are generally even larger. This phenomenon is consistent to the decreasing robustness in Figure 2 for capacity 5. In general, for a fixed capacity, increasing c increases the natural and adversarial margins of the majority of the data, while it also increases the portion of data of negative adversarial margins.

The cost for this highly confident prediction is the robustness near the decision boundary at convergence, i.e., the portion of the data points in the natural and sensibly reversed stage. When we consider the margin on the training set, we find that there is a linear relationship between the portion of the data points having negative adversarial margins and the test accuracy. In practice, such a portion can be an indicator about the robustness of the model or the sufficiency of the model capacity, without directly testing the model. Interestingly, while the natural accuracy positively correlated to the c value, the robustness does not show clear negative correlation to c. Rather, as c become closer to 0.7, more robust result the model shows. As we see that c = 0.8 has better robustness than c ≤ 0.5, in CIFAR, the main reason of the observed trade-off could attribute to the influential adversarial perturbations.

For the CIFAR dataset, we also try to train a robust model with a small capacity. We intentionally choose the CNNs model that is used in Experiment 1 for MNIST. This model can be not enough for adversarial training on CIFAR. We report the results in Table 5 . Given the serious lack of model capacity, the decrease in natural accuracy of SENSE models compared with the naturally trained model is not serious, while achieving better robust accuracy numbers. Note that none of our models collapse, and therefore the possibility of sensible adversarial training is not sensitive to the choice of c even in the lack of model capacity.

6. CONCLUDING REMARKS:

In this paper, we proposed a sensible adversary which is useful for learning a defense model, keeping a high natural accuracy simultaneously. We theoretically establish that the Bayes rule is most robust under the framework of sensible adversarial learning. Our learning algorithm is efficient and stable, and not sensitive to the choice of the main hyperparameter c. Also, c has a clear meaning as the lowest prediction-probability bound. Our empirical experiments yield state-of-the-art results of adversarial learning on the CIFAR10 and MNIST datasets. In addition, we showed that the sensible approach can effectively deal with the lack of model capacity. This is because by paying a robust accuracy on a certain area, the algorithm protects the model from being collapsed by influential adversarial examples. Furthermore, the sensible adversarial learning trains a model to have high prediction margins on both natural and adversarial examples.

We now mention several future directions for research on sensible adversary. One remaining theoretical problem is to develop generalization error bounds for sensible adversary learning, so that we can theoretically justify our empirical performance. In fact, in this work, we did not tackle the lack of sample size. In particular, as our algorithm tends to produce a model with large natural and adversarial prediction margins, in the lack of sample size, it is not clear if this large margins are always beneficial. Therefore, there is much remaining work to be done to theoretically understand the high margin tendency of the models trained with sensible adversarial examples with relation to generalization.

Appendices:

A ADDITIONAL RELATED WORK Bubeck et al. (2018) conjecture that learning a robust model is information theoretically possible but computationally intractable. They introduce an example which is not robustly learnable in polynomial time. Our view is consistent with Bubeck et al. (2018) in that a robust model exists, but we search for an efficient algorithm to estimate a robust model in a reasonable time. Su et al. (2018) compared various naturally trained models on ImageNet, and found that the trade-off varies among different model architectures. Also, they empirically discovered that more accurate models tend to be less robust when the models are trained with natural examples. Stutz et al. (2019) demonstrate that given a large training set, adversarial training can produce a robust model that is as accurate as a naturally trained model. Our work to maximize the synergistic effect between natural and adversarial accuracy is consistent to their demonstration. Also, they show that most of PGD attacks are offmanifold of the original data, and by on-manifold adversarial training, the natural accuracy can be improved.

If we see the restricted space X as the data manifold, and X c as off-manifold, our sensible framework aligns with their view, given the data manifolds of different classes are separated by at least 2 . Kurakin et al. (2016b) suggest adversarial learning that trains with data randomly divided into two parts, a natural and adversarial set. We divide data in a data adaptive way into three parts including a sensibly reversed adversarial set. The relationship between their approach and our algorithm is discussed more in Appendix F.

B PROOFS OF THEOREMS:

To save the space, we use η(k|x) to denote P(Y = k|X = x).

EQUATION

The last equality is by

1 f B (x) =k,f (x) =k = (1 -1 f B (x)=k )1 f (x) =k .

The first term R std (f ) is uniquely minimized by the Bayes rule f B . The second term is always non-negative, and is zero when

f = f B . Therefore, R s rob (f ) is uniquely minimized by f B .

Theorem 2. (Restated) Let SX, be an -ball centered at X. Then for any f ∈ F and for any set A ⊂ X \ DB(f B , ),

P ∃ x ∈ SX, s.t. f B (x ) = Y, X ∈ A ≤ P ∃ x ∈ SX, s.t. f (x ) = Y, X ∈ A . Proof. For any set A ⊂ X \DB(f B , ), P(∃x ∈ SX, s.t. f B (x ) = Y, X ∈ A) = P(f B (X) = Y, X ∈ A).

Note that on any subset B ⊂ X , the Bayes rule has the least error probability. Therefore, P(f

B (X) = Y, X ∈ A) ≤ P(f (X) = Y, X ∈ A) ≤ P(∃x ∈ SX, s.t. f (x ) = Y, X ∈ A). The last inequality is trivial because if f (X) = Y ⇒ ∃x ∈ SX, s.t. f (x ) = Y . Theorem 3. (Restated) Let A = f ∈ F P f (x) = f B (x), ∀x ∈ SX, (f B ) = 1 and Rs rob (f ) = EP X,Y [ s rob, (f, X, Y )].

Then, for any > 0, Rs rob (f ) is only minimized by any f ∈ A . Furthermore, if B( X , ) ⊃ X , f B is the unique minimizer of Rs rob (f ).

Proof. The sensible risk of f w.r.t. the restricted distribution corresponding to ( 6) is

EQUATION

because 8) is always non-negative. Therefore, only functions in A = f ∈ F| P f (X) = f B (X) = 1 need to be considered as possible minimizers of Rs rob (f ). Note that A ⊂ A. By the definition, we know that f ∈ A \ A if and only if

P(Y = k|X = x) = P(Y = k|X = x) for x ∈ X . The minimum of Rs rob (f ) is achieved by f B with the minimum value R * = Rstd (f B ) + 0. Therefore, any function f that Rstd (f ) > R * cannot achieve the minimum of Rs rob (f ) because the term (1 ∃x ∈Sx, (f B ) s.t. f (x ) =f B (x) -1 f (x) =f B (x) ) in (

i) P f (X) = f B (X) = 1 ii) P f (x) = f B (x), ∀x ∈ SX, (f B ) < 1 Therefore, for f ∈ A\A , ∃A ⊂ X s.t. P X ∈ A > 0 and ∃x ∈ Sx, (f B ) for ∀x ∈ A s.t. f (x ) = f B (x ).

For this f , the equation in (8) can be written as Rs rob (f ) = R * + α for some α ≥ 0. Now we show that α > 0.

Note that by the definition of the Bayes rule, P(y

= f B (x)|X = x) ≥ 1 K . Otherwise, 1 = K k=1 P(y = k|X = x) ≤ KP(y = f B (x)|X = x) < 1,

which is contradict. Then, for the f ∈ A \ A and A ⊂ X that are described above,

(1 ∃x ∈Sx, (f B ) s.t. f (x ) =f B (x) -1 f (x) =f B (x) )η(f B (x)|x)d P(x) ≥ 1 K X (1 ∃x ∈Sx, (f B ) s.t. f (x ) =f B (x) -1 f (x) =f B (x) )d P(x) = 1 K X (1 ∃x ∈Sx, (f B ) s.t. f (x ) =f B (x) )dP(x) by f ∈ A ≥ 1 K A (1 ∃x ∈Sx, (f B ) s.t. f (x ) =f B (x) )dP(x) = P(A) K > 0.

Therefore, α > 0. Note that for any f ∈ A , the second term in ( 8) is zero by the definition of A . Therefore, first result of the theorem is proved. Furthermore, for such that B( X , ) ⊃ X , A = {f B }. Therefore, when B( X , ) ⊃ X , f B is the unique minimizer of Rs rob (f ). Theorem 4 says that for fr ∈ A, which behaves as the Bayes rule f B on the restricted support, the corresponding sensible adversarial risk EP X,Y [ (f ( X| fr , Y )] is minimized only by fr. This implies that if we do not have any information about the Bayes rule on X c , the sensibly optimal model w.r.t. PX,Y can be arbitrary on X c although this optimal model is the Bayes rule on X . Our algorithm deals with this arbitrariness by searching for a better reference function in each iteration. As a current model is used as a reference function, i.e., the estimation of the defense model and the reference model is identical, the algorithm essentially pursues sensibleness on X and robustness on X c of the trained model. Note that on X , sensibleness a sufficient condition for natural accuracy.

Theorem 4. Let A = f ∈ F| P f (X) = f B (X) = 1 ,

Proof. By using the same way to derive ( 6) and ( 8) and noting that fr(x) = f B (x) on X , we get

EQUATION

Because fr(x) = f B (x) on X , Rstd (fr) = Rstd (f B ). This implies fr minimizes Rs rob (f |fr) because Rstd (f B ) is the minimum of Rstd (f ) for all f ∈ F and the second term in ( 9) is non-negative. This implies any function f s.t. Rstd (f ) > Rstd (f B ) cannot achieve the minimum of Rs rob (f |fr). Therefore, as a minimizer of Rs rob (f |fr), we only need to consider f ∈ A. Note that 1 f (x) =f B (x) = 0 for f ∈ A on X . Therefore, by letting R * std be Rstd (f B ), the equation in ( 9) can be written as

EQUATION

Note that η(f B (x)|x) is positive on X . Therefore, for f ∈ A to minimize (10), it must satisfy that P ∃x ∈ SX, (fr) s.t. f (x ) = fr(x ) = 0. This essentially says f should be fr.

C ALGORITHM:

Algorithm 2 Sensible adversarial training for p norm restriction

Input: Initialized f = f θ , c ∈ (0, 1), step number and sizes K, η 1 , η 2 , data X (0) adv = X 2: repeat for i = 1, ..., m, s.t. f (x (0) i,adv ) = y i 4: for k = 1, ..., K x (k) i,adv ← Π Bp(xi, ) (η 1 ∇x (f (x (k-1) i,adv ),yi) ∇x (f (x (k-1) i,adv ),yi) p + x (k-1) i,adv ), Π: the projection operator 6: if (f, x (k) i,adv , y i ) > log 1 c (sensible reversion) x (K) i,adv = x (k-1) i,adv 8: break θ ← θ -η 2 m i=1 ∇ θ (f, x (K) i,adv , y i )/m 10: until training converged

Algorithm 2 is a straightforward extension of Algorithm 1 from ∞ norm to p norm. In addition, we present another way to realize the sensible reversion. After stepping back when the loss exceeds the threshold, we can add a random noise as the next step instead of just breaking the iteration as in Algorithm 1 and 2. Considering the nature of (mini-) batch leaning, this random noise does not add much computational cost because until every loss exceeds the threshold, the for loop would keep calculating the forward and backward loop with fixed perturbation for the reversed examples. This algorithm for ∞ norm is presented in Algorithm 3, of which the objective function is also (5). Algorithm 3 also reverses the adversarial example when the loss is greater than log 1 c . The only difference is that instead of breaking the iteration after the reversion, it considers a random noise as the next step. This can lead to more effective search for the local loss maximizer in the ball x, (f ) in definition 4. The extension of Algorithm 3 to p norm is straight forward, and thus omitted.

Algorithm 3 Sensible adversarial training for ∞ norm restriction:

Input: Initialized f = f θ , c ∈ (0, 1), step number K, step sizes η 1 , η 2 , data X (0) adv = X repeat 3: for i = 1, ..., m, s.t. f (x (0) i,adv ) = y i for k = 1, ..., K xi ← Π B(xi, ) (step (k) i + x (k-1)

i,adv ), Π: the projection operator 6: Once an incorrectly classified natural example becomes correctly classified, its full PGD attack can pose a sudden large gradient for the model update. Then, when it is incorrectly classified again, the loss on it suddenly reduces to the natural loss that is distinctly smaller than the large loss value on its full PGD attack. Therefore, whenever an example changes its state between a correctly and incorrectly classified example, i.e., the full PGD and natural stage, the corresponding loss can fluctuate making the learning unstable.

EQUATION

However, if we add sensible reversion step, particularly if c ≥ 0.5, this fluctuation does not occur. Between the two stages, the sensibly reversed stage provides a kind of cushion between the natural and full PGD stage, preventing the sudden change in the loss value as described. Until Sx, (f ) = Sx, , the sensibly adversarial perturbation for x is adapted to make the loss of the current function approximately equal to log 1 c . As the abstraction in Figure 4 , it is like to have a virtually extended area at pf,y (x) = c on the loss function that is locally flat on {x s |Sx, (f ) = Sx, }, i.e., on a set of xs that are in the sensibly reversed stage. However, in spite of the existence of such a flat loss area, the model still can learn with xs in sensibly reversed stage. This is obvious because the cross-entropy loss has its non-zero gradient when it is log 1 c . Interestingly, when the model is updated in a way to decrease the loss of the previous xs , the new sensible perturbation is again adapted to have the loss approximately equal to log 1 c . Therefore, the course of training directs the model to have sensible adversarial examples in the full PGD stage. Furthermore, as long as c > 0, even when xs is in a sensibly reversed stage, it still has perturbation in an adapted -ball. This helps to obtain a robustness although it is not on a full ball.

Note that for c ≥ 0.5, the learning is very stable because only natural examples can overpower the training;

The large loss values are only achievable by sensible attack in the natural stage. Any single full PGD attack cannot dominate the next update of the training because the loss function has relatively small gradients on the full PGD stage. Therefore, our algorithms not only effectively ignore any influential full PGD attacks that may overpower the next update but also train a model that allows as many sensible attacks to be full PGD attacks. In other words, our algorithms can stably learn a robust model.

E HIGH MARGIN PROPERTY OF SENSIBLE ADVERSARIAL TRAINING:

In sensible adversarial learning, the natural accuracy clearly takes priority over the adversarial accuracy. The perturbed example is not allowed to cross the decision boundary of the Bayes rule in (2) or to reach the vicinity of the decision boundary of a reference function in (4). As mentioned, when the cross-entropy loss is used, pf,y (x) ≥ c is a prerequisite for adding any adversarial perturbation on x. We note that this condition provides Therefore, the priority of natural accuracy hints that the learning will prevent the natural margin from being sacrificed for the sake of adversarial robustness. We note that the natural margin of x is an upper bound of its adversarial margin. Therefore, if a model cannot confidently predict a natural example x, neither can the model confidently predict any adversarial examples of x.

Experiment 1 We investigate the margins of the models trained in Experiment 2, to understand the effect of c. In Figure 5 we draw density plots of the margins on the test set for varying c for the fixed capacities. Overall, we see that a larger c results in a larger adversarial margins and natural margins. In Figure 6 , we also draw the margins but for varying model capacity. In general, for a each c, a smaller capacity has more data points of negative margins. However, for naturally trained models, i.e., the models with c = 1.0, a larger model has smaller adversarial margins. This is consistent to the observation of Su et al. (2018) that accurate models tend to be less robust when the models are trained with natural examples. On the other hand, although not displayed, the plots corresponding to c = 0.9 are essentially similar to the plots in the second low. This implies even for large c, our method is not like natural learning; The models trained with c = 0.9 still have larger natural and adversarial margins.

Note that in Figure 3 , the regular PGD model of capacity 5 has only few negative adversarial margins. Instead, their natural margins significantly smaller than those of the natural models. In contrast, our model has negative adversarial margins for a few more data points. Note that the majority of both natural and adversarial margins of the our models are significantly larger than that of the regular PGD models. For capacity 1,2 and 3, the regular PGD models collapse having small "mean" of adversarial losses. On the contrary, our models deal with the lack of model capacity by letting more portion of examples to have negative margins. As demonstrated by Figure 5 , in our methods the mean adversarial loss can be arbitrarily large. Instead, it maintains a large portion of points having relatively large adversarial margins, i.e., being far from decision boundaries.

The flip side of the advantage of high margin is the possibility of over fitting. For c ≥ 0.5 in Figure 7 , the best robust accuracies are not achieved when capacity is 5 but when capacity is 3. On the contrary, the PGD method achieves better robustness as the capacity increases. Therefore, there is a possibility of over-fitting problem that arises form the high margin property of sensible adversarial learning. Experiment 2 In Figure 8 , we draw the plot of natural and adversarial margins on the testset of CIFAR10.

Compared with the distribution of the TRADE model, the natural margin of the SENSE model is large. For the adversarial margins, the SENSE model has two clearly separated clusters; one is of negative margins and the other is of positive margins. Instead, the positive margins are distributed on the larger values. We remark that this phenomenon is consistent to the sensible idea to allow to be fooled near the decision boundary of the Bayes rule. When the capacity is not enough, the concept of the decision boundary is not of the Bayes rule. Its is projected to an inevitable boundary of a model with nice natural performance, caused by the lack of model capacity. The portion of negative adversarial margin is a cost for sensibility and with this cost, the model can obtain robustness as much as possible given a model capacity. We see the portion of adversarial margins of the SENSE model in the negative area in Figure 8 is not small. This may imply the current model capacity and the sample size for SENSE are not enough.

F COMPARISON WITH OTHER METHODS:

The sensible selection and reversion in our approach distinguish sensible learning from other approaches that balance between the natural and adversarial accuracy. The objective function of our algorithm can be rewritten as follows. For f ∈ F ,

EQUATION

where s (f, x, y) is defined in (5). 

EQUATION

)

where xi is an adversarial example of xi.

Our approach is similar in that we also divide the data for different usage. However, we divide the data by not a random but an adaptive way, so that s (f, xi, yi) ≥ s (f, xj, yj) for xi ∈ A f and xj ∈ C f . Also, we have an additional set other than a natural and fully adversarial set. This additional set, a sensibly reversed adversarial set, plays an important role in allowing a data point smoothly changes its identity between a natural example and a full adversarial example. Note that we do not have any weight controllers like λ in (12). The hyperparameter c itself controls the importance of the natural accuracy in comparison to the adversarial accuracy. 

EQUATION

This formulation shows several key differences between TRADE and SENSE. First, (13) uniformly restricts the perturbation norm to for all data points, whereas (11) selects the sets for A f , B f and C f and restricts the perturbation in different ways. Second, the main parameter β in (13) controls the importance of the natural accuracy in comparison to the smoothness of the model. However, the main parameter c in (11) directly controls the lower bound of the natural and adversarial loss of the individual data. This is the lower bound on the prediction probability c if is the cross entropy loss. Third, the term (f (X ), f (X)) in ( 13) leads the model to be smooth to all directions in the input space, but s (f, x, y) in ( 11) leads to be close to one. Therefore, intuitively, TRADE achieves robustness by obtaining smoothness of the model, whereas SENSE achieves robustness by reformulating (1) in a way to promote high confidency of the robust prediction. This may provide an intuition on the plot of the margin density in Figure 8 . By this intuition, we apply PGD attacks with larger perturbation of the training . We apply the attacks on our trained models and the TRADE model by Zhang et al. (2019) for MNIST and CIFAR10. The results are in Table 6 and Table 7 . The sketch of the proof on the standard and adversarial robust risks in (e) and (f) in Figure 9 We first calculate the three classes of functions which minimize natural, adversarial robust, and sensibly robust risk respectively, w. First, the minimizers of each risk w.r.t. PX,Y are as following.

G ADDITIONAL INFORMATION ABOUT CHEESE HOLE DISTRIBUTION:

1) Let FB be a set of naturally optimal functions w.r.t PX,Y :

FB = {f ∈ F|f (x) = sign(x1 -0.5) for (x1, x2) ∈ X }

2) Let Fs rob be a set of sensibly optimal functions w.r.t PX,Y :

Fs rob = {f ∈ F|f (x) = sign(x1 -0.5) for (x1, x2) ∈ B( X , )}.

3) Let Frob be a set of robustly optimal functions w.r.t PX,Y :

EQUATION

Second, we consider the worst case standard risk w.r.t. PX,Y for each class above.

1) max f ∈ FB R std (f ): Although p = 0.5, due to the symmetry of the shape of X , max f ∈ FB R std (f ) is the area on X \ X , the area outside the small nine squares.

2)

max f ∈ F s rob R std (f ): For the same reason above, max f ∈ F s rob R std (f ) is the area on X \ B( X ,

), the area outside the small nine squares extended by .

3) max f ∈ Frob R std (f ): When < α/2 or ≥ 3α/4, we consider the deviated line on B( X , ), and regard the model outside B( X , ) as incorrect. The risk is calculated easily by using the fact that the risk of any f ∈ Frob on is B( X , ) is 3 × min(a + 2 , 2a) × (1 -p) × min(2 , 0.5)/0.5. When α/2 ≤ < 3α/4, since B( X , ) covers X , the worst case functions are in a form of f (x) = sign(x1 -0.5 + c ) for some c on the entire X . For each s.t. 3α/2 + < x1 < 0.5 -, it is easy to find the corresponding c .

Last, for the worst case adversarial robust risk w.r.t. PX,Y , we can calculate the risks in a similar way to above.

H ADDITIONAL INFORMATION ABOUT EXPERIMENTS H.1 EXPERIMENT 1:

We consider the MNIST and CIFAR-10 dataset (LeCun et al., 2010; Krizhevsky & Hinton, 2009) .

Training For each dataset, we initialize our model by a naturally trained model. Then, we train the initialized model with sensible adversarial examples with the specifications in Table 8 . (Dong et al., 2018) . In Experiment 1, we consider adversarial perturbations with ∞-norm less than , where = 0.3 for the MNIST dataset and = 8/255 for the CIFAR10 dataset.

We attack our models with the white-box attacks. We use the attacks implemented in Foolbox (Rauber et al., 2017 ), Advertorch (Ding et al., 2019b) , and Adversarial Robustness 360 Toolbox (ART) (Nicolae et al., 2018) . The attack specifications are in Table 9 . The options that are not listed in the table are kept as default of the attack generating functions. We set the step size for MIFGSM as slightly smaller than 8 255 × 1 40 in order to keep the adversarial example from having the perturbation norm greater than = 8/255. For the performance of TRADE on C&W40, we apply the C&W40 attack with the same specifications in Table 9 . For the other results of TRADE in Table 1 , we refer to Zhang et al. (2019) . For PGD attacks, we draw accuracy and loss plots for increasing step numbers in Figure 10 and Figure 11 . They show that for the chosen step sizes, the chosen step numbers are enough to generate proper local maximizer of the PGD objective function. Particularly, for CIFAR, we use the step size 2/255 rather than × 2/255 as it is more efficient.

PGD attack serenity check:

We test with 100 random restarts, and for the step size 0.01, the step number 500 seem enough by Figure 10 . Testing with black-box attacks We attack our models with PGD40 and MIFGSM swith the specifications in Table 9 . As the Foolbox implementation for MIGSM only returns the successful attacks on the generating model, we only apply these attacks on the defense model. We note that the argument return early of MIFGSM is set to False as in Table 9 . For TRADE, we use the models by Zhang et al. (2019) for both MNIST and CIFAR10.

Figure 11: The convergence check for the PGD attacks on the CIFAR model. We used a step size 0.01 and for CIFAR, 2/255. We can see that K = 100 is enough to achieve the lowest point by counting the worst case of random restarts.

H.2 EXPERIMENT 2:

Model Architecture We conduct Experiment 2 on the MNIST dataset (LeCun et al., 2010) . We consider a sequence of CNNs with the increasing number of kernels. A network of capacity q has two convolutional layers with 2 (d-1) and 2 d filters respectively, followed by a fully connected linear layer of 2 (d+4) units. Each layer is activated by ReLU. Each convolutional layer is followed by 2 × 2 a max-pooling layer. The size of all convolutional filters is 5 × 5.

With a similar sequence of CNNs, Madry et al. (2017) investigate the model behavior when the capacity increases. They have capacity scale 1,2,4,8 and 16. In their experiment, capacity scale 1 and 2 collapse. our capacities 2 and 3 are comparable to the capacity scale 1 and 2 by Madry et al. (2017) . Likewise, our capacities 4 and 5 are comparable to their capacity scale 4 and 8 1 . Therefore, our result, which shows the PGD models of capacity 1,2 and 3 collapse, is consistent to the result by Madry et al. (2017) .

Training We train the sequence of MNIST models with sensibly adversarial example with = 0.3, η1 = 0.05 and K = 10 for varying c ∈ {0.0, 0.1, • • • , 0.9}. The initial learning rate η2 is 0.01, and we train for 500 epochs. When training the PGD models, we use the same hyperparameters except c.

Testing The MNIST models are tested with ∞ PGD attacks of = 0.3 with the step number K = 40 and step size η1 = 0.01. We generate the attacks by using a Python package Advertorch by Ding et al. (2019b) .

H.3 ADDITIONAL EXPERIMENT FOR TABLE 6 AND TABLE 7:

We conduct our additional experiment on the MNIST and CIFAR-10 dataset (LeCun et al., 2010; Krizhevsky & Hinton, 2009) . For each dataset, we consider the SENSE model trained in Experiment 1 and the TRADE model by Zhang et al. (2019) . We note that for each dataset, the TRADE and SENSE model share the same architecture. On theses models, we apply the PGD attacks with perturbations larger than the training . We generate the PGD attacks, using a Python package Advertorch by Ding et al. (2019b) with the following attack specifications.

MNIST Let 0 = 0.3, which is the training for each model. For δ ∈ {1.1, 1.2, 1.3}, we generate the ∞ PGD attacks of = δ 0 with the step number K = 40 × δ and step size η1 = 0.01.

:

Our models could be slightly smaller than the counterparts byMadry et al. (2017) because our max-pooling layers do not apply any padding.

Figures and Tables:

Figure 1: Cheese holes distribution. (a) The outer square is the support of the underlying true distribution P X,Y , but the sampling is restricted on the small squares X × Y with distribution PX,Y . (b) The worst case naturally optimal model w.r.t. PX,Y . (c) The worst case robustly optimal model w.r.t. PX,Y . (d) The worst case sensibly robust model w.r.t. PX,Y .

Therefore, our algorithms generate a sensible adversarial example xs | fr=f in three different ways: (1) xs = x for x ∈ A f , (2) xs = xp for x ∈ B f , and (3) xs = xp for x ∈ C f , where xp is a full K-step PGD attack. Therefore, sensible adversarial training inherently involves a set selection mechanism. The sensible adversarial loss s (f, x, y) can be written as

Figure 2: The natural and robust accuracies of our models for the varying parameter c.

Figure 3: The prediction margins at convergence of capacity 5 on the test set. The natural margin of a model f at (x, y) is log pf,y (x) -max y =y log pf,y (x). The adversarial margin is calculated by log pf,y (x p ) -max y =y log pf,y (x p ), where xp is a full PGD attack.

and take a reference function fr ∈ A. Consider extended sensible adversarial examples, with c < 1 and as the 0-1 loss. Then, for any > 0, Rs rob (f |fr) = EP X,Y [ (f ( X| fr , Y )] is uniquely minimized by fr.

K) i,adv , y i )/m until training converged D THE LANDSCAPE FOR SENSIBLE ROBUST OPTIMIZATION Note that when the original examples are incorrectly classified, full PGD examples can be very influential in regular adversarial training. If we do not add any perturbations on these potentially influential examples and add full PGD perturbations on the other examples, the resultant examples are equivalent to the sensible adversarial examples when c = 0. Experiment 1 shows this small change keeps the models from collapsing, demonstrating how influential the PGD perturbations on the incorrectly classified natural examples. However, an adversarial training with the sensible adversarial examples with c = 0, the empirical loss can largely fluctuate during the training;

Figure 4: An abstraction of the sensible adversarial loss when c ≥ 0.5. When x is in the sensibly reversed stage for a current model f , the loss of xs is approximately log 1 c . Although the loss is approximately the same while x stays in this stage, the model updates in a way to pushes the sensible adversarial example to become a full PGD attack as the arrow.

Figure 5: The prediction margins at convergence of the models in Experiment 1 on the test set.

Figure 6: The prediction margins at convergence of the models in Experiment 1 on the test set.

Zhang et al. (2019) investigate the Bayes decision boundary to resolve the trade-off problem. They propose TRADE, of which the objective function is for β > 0,

Figure 7: Another visualization of Figure 2. The name of each panel denotes the hyperparameter c. -Inf denotes the model trained with the PGD method.

Figure 8: Adversarial and natural prediction margins on CIFAR10 of the SENSE and TRADE model. The margins are calculated by M (f, x, y) = log py (x) -max y =y log py (x) = s y (x)max y =y s y (x), where s denote a score function of f , i.e., the output of the neural network.

Figure 9 (e) compares the worst-case standard risk R std ( f * rob ) and R std ( f B * ). Although R std ( f * rob ) does not consistently decrease as increases, it is always much smaller than R std ( f B * ). Therefore, pursuit of robustness leads to a more naturally accurate classifier. Figure 9 (e) shows that, as increases from = α/2 to 1, the standard risk of the robustly optimal model gradually increases whereas the sensibly robust model keeps zero risk. Figure 9 (f) demonstrates the robustness against -ball attacks of f B * nat (black), f * rob (blue), and f s * rob (red). Although sensibly robust models have large adversarial robustness increasing to 1 as increases, this is because more and more adversarial examples can cross the border, while the model keeps its decision boundary consistent to the class border line. On the other hand, adversarially robust functions have constant robust risk for > 1/4. This is because the robust functions predict as y = 1 for every x ∈ X .

Figure 9: Cheese holes distribution. (e) and (f) The natural and robust risk when p = 0.55. The black, blue, and red colors are the worst cases of naturally, adversarially, and sensibly robust functions. (g) The robustly optimal model when > 0.25. (h) The sensibly robust model when > 1/12.

r.t PX,Y . Then for each class, we consider the worst case function from each class, in that the function maximizes the standard risk w.r.t PX,Y . The corresponding standard risks are in Figure 9 (e).Likewise, for each class, we consider the worst case function from each class, in that the function maximizes the adversarial robust risk w.r.t PX,Y . The corresponding adversarial robust risks are in Figure1(f).

Figure10: The convergence check for the PGD attacks on the MNIST model. We used a step size 0.01. We can see that K = 500 is enough to achieve the lowest point by counting the worst case of random restarts.

The test result on natural examples and ∞ -attacks for CIFAR10 ( = 8/255). The PGD attacks are generated with 20 random restarts and counted the worst case only.

Table content: <table><tr><td>NAT</td><td colspan="6">CW40 DeepFool FGSM LBFGS MIFGSM PGD100</td></tr><tr><td colspan="2">SENSE 91.51 67.01</td><td>78.89</td><td>72.72</td><td>85.94</td><td>68.87</td><td>57.23</td></tr><tr><td colspan="2">TRADE 84.92 62.19</td><td>61.38</td><td>61.06</td><td>81.58</td><td>57.95</td><td>54.72</td></tr></table>

The transfer attack results between the TRADE and SENSE model on CIFAR10. The ∞ PGD40 and MIFGSM attacks are generated. The subscripts of the column names denote the generating model. The denominator and numerator in each cell are the number of adversarial attacks and correct predictions respectively ( = 8/255).Defence model PGD SEN SE PGD T RADE MIFGSM SEN SE MIFGSM T RADE

Table content: <table><tr><td>TRADE</td><td>7831/10000 5655/10000</td><td>1584/3113</td><td>5888/10000</td></tr><tr><td>SENSE</td><td>6499/10000 6961/10000</td><td>6887/10000</td><td>2916/4112</td></tr></table>

MNIST: test results of our models on natural examples and ∞ based attacks ( = 0.3).

Table content: <table><tr><td colspan="4">Defence model Natural PGD500 C&amp;W40</td></tr><tr><td>SENSE</td><td>99.51</td><td>91.74</td><td>96.02</td></tr><tr><td>TRADE</td><td>99.48</td><td>93.30</td><td>96.90</td></tr></table>

The sensitivity of c on CIFAR models of WideResNet CIFAR WideResNet c=0.0 c=0.3 c=0.5 c=0.6 c=0.7 c=0.8 We train train WideResNet-34-10 with several different c values. Except that for c = 0.8 we stopped learning at 120 epoch, other models are trained for 300 epochs. We report the adversarial accuracy against and PGD100 attacks with a step size η 1 = 2/255 for the first 2000 test examples, with random 20 restarts. The results are in Table4.

Table content: <table><tr><td>Natural data</td><td colspan="2">82.88 86.76 90.42 90.87 91.51 92.35</td></tr><tr><td>PGD100</td><td colspan="2">43.70 46.90 50.95 55.90 57.80 55.60</td></tr><tr><td colspan="3">Table 5: The sensitivity of c on CIFAR models with CNNs</td></tr><tr><td colspan="3">CIFAR CNNs c=0.1 c=0.5 c=0.9 natural training</td></tr><tr><td>NAT</td><td colspan="2">66.26 75.70 82.02 80.85</td></tr><tr><td>PGD40</td><td>26.67 20.26 3.95</td><td>0.00</td></tr><tr><td>CIFAR:</td><td/><td/></tr></table>

MNIST: test results of the our models on natural examples and ∞ based attacks.

Table content: <table><tr><td>Defence model</td><td>= 0.3</td><td>= 0.33</td><td>= 0.36</td><td>= 0.39</td></tr><tr><td>SENSE</td><td>96.46</td><td>92.89</td><td>83.15</td><td>63.51</td></tr><tr><td>TRADE</td><td>96.72</td><td>90.56</td><td>46.94</td><td>11.66</td></tr></table>

CIFAR: test results of the our models on natural examples and ∞ based attacks.

Table content: <table><tr><td>Defence model</td><td>= 10/255</td><td>= 12/255</td><td>= 14/255</td><td>= 16/255</td></tr><tr><td>SENSE</td><td>62.63</td><td>60.39</td><td>58.05</td><td>55.09</td></tr><tr><td>TRADE</td><td>47.61</td><td>39.05</td><td>31.57</td><td>25.15</td></tr></table>

The learning specifications for the SENSE models in experiment 1

Table content: <table><tr><td>Dataset</td><td/><td>η 1</td><td>K c</td><td colspan="2">Initial η 2 Epoch</td></tr><tr><td>MNIST</td><td colspan="2">0.3 0.05</td><td colspan="2">10 0.5 0.01</td><td>500</td></tr><tr><td>CIfAR10</td><td>8 255</td><td>8 255 × 2 10</td><td colspan="2">10 0.7 0.1</td><td>300</td></tr><tr><td colspan="6">Testing with white-box attacks For white-box attacks, we consider PGD (Madry et al., 2017), C&amp;W</td></tr><tr><td colspan="6">(Ding et al., 2019a), DeepFool (Moosavi-Dezfooli et al., 2016), FGSM (Kurakin et al., 2016a), LBFGS (Taba-</td></tr><tr><td>cof &amp; Valle, 2016), and MIFGSM</td><td/><td/><td/><td/></tr></table>

The white-box attack specifications. We denote the step size and step number by η 1 and K.

Table content: <table><tr><td>Dataset</td><td>Attack</td><td>η 1</td><td>K</td><td colspan="2">Python package Function</td></tr><tr><td>MNIST</td><td>PGD500</td><td>0.01</td><td>500</td><td>Advertorch</td><td>LinfPGDAttack</td></tr><tr><td>MNIST</td><td>C&amp;W40</td><td>0.01</td><td>40</td><td>ART</td><td>CarliniLInfMethod</td></tr><tr><td colspan="2">CIfAR10 PGD100 CIfAR10 C&amp;W40 CIfAR10 DeepFool</td><td>2 255 255 × 1 8 20 default</td><td>100 40 default</td><td>Advertorch ART Foolbox</td><td>LinfPGDAttack CarliniLInfMethod DeepFoolLinfinityAttack</td></tr><tr><td>CIfAR10 CIfAR10</td><td>FGSM LBFGS</td><td>8 255 default</td><td>1 default</td><td>Advertorch Foolbox</td><td>GradientSignAttack LBFGSAttack</td></tr><tr><td colspan="2">CIfAR10 MIFGSM</td><td>8 255 × 1 40.001</td><td>40</td><td>Foolbox</td><td>MomentumIterativeAttack (distance=Linfinity,</td></tr><tr><td/><td/><td/><td/><td/><td>return early=False)</td></tr></table>

Peer review (iclr2020):
The paper proposes the notion of a *sensible* adversary that does not perturb data points on which the Bayes-optimal classifier is incorrect. The authors then provide theory showing that minimizing robust risk against such a sensible adversary yields the Bayes-optimal classifier, which addresses the question about standard vs. robust risk posed in prior work. On the experimental side, the authors then introduce a simple yet effective variation of adversarial training / robust optimization. Instead of maximizing the loss over the perturbation set, the proposed variant stops as soon as the loss exceeds a certain threshold. This can be seen as a variant of gradient clipping that reduces the influence of examples with a very high loss. The authors show that their modification yields an 8 - 9% improvement in robust accuracy on CIFAR-10, which gives state-of-the-art performance. I find the empirical improvements achieved with their modification of PGD-style adversarial training very interesting and recommend accepting the paper. However, it is not clear to me how well the theory is connected to the empirical findings. Moreover, there are additional experiments the authors can conduct to investigate the performance of their method more thoroughly. Concretely: - The theory relies on the *reference model* f_r being the Bayes-optimal classifier, while the experiments use the current model as reference model. Especially early in training, the current model performs significantly worse than a Bayes-optimal classifier. Moreover, it is unclear if the proposed training modification is effective if the reference model f_r is a separate classifier. It would be interesting to use a separately trained CNN (standard training without any robustness interventions) as a reference model to see if the training modification still yields improvements. - If the improvements of the proposed method come from the loss of a few adversarial examples dominating the overall loss in a batch, it would be interesting to measure and plot the loss distribution over examples in a batch with experiments. - As a baseline, comparing the proposed approach to clipping the loss or gradients of each example would be interesting. - In the robustness evaluation, have you experimented with randomly-restarted PGD? - To ensure that PGD works as intended, it would be helpful to see a plot of PGD iteration vs. adversarial accuracy. - It would be interesting to see accuracy numbers (standard and robust) for models trained with different values for the parameter c. This would also provide information about the sensitivity of this hyperparameter. Further comments: - I encourage the authors to release their code and pre-trained models in a format that is easy for other researchers to build on (e.g., PyTorch model checkpoints). - Page 3, *Note that R_rob(f^B) = P(X_1 [...]* - should this be X instead of X_1? - Line 9 of Algorithm 1: should the sum go from 1 to m? - Equation 5: is *x <= log 1/c* in the subscript a typo? - Page 8: *Our model achieves 91.51natural accuracy.*: percent symbol and space missing