Improving Adversarial Robustness Requires Revisiting Misclassified Examples
Authors: Yisen Wang
Difan Zou
Jinfeng Yi
James Bailey
Xingjun Ma
Quanquan Gu
Venue:  (2020)
Keywords: Robustness
Adversarial Defense
Adversarial Training

Abstract: Deep neural networks (DNNs) are vulnerable to adversarial examples crafted by imperceptible perturbations. A range of defense techniques have been proposed to improve DNN robustness to adversarial examples, among which adversarial training has been demonstrated to be the most effective. Adversarial training is often formulated as a min-max optimization problem, with the inner maximization for generating adversarial examples. However, there exists a simple, yet easily overlooked fact that adversarial examples are only defined on correctly classified (natural) examples, but inevitably, some (natural) examples will be misclassified during training. In this paper, we investigate the distinctive influence of misclassified and correctly classified examples on the final robustness of adversarial training. Specifically, we find that misclassified examples indeed have a significant impact on the final robustness. More surprisingly, we find that different maximization techniques on misclassified examples may have a negligible influence on the final robustness, while different minimization techniques are crucial. Motivated by the above discovery, we propose a new defense algorithm called {\em Misclassification Aware adveRsarial Training} (MART), which explicitly differentiates the misclassified and correctly classified examples during the training. We also propose a semi-supervised extension of MART, which can leverage the unlabeled data to further improve the robustness. Experimental results show that MART and its variant could significantly improve the state-of-the-art adversarial robustness.

Published as a conference paper at ICLR 2020
IMPROVING ADVERSARIAL ROBUSTNESS REQUIRES REVISITING MISCLASSIFIED EXAMPLES
Jinfeng YF, James Bailey', Xingjun Ma"! Quanquan Gu²t SSannghai Jiao Tong University University of California, Los Angles JJD.com "The University of Melbourne
Yisen Wang'; Difan Zou?;
sexangyisen&gmail. com, (knowzou, qgu) 8cs. uula edu, yijinfenge jd. com, (baileyj, xingjun, ma ) 8unimelb. edu.au
ABSTRACT
Deep neural networks (DNNs) are vulnerable adversarial examples crafted by imperceptible perturbations. A range of defense techniques have been proposed to improve DNN robustness to adversarial examples, among which adversarial training has been demonstrated to be the most effective. Adversarial training is often formulated as a min-max optimization problem, with the inner maximization for generating adversarial examples. However, there exists a simple, yet easily overlooked fact that adversarial examples are only defined on correctly classified (natural) examples, but inevitably, some (natural) examples will be misclassified during training. In this paper. we investigate the distinctive influence of misclassi- fied and correctly classified examples on the final robustness of adversarial training. Specifically, we find that misclassified examples indeed have significant impact on the final robustness. More surprisingly, we find that different maximization techniques on misclassified examples may have negligible influence on the final robustness, while different minimization techniques are crucial. Motivated by the above discovery, propose new defense algorithm called Misclassification Aware adveRsarial Training (MART), which explicitly differentiates the misclas- sified and correctly classified examples during the training. We also propose a semi-supervised extension of MART, which can leverage the unlabeled data to further improve the robustness. Experimental results show that MART and its variant could significantly improve the state-of-the-ar adversarial robustness.
INTRODUCTION
Despite their great success in applications such as computer vision (He ct al.
2016). speech recogni-
tion (Wang al. 2017) and natural language processing (Devlin et al. 2018 Zeng et al. 2019), deep neural networks (DNNs) are extremely vulnerable to adversarial examples crafted by adding small adversarial perturbations
natural examples (Szegedy et al. 2013 Goodfellow al., 2015 Wu let al. 2020). Given a DNN classifier he with parameter 0 and correctly classified natural example with class label y (h) - y), an adversarial example x' can be generated by perturbing such that (x') # i.e., the natural example is correctly classified before perturbation but misclassified after perturbation. The perturbation required for misclassification is often small and bounded by an T-pnormm ||x' xI|p < which keeps x' within the e-ball centered at x, so that it is visually the "same" for human observers. This vulnerability of DNNs raises serious security concerns about their practicability in security critical applications (Chen et al. 2015 Kurakin et al., 2016 Jiang et al.)
2019 Finlayson et al. 2019 Ma et al. 2019).
Compared with pre/post-processiee denoising (Guo et al.) 2018; Liao et al. 2018; [Samangouei et al. 2018 Bai et al. 2019) and adversarial detection (Feinman et al. 2017 Ma et al. 2018 Lee et al. 2018), several defense techniques have been proposed to train DNNs that are inherently robust to adversarial examples including defensive distillation (Papernot et al. 2016), gradient regularization (Gu & Rigazio 2014 Papernot et af. 2017 Ross & Doshi- Velez (2018] Tramer et al.] 2018), model compression (Das et al. 2018 Liu et al. 2018 and activation pruning (Dhillon al. 2018 Rakin al. 2018), among which
methods such as feature squeezing [uu et al. 2017). input
" "Equal contribution. 'Corresponding author.

---

Published as a conference paper at ICLR 2020
45.0
45.0
45.0
Robustness
42.5
Test Robustness
Test Robustha
40.0
40.0
40.0
37.5
37.5
37.5
Perturb (PGD 5 us
s5.0
PGD
35.0
CE US"
32.5
Not perturb
Not parturb
32.5
FGSM on s*
FGSM on s
32.5
on
on
30.0
60 Training Epoch
60
100
30.0
20
Training Epoch
Training Epoch
(a) Not perturb
(b) Inner maximization
(c) Outer minimization
Figure 1: The distinctive influence of misclassified examples (S versus correctly classified ones (S*) on the robustness of adversarial training. We test the white-box robustness of different strategies on either subset of examples: (a) using them directly for training ("not perturb")) (b) using weak attack (FGSM) in the inner maximization; and (c) using "regularized CE" loss in the outer minimization.
adversarial training has been demonstrated to be the most effective (Athalye et al. 2018). Adversarial training can be regarded as a data augmentation technique that trains DNNs on ad versariial examples, and can be viewed as solving the following min-max optimization problem (Madry et al., 2018):
min
max X Se
E(ha (x,)) U.),
(1)
where n is the number of training examples and E(-) is the classification loss, such as the commonly used cross-entropy (CE) loss. The inner maximization generates adversarial examples that can be used by the outer minimization to train robust DNNs. Recently, adversarial training with adversarial examples generated by Projected Gradient Descent (PGD) (Madry et al.] (018)) has been demonstrated to be the only method that can train moderately robust DNNs without being fully attacked (Athalye) let al. 2018). However, there is still a significant gap between adversarial robustness (test accuracy on adversarial examples) and natural accuracy (test accuracy on natural examples), even for simple image datasets like CIFAR-10 (Krizhevsky & Hinton 2009).
Compared with natural training (on natural examples), training adversarially robust DNNs is partic- ularly difficult (Madry et al., 2018). Nakkiran (2019) showed that a model requires more capacity to be robust (i.e. simple models can have high natural accuracy but are less likely to be robust). In
addition, the sample complexity of adversarial training can be significantly higher than that of natural training, that is, training robust DNNs tends to require more data either labeled (Schmidt et al. 2018) or unlabeled ones (Uesato et al., 2019: Carmon et al., | Najafi et al. 2019 /hai et al. 2019). Moreover, Tsipras et al. (2019): Zhang et al. (2019) demonstrated that adversarial robustness may be inherently at odds with natural accuracy. Parallel to these studies, in this paper, we provide some new insights on the adversarial examples used for adversarial training.
Recall that the formal definition of an adversarial example is conditioned on it being correctly classi- fied" (Carlii et al. 2019). From this perspective, adversarial examples generated from misclassified examples are "undefined". Most adversarial training variants neglect this distinction, where all training examples are treated equally in both the maximization and the minimization processes, regardless of whether not they are correctly classified. The only exception we are aware of is Ding ctt al. (2018), which proposes to use maximal margin optimization for correctly classified examples. Yet they did not pay sufficient attention misclassified examples. A deeper understanding about the influence of misclassified and correctly classified examples on the robustness is still missing in the literature. Therefore, we raise the following questions:
Are the adversarial examples generated from i) misclassified and ii) correctly classified examples, equally important for adversarial robustness? If not, how can one make better use of the difference to improve robustness?
In this paper, investigate this intriguing, yet thus far overlooked aspect of adversarial training, and find that misclassified and correctly classified examples exhibit a distinctive influence on the final robustness. To illustrate this phenomenon, we conduct a proof-of-concept experiment on CIFAR- 10 in a white-box setting with Lox maximum perturbation e = 8/255. We first train an 8-layer
In this paper, both "correctly classified" and "misclassified" refer predictions natural training examples.
2

---

Published as a conference paper at ICLR 2020
Convolutional Neural Network (CNN) using standard adversarial training with 10-step PGD (PGD!0) and step size e/4, then use this network (87% training accuracy) to select two subsets of natural training examples to investigate: 1) subset of misclassified examples S (13% of training data), and 2) a subset of correctly classified examples S+ (also 13% of training data, = -S- Using these two subsets, we explore different ways to re-train the same network, and evaluate its robustness against white-box PGD (step size e/ 10) attacks on the test dataset.
In Figure [(a). we find that misclassified examples have a significant impact on the final robustness. Compared with standard adversarial training (dashed blue line), the final robustness drops drastically, if examples in subset S are not perturbed (solid green line) during adversarial training (other examples are still perturbed by PGD 19). In contrast, the same operation on subset S only slightly affects the final robustness (solid orange line). Previous work has found that removing small proportion of training examples does not reduce the robustness (Ding et al. (2019), which seems to be true for correctly classified examples, but apparently not true for misclassified examples.
To further understand the distinctive influence of misclassified and correctly classified examples, we test different techniques on them within either the maximization or the minimization process of adversarial training. Firstly, we apply different maximization techniques while keeping the minimization loss CE unchanged. As shown in Figure [(b) the final robustness is barely affected when
weak attack (e.g., Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2015)) to perturb misclassified examples S (all other training examples are still perturbed by PGDD"). This suggests that different maximization techniques negligible influence on the final robustness, provided that the inner maximization problem is solved to
misclassified examples S may have
a moderate precision (Wang al. 2019). However, for subset S+, a weak attack for the maximization tends to degenerate the robustness. Secondly, we test different minimization techniques with the inner maximization still solved by PGD Interestingly, we find that different minimization techniques on misclassified examples make a significant difference to the final robustness. As shown in Figure [[c] compared with standard adversarial training (dashed blue line) with the CE loss, the final robustness is significantly improved when the outer minimization on misclassified examples is "regularized" (solid green line) by an additional term (a KL-divergence term that was used previously in Zheng et al. (2016); Zhang et al. (2019)). The same regularization applied to correctly classified examples also helps the final robustness (solid orange line), though not as significantly as for misclassified examples.
Motivated by the above observations, we reformulate the adversarial risk to incorporate an explicit differentiation of misclassified examples in form of regularization. We then propose defense algorithm to achieve this in a dynamic way during adversarial training. Our main contributions are:
We investigate the distinctive influence of misclassified and correctly classified examples on the final robustness of adversarial training. We find that the manipulation on misclassified examples has more impact on the final robustness, and the minimization techniques are more crucial than maximization ones under the min-max optimization framework.
We propose a regularized adversarial risk which incorporates an explicit differentiation of misclas- sified examples regularizer. Based on that, further propose a new defense algorithm, called Misclassification Aware adveRsarial Training (MART).
Experimentally, we show that adversarial robustness can be significantly improved over the state- of-the-art, by a specific focus on misclassified examples. It also helps improve recently proposed adversarial training with unlabeled data.
2
MISCLASSIFICCTTION AWARE ADVERSARIAL RISK
In this section, we propose regularized adversarial risk that incorporates an explicit differentiation of misclassified examples.
2.1
PRELIMINARIES
We first define some notations. We use lower case and lower case bold face to denote scalars and vectors, respectively. We use upper case calligraphic symbols to denote sets.
For a K-class (K 2 2) classification problem, given a dataset {xx, y.) y.).=-,..... with x; E Rd as natural example and yi E {I,..., K} as its associated label, DNN classifier he with model
3

---

Published as a conference paper at ICLR 2020
parameter 0 predicts the class of input example x;:
hx(x,) arg max P&(X,, 0), Pi (x,, 0) = (xp(zx(x; 0))/ k=1...,KK
exp(z (x;; 0)),
(2)
k=11
where Z (x,, 0) is the logits output of the network with respect to class k and pk (x,, 0) is the probability (softmax on logits) of x; belonging to class k.
The adversarial risk (Madry et al. 2018 dataset {(x,, (xx,)1)) v.) and classifier he can be defined with respect to the 0-1 (Zhang et al., 2019) as:
R(ha) 1 Σ max
( (x() # Ut),
(3)
(x)
where 1 (.) is the indicator function and B, = (x -x
x p S ) denotes the -nom ball centered at x; with radius 6. We will focus on the Tige- in this paper.
2.2 MISCLASSIFICATION AWARE REGULARIZATION
Note that the adversarial risk in 3 is defined on adversarial examples within the e-ball of all natural examples, regardless of whether they are correctly classified (ho(x.) \/s) misclassified (x,( * y:) by the current model hg. To differentiate, we reformulate the adversarial risk based on the prediction of the current network heg Specifically, natural training examples can be divided into two subsets with respect to he, with one subset of correctly classified examples (8%,)) and one subset of misclassified examples (Sh1g)
Stee = i [m], (x;) = 34 and 5%o = (f: ho),ho(x)) + vz)-
Then we are going to define adversarial risk separately for correctly classified and misclassified examples. As we observed in Figure [[c] regularization on misclassified examples can significantly improve robustness. Therefore, for misclassified examples, we formulate the adversarial risk as:
R (he. x,) == (ha (x() * ue) + (he (x,) o ho hx()),
(4)
where the adversarial example x is generated by solving
X arg max 1 (x) + v:).
(5)
(x.)
We remark that the first and second terms on the R.H.S. of 4 correspond to the standard adversarial risk and the regularization term respectively. Moreover, would like to clarify that the regularization term 1 (he (x,) 전 he (x()) aims to encourage the output of neural network to be stable against misclassified adversarial examples. For misclassified examples, direct minimization of the standard adversarial risk may be too hard, as themselves cannot be classified correctly, even without any perturbations. A similar idea has been used in stability training on all training examples (Zheng et al.) 2016 Kannan al. 2018 Zhang al. 2019).
Then we consider correctly classified examples. As can observed in Figure [[c] regularization on correctly classified examples cannot provide as significant improvement as achieved by that on misclassified ones. Moreover, in this case it can be found that 1 (x,) * (x)) = (<)((x) # y:) since have (x.) yı, which implies that the regularizer has exactly same form as the adversarial risk. Therefore, for correctly classified example, we simply use the standard adversarial risk, i.e.,
R (ho, xx) == xxEB,(x)
1 (ho (x() * ys) (ho())) (x)) # vi),
where the adversarial example X is defined in (5).
Finally, combining the proposed two adversarial risks for correctly classified examples and misclas- sified examples in an adversarial training framework, we can train network that minimizes the following risk:
ming Rmise (ho): 1 (EiEs, R R+oh x;) + Eiss,, R (hoh, xi)
- (1(he(x() # #v)) ((h((x,) * ho(x()) 1hh (x.) # w))
(7)
where x is defined in and the second equality follows from the definition of S, and Spp. The new risk defined above is a regularized adversarial risk with regularization term 1/n Σ 1 (x) + ho())) ( (x;: y:), which call the misclassification aware regularization.
4

---

Published as a conference paper at ICLR 2020
3 PROPOSED DEFENSE: MISCLASSIFICATION AWARE ADVERSARIAL TRAINING (MART)
In the previous section, we derived the misclassification aware adversarial risk based on 0-1 loss. However, optimization over (-1 loss is intractable in practice. We next propose a Misclassification Aware adveRsarial Training (MART) algorithm, by replacing the 0-1 losses with proper surrogate loss functions which are both physical meaningful and computationally tractable. Following that, we further analyze the difference of MART to existing work, and propose a remi-supervised extension.
3.1 THE PROPOSED DEFENSE ALGORITIM
Surrogate Loss for Outer Minimization. As presented in 7 the minimization consists of three indicator functions: (1) Il (ho (h)))() # yr): (2) I (ho (x;) # ho x))): and (3) (ho(x:) # yz)-
For the first indicator function 1. (ha (x/) * yi). we propose to use a boosted cross entropy (BCE) loss the surrogate loss, instead of the commonly used CE loss in (Madry al. 2018 Wang et al.) 2019). This is largely because classifying adversarial examples requires a stronger classifier than natural examples, as the presence of adversarial examples makes the classification decision boundary become more complicated. This is pointed out by (Madry et al. (0118), where they increase the model capacity for a stronger classifier. The benefit of using BCE compared to CE will shortly be presented in the experiment section. The proposed BCE loss is defined as:
BCE BCE(p(X), U)) = log (Py, (X,, 0)) llog (1 ma- RA
max Pie (x 0)),
(8)
where pa (x/, O) is the probability output defined in (2) the first term log (P (X 0)) is the com- monly used CE loss, denoted CE(p(X', f), ya), and the second term log (1 max P& (K{ 0)) is margin term used to improve the decision margin of the classifier. A similar idea has been used for improving adversarial strength by Carlini & Wagner (017). Note that BCE is just a simple boost that works well in experiments and other boosted losses could also work here.
For the second indicator function 1 (he (x,) * he (x))). we can use KL divergence as the surrogate loss function (Zhang al. 2019 Zheng et al. (0166), since he (x.) * he (x() implies that adversarial examples have different output distributions to that of natural examples. Thus, we have
K
Pk (x 0) P& (x
(L(p(x, 0)) = Σ (x 0) log
The third indicator function (x:) + ya) is condition that emphasizes learning on misclassified examples. However, the condition cannot be directly optimized if we conduct a hard decision during the training process (Ding al. (2019) uses hard decision and does not optimize the condition). Instead, we propose to use a soft decision scheme by replacing 1 (he (x,) * y1 with the output probability Pv. (x,, 0). This will be large for misclassified examples and small for correctly classified examples.
Surrogate Loss for Inner Maximization. The goal of inner maximization is to generate adversarial example X for natural example x; by solving (5). Therefore, we aim to find surrogate loss function for the indicator function (xa(x() * y:). Here, we leverage the commonly used CE loss as the surrogate loss and find the adversarial example x as follows:
arg max CE(p(x,,0) 0), x (x.)
(10)
Following our findings in Figure [(b) that a strong attack can help robustness (though negligible
for misclassified examples), we propose to use the (strong) PGD attack to maximize the CE loss for
both the correctly classified and misclassified examples, the same as standard adversarial training.
Note that other surrogate loss functions, such those exploited in (Athalye et al. 2018 Carlini & Wagner 2017) for adversarial attack, could also be used here.
We slightly abuse the notation of X as the maximizer of the surrogate loss rather than the 0-1 loss in
5

---

Published as a conference paper at ICLR 2020
The Overall Objective. Based on the surrogate loss functions, state the final objective function for our proposed Misclassffication Aware adveRsarial Training (MART) defense:
2 [(x,, Ms. 0),
MART (0)
(11)
7
where E(x,, Mr: 0) is defined as
[(x,, Vi, 0) = (C u)) + X (L,(, ()),p(X(, 0)) Py, (x,, 0))
Here the adversarial example x is generated by (10), and À is tunable scaling parameter that balances the two parts of the final loss, and fixed for all training examples. The complete training procedure of MART is described in Appendix A
3.2 RELATION TO EXISTING WORK
In this section, we briefly discuss the difference between our MART and existing defense methods including standard adversarial training (Standard) (Madry et al.] (2018), logit pairing methods (Kannan al. 2018), max- margin adversarial training (MMA) (Ding et al. 2018 and TRADES (Zhang et al. 2019 as presented in Table
Table 1: Loss function comparison with existing work. The adversarial example x' is generated by (10p for all defense methods except TRADES and MMA. The adversarial example TRADES is generated by maximizing its regularization term (KL-divergence), and the adversarial example in MMA is generated by solving (10) with different perturbation limit (i.e., c).
Defense Method
Loss Function
Standard
CE(p(X', 0).
ALP
(E(p(X(,0).)) A- |p(x 10 0) - p(x, 0)
CLP
CE(p(x,0),y) + +X-1I=(X(000)
p(x,0)
TRADES
MMA
CE(p(X', 0). y) ((ha(x))= = + ().p(x,0).v)
(he + y)
MART
BCE(p 0). - - p ()) P (x 0))
Specifically, the Standard algorithm was designed to minimize the standard adversarial loss, i.e., cross-entropy loss on adversarial examples. Logit pairing methods, consisting of adversarial logit pairing (ALP) and clean logit pairing (CLP), introduce regularization term enclosing both natural examples and their adversarial counterparts. The objective function of TRADES is also a linear combination of natural loss and regularization terms on the output probabilities corresponding to natural examples and their ad advesaaiial counterparts using KL divergence. However, none of these algorithms differentiates the misclassified examples and correctly classified examples.
The most relevant work is MMA, which proposes to use maximal margin optimization for correctly classified examples while keeping the optimization on misclassified examples unchanged. Specifically, for correctly classified examples, MMA adopts cross-entropy loss adversarial examples, which are generated by solving (TO) with example-dependent perturbation limit. For misclassified examples, MMA directly applies cross-entropy loss natural examples. We emphasize that our MART is different from MMA in the following aspects: (I) MMA performs hard decision to identify misclassified examples from training data, while MART uses soft decision scheme training data based on the corresponding output probabilities (p(x', ()), which can be jointly learned during the training process; (2) for correctly classified examples, MMA adopts cross-entropy loss adversarial examples with different perturbation limits, while MART utilizes the proposed BCE loss on the adverarial examples with the same perturbation limit; (3) for misclassified examples, MMA adopts cross-entroes cross-entropy loss on natural examples, while MART adopts regularized adversarial loss involving both adversarial and natural examples. Because of these differences, we later will show that MART outperforms MMA in the experiments.
3.3 SEMI-SUPERVISED EXTENSION WITH UNLABELED DATA
Recent work has shown that semi- supervised learning with additional unlabeled data improve the adversarial robustness (Uesato et al. 2019 Carmon et al.] 2019, Najafi et al.) 2019 Zhai et al.) 2019). Specifically, the training loss function applied in these semi-supervised learning methods is typically

---

Published as a conference paper at ICLR 2020
defined as weighted sum of the supervised loss (loss the labeled data) and the unsupervised one (loss on the unlabeled data), i.e.,
( Sup (0) + y Lumsup (
where 7 is the weight of unsupervised loss. As pointed out in Uesato et al. (019)) there are multiple choices of the unsupervised loss function Lunsup (0), leading to different defense methods, among which the most effective defense method is UAT++ In particular, UAT++ first trains natural model on labeled data, and then use this model to generate pseudo labels for unlabeled data. Moreover, given a training data (x, (which can either labeled or unlabeled data), the supervised and unsupervised loss functions adopted in UAT++ are defined asF
pUAT+ (x, y: 0) = uunsup (x, y; 0) max CE(p(x',0) + + max (L(p(x, 0)ll(xx,), 0)), X'EB.
AIAT++
X'EB.
where À is a tunable hyperparameter. A similar idea was also proposed in a concurrent work (Carmon al. 2019), leading to another semi-supervised defense method called RST. The first stage of RST is also to generate pseudo labels for unlabeled data by training natural model on the labeled data. Then in the second stage, RST applies TRADES loss to train the robust model based on both labeled and unlabeled data, i.e., given training data (x, y), the supervised and unsupervised loss functions adopted in RST defined as
'sup (x, y; 0)
ERST (x, y; 0) CE(p(x, 0), X max (L(p(x, 0) x'EB.
As pointed out in Figure [(b) and the following experiment section, the maximization technique has a neglectable influence on the robustness. Therefore, the major difference between UAT++ and RST is the objective function for minimization. Considering MART is also an objective function, thus could be easily combined with semi-superrvsse learning with unlabeled data. Following RST. we propose the following cuni-super version of MART:
MAKT
pMAKT
(x,, ys: 0) +
MART "usuup
(x4, yoi 0)
seme
sup
SESup
iE
with supervised and unsupervised loss function defined as follows,
(MARTT (x, y/; 0) = MMRTT (x, y; 0) BCE(p( 0), + + KL(p(x, ()) (1 (x 0)), sup
where the adversarial example x' is generated by solving (TO). and Suup and Sunsup denote the set of labeled data and unlabeled data respectively.
EXPERIMENTS
In this section, we first conduct set of experiments to provide a comprehensive understanding of our proposed defense MART, and then evaluate its robustness on benchmark datasets in both white-box and black-box settings. Finally, we benchmark the state-of-the-art robustness and explore using unlabeled data for further improvement.
4.1 UNDERSTANDING THE PROPOSED MART
Here, we investigate MART from 4 different perspectives: (1) removing components of the MART loss function, (2) replacing components the MART loss function, (3) misclassification aware loss on certain proportions of training data, and (4) sensitivity to regularization parameter A.
Experimental Setup. We train ResNet- 18 (He et al., 2016) with different variants of MART on CIFAR-10 Krizhevsky & Hinton| 2009). All the models are trained using SGD with momentum 0.9, weight decay x 10-1 and an initial learning rate of 0. which is divided by 10 at the 75-th and 90-th epoch. All natural images are normalized into [O, 1]. and simple data augmentations including pixel padding with 32 x 32 random crop and random horizontal flip. The maximum perturbation = //255 and parameter X 6. The training attack is PGD 10 with random start and step size e/4, while the test attack is PGD20 with random start and step size e/10.
Removing Components of MART. Recalling the objective function of MART in 11) it has three terms in the loss function: BCE, KL and 1 - p As illustrated in Figure ((a) removing -p or KL.
The adversarial example x' in KL term is reused the one in CE term for efficiency in Uesato ct al. 2019 $ For simplicity, we use these abbreviations in Section 4.1

---

Published as a conference paper at ICLR 2020
Test Rabusness
Rict-oiee 160%
DCRrLaRd
RL-l-siie 75%
0000 B00t
Richision 50% das
Mox
Richotiee 0%
Treeing Epach
Touiing Eprach
Trairing Epreds
(a) Removing
(b) Replacing
(c) Training data (%)
(d) Regularization
Figure 2:
The comprehensive ablation experiments of MART. In each plot, the dashed blue line represents the original MART method.
or both all leads to a significant robustness degradation. In particular, we found that the soft decision term 1 p has constant robustness improvement throughout the training process, while the KL term can help mitigate overfitting at a later stage of training (after 80 epochs). When the two terms are combined together, they boost the final robustness considerably without causing overfitting.
Replacing Components of MART. As we show in Figure 2(b) when the BCE component is either replaced by CE term or redefined on natural examples (Xnat), the final robustness decreases by a substantial amount. It suggests that learning with CE instead of our proposed BCE suffers from insufficient learning with lower robustness throughout the entire training process. On the other hand, learning with BCE on natural examples exhibits severe overfitting at the later stage (solid green line). We did not observe any benefit when replacing CE by KL. the inner maximization of adversarial min-max framework (solid red line), an observation that is consistent with Figure ((b)
Ablation on Training Data. Here, we show the contribution of our proposed misclassification aware regularization (e.g., the KL . (1
p) term in (TT) to the final robustness with respect to the training data. Specifically, we gradually increase the proportion of training examples that are trained using the proposed misclassification aware regularization term, and display the corresponding robustness in Figure 2(c) The training examples using the proposed regularization are randomly selected, and the BCE term is still defined all training (adversarial) examples. As can be observed, the robustness can be improved steadily when the proposed regularization is applied on more data. This verifies the benefit of the differentiation of correctly classified and misclassified examples.
Sensitivity to Regularization Parameter X. We further investigate the parameter À in MART objective function defined in (TT) which controls the strength of the regularization. We also test the regularization parameter À of TRADES (please refer to Table [].. We present the results in Figure [ for different À E [1/2, 50). By explicitly differentiating the misclassified and correctly classified examples, MART achieves good stability and robustness across different choices of X, which is also consistently better and more stable than that of TRADES.
4.2 ROBUSTNESS EVALUATION AND ANALYSIS
In this part, we evaluate the robustness of MART on both MNIST (LeCun et al. 1998) and CIFAR-10 (Krizhevsky & Hinton 2009) datasets against various white-box and black-box attacks.
Baselines (1) Standard (addry et al. 2018) (2) MMA (Ding et al. 2019); (3) Dynamic (Wang
ct 2019); and (4) TRADES (Zhang et al. 2019). We only compare with adversarial training variants, since they are the most effective defense to date (Athalye al. 2018).
Defense Settings. For MNIST, all defense models are built on 4-layer CNN and trained using SGD with momentum 0.9. The initial learning rate is 0.01 and divided by 10 at the 20-th and 40-th epoch. For CIFAR-10, we use SGD with momentum 0.9, weight decay 3.5 x 10-3 and an initial learning rate of 0.01, which is divided by 10 the 75-th and 90-th epoch. For the training attack, it is also the 1G with random start and step size e/4. The perturbation limit = 0.3 for MNIST, and e = 8/255 for CIFAR-10. For MART. we set configured as per their original papers: max margin is set to 0.45 (MNIST) or 12/255 (CIFAR-10) for MMA, maximum criterion value Cmax 0.5 for Dynamic and À = for TRADES.
5. Hyperparameters of the baselines are
$ For clarity, we set the ticks on x-axis in uniform interval for different Stondard, MMA and TRADES have been introduced in Section 3.2 Dynamic is the adversarial training with criterion that dynamically controls the convergence quality of the inner maximization.
8

---

Published as a conference paper at ICLR 2020
Table White-box robustness (accuracy (%) on white-box test attacks) on MNIST and CIFAR-10.
MNIST
CIFAR-I0
Defense
Natural
FGSM
PGD
CW 00
Natural
FGSM
PGD
CW 00
Standard
99.11
97.17
94.62
94.25
84.44
61.89
47.55
45.98
MMA
98.92
97.25
95.25
94.77
84.76
62.08
48.33
45.77
Dynamic
98.96
97.34
95.27
94.85
83.33
62.47
49.40
46.94
TRADES
99.25
96.67
94.58
94.03
82.90
62.82
50.25
48.29
MART
98.74
97.87
96.48
96.10
83.07
65.65
55.57
54.87
Table 3: Black-box robustness (accuracy (%) on black-box test attacks) on MNIST and CIFAR-10.
MNIST
CIFAR-I0
Defense
FGSM
PGD 10
PGD
CW
FGSM
PGD 10
PGD
CW oc
Standard
96. 12
95.73
95.47
96.34
79.98
80.27
80.01
80.85
MMA
96.11
95.94
95.81
96.87
80.28
80.52
80.48
81.32
Dynamic
97.60
96.25
95.82
97.03
81.37
81.71
81.38
82.05
TRADES
97.49
96.03
95.73
97.20
81.52
81.73
81.53
82.11
MART
97.77
96.96
96.97
98.36
82.75
82.93
82.70
82.95
White-box Robustness. We evaluate the robustness of all defense models against three types of attacks for both MNIST and CIFAR-10: FGSM, PGI)20 (20-step PGD with step size e/10), and CW oo (Ioz version of CW optimized by PGD). All attacks have full access model parameters and are constrained by the same perturbation limit e. The white-box robustness of all defense models reported in Table where "Natural" denotes the accuracy on natural test images. Our proposed defense MART achieves the best robustness against all three types of attacks on both MNIST and CIFAR-10. Compared with MNIST, the robustmess improvements of MART over other baselines more significant CIFAR-10. This is because adversarial training CIFAR-I0 10 is more challenging problem that may have more misclassified examples during training, and MART can better handle those misclassified examples due to its regularization term in (II.. Note that the robustness improvement MART is not caused by the so-called "obfuscated gradients" (Athalye et al. 2018). This can be verified by two phenomenons: (I) strong test attacks (e.g., PGD have higher success rates (lower accuracies) than weak test attacks (e.g., FGSM), and (2) white-box test attacks have higher success rates than back-box test attacks (comparing Table with Table 3). Besides, we conduct an additional check using a gradient- attack SPSA (Uesato et al. (018). SPSA attack does obtain lower accuracy than gradient-based attacks like PGD, which confirms that the robustness of MART trained models are not due to gradient masking.
Black-box Robustness. Black-box test attacks are crafted from the natural test images by attacking a surrogate model with an architecture that is either copy of the defense model (MNIST) or more complex ResNet-50 model (CIFAR-10), Both surrogate models are trained separately from the defense models on the original training sets. The attacking methods used here are: FGSM, PGD 10 2GD20, and CW c. The black-box box robustness of all defense models are reported Table 3 Again, the proposed defense MART achieves higher robustness than other baselines. Compared with the white-box results, defense methods achieve much better robustness against black-box attacks, close to the natural accuracy. This suggests that adversarial training is indeed very practical choice for defense scenarios where the target model can be kept secret from potential attackers. is also observed that robustness strong attacks like CW is higher than weak attacks like FGSM, which indicates that strong attacks have less transferability than weak attacks (Madry et al. 2018
4.3 BENCHMARKING THE STATE- -HEEE-OO---EAAAAEE ROBUSTNESS
In this part, conduct more experiments large-capacity network WideResNet (Zagoruyko & Komodakis 2016) to benchmark the state-of-the-art robustness, and also explore using unlabeled data for further robustness boost.
Performance on WideResNet. We employ WideResNNet-4-111 10 (depth 34 and width 10) to explore the full power of our proposed MART defense method, and also benchmark the oot-o---t-ht-rr--- robustness on CIFAR-1O. The robustness all defense models are tested against white-bon box FGSM, 2GD20 1GD100 and CW attacks, under the same settings Section 4.1 We report the robustness

---

Published as a conference paper at ICLR 2020
Table 4: White-box robustness (%) on CIFAR-IO using the WideResNet-33-10.
FGSM
PGD
PGD 100
CW cc
Defense
Natural
Best
Last
Best
Last
Best
Last
Best
Last
Standard
87.30
56.10
56.10
52.68
49.31
51.55
49.03
50.73
48.47
Dynamic
84.51
63.53
63.53
55.03
51.70
54.12
50.07
51.34
49.27
TRADES
84.22
64.70
64.70
56,40
53.16
55.68
51.27
51.98
51.12
MART
84.17
67.51
67.51
58.56
57.39
57.88
55.04
54.58
54.53
Table 5: White-box robustness (%) WideResNet with additional unlabeled data.
a) W4deResNet-34-88 with 100K unlabeled data b) WideResNet-28-100 10 with 500K unlabeled data
Defense
Natural
PGD
Defense
Natural
PGD
+
86.04
59.41
UAT+
86.21
62.76
RST
88.24
59.60
RST
89.70
63.10
MART
86.68
61.88
MART 86.30
65.04
of both the best and the last epoch models obtained during training in Table For each defense method against each attack, the "best" refers to the highest robustness that ever achieved at different checkpoints. Specifically, against FGSM attack, the best model was found at the last epoch (e.g., "best" is also "last") for all defense methods, while against PGD 20 PGD 100 and CW attacks, the best model was found at the epoch right after the first time learning rate decay (i.e., epoch 76). Our proposed MART outperforms all baseline methods in terms of the robustness of both the best and the last epoch models. Particularly under the most common comparison setting (against PGD attacks on CIFAR-10 MART improved ~ 8% over Standard, and ~ 4% even over TRADES for the last epoch model. A similar trend of improvement is also observed for the best epoch model results. Considering the worst case accuracies against all attacks, MART still gains ~ 6% and ~ 3.5% robustness improvement over Standard and TRADES respectively.
Boosting with Additional Unlabeled Data. Here, we evaluate the proposed semi-supervised version of MART and show that it can also benefit from additional unlabeled data and achieves better robustness. Following the exact settings in UAT++ (Uesato et al. 2019) and RST (Carmon et al. 2019), we compare the robustness of MART with them on WideResNet-344 and WideResNNt-22 10 against 2GD (FGSM2)) respectively (in the same setting reported in their paper). The dataset is CIFAR-10 with 100K and 500K unlabeled data extracted from the 80 Million Tiny Images dataset (Torralba et al. 2008). As confirmed in Table our proposed defense MART can also benefit from unlabeled data, and further improves the UAT+ and RST defenses. This again verifies the benefit of differentiating misclassified and correctly classified examples for improving robustness, and further demonstrates the superiority of our proposed method.
5 CONCLUSION AND FUTURE WORK
In this paper, we investigated the interesting observation that misclassified examples have a recog- nizable impact on the final robustness adversarial training, especially for the outer minimization process. Based on this observation, we designed misclassification aware adversarial risk, which is formulated as adding an misclassification aware regularization to the standard adversarial risk. Following the regularized adversarial risk, proposed new defense algorithm, called Misclassifi- cation Aware udveRsarial Training (MART), with appropriate surrogate loss functions. Experimental results demonstrated that MART can achieve significantly improved adversarial robustness with respect to the stateoof-the-at and can also achieves better robustness with additional unlabeled data.
In the future, we plan investigate the effect of differentiation of correctly classificd/miscaassieedd training examples in the recently proposed certificd/provable robustness framework (Cohen et al. 2019 Salman et al. 2019) and explore the potential improvements brought by the differentiation of training examples.
ACKNOWLEDGEMENT
We thank the anonymous reviewers and area chair for their helpful comments. Part of the experiments were done on JD AI Platform "NeuHub" when Yisen Wang was at JD.com.
|| Explanaiions about the "best" and "last" results are in Appendix B
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICMI., 2018.
Yang Bai, Yan Feng, Yisen Wang. Tao Dai, Shu- Tao Xia, and Yong Jiang. Hilbert-basee generative defense for adversarial examples. In ICCV, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In S&P, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, and Aleksander Madry. On evaluating adversarial robustness. arXiv preprint 1rXiv:1902.06705, 2019.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John Duchi. Unlabeled data improves adversarial robustness. arXiv preprint 1905. 2019.
Chenyi Chen, Ari Seff, Alain Kornhause,, and Jianxiong Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In CVPR, 2015.
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv: (902:.0/9000000 1902.02918, 2019.
Nilaksh Das, Madhuri Shanbhogue, Shang- Chen, Fred Hohman, Siwei Li, Li Chen, Michael Kounavis, and Duen Horng Chau. Compression to the rescue: Defending from adversarial attacks across modalities. In KDD, 2018
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810. 04805, 2018.
Guneet S Dhillon, Kamyar Azizzadenesheli, Zachary C Lipton, Jeremy Bernstein, Jean Kossaifi, Aran Khanna, and Anima Anandkumar. Stochastic activation pruning for robust adversarial defense. ICLR, 2018
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Max-margin adversarial (mma) training: Direct input space margin maximization through adversarial training. arXiv preprint 1rXiv:1112.02637, 2018
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang. On the sensitivity of adversarial robustness input data distributions. ICLR, 2019.
Reuben Feinman, Ryan Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint IrXiv:1703.0000 170.000410 2017.
Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L. Beam, and Isaac S Kohane, Adversarial attacks on medical machine learning. Science, 36366433) 2019.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint 14X1v:1412.5006, 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using imput transformations. In ICLR, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In CVPR, 2016.
Linxi Jiang, Xingjun Ma, Shaoxiang Chen, James Bailey, and Yu-Gang Jiang. Black-box adversarial attacks on video recognition models. ACM MM, pp. 864-872, 2019.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv: arXiv:1803.0668 2018.
11

---

Published as a conference paper at ICLR 2020
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, University of Toronto, 2009.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial examples in the physical world. 2016.
arXiv preprint
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2
1998.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, pp. 7167-7177,
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In CVPR, 2018.
Qi Liu, Tao Liu, Zihao Liu, Yanzhi Wang. Yier Jin, and Wujie Wen. Security analysis and en- hancement of model compressed deep learning systems under adversarial attacks. arXiv preprint 1rXii::802.05/9/ 2018.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song. Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018.
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian James Bailey, and Feng Lu. Understanding adversarial attacks on deep learning based medical image analysis systems. arXiv preprint arXiv:190..10456. 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations in learning from incomplete data. arXiv preprint arXiv:1905. 2019.
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv preprint /01.v0290. 2019.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense adversarial perturbations against deep neural networks. In S&P, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Asia CCS, 2017.
Adnan Siraj Rakin, Jinfeng Yi, Boqing Gong, and Deliang Fan. Defend deep neural networks against adversarial examples via fixed anddynamic quantized activation functions. arXiv preprint arXiv:1807.06714. 2018.
Andrew Slavin Ross and Finale Doshi Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In AAAI, 2018
Hadi Salman, Greg Yang. Jerry Li, Pengchuan Zhang, Huan Zhang, Ilya Razenshteyn, and Sebastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. arXiv preprint arXiv:/906. 045844 2019.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-GAN: Protecting classifiers against adversarial attacks using generative models. In ICLR, 2018.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. In NewrIPS, 2018
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.61991 2013.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 30(1 (1:11):190 1958 2008
12

---

Published as a conference paper at ICLR 2020
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, lan Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be odds with accuracy. In ICLR, 2019.
Jonathan Uesato, Brendan O' O'Donoghue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks. arXiv preprint /.02.0060000066 2018.
Jonathan Uesato. Jean- nan-Baptiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhusscin Fawzi, Push- meet Kohli, et al. Are labels required for improving adversarial robustness? arXiv preprint 1905. 2019.
Yisen Wang, Xuejiao Deng, Songbai Pu, and Zhiheng Huang. Residual convolutional ctc networks for automatic speech recognition. arXiv preprint 17XIv:1702.07793, 2017.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICMI., 2019.
Dongxian Wu, Yisen Wang. Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter:
On the transferability of adversarial examples generated with resnets. In ICLR, 2020.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint 17Xvv:170.001155, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146. 2016.
Min Zeng. Yisen Wang, and Yuan Luo. Dirichlet latent variable hierarchical recurrent encoder-decoder in dialogue generation. In EMNLP, 2019.
Runtian Zhai, Tianle Cai, Di He, Chen Dan, Kun He, John Hopcroft, and Liwei Wang. Adversarially robust generalization just requires more unlabeled data. arXiv preprint arXiv: 00555, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In CVPR, 2016.
13

---

Published as a conference paper at ICLR 2020
A MART ALGORITHM
Algorithm Miselassification Aware adveRsarial Training (MART)
1: Input: Training data (xi, ys ti....... outer iteration number To. inner iteration number Tr, maximum perturbation 6, step size for inner optimization fjr, step size for outer optimization dj0 2: Initialization: Standard random initialization of he
3: for t = .,..., To do
4: Uniformly sample minibatch of training data B(t)
5: for x, E B(() do
6: x x; t ç. with 1))
2 is uniform distribution
7:
for s = ,...., Ti do
8:
x t (x,) (x} + 111 sign (Vx CE(p(x, 0), 9)))
-I(()) is the projection operator
9:
end for x f-x
11:
end for
12:
0 0+00 no Ex, VoL(x, yr &;;0)
13:
end for
14:
Output: Robust classifier he
B EXPLANATIONS ABOUT THE RESULTS OF TRADES
To make fair comparison with the latest method TRADES (hhag et al., 2019), our code built upon the TRADES framework. We would like point out that the robustness reported in their paper the b robustness that occurred during training. To avoid questioning our results, we have reported both the *best and the *last* results in Table 4 and the *best* results for TRADES match their original paper.
14


Reviews:
[{"review_id": "Hyl_k_T2YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n=======\nNeural Networks (NN) have been shown to be susceptible to various adversarial attacks i.e. if we perturb the \"x\" just a little, the output prediction changes. So, there has been much research devoted to how we can make NNs robust to such attacks. Typically, the adversarial examples that are used to train adversarially robust methods are generated from the correctly classified examples. \n\nThis paper first shows empirically that the adversarial examples generated from misclassified examples by the model h_{\\theta} are as important as the ones generated from correctly classified examples on the CIFAR dataset. The authors then propose a novel objective for the adversarial risk that incorporates the misclassified examples as a regularizer. Next, a surrogate convex objective is derived which is optimized to come up with a new kind of adversarial training called misclassification aware adversarial training (MART). \n\n\nOriginality:\n==========\nThe proposed objective seems novel to me compared to the existing methods, perhaps less so to someone who is an expert in adversarial methods. That said, I definitely found the addition of a misclassification-based regularization term as intriguing. \n\nHowever, I was wondering that maybe the previous approaches did not distinguish between correctly vs incorrectly classified examples while generating adversarial examples was because that is a more general setting. This is so because the idea of classification is tied to a certain model h_{\\theta} and hence is not model-agnostic by definition. Perhaps the strength of the approach by Madry et. al. 18, which is the closest work to this paper, is in being model-agnostic. I hope I am not missing something!  \n\n\nQuality:\n=======\nThe paper is technically sound and is a solid contribution to the literature on adversarial robustness. The motivation,  experimental results, and ablation studies are all very well executed. The results are shown on MNIST and CIFAR-10 image datasets and it outperforms a host of competitive baseline algorithms.\n\n\nClarity:\n=======\nThe paper is well organized and is very well written. The experimental results are very thorough and well explained.\n\n\nSignificance:\n============\nThe paper solves an important problem of dealing with adversarial attacks on deep neural networks. Further, the solution they propose is novel and a seems like a significant improvement over the extend state-of-the-art.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxsqDY7cH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper improves adversarial training by introducing two modifications to the loss function: (i) a \"boosted\" version of the cross-entropy loss that involves a term similar to a large-margin loss, and (ii) weighting the adversarial loss differently depending on how correctly classified an example is. When put together, these modifications achieve state-of-the-art robustness on CIFAR-10, improving over the previously best robust accuracy by about 3.5%. The authors perform multiple ablation studies and demonstrate that their modified loss function also improves when additional unlabeled data is added (again achieving state-of-the-art robustness).\n\nI recommend accepting the paper. The modifications for the loss function are well motivated and improve over the state of the art by a non-trivial amount. Moreover, the authors nicely put their loss function in the context of prior work.\n\nAdditional comments:\n\n- In Table 4, are the \"best\" columns the best checkpoint for the respective column (potentially different checkpoints for different columns) or does \"best\" refer to a single model (for each row)?\n\n- Is 65.04% (Table 5 b) now the best published robust accuracy on CIFAR-10 (at least to best of the authors' knowledge)? If so, it may be helpful to indicate this to the reader.\n\n- In Figure 2d, it could be insightful to expand the plot further to see the regime where the performance of MART drops substantially.\n\n- In Figure 1, the three plots would be easier to compare if the y-axes were the same.\n\n- From Figure 2, it looks like the gain from the BCE loss is as large as the gain from treating misclassified examples differently. Is this correct?\n\n- I strongly encourage the authors to release their models in a format that is easy for other researchers to use (e.g., PyTorch model checkpoints). This will make it substantially easier for future work to build on the results in this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1eyg-mJ9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper essentially presented a viewpoint, i.e. misclassified examples may have a significant impact on the final robustness.\nThe authors conducted a series of qualitative experiments to verify \n1) Misclassified examples have more impact on the final robustness than correctly classified examples.\n2) For misclassified examples, different maximization techniques may have a negligible influence, but minimization techniques play a critical role on the final robustness.\n3) The authors proposed a new defense algorithm which focus on generating adversarial examples from misclassified examples during the training. The algorithm was shown to improve the final robustness by revisiting these adversarial misclassified examples.\n\nGenerally speaking, the idea, though somewhat straightforward and less elegant, is reasonably presented and also well-motivated. Empirical validations seemed to support the idea and indicated that the proposed approach could improve the adversarial robustness.\n\n\nThere are several major concerns with the paper as follows:\n\n1.\tIt is good and reasonable to put more emphasis on the misclassification examples, since it is likely that the region of the “perturbation” largely overlaps with the region of the misclassified examples.\n\n2.\tHowever, this may be dealt with in a more elegant or systematic way. In the viewpoint of the reviewer, actually a different emphasis can be imposed on each different data point including both the correctly-classified and mis-classified samples. In another word, a more systematical extension or a metric may actually be developed emphasizing more on mis-classified samples. It is suspected that different samples within the set of mis-classified samples (or even in the set correctly-classified samples) could also have a different influence. It is highly likely that a more elegant and mathematic way can be designed for this purpose. \n\n3.\tFurther to point 2, in particular, the regulation w.r.t the correctly classified examples may not be ignored. It is interesting to consider differently the correctly-classified examples due to the trade-off between robustness and standard accuracy. This can also be seen in another submission of ICLR2020 (titled Sensible adversarial learning”) which actually discards the mis-classified samples. I would like to see some additional comments, clarification, or discussions on this point.\n\n4.\tI am curious to know if outliers would be over-emphasized by the proposed idea. Some discussion or even some illustrations on a synthetic case would be interesting. Will the existence of outliers affect the robustness?\n\n5.\tIn Table 2, the best and the last results on FGSM are totally identically, I wonder if this is a wrong copy, which should be further clarified and discussed.\n\n6.\t In the Unlabeled Data experiment part, it is better to compare the results with both UAT++ and RST at the same time rather than separately.\n\n7.\tEquation (4) shows that the two parts on the right side are added together, is that right? This is inconsistent with Equation (7) and the following description.\n\n8.\tIn 3.2, the first paragraph, “max-margin adversarial training (MMA) (Ding et al., 2019)” appears to be wrong. The correct reference is “GavinWeiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Max-margin adversarial (mma) training: Direct input space margin maximization through adversarial training. arXiv preprint arXiv:1812.02637, 2018.”.\n\n\n==================\nI have read carefully the response from the authors. The newly added results appear satisfactory to me and I  am generally happy with the clarifications. I have then  adjusted my rating accordingly.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents modifications to the adversarial training loss that yield improvements in adversarial robustness.  While some reviewers were concerned by the lack of mathematical elegance in the proposed method, there is consensus that the proposed method clears a tough bar by increasing SOTA robustness on CIFAR-10. 
Decision: Accept (Poster)