Title: IMPROVING ADVERSARIAL ROBUSTNESS REQUIRES REVISITING MISCLASSIFIED EXAMPLES

Abstract: Deep neural networks (DNNs) are vulnerable to adversarial examples crafted by imperceptible perturbations. A range of defense techniques have been proposed to improve DNN robustness to adversarial examples, among which adversarial training has been demonstrated to be the most effective. Adversarial training is often formulated as a min-max optimization problem, with the inner maximization for generating adversarial examples. However, there exists a simple, yet easily overlooked fact that adversarial examples are only defined on correctly classified (natural) examples, but inevitably, some (natural) examples will be misclassified during training. In this paper, we investigate the distinctive influence of misclassified and correctly classified examples on the final robustness of adversarial training. Specifically, we find that misclassified examples indeed have a significant impact on the final robustness. More surprisingly, we find that different maximization techniques on misclassified examples may have a negligible influence on the final robustness, while different minimization techniques are crucial. Motivated by the above discovery, we propose a new defense algorithm called Misclassification Aware adveRsarial Training (MART), which explicitly differentiates the misclassified and correctly classified examples during the training. We also propose a semi-supervised extension of MART, which can leverage the unlabeled data to further improve the robustness. Experimental results show that MART and its variant could significantly improve the state-of-the-art adversarial robustness.

1. INTRODUCTION:

Despite their great success in applications such as computer vision (He et al., 2016) , speech recognition (Wang et al., 2017) and natural language processing (Devlin et al., 2018; Zeng et al., 2019) , deep neural networks (DNNs) are extremely vulnerable to adversarial examples crafted by adding small adversarial perturbations to natural examples (Szegedy et al., 2013; Goodfellow et al., 2015; Wu et al., 2020) . Given a DNN classifier h θ with parameter θ and a correctly classified natural example x with class label y (h θ (x) = y), an adversarial example x can be generated by perturbing x such that h θ (x ) = y, i.e., the natural example is correctly classified before perturbation but misclassified after perturbation. The perturbation required for misclassification is often small and bounded by an L p -norm xx p ≤ , which keeps x within the -ball centered at x, so that it is visually the "same" for human observers. This vulnerability of DNNs raises serious security concerns about their practicability in security critical applications (Chen et al., 2015; Kurakin et al., 2016; Jiang et al., 2019; Finlayson et al., 2019; Ma et al., 2019) .

Compared with pre/post-processing methods such as feature squeezing (Xu et al., 2017) , input denoising (Guo et al., 2018; Liao et al., 2018; Samangouei et al., 2018; Bai et al., 2019) and adversarial detection (Feinman et al., 2017; Ma et al., 2018; Lee et al., 2018) , several defense techniques have been proposed to train DNNs that are inherently robust to adversarial examples including defensive distillation (Papernot et al., 2016) , gradient regularization (Gu & Rigazio, 2014; Papernot et al., 2017; Ross & Doshi-Velez, 2018; Tramèr et al., 2018) , model compression (Das et al., 2018; Liu et al., 2018) and activation pruning (Dhillon et al., 2018; Rakin et al., 2018) , among which adversarial training has been demonstrated to be the most effective (Athalye et al., 2018) . Adversarial training can be regarded as a data augmentation technique that trains DNNs on adversarial examples, and can be viewed as solving the following min-max optimization problem (Madry et al., 2018) :

EQUATION

where n is the number of training examples and (•) is the classification loss, such as the commonly used cross-entropy (CE) loss. The inner maximization generates adversarial examples that can be used by the outer minimization to train robust DNNs. Recently, adversarial training with adversarial examples generated by Projected Gradient Descent (PGD) (Madry et al., 2018) has been demonstrated to be the only method that can train moderately robust DNNs without being fully attacked (Athalye et al., 2018) . However, there is still a significant gap between adversarial robustness (test accuracy on adversarial examples) and natural accuracy (test accuracy on natural examples), even for simple image datasets like CIFAR-10 ( Krizhevsky & Hinton, 2009) .

Compared with natural training (on natural examples), training adversarially robust DNNs is particularly difficult (Madry et al., 2018) . Nakkiran (2019) showed that a model requires more capacity to be robust (i.e., simple models can have high natural accuracy but are less likely to be robust). In addition, the sample complexity of adversarial training can be significantly higher than that of natural training, that is, training robust DNNs tends to require more data either labeled (Schmidt et al., 2018) or unlabeled ones (Uesato et al., 2019; Carmon et al., 2019; Najafi et al., 2019; Zhai et al., 2019) . Moreover, Tsipras et al. (2019) ; Zhang et al. (2019) demonstrated that adversarial robustness may be inherently at odds with natural accuracy. Parallel to these studies, in this paper, we provide some new insights on the adversarial examples used for adversarial training.

Recall that the formal definition of an adversarial example is conditioned on it being correctly classified * (Carlini et al., 2019) • Experimentally, we show that adversarial robustness can be significantly improved over the stateof-the-art, by a specific focus on misclassified examples. It also helps improve recently proposed adversarial training with unlabeled data.

2. MISCLASSIFICATION AWARE ADVERSARIAL RISK:

In this section, we propose a regularized adversarial risk that incorporates an explicit differentiation of misclassified examples.

2.1. PRELIMINARIES:

We first define some notations. We use lower case and lower case bold face to denote scalars and vectors, respectively. We use upper case calligraphic symbols to denote sets.

For a K-class (K ≥ 2) classification problem, given a dataset {(x i , y i )} i=1,...,n with x i ∈ R d as a natural example and y i ∈ {1, . . . , K} as its associated label, a DNN classifier h θ with model parameter θ predicts the class of an input example x i :

EQUATION

where z k (x i , θ) is the logits output of the network with respect to class k, and p k (x i , θ) is the probability (softmax on logits) of x i belonging to class k.

The adversarial risk (Madry et al., 2018 ) on dataset {(x i , y i )} i=1,...,n and classifier h θ can be defined with respect to the 0-1 loss (Zhang et al., 2019) as:

EQUATION

where 1(•) is the indicator function and B (x i ) = {x : xx i p ≤ } denotes the L p -norm ball centered at x i with radius . We will focus on the L ∞ -ball in this paper.

2.2. MISCLASSIFICATION AWARE REGULARIZATION:

Note that the adversarial risk in ( 3) is defined on adversarial examples within the -ball of all natural examples, regardless of whether they are correctly classified (h θ R

(x i ) = y i ) or misclassified (h θ (x i ) = y i )

S + h θ = {i : i ∈ [n], h θ (x i ) = y i } and S - h θ = {i : i ∈ [n], h θ (x i ) = y i }.

EQUATION

) where the adversarial example x i is generated by solving

x i = arg max

EQUATION

We remark that the first and second terms on the R.H.S. of (4) correspond to the standard adversarial risk and the regularization term respectively. Moreover, we would like to clarify that the regularization term 1(h θ (x i ) = h θ (x i )) aims to encourage the output of neural network to be stable against misclassified adversarial examples. For misclassified examples, direct minimization of the standard adversarial risk may be too hard, as themselves cannot be classified correctly, even without any perturbations. A similar idea has been used in stability training on all training examples (Zheng et al., 2016; Kannan et al., 2018; Zhang et al., 2019) .

Then we consider correctly classified examples. As can be observed in Figure 1 (c), regularization on correctly classified examples cannot provide as significant improvement as achieved by that on misclassified ones. Moreover, in this case it can be found that

1(h θ (x i ) = h θ (x i )) = 1(h θ (x i ) = y i )

since we have h θ (x i ) = y i , which implies that the regularizer has exactly same form as the adversarial risk. Therefore, for correctly classified example, we simply use the standard adversarial risk, i.e., R + (h θ , x i ) := max

EQUATION

where the adversarial example x i is defined in (5).

Finally, combining the proposed two adversarial risks for correctly classified examples and misclassified examples in an adversarial training framework, we can train a network that minimizes the following risk:

EQUATION

) where x i is defined in (5) and the second equality follows from the definition of S h + θ and S h - θ . The new risk defined above is a regularized adversarial risk with regularization term 1/n

n i=1 1(h θ (x i ) = h θ (x i )) • 1(h θ (x i ) = y i

), which we call the misclassification aware regularization.

3. PROPOSED DEFENSE: MISCLASSIFICATION AWARE ADVERSARIAL TRAINING (MART):

In the previous section, we derived the misclassification aware adversarial risk based on 0-1 loss. However, optimization over 0-1 loss is intractable in practice. We next propose a Misclassification Aware adveRsarial Training (MART) algorithm, by replacing the 0-1 losses with proper surrogate loss functions which are both physical meaningful and computationally tractable. Following that, we further analyze the difference of MART to existing work, and propose a semi-supervised extension.

3.1. THE PROPOSED DEFENSE ALGORITHM:

Surrogate Loss for Outer Minimization. As presented in (7), the minimization consists of three indicator functions: (1)

1(h θ (x i ) = y i ); (2) 1(h θ (x i ) = h θ (x i )); and (3) 1(h θ (x i ) = y i ).

For the first indicator function 1(h θ (x i ) = y i ), we propose to use a boosted cross entropy (BCE) loss as the surrogate loss, instead of the commonly used CE loss in (Madry et al., 2018; Wang et al., 2019) . This is largely because classifying adversarial examples requires a stronger classifier than natural examples, as the presence of adversarial examples makes the classification decision boundary become more complicated. This is pointed out by (Madry et al., 2018) , where they increase the model capacity for a stronger classifier. The benefit of using BCE compared to CE will shortly be presented in the experiment section. The proposed BCE loss is defined as:

EQUATION

where p k (x i , θ) is the probability output defined in (2), the first term -log p yi (x i , θ) is the commonly used CE loss, denoted CE(p(x i , θ), y i ), and the second term

-log 1 -max k =yi p k (x i , θ)

is a margin term used to improve the decision margin of the classifier. A similar idea has been used for improving adversarial strength by Carlini & Wagner (2017) . Note that BCE is just a simple boost that works well in our experiments and other boosted losses could also work here.

For the second indicator function 1(h θ (x i ) = h θ (x i )), we can use KL divergence as the surrogate loss function (Zhang et al., 2019; Zheng et al., 2016) , since h θ (x i ) = h θ (x i ) implies that adversarial examples have different output distributions to that of natural examples. Thus, we have

EQUATION

)

The third indicator function 1(h θ (x i ) = y i ) is a condition that emphasizes learning on misclassified examples. However, the condition cannot be directly optimized if we conduct a hard decision during the training process (Ding et al. ( 2019) uses hard decision and does not optimize the condition). Instead, we propose to use a soft decision scheme by replacing 1(h θ (x i ) = y i ) with the output probability 1 -p yi (x i , θ). This will be large for misclassified examples and small for correctly classified examples.

Surrogate Loss for Inner Maximization. The goal of inner maximization is to generate adversarial example x i for natural example x i by solving (5). Therefore, we aim to find a surrogate loss function for the indicator function 1(h θ (x i ) = y i ). Here, we leverage the commonly used CE loss as the surrogate loss and find the adversarial example x i † as follows:

x i = arg max

EQUATION

)

Following our findings in Figure 1 Note that other surrogate loss functions, such as those exploited in (Athalye et al., 2018; Carlini & Wagner, 2017) for adversarial attack, could also be used here.

The Overall Objective. Based on the surrogate loss functions, we can state the final objective function for our proposed Misclassification Aware adveRsarial Training (MART) defense:

EQUATION

where (x i , y i , θ) is defined as

(x i , y i , θ) := BCE p(x i , θ), y i + λ • KL p(x i , θ) p(x i , θ) • 1 -p yi (x i , θ) .

Here the adversarial example x i is generated by (10), and λ is a tunable scaling parameter that balances the two parts of the final loss, and is fixed for all training examples. The complete training procedure of MART is described in Appendix A.

3.2. RELATION TO EXISTING WORK:

In this section, we briefly discuss the difference between our MART and existing defense methods including standard adversarial training (Standard) (Madry et al., 2018) 

+ λ • p(x , θ) -p(x, θ) 2 2 CLP CE(p(x, θ), y) + λ • p(x , θ) -p(x, θ) 2 2 TRADES CE(p(x, θ), y) + λ • KL p(x, θ)||p(x , θ) MMA CE(p(x , θ), y) • 1(h θ (x) = y) + CE(p(x, θ), y) • 1(h θ (x) = y) MART BCE(p(x , θ), y) + λ • KL(p(x, θ)||p(x , θ)) • (1 -p y (x, θ))

Specifically, the Standard algorithm was designed to minimize the standard adversarial loss, i.e 

3.3. SEMI-SUPERVISED EXTENSION WITH UNLABELED DATA:

Recent work has shown that semi-supervised learning with additional unlabeled data can improve the adversarial robustness (Uesato et al., 2019; Carmon et al., 2019; Najafi et al., 2019; Zhai et al., 2019) . Specifically, the training loss function applied in these semi-supervised learning methods is typically defined as a weighted sum of the supervised loss (loss on the labeled data) and the unsupervised one (loss on the unlabeled data), i.e.,

L(θ) = L sup (θ) + γ • L unsup (θ),

where γ > 0 is the weight of unsupervised loss. As pointed out in Uesato et al. (2019) , there are multiple choices of the unsupervised loss function L unsup (θ), leading to different defense methods, among which the most effective defense method is UAT++. In particular, UAT++ first trains a natural model on labeled data, and then use this model to generate pseudo labels for unlabeled data. Moreover, given a training data (x, y) (which can be either labeled or unlabeled data), the supervised and unsupervised loss functions adopted in UAT++ are defined as ‡ UAT++ sup

(x, y; θ) = UAT++ unsup (x, y; θ) = max x ∈B CE(p(x , θ), y) + λ • max x ∈B KL(p(x, θ)||p(x , θ)),

where λ is a tunable hyperparameter. A similar idea was also proposed in a concurrent work (Carmon et al., 2019) , leading to another semi-supervised defense method called RST. The first stage of RST is also to generate pseudo labels for unlabeled data by training a natural model on the labeled data.

Then in the second stage, RST applies TRADES loss to train the robust model based on both labeled and unlabeled data, i.e., given a training data (x, y), the supervised and unsupervised loss functions adopted in RST is defined as

RST sup (x, y; θ) = RST unsup (x, y; θ) = CE(p(x, θ), y) + λ • max x ∈B KL(p(x, θ)||p(x , θ)).

As we pointed out in Figure 1 (b) and the following experiment section, the maximization technique has a neglectable influence on the robustness. Therefore, the major difference between UAT++ and RST is the objective function for minimization. Considering MART is also an objective function, it thus could be easily combined with semi-supervised learning with unlabeled data. Following RST, we propose the following semi-supervised version of MART:

L MART semi (θ) = i∈Ssup MART sup (x i , y i ; θ) + γ • i∈Sunsup MART unsup (x i , y i ; θ)

with supervised and unsupervised loss function defined as follows, 

4. EXPERIMENTS:

In this section, we first conduct a set of experiments to provide a comprehensive understanding of our proposed defense MART, and then evaluate its robustness on benchmark datasets in both white-box and black-box settings. Finally, we benchmark the state-of-the-art robustness and explore using unlabeled data for further improvement.

4.1. UNDERSTANDING THE PROPOSED MART:

Here, we investigate MART from 4 different perspectives: (1) removing components of the MART loss function, (2) replacing components of the MART loss function, (3) misclassification aware loss on certain proportions of training data, and (4) sensitivity to regularization parameter λ.

Experimental Setup. We train ResNet-18 (He et al., 2016) with different variants of MART on CIFAR-10 (Krizhevsky & Hinton, 2009) . All the models are trained using SGD with momentum 0.9, weight decay 2 × 10 -4 and an initial learning rate of 0.1, which is divided by 10 at the 75-th and 90-th epoch. All natural images are normalized into [0, 1], and simple data augmentations including 4-pixel padding with 32 × 32 random crop and random horizontal flip. The maximum perturbation = 8/255 and parameter λ = 6. The training attack is PGD 10 with random start and step size /4, while the test attack is PGD 20 with random start and step size /10.

Removing Components of MART. Recalling the objective function of MART in (11) , it has three terms in the loss function: BCE, KL and 1 -p § . As illustrated in Figure 2(a) , removing 1 -p or KL ‡ The adversarial example x in KL term is reused the one in CE term for efficiency in Uesato et al. (2019) § For simplicity, we use these abbreviations in Section 4.1 or both all leads to a significant robustness degradation. In particular, we found that the soft decision term 1 -p has a constant robustness improvement throughout the training process, while the KL term can help mitigate overfitting at a later stage of training (after 80 epochs). When the two terms are combined together, they boost the final robustness considerably without causing overfitting.

Replacing Components of MART. As we show in Figure 2 (b), when the BCE component is either replaced by a CE term or redefined on natural examples (x nat ), the final robustness decreases by a substantial amount. It suggests that learning with CE instead of our proposed BCE suffers from insufficient learning with lower robustness throughout the entire training process. On the other hand, learning with BCE on natural examples exhibits severe overfitting at the later stage (solid green line). We did not observe any benefit when replacing CE by KL in the inner maximization of adversarial min-max framework (solid red line), an observation that is consistent with Figure 1 (b).

Ablation on Training Data. Here, we show the contribution of our proposed misclassification aware regularization (e.g., the KL • (1 -p) term in ( 11)) to the final robustness with respect to the training data. Specifically, we gradually increase the proportion of training examples that are trained using the proposed misclassification aware regularization term, and display the corresponding robustness in Figure 2 (c). The training examples using the proposed regularization are randomly selected, and the BCE term is still defined on all training (adversarial) examples. As can be observed, the robustness can be improved steadily when the proposed regularization is applied on more data. This verifies the benefit of the differentiation of correctly classified and misclassified examples.

Sensitivity to Regularization Parameter λ. We further investigate the parameter λ in MART objective function defined in (11) which controls the strength of the regularization. We also test the regularization parameter λ of TRADES (please refer to Table 1 ). We present the results in Figure 2 , 50] . By explicitly differentiating the misclassified and correctly classified examples, MART achieves good stability and robustness across different choices of λ, which is also consistently better and more stable than that of TRADES.

(d) ¶ for different λ ∈ [1/2

4.2. ROBUSTNESS EVALUATION AND ANALYSIS:

In this part, we evaluate the robustness of MART on both MNIST (LeCun et al., 1998) et al., 2019) ; and (4) TRADES (Zhang et al., 2019) . We only compare with adversarial training variants, since they are the most effective defense to date (Athalye et al., 2018) .

Defense Settings. For MNIST, all defense models are built on a 4-layer CNN and trained using SGD with momentum 0.9. The initial learning rate is 0.01 and divided by 10 at the 20-th and 40-th epoch. For CIFAR-10, we use SGD with momentum 0.9, weight decay 3.5 × 10 -3 and an initial learning rate of 0.01, which is divided by 10 at the 75-th and 90-th epoch. For the training attack, it is also the PGD 10 with random start and step size /4. The perturbation limit = 0.3 for MNIST, and = 8/255 for CIFAR-10. For MART, we set λ = 5. Hyperparameters of the baselines are configured as per their original papers: max margin is set to 0.45 (MNIST) or 12/255 (CIFAR-10) for MMA, maximum criterion value c max = 0.5 for Dynamic and λ = 4 for TRADES. 11) . Note that the robustness improvement of MART is not caused by the so-called "obfuscated gradients" (Athalye et al., 2018) . This can be verified by two phenomenons: (1) strong test attacks (e.g., PGD 20 ) have higher success rates (lower accuracies) than weak test attacks (e.g., FGSM), and (2) white-box test attacks have higher success rates than back-box test attacks (comparing Table 2 with Table 3 ). Besides, we conduct an additional check using a gradient-free attack SPSA (Uesato et al., 2018) . SPSA attack does not obtain lower accuracy than gradient-based attacks like PGD, which confirms that the robustness of MART trained models are not due to gradient masking.

Black-box Robustness. Black-box test attacks are crafted from the natural test images by attacking a surrogate model with an architecture that is either a copy of the defense model (MNIST) or a more complex ResNet-50 model (CIFAR-10). Both surrogate models are trained separately from the defense models on the original training sets. The attacking methods used here are: FGSM, PGD 10 , PGD 20 , and CW ∞ . The black-box robustness of all defense models are reported in Table 3 . Again, the proposed defense MART achieves higher robustness than other baselines. Compared with the white-box results, all defense methods achieve much better robustness against black-box attacks, even close to the natural accuracy. This suggests that adversarial training is indeed a very practical choice for defense scenarios where the target model can be kept secret from potential attackers. It is also observed that robustness on strong attacks like CW ∞ is higher than weak attacks like FGSM, which indicates that strong attacks have less transferability than weak attacks (Madry et al., 2018) .

4.3. BENCHMARKING THE STATE-OF-THE-ART ROBUSTNESS:

In this part, we conduct more experiments on a large-capacity network WideResNet (Zagoruyko & Komodakis, 2016) to benchmark the state-of-the-art robustness, and also explore using unlabeled data for further robustness boost. Boosting with Additional Unlabeled Data. Here, we evaluate the proposed semi-supervised version of MART and show that it can also benefit from additional unlabeled data and achieves better robustness. Following the exact settings in UAT++ (Uesato et al., 2019) and RST (Carmon et al., 2019) , we compare the robustness of MART with them on WideResNet-34-8 and WideResNet-28-10 against PGD 20 (FGSM 20 ) respectively (in the same setting as reported in their paper). The dataset is CIFAR-10 with 100K and 500K unlabeled data extracted from the 80 Million Tiny Images dataset (Torralba et al., 2008) . As confirmed in Table 5 , our proposed defense MART can also benefit from unlabeled data, and further improves the UAT++ and RST defenses. This again verifies the benefit of differentiating misclassified and correctly classified examples for improving robustness, and further demonstrates the superiority of our proposed method.

5. CONCLUSION AND FUTURE WORK:

In this paper, we investigated the interesting observation that misclassified examples have a recognizable impact on the final robustness of adversarial training, especially for the outer minimization process. Based on this observation, we designed a misclassification aware adversarial risk, which is formulated as adding an misclassification aware regularization to the standard adversarial risk. Following the regularized adversarial risk, we proposed a new defense algorithm, called Misclassification Aware adveRsarial Training (MART), with appropriate surrogate loss functions. Experimental results demonstrated that MART can achieve significantly improved adversarial robustness with respect to the state-of-the-art, and can also achieves better robustness with additional unlabeled data.

In the future, we plan to investigate the effect of differentiation of correctly classified/misclassified training examples in the recently proposed certified/provable robustness framework (Cohen et al., 2019; Salman et al., 2019) and explore the potential improvements brought by the differentiation of training examples.

ACKNOWLEDGEMENT:

We thank the anonymous reviewers and area chair for their helpful comments. Part of the experiments were done on JD AI Platform "NeuHub" when Yisen Wang was at JD.com. Uniformly sample a minibatch of training data B (t) 5:

A MART ALGORITHM:

for x i ∈ B (t) do 6:

x i = x i + • ξ, with ξ ∼ U(-1, 1)

# U is a uniform distribution 7:

for s = 1, . . . , T I do 8:

x i ← Π B (xi) x i + η I • sign(∇ x i CE(p(x i , θ), y i )) # Π(•) is the projection operator 9: end for 10:

x i ← x i 11:

end for 12:

θ ← θ -η O xi∈B (t) ∇ θ L(x i , y i , x i ; θ) 13: end for 14: Output: Robust classifier h θ

B EXPLANATIONS ABOUT THE RESULTS OF TRADES:

To make a fair comparison with the latest method TRADES (Zhang et al., 2019) , our code is built upon the TRADES framework. We would like to point out that the robustness reported in their paper is the *best* robustness that occurred during training. To avoid questioning of our results, we have reported both the *best* and the *last* results in Table 4 , and the *best* results for TRADES match their original paper.

:

* In this paper, both "correctly classified" and "misclassified" refer to predictions on natural training examples.

† We slightly abuse the notation of x i as the maximizer of the surrogate loss rather than the 0-1 loss in (5).

¶ For clarity, we set the ticks on x-axis in uniform interval for different λ. Standard, MMA and TRADES have been introduced in Section 3.2. Dynamic is the adversarial training with a criterion that dynamically controls the convergence quality of the inner maximization.

Explanations about the "best" and "last" results are in Appendix B.

Figures and Tables:

Figure 1: The distinctive influence of misclassified examples (S -) versus correctly classified ones (S + ) on the robustness of adversarial training. We test the white-box robustness of different strategies on either subset of examples: (a) using them directly for training ("not perturb"); (b) using weak attack (FGSM) in the inner maximization; and (c) using "regularized CE" loss in the outer minimization.

Then we are going to define adversarial risk separately for correctly classified and misclassified examples. As we observed in Figure 1(c), regularization on misclassified examples can significantly improve robustness. Therefore, for misclassified examples, we formulate the adversarial risk as:

(b) that a strong attack can help robustness (though is negligible for misclassified examples), we propose to use the (strong) PGD attack to maximize the CE loss for both the correctly classified and misclassified examples, the same as standard adversarial training.

y; θ) = MART unsup (x, y; θ) = BCE(p(x , θ), y) + λ • KL(p(x, θ)||p(x , θ)) • (1 -p y (x, θ)),where the adversarial example x is generated by solving (10), and S sup and S unsup denote the set of labeled data and unlabeled data respectively.

Figure 2: The comprehensive ablation experiments of MART. In each plot, the dashed blue line represents the original MART method.

and CIFAR-10 (Krizhevsky & Hinton, 2009) datasets against various white-box and black-box attacks. Baselines. (1) Standard (Madry et al., 2018); (2) MMA (Ding et al., 2019); (3) Dynamic (Wang

. From this perspective, adversarial examples generated from misclassified examples are "undefined". Most adversarial training variants neglect this distinction, where all training examples are treated equally in both the maximization and the minimization processes, regardless of whether or not they are correctly classified. The only exception we are aware of is Ding et al. (2018), which proposes to use maximal margin optimization for correctly classified examples. Yet they did not pay sufficient attention to misclassified examples. A deeper understanding about the influence of misclassified and correctly classified examples on the robustness is still missing in the literature. Therefore, we raise the following questions: Convolutional Neural Network (CNN) using standard adversarial training with 10-step PGD (PGD 10 ) and step size /4, then use this network (87% training accuracy) to select two subsets of natural training examples to investigate: 1) a subset of misclassified examples S -(13% of training data), and 2) a subset of correctly classified examples S + (also 13% of training data, |S + | = |S -|).Using these two subsets, we explore different ways to re-train the same network, and evaluate its robustness against white-box PGD 20 (step size /10) attacks on the test dataset.

Table content: <table><tr><td>In Figure 1(a), we find that misclassified examples have a significant impact on the final robustness.</td></tr><tr><td>Compared with standard adversarial training (dashed blue line), the final robustness drops drastically,</td></tr><tr><td>if examples in subset S -are not perturbed (solid green line) during adversarial training (other</td></tr><tr><td>examples are still perturbed by PGD 10 ). In contrast, the same operation on subset S + only slightly</td></tr><tr><td>affects the final robustness (solid orange line). Previous work has found that removing a small</td></tr><tr><td>proportion of training examples does not reduce the robustness (Ding et al., 2019), which seems to be</td></tr><tr><td>true for correctly classified examples, but is apparently not true for misclassified examples.</td></tr><tr><td>To further understand the distinctive influence of misclassified and correctly classified examples,</td></tr><tr><td>we test different techniques on them within either the maximization or the minimization process</td></tr><tr><td>of adversarial training. Firstly, we apply different maximization techniques while keeping the</td></tr><tr><td>minimization loss CE unchanged. As shown in Figure 1(b), the final robustness is barely affected</td></tr><tr><td>when we use a weak attack (e.g., Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015))</td></tr><tr><td>to perturb misclassified examples S -(all other training examples are still perturbed by PGD 10 ).</td></tr><tr><td>This suggests that different maximization techniques on misclassified examples S -may have a</td></tr><tr><td>negligible influence on the final robustness, provided that the inner maximization problem is solved to</td></tr><tr><td>a moderate precision (Wang et al., 2019). However, for subset S + , a weak attack for the maximization</td></tr><tr><td>tends to degenerate the robustness. Secondly, we test different minimization techniques with the inner</td></tr><tr><td>maximization still solved by PGD 10 . Interestingly, we find that different minimization techniques on</td></tr><tr><td>misclassified examples make a significant difference to the final robustness. As shown in Figure 1(c),</td></tr><tr><td>compared with standard adversarial training (dashed blue line) with the CE loss, the final robustness</td></tr><tr><td>is significantly improved when the outer minimization on misclassified examples is "regularized"</td></tr><tr><td>(solid green line) by an additional term (a KL-divergence term that was used previously in Zheng</td></tr><tr><td>et al. (2016); Zhang et al. (2019)). The same regularization applied to correctly classified examples</td></tr><tr><td>also helps the final robustness (solid orange line), though not as significantly as for misclassified</td></tr><tr><td>examples.</td></tr><tr><td>Motivated by the above observations, we reformulate the adversarial risk to incorporate an explicit</td></tr><tr><td>differentiation of misclassified examples in a form of regularization. We then propose a new defense</td></tr><tr><td>algorithm to achieve this in a dynamic way during adversarial training. Our main contributions are:</td></tr><tr><td>• We investigate the distinctive influence of misclassified and correctly classified examples on the</td></tr><tr><td>final robustness of adversarial training. We find that the manipulation on misclassified examples</td></tr><tr><td>has more impact on the final robustness, and the minimization techniques are more crucial than</td></tr><tr><td>maximization ones under the min-max optimization framework.</td></tr><tr><td>• We propose a regularized adversarial risk which incorporates an explicit differentiation of misclas-</td></tr><tr><td>sified examples as a regularizer. Based on that, we further propose a new defense algorithm, called</td></tr><tr><td>Misclassification Aware adveRsarial Training (MART).</td></tr><tr><td>Are the adversarial examples generated from i) misclassified and ii) correctly</td></tr><tr><td>classified examples, equally important for adversarial robustness? If not, how can</td></tr><tr><td>one make better use of the difference to improve robustness?</td></tr></table>

, logit pairing methods(Kannan et al., 2018), max-margin adversarial training (MMA)(Ding et al., 2018) and TRADES(Zhang et al., 2019), as presented in Table1. Loss function comparison with existing work. The adversarial example x is generated by (10) for all defense methods except TRADES and MMA. The adversarial example in TRADES is generated by maximizing its regularization term (KL-divergence), and the adversarial example in MMA is generated by solving (10) with different perturbation limit (i.e., ).

Table content: <table><tr><td>Defense Method</td><td>Loss Function</td></tr><tr><td>Standard</td><td>CE(p(</td></tr></table>

White-box robustness (accuracy (%) on white-box test attacks) on MNIST and CIFAR-10. We evaluate the robustness of all defense models against three types of attacks for both MNIST and CIFAR-10: FGSM, PGD 20 (20-step PGD with step size /10), and CW ∞ (L ∞ version of CW optimized by PGD). All attacks have full access to model parameters and are constrained by the same perturbation limit . The white-box robustness of all defense models are reported in Table2, where "Natural" denotes the accuracy on natural test images. Our proposed defense MART achieves the best robustness against all three types of attacks on both MNIST and CIFAR-10. Compared with MNIST, the robustness improvements of MART over other baselines are more significant on CIFAR-10. This is because adversarial training on CIFAR-10 is a more challenging problem that may have more misclassified examples during training, and MART can better handle those misclassified examples due to its regularization term in (

Table content: <table><tr><td/><td/><td colspan="2">MNIST</td><td/><td/><td colspan="2">CIFAR-10</td><td/></tr><tr><td>Defense</td><td>Natural</td><td>FGSM</td><td>PGD 20</td><td>CW ∞</td><td>Natural</td><td>FGSM</td><td>PGD 20</td><td>CW ∞</td></tr><tr><td>Standard</td><td>99.11</td><td>97.17</td><td>94.62</td><td>94.25</td><td>84.44</td><td>61.89</td><td>47.55</td><td>45.98</td></tr><tr><td>MMA</td><td>98.92</td><td>97.25</td><td>95.25</td><td>94.77</td><td>84.76</td><td>62.08</td><td>48.33</td><td>45.77</td></tr><tr><td>Dynamic</td><td>98.96</td><td>97.34</td><td>95.27</td><td>94.85</td><td>83.33</td><td>62.47</td><td>49.40</td><td>46.94</td></tr><tr><td>TRADES</td><td>99.25</td><td>96.67</td><td>94.58</td><td>94.03</td><td>82.90</td><td>62.82</td><td>50.25</td><td>48.29</td></tr><tr><td>MART</td><td>98.74</td><td>97.87</td><td>96.48</td><td>96.10</td><td>83.07</td><td>65.65</td><td>55.57</td><td>54.87</td></tr><tr><td colspan="9">Table 3: Black-box robustness (accuracy (%) on black-box test attacks) on MNIST and CIFAR-10.</td></tr><tr><td/><td/><td colspan="2">MNIST</td><td/><td/><td colspan="2">CIFAR-10</td><td/></tr><tr><td>Defense</td><td>FGSM</td><td>PGD 10</td><td>PGD 20</td><td>CW ∞</td><td>FGSM</td><td>PGD 10</td><td>PGD 20</td><td>CW ∞</td></tr><tr><td>Standard</td><td>96.12</td><td>95.73</td><td>95.47</td><td>96.34</td><td>79.98</td><td>80.27</td><td>80.01</td><td>80.85</td></tr><tr><td>MMA</td><td>96.11</td><td>95.94</td><td>95.81</td><td>96.87</td><td>80.28</td><td>80.52</td><td>80.48</td><td>81.32</td></tr><tr><td>Dynamic</td><td>97.60</td><td>96.25</td><td>95.82</td><td>97.03</td><td>81.37</td><td>81.71</td><td>81.38</td><td>82.05</td></tr><tr><td>TRADES</td><td>97.49</td><td>96.03</td><td>95.73</td><td>97.20</td><td>81.52</td><td>81.73</td><td>81.53</td><td>82.11</td></tr><tr><td>MART</td><td>97.77</td><td>96.96</td><td>96.97</td><td>98.36</td><td>82.75</td><td>82.93</td><td>82.70</td><td>82.95</td></tr><tr><td colspan="2">White-box Robustness.</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

White-box robustness (%) on CIFAR-10 using the WideResNet-34-10.

Table content: <table><tr><td/><td/><td colspan="2">FGSM</td><td colspan="2">PGD 20</td><td colspan="2">PGD 100</td><td colspan="2">CW ∞</td></tr><tr><td>Defense</td><td>Natural</td><td>Best</td><td>Last</td><td>Best</td><td>Last</td><td>Best</td><td>Last</td><td>Best</td><td>Last</td></tr><tr><td>Standard</td><td>87.30</td><td colspan="8">56.10 56.10 52.68 49.31 51.55 49.03 50.73 48.47</td></tr><tr><td>Dynamic</td><td>84.51</td><td colspan="8">63.53 63.53 55.03 51.70 54.12 50.07 51.34 49.27</td></tr><tr><td>TRADES</td><td>84.22</td><td colspan="8">64.70 64.70 56.40 53.16 55.68 51.27 51.98 51.12</td></tr><tr><td>MART</td><td>84.17</td><td colspan="8">67.51 67.51 58.56 57.39 57.88 55.04 54.58 54.53</td></tr></table>

White-box robustness (%) on WideResNet with additional unlabeled data. best and the last epoch models obtained during training in Table4. For each defense method against each attack, the "best" refers to the highest robustness that ever achieved at different checkpoints. Specifically, against FGSM attack, the best model was found at the last epoch (e.g., "best" is also "last") for all defense methods, while against PGD 20 , PGD 100 and CW ∞ attacks, the best model was found at the epoch right after the first time learning rate decay (i.e., epoch 76). Our proposed MART outperforms all baseline methods in terms of the robustness of both the best and the last epoch models. Particularly under the most common comparison setting (against PGD 20 attacks on CIFAR-10), MART improved ∼ 8% over Standard, and ∼ 4% even over TRADES for the last epoch model. A similar trend of improvement is also observed for the best epoch model results. Considering the worst case accuracies against all attacks, MART still gains ∼ 6% and ∼ 3.5% robustness improvement over Standard and TRADES respectively.

Table content: <table><tr><td colspan="3">a ) WideResNet-34-8 with 100K unlabeled data</td><td colspan="3">b ) WideResNet-28-10 with 500K unlabeled data</td></tr><tr><td colspan="3">Defense Natural PGD 20</td><td colspan="3">Defense Natural PGD 20</td></tr><tr><td>UAT++</td><td>86.04</td><td>59.41</td><td>UAT++</td><td>86.21</td><td>62.76</td></tr><tr><td>RST</td><td>88.24</td><td>59.60</td><td>RST</td><td>89.70</td><td>63.10</td></tr><tr><td>MART</td><td>86.68</td><td>61.88</td><td>MART</td><td>86.30</td><td>65.04</td></tr><tr><td>of both the</td><td/><td/><td/><td/><td/></tr></table>

Algorithm 1 Misclassification Aware adveRsarial Training (MART) 1: Input: Training data {x i , y i } i=1,...,n , outer iteration number T O , inner iteration number T I , maximum perturbation , step size for inner optimization η I , step size for outer optimization η O 2: Initialization: Standard random initialization of h θ 3: for t = 1, . . . , T O do

Table content: <table><tr><td>4:</td></tr></table>

Peer review (iclr2020):
The paper essentially presented a viewpoint, i.e. misclassified examples may have a significant impact on the final robustness. The authors conducted a series of qualitative experiments to verify 1) Misclassified examples have more impact on the final robustness than correctly classified examples. 2) For misclassified examples, different maximization techniques may have a negligible influence, but minimization techniques play a critical role on the final robustness. 3) The authors proposed a new defense algorithm which focus on generating adversarial examples from misclassified examples during the training. The algorithm was shown to improve the final robustness by revisiting these adversarial misclassified examples. Generally speaking, the idea, though somewhat straightforward and less elegant, is reasonably presented and also well-motivated. Empirical validations seemed to support the idea and indicated that the proposed approach could improve the adversarial robustness. There are several major concerns with the paper as follows: 1. It is good and reasonable to put more emphasis on the misclassification examples, since it is likely that the region of the “perturbation” largely overlaps with the region of the misclassified examples. 2. However, this may be dealt with in a more elegant or systematic way. In the viewpoint of the reviewer, actually a different emphasis can be imposed on each different data point including both the correctly-classified and mis-classified samples. In another word, a more systematical extension or a metric may actually be developed emphasizing more on mis-classified samples. It is suspected that different samples within the set of mis-classified samples (or even in the set correctly-classified samples) could also have a different influence. It is highly likely that a more elegant and mathematic way can be designed for this purpose. 3. Further to point 2, in particular, the regulation w.r.t the correctly classified examples may not be ignored. It is interesting to consider differently the correctly-classified examples due to the trade-off between robustness and standard accuracy. This can also be seen in another submission of ICLR2020 (titled Sensible adversarial learning”) which actually discards the mis-classified samples. I would like to see some additional comments, clarification, or discussions on this point. 4. I am curious to know if outliers would be over-emphasized by the proposed idea. Some discussion or even some illustrations on a synthetic case would be interesting. Will the existence of outliers affect the robustness? 5. In Table 2, the best and the last results on FGSM are totally identically, I wonder if this is a wrong copy, which should be further clarified and discussed. 6. In the Unlabeled Data experiment part, it is better to compare the results with both UAT++ and RST at the same time rather than separately. 7. Equation (4) shows that the two parts on the right side are added together, is that right? This is inconsistent with Equation (7) and the following description. 8. In 3.2, the first paragraph, “max-margin adversarial training (MMA) (Ding et al., 2019)” appears to be wrong. The correct reference is “GavinWeiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Max-margin adversarial (mma) training: Direct input space margin maximization through adversarial training. arXiv preprint arXiv:1812.02637, 2018.”. ================== I have read carefully the response from the authors. The newly added results appear satisfactory to me and I am generally happy with the clarifications. I have then adjusted my rating accordingly.