Title: LEARNING WITH PROTECTION: REJECTION OF SUSPICIOUS SAMPLES UNDER ADVERSARIAL ENVIRONMENT

Abstract: We propose a novel framework for avoiding the misclassification of data by using a framework of learning with rejection and adversarial examples. Recent developments in machine learning have opened new opportunities for industrial innovations such as self-driving cars. However, many machine learning models are vulnerable to adversarial attacks and industrial practitioners are concerned about accidents arising from misclassification. To avoid critical misclassifications, we define a sample that is likely to be mislabeled as a suspicious sample. Our main idea is to apply a framework of learning with rejection and adversarial examples to assist in the decision making for such suspicious samples. We propose two frameworks, learning with rejection under adversarial attacks and learning with protection. Learning with rejection under adversarial attacks is a naive extension of the learning with rejection framework for handling adversarial examples. Learning with protection is a practical application of learning with rejection under adversarial attacks. This algorithm transforms the original multi-class classification problem into a binary classification for a specific class, and we reject suspicious samples to protect a specific label. We demonstrate the effectiveness of the proposed method in experiments.

1. INTRODUCTION:

Recent machine learning models have achieved great success in applications such as self-driving cars (Bojarski et al., 2016) and medical diagnosis (Litjens et al., 2017) . In a classification problem, we sometimes face the situation where the misclassification of a specific label can result in serious accidents. This problem arises in various situations, and below we provide examples. Example 1 (Object Detection for Self-Driving Cars). For self-driving cars, object detection is crucial. For protecting the life of pedestrians, self-driving cars need to be able to detect human or suspend decision making if it detects an object that might be a human. Example 2 (Medical Diagnosis). Cancer is one of the leading causes of death globally, and machine learning algorithms for detecting cancer from images have been gaining attention. This is done by detecting abnormalities in an image that might indicate cancer.

As shown above, we want the labeling of data to be conservative for certain situations; for example, in autonomous driving, it is better to misclassify a non-human object as a human than to risk misclassifying a human. In this paper, we introduce two novel frameworks called learning with rejection under adversarial attacks and learning with protection to protect a specific class from critical misclassification. Our algorithm is based on the techniques of learning with rejection and adversarial examples. Learning with rejection under adversarial attacks is a naive extension of learning with rejection to hold a decision when we face a suspicious sample that is vulnerable to adversarial attack. Learning with protection is an application of learning with rejection under adversarial attack. The purpose is to protect a specific class from misclassification. Both frameworks are based on the existing work of learning with rejection and adversarial examples.

Learning with rejection is a classification scenario where the learner is given the option to reject an instance instead of predicting its label. The purpose of this framework is to prevent critical misclassification, where rejecting labeling a datum incurs a lower cost than misclassification. In this field, considerable theoretical and empirical analysis has been conducted (Bartlett & Wegkamp, 2008; Cortes et al., 2016a; b; Grandvalet et al., 2009; Herbei & Wegkamp, 2006; Yuan & Wegkamp, 2010) . Learning with rejection is one ultimate solution against data with high uncertainty. However, the existing frameworks have not considered adversarial examples, which refers to a well-designed attack to mislead people to misclassification.

In the industrial application of machine learning, it has been observed that many machine learning models are vulnerable to adversarial attacks. Threat model of adversarial attacks specifies the capabilities of the adversary and classifies adversary attacks to two types of attacks, white-box attack and black-box attack. Adversarial inputs that result in machine learning models returning incorrect outputs are called adversarial examples. Several previous studies have artificially generated adversarial examples by adding small perturbations that are imperceptible to humans (Goodfellow et al., 2015; Gu & Rigazio, 2014; Huang et al., 2015; Carlini & Wagner, 2017) . Methods for protecting against these adversarial examples are also being proposed. Among them, adversarial training is one of the most effective defenses (Szegedy et al., 2014; Goodfellow et al., 2015; Shaham et al., 2015; Carlini & Wagner, 2016; Papernot et al., 2016; Xu et al., 2017; Madry et al., 2018; Buckman et al., 2018; Kannan et al., 2018; Pang et al., 2018; Wong & Kolter, 2018; Tramèr et al., 2018) .

However, most of the defense method fails to avoid misclassification under adversarial attacks. Carlini & Wagner (2017) defeated representative methods for detection of adversarial examples. Athalye et al. (2018) reports that several defense algorithms for white-box setting fail when the attacker uses a carefully designed gradient-based method. Besides, Shafahi et al. (2019) ; Gilmer et al. (2019) shows that adversarial examples are inevitable in some cases. However, we sometimes face situations in which misclassification can result in fatal accidents. For that purpose, we propose learning with rejection option under adversarial attack. The main idea of our method is to hold a decision when we get suspicious samples. As an application of learning with a rejection option, we propose a method for preventing the misclassification of data for a specific label under adversarial examples. This method considers situations where we must protect our model from adversarial examples by any means. In addition, It should be noted that our method can be combined with other existing methods such as adversarial training. In the following section, we describe the problem setting. In Sections 3 and 4, we propose and describe our algorithm, and in Section 5, we show the experimental results.

2. PROBLEM SETTING:

We use the standard settings for binary classification problems, i.e., training and test data points are i.i.d. and sampled from some unknown distribution D over X × {-1, +1}, where X denotes the input space. The goal is to learn a classifier h : X → {-1, +1} that assigns a label ŷ(x) to a datum x as ŷ(x) = sign(h(x)). Let L be a loss function L : R × {-1, +1} → R. The optimal classifier h * is given by h * = arg min h∈F E[L(h(X), Y )], where F is a set of measurable functions.

When we train a classifier with training samples, we can naively replace the expectation with the corresponding sample averages. For a hypothesis set H, let us define an estimator of the optimal classifier h * as ĥ = arg min h∈H Ê[L(h(X), Y )], where Ê denotes the averaging operator over training data. Although an estimator ĥ converges to h * in many cases with infinite samples, an estimator ĥ might return result that differ greatly from h * in a case with finite samples. In addition, when h(x) is not smooth at around x, it might become difficult to estimate the function. These can cause serious problems in real-world applications, such as traffic accidents arising from misclassification by the algorithms used in self-driving cars. To make our algorithm's inference more robust, we allow a learner to reject such suspicious samples. Let R denote rejection. For any given instance x ∈ X , the learner has the option of abstaining from assigning a label or rejecting that instance and returning the symbol R , or assigning the label ŷ ∈ {-1, +1}. If the learner rejects an instance, then they incur some loss c(x) ∈ R; if it does not reject but assigns an incorrect label, then it incurs a cost of one; otherwise, it suffers no loss. Thus, the learner outputs a rejection function r : X → R, that determines the points x ∈ X to be rejected according to r(x) ≤ 0. Let us denote a loss function with rejection option L R : R × {-1, +1} → R. For example, a loss function can be defined as follows: There are two frameworks used in the learning with rejection model: the confidence-based and separation-based approaches Cortes et al. (2016a) . The separation-based approach is first formulated by Cortes et al. (2016a) . This formulation is a generalization of confidence-based approach (Bartlett & Wegkamp, 2008) . In the separation-based approach, we train a classifier and a rejection function simultaneously. In confidence-based approach, given the conditional probability p(y = +1|x) and a cost function c(x), we only train a classifier and obtain a rejection function automatically after training a classifier. Although the confidence-based approach is a special case of the separation-based approach, the method of confidence-based approach is easy to implement. In this paper, we follow the confidence-based approach and correspond the cost function to vulnerability for adversarial examples.

L R (h(x), y) = 1 yh(x)≤0 1 r(x)≥0 + c(x)1 r(x)≤0 .

Our goal is to obtain the optimal classifier h * , which minimizes the classification risk with rejection option, i.e., h

* = arg min h∈H E[L R (h(X), Y )].

3. LEARNING WITH REJECTION UNDER ADVERSARIAL ATTACKS:

In this section, we describe a method to hold a decision against suspicious samples.

3.1. STRATEGY FOR REJECTING SUSPICIOUS SAMPLES:

To reject suspicious samples, we define which samples are suspicious. We introduce a concept of a p-norm suspicious sample. Let • p be p -norm. We refer a sample x as suspicious based on the value of p -norm if there exits a sample x such that xx p ≤ ε and ĥ(x ) returns a different class from ĥ(x). If a sample is suspicious, we refrain from assigning a class label.

Suppose that we have a estimator f (x) of the class conditional probability η(x) = p(y = +1|x) and a classifier ĥ(x) = sign f (x) -1 2 . Let B ∞ x ( ) denote the p ball centered at x ∈ X with radius , i.e., B p

x ( ) = {x ∈ X : xx p ≤ }. We calculate z * (x) ∈ B p x ( ) as follows:

z * (x) = arg max x ∈B p x ( )

L( ĥ(x ), x).

By defining z * (x) as in the above equation, we can easily calculate it with the techniques used in adversarial examples. If ĥ(z * (x)) ĥ(x) ≤ 0 for a sample x, we regard the sample as suspicious one.

We illustrate the concept in Figure 1 .

3.2. RELATIONSHIP WITH LEARNING WITH REJECTION:

To interpret the above strategy, we consider the relationship between the strategy and learning with rejection. First, we consider the ideal situation, where we know the true value of the conditional class probability η(x) = p(y = +1|x). In confidence-based approach, we can determine the rejection function r(x) corresponding to the rejection cost as shown by Cortes et al. (2016a) . It is known that the classifier h * defined for any x ∈ X by h * (x) = η(x) -1 2 is optimal. For any x ∈ X , the misclassification cost for h * is E 1 yh(x)≤0 |x = min{η(x), 1 -η(x)}. The optimal rejection function r * (x) should therefore be defined such that r * (x) ≤ 0 if and only if

min{η(x), 1 -η(x)} ≥ c(x) ⇔ 1 -max{η(x), 1 -η(x)} ≥ c(x) ⇔ |h(x)| ≤ 1 2 -c(x).

Therefore, the optimal rejection function r * (x) is given as r

* (x) = |η(x) -1 2 | -1 2 -c(x)

. Thus, in confidence-based approach, we can determine the optimal rejection function r * (x) if we know the conditional probability η(x) and the rejection cost c(x).

To reject suspicious samples, we relate the rejection cost c(x) with uncertainty of data x. One naive idea is to set c(x) as a monotonically decreasing function representing the smoothness of η(x). When |η(x) -η(x )| is zero for x, x ∈ R d , c(x) takes its largest value; when max |f (x) -f (x )| is large, c(x) is small. This means that, if a sample x is suspicious, then the cost of rejection is low, and we can reject the sample easily; conversely, if a sample x is not suspicious, then the cost of rejection is high. When the cost is high, we hesitate to reject the sample. We assumed this condition could be met when designing our algorithm. As we show later, we can easily find such c(x) and reject suspicious samples without expending much energy in determining the value of c(x). In our strategy, we reject a sample x such that

η(z * (x)) - 1 2 η(x) - 1 2 ≤ 0.

Therefore, r * (x) ≤ 0 if and only if

η(z * (x)) ≤ 1 2 and η(x) ≥ 1 2 or η(z * (x)) ≥ 1 2 and η(x) ≤ 1 2 ⇔ η(x) -η(x) + η(z * (x)) ≤ 1 2 and η(x) ≥ 1 2 or η(x) -η(x) + η(z * (x)) ≥ 1 2 and η(x) ≤ 1 2 ⇔ 0 ≤ η(x) - 1 2 ≤ η(x) -η(z * (x)) or η(x) -η(z * (x)) ≤ η(x) - 1 2 ≤ 0 (1) Let us assume that η(x) -η(z * (x)) ≥ 0 when η(x) -1 2 ≥ 0 and η(x) -η(z * (x)) ≤ 0 when η(x) -1 2 ≤ 0. Then, (1) insists that r * (x) ≤ 0 if and only if |h(x)| ≤ |η(x) -η(z * (x))|.

Therefore, the following relationship holds for a sample x such that |η(x) -η(z

* (x))| ≤ 1 2 : c(x) = 1 2 -|η(x) -η(z * (x))|.

Thus, in our strategy, the cost function is a decreasing function of the smoothness of η(x).

Next, we show a relationship between the cost function and the degree of perturbation ε. Let us assume η(x) is Lipschitz continuous, with p -norm, and

|η(x) -η(x )| ≤ λ x -x p = λε,

where x ∈ X , x ∈ B p x ( ), and λ > 0 is a positive constant. From Lipschitz continuity, c(x) ≥ 1 2 -λε holds. This means that the cost of rejection will be small when ε is a large value. Therefore, a learner can reject a sample with low cost when the learner is afraid of the misclassification and set the possible perturbation of x with p -norm large.

3.3. ALGORITHM:

Based on the above idea, we develop an algorithm for rejecting suspicious samples. To discuss learning the rejection in the framework of confidence-based approach, we construct an estimator f of η(x) = p(y = +1|x) and a classifier ĥ(x) = f (x) -1 2 . Then, we define pseudo loss function L for two samples x ∈ X and x ∈ B p

x ( ) as a binary loss function for a pair of (x , ŷ(x), where ŷ(x) = sign( ĥ(x)). For example, if we assume logistic loss, it can be defined as

L( f (x ), ŷ(x)) = -1 ŷ(x)=-1 log f (x) -1 ŷ(x)=+1 log 1 -f (x) .

Using the pseudo loss, we calculate z * (x) for a sample x as follows:

z * (x) = arg max x ∈B p x ( )

L( ĥ(x ), ŷ(x)).

Because this optimization is hard to calculate, we apply techniques of adversarial examples as a heuristics. Then, we reject a sample x if and only if ĥ(z * (x)) ĥ(x) ≤ 0. One of the simplest methods for generating adversarial examples is the fast gradient sign method (FGSM) Goodfellow et al. (2015) , which is a fast single-step attack that maximizes the loss function in the linear approximation. The perturbation under the FGSM is calculated as follows:

δ = ε • sign(∇ x L(h(x), y)).

The projected gradient descent method (PGD) Madry et al. ( 2018) is an iterative variant of the FGSM. The perturbation under the PGD at time step s + 1 is calculated as follows:

δ (s+1) = P ε (δ (s) + α • sign(∇ x L(h(x + δ (s) ), t)))

, where α denotes a single step and P ε (•) denotes the projection onto the p -ball with radius ε.

4. LEARNING WITH PROTECTION:

In this section, we propose a practical and easily implemented algorithm as an application of learning with rejection under adversarial attack. In many cases, we want to avoid misclassification for a specific class. In self-driving cars, we must be able to detect human beings to avoid serious accidents.

In medical diagnosis, we should not miss a dangerous disease. We thus propose an algorithm to protect a specific class from misclassification in a multi-class classification problem.

We extend the previous problem setting of binary classification to that of multi-class classification. Let X and Y be the feature and label spaces, respectively, and suppose that there is an unknown distribution D over X ×Y. We assume that the label space has K ≥ 2 labels, i.e., Y = {1, 2, 3, ..., K}. We define the class for which we want to avoid misclassification as the defense target class t ∈ Y. For example, if we want to avoid misclassification of 1, we designate the defense target class as t = 1. We call the algorithm for protecting a defense target class based on the following idea learning with protection.

4.1. PROTECTING A DEFENSE TARGET CLASS:

Let F be the set of measurable functions from X × Y to R. For a function f ∈ F, we define a classifier h f : X × Y → R as follows:

y f (x) = arg max k∈[K] f (x, y = k).

Let L(f ; x, y) be the loss function of a function f ∈ F for data (x, y). The goal of the learning problem is to find f ∈ F such that the population risk R(f

) := E (x,y)∈D [L(f ; x, y)] is minimized.

This formulation is the standard setting of multi-class classification problem.

From the original formulation of the multi-class classification problem, we construct a binary classification problem to protect the defense target class t from misclassification. Let us define a function y b f : X → {-1, +1} for a function f to transform the prediction of multi-class classification to that of binary classification as

y b f (x) = +1 y f (x) = t -1 y f (x) = t .

For a pair (x, y b f (x)), we calculate a new feature z * (x) as follows:

EQUATION

where Lf (f ; x , y b f (x)) is a binary loss function for a pair (x, y b f (x)). This loss function corresponds to the pseudo loss function introduced in the previous section. For example, we can define 0-1 loss as follows:

L(f ; x , y b

Algorithm 1 Learning with Protection:

Input: Trained classier ĥ, defense target class t ∈ [K], and test dataset {x i } n i=1 . Construct a pseudo loss function (3) for test dataset {x i } n i=1 . Compute z * (x i ) for a sample x i by (2) by method of adversarial example. Reject a sample x i if ĥ(x i ) = t and ĥ(z * (x i )) = t

4.2. ALGORITHM WITH CROSS-ENTROPY LOSS:

Various algorithms can be used to implement the above framework. In this section, we consider a cross-entropy loss function and define an algorithm for protecting the defense target class.

Let H be a hypothesis set. First, we train a model f ∈ H with the cross-entropy loss by any of the suitable methods for classification problems such as adversarial training. After training g, we obtain f , the function that minimizes the cross-entropy loss. The trained model is an estimator of the risk minimizer f * on population. Second, we use the logistic loss for L, i.e.,

L(f ; x , y b f (x)) = -1 y b f (x)=-1 log exp (f (x , y = t)) k∈[K] exp (f (x , y = k)) -1 y b f (x)=+1 log 1 - exp (f (x , y = t)) k∈[K] exp (f (x , y = k))

.

(3)

The intuition of this loss function is as follows. Let

g * (x , t) = exp(f * (x ,y=t)) k∈[K] exp(f * (x ,y=k))

. Under a cross entropy loss function, we can interpret g * (x , k) as p(y = k|x ) with g * . Therefore, we can apply the log loss function function with g * (x , t) = p(y = t|z) and p(y = t|z) = 1 -p(y = t|z) = 1 -g * (x , t). As mentioned above, to find a feature z * from (2), we can apply methods of adversarial examples as a heuristics. We show the pseudo algorithm in Algorithm 1.

5. EXPERIMENTS:

To demonstrate the effectiveness of proposed method, we conducted experiments using benchmark data. For the benchmark data, we use the CIFAR-10 1) (Krizhevsky, 2009) and ImageNet-100 2) (Kang et al., 2019) dataset. The CIFAR-10 dataset consists of 32 × 32 color images in 10 classes, with 6, 000 images per class. There are 50, 000 training images and 10, 000 test images. The ImageNet-100 dataset consists of the 100-class subset of ImageNet-1K (Deng et al., 2009; Russakovsky et al., 2015) containing every 10th class by WordNet ID order. As we explained above, learning with protection rejects suspicious samples after we train a model f to protect defense target class. For CIFAR-10, we set airplane class as defense target class. For ImageNet-A, we set the first 10 classes in the order of WordNet ID as defense target class. Our method allow various training method to estimate f * . When we train the model f , we used both standard training method and adversarial training (Madry et al., 2018) . After trainning a model, we adopt 30-step ∞ bounded PGD attack to find the feature z * . For each experiment, we output the accuracy, precision, recall, rejection rate, and true rejection. Let TP, TN, FP, and FN be the numbers of true positive, true negative, false positive, and false negative of a classification result. Then, accuracy, precision, and recall are defined as accuracy = TP+TN TP+TN+FP+FN , precision = TP TP+FP , and recall = TP TP+FN . Let true accept (TA) be an outcome where the rejection function correctly accepts data such that a sample x does not belong to the defense target class, i.e., h(x) = t. Let true reject (FR) be an outcome where the rejection function correctly rejects data such that a sample x belongs to the defense target class, i.e., h(x) = t. FA and FR can be defined similarly (Ni et al., 2019) . Then, rejection rate and precision of rejection are defined as rejection rate = TR+FR TR+TA+FR+FA and precision of rejection = TR TR+FR .

1) See https://www.cs.toronto.edu/~kriz/cifar.html.

2) See http://image-net.org/index. Adversarial training: We use ∞ bounded projected gradient descent (PGD) method to generate training samples. We select a attack target class for each image uniformly at random from the set of incorrect classes. For distortion size ε, we set as ε = 8/255 and apply random scaling Uniform(0, ε) to improve performance against smaller distortions. We use 10 optimization steps and step size ε/ √ steps as described in (Kang et al., 2019) . We update the model by stochastic gradient descent (SGD) method using only the adversarial images (no clean images).

5.1. PROTECTION AGAINST ARTIFICIAL ADVERSARIAL EXAMPLES:

We evaluate our learning with protection method on the CIFAR-10 and ImageNet-100 validation sets against artificial adversarial examples with the defense target class.

protection against expected attack: We distort the inputs by 50-step ∞ bounded PGD attack and apply learning with protection method. The single step size of PGD is calculated by ε/ √ steps. The results are shown in Table 1 and Appendix protection against unexpected attack: Although in our learning with protection method, a specific adversarial attack method is used to find the feature z * (x i ) for a sample x i (we adopt ∞ bounded PGD in this experiment), the input might be distorted by different attack methods. In order to evaluate the performance against such unexpected adversaries, we conducted experiments against 50-step l 2 bounded PGD attack. The results are shown in Table 2 and Appendix We can see from the table that our method achieves 100% in the recall, i.e., we can reject all samples such that y = t and h(x) = t. As shown in Appendix B.1, models with adversarial training outperforms models without adversarial training. It is because our method only reduce the false negative, and it is difficult to improve the performance of a model when the number of the true positive is small. When we use adversarial training, we can increase the recall with the small number of rejection. For example, in Table 2 , when attack is 2 bounded PGD (ε = 80) for CIFAR-10, recall increases about 20% by only reject 9.14% of all samples. On the other hand, On the other hand, when we reject too many samples, the precision of rejection dramatically drops. Therefore, we need to control the perturbation ε not to reject samples unnecessarily.

5.2. PROTECTION AGAINST NATURAL ADVERSARIAL EXAMPLES:

Natural adversarial examples which are introduced in (Hendrycks et al., 2019) are defined as unmodified and naturally occurring examples that cause classifier accuracy to degrade drastically. In order to evaluate the proposed method for such real hard samples, we conduct experiments on ImageNet-A 3)

3) See https://github.com/hendrycks/natural-adv-examples. We first trained the model on the training set of ImageNet-1K dataset and evaluate the performance on ImageNet-A with our learning with protection method at test time. Due to the difference in the number of classes, only 200 of its 1,000 logits are used in test time. We set the first 20 classes in the order of WordNet ID as the defense target class. The results are shown in Table 3 .

As shown in Table 3 and Appendix B.2, the proposed method works well for natural adversarial samples. For example, when the distortion size is 2/255, the recall increases about 20% by rejecting only 30% of all samples. On the other hand, the rejection rate of the models without adversarial training is high even when the size of the perturbation ε is small.

A TRAINING HYPERPARAMETERS:

For CIFAR-10, we trained on a single NVIDIA V100 GPU for 200 epochs with batch size 32, initial learning rate 0.1, momentum 0.9, and weight decay 10 -4 . We decayed the learning rate at epochs 100 and 150. For ImageNet-100, we trained on machines with 8 NVIDIA V100 GPUs using standard data augmentation used in (He et al., 2015) . We ran synchronized SGD (Goyal et al., 2017) for 90 epochs with batch size 32 × 8 and a learning rate schedule with 5 "warm-up" epochs and a decay at epochs 30, 60, and 80 by a factor of 10. 

B FULL EXPERIMENTAL RESULTS:



B.1 PROTECTION AGAINST ARTIFICIAL ADVERSARIAL EXAMPLES:



:

f (x)) = 1 y b f (x)=-1 1 y b f (x )=+1 + 1 y b f (x)=+1 1 y b f (x )=-1 .Then, for this binary classification problem derived from multi-class classification problem, we apply an algorithm for rejection. If y f (z * (x)) = t for a sample x such that y f (x) = t, then we reject the sample because the sample x predicted as a class that is not t might be a class t in neighborhood of x with p -norm. We call algorithms based on the abeve idea learning with protection.

CONCLUSIONIn this paper, we proposed learning with rejection under adversarial attacks for avoiding the misclassification. The proposed method suspend decision making when a sample is p -norm suspicious. By extending the strategy, we proposed an easily implementable algorithm called learning with protection. This algorithm prevents a defense target class from misclassification. We showed the performance of learning with protection using benchmark datasets. From the results, we confirmed that our method successfully caught samples that would have misclassified if we did not reject it.

Figures and Tables:

Figure 1: suspicious samples: suspicious samples have a risk of misclassification against the perturbations in p ball.

B.1.

B.1.

Calculation of Adversarial Examples: Here, we describe the two simple first-order methods for calculating adversarial examples used in this paper. Although various types of adversarial examples have been proposed, hereinafter adversarial examples refer to perturbation-based methods.

Table content: <table/>

Results of adversarial trained model against ∞ bounded PGD attack. Results of standard trained model are shown in Appendix B.1. We show the accuracy (Acc), precision (Pre), recall (Rec), rejection rate (Rej), and precision of rejection (PR). For CIFAR-10, we use the ResNet-56(He et al., 2015) architecture. For ImageNet-100, we use the ResNet-50(He et al., 2015) architecture with 224 × 224 resolution as implemented in torchvision. We describe training hyperparameters in Appendix A.

Table content: <table><tr><td/><td/><td colspan="2">CIFAR-10</td><td/><td/><td/><td/><td/><td colspan="2">ImageNet-100</td><td/></tr><tr><td colspan="2">Attack Protection</td><td>Acc</td><td>Pre</td><td>Metric Rec</td><td>Rej</td><td>PR</td><td colspan="2">Attack Protection</td><td>Acc</td><td>Pre</td><td>Metric Rec</td><td>Rej</td><td>PR</td></tr><tr><td/><td>without</td><td colspan="4">98.15 94.05 87.00 0.00</td><td>0.00</td><td/><td>without</td><td colspan="4">97.72 88.14 89.20 0.00</td><td>0.00</td></tr><tr><td>clean</td><td>4/255</td><td colspan="5">99.18 94.05 97.42 04.66 22.96</td><td>clean</td><td>4/255</td><td colspan="4">98.49 88.14 98.67 12.30 7.74</td></tr><tr><td/><td>16/255</td><td colspan="5">97.40 94.05 100.0 78.86 1.65</td><td/><td>16/255</td><td colspan="4">91.50 88.14 100.0 85.20 1.26</td></tr><tr><td/><td>without</td><td colspan="4">94.80 80.61 63.20 0.00</td><td>0.00</td><td/><td>without</td><td colspan="4">94.16 71.76 68.60 0.00</td><td>0.00</td></tr><tr><td>4/255</td><td>4/255</td><td colspan="5">98.13 80.61 97.53 10.07 34.96</td><td>4/255</td><td>4/255</td><td colspan="4">96.60 71.76 98.00 16.37 18.18</td></tr><tr><td/><td>16/255</td><td colspan="5">91.03 80.61 100.0 83.06 4.43</td><td/><td>16/255</td><td colspan="4">79.48 71.76 100.0 86.15 3.62</td></tr><tr><td/><td>without</td><td colspan="3">84.94 12.01 8.00</td><td>0.00</td><td>0.00</td><td/><td>without</td><td colspan="4">88.34 40.10 33.60 0.00</td><td>0.00</td></tr><tr><td>16/255</td><td>4/255</td><td colspan="3">85.67 12.01 8.93</td><td colspan="2">2.15 48.37</td><td>16/255</td><td>4/255</td><td colspan="4">88.80 40.10 35.59 0.91 60.87</td></tr><tr><td/><td>16/255</td><td colspan="5">46.92 12.01 100.0 88.96 10.34</td><td/><td>16/255</td><td colspan="4">61.66 40.10 96.00 85.85 7.51</td></tr><tr><td colspan="3">Neural network model:</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Results of adversarial trained model against 2 bounded PGD attack. Results of standard trained model are shown in Appendix B.1. We show the accuracy (Acc), precision (Pre), recall (Rec), rejection rate (Rej), and precision of rejection (PR).

Table content: <table><tr><td/><td/><td colspan="2">CIFAR-10</td><td/><td/><td/><td/><td/><td colspan="2">ImageNet-100</td><td/></tr><tr><td colspan="2">Attack Protection</td><td>Acc</td><td>Pre</td><td>Metric Rec</td><td>Rej</td><td>PR</td><td colspan="2">Attack Protection</td><td>Acc</td><td>Pre</td><td>Metric Rec</td><td>Rej</td><td>PR</td></tr><tr><td/><td>without</td><td colspan="4">98.15 94.05 87.00 0.00</td><td>0.00</td><td/><td>without</td><td colspan="4">97.72 88.14 89.20 0.00</td><td>0.00</td></tr><tr><td>clean</td><td>4/255</td><td colspan="5">99.18 94.05 97.42 04.66 22.96</td><td>clean</td><td>4/255</td><td colspan="4">98.49 88.14 98.67 12.30 7.74</td></tr><tr><td/><td>16/255</td><td colspan="5">97.40 94.05 100.0 78.86 1.65</td><td/><td>16/255</td><td colspan="4">91.50 88.14 100.0 85.20 1.26</td></tr><tr><td/><td>without</td><td colspan="4">95.56 84.49 68.10 0.00</td><td>0.00</td><td/><td>without</td><td colspan="4">93.18 66.46 64.20 0.00</td><td>0.00</td></tr><tr><td>80</td><td>4/255</td><td colspan="5">98.45 84.49 97.70 9.14 33.15</td><td>600</td><td>4/255</td><td colspan="4">95.60 66.46 93.86 16.67 18.81</td></tr><tr><td/><td>16/255</td><td colspan="5">93.02 84.49 100.0 82.10 3.89</td><td/><td>16/255</td><td colspan="4">75.42 66.46 100.0 86.13 4.12</td></tr><tr><td/><td>without</td><td colspan="4">85.10 16.16 11.70 0.00</td><td>0.00</td><td/><td>without</td><td colspan="4">87.78 37.24 32.40 0.00</td><td>0.00</td></tr><tr><td>320</td><td>4/255</td><td colspan="5">86.27 16.16 14.10 3.84 44.27</td><td>2400</td><td>4/255</td><td colspan="4">88.24 37.24 34.25 0.65 81.82</td></tr><tr><td/><td>16/255</td><td colspan="5">55.24 16.16 100.0 86.44 10.22</td><td/><td>16/255</td><td colspan="4">56.47 37.24 96.43 86.49 7.62</td></tr></table>

Results of ImageNet-A. Full experimental results are shown in Appendix B.2. We show the accuracy (Acc), precision (Pre), recall (Rec), rejection rate (Rej), and precision of rejection (PR). /255 80.23 39.98 39.15 29.72 31.18 42.88 42.88 100.0 89.99 18.43 ε = 4/255 78.46 39.98 72.40 60.27 24.23 42.88 42.88 100.0 89.99 18.43 ε = 8/255 52.41 39.98 99.42 85.60 19.05 42.88 42.88 100.0 89.99 18.43 dataset. ImageNet-A contains 7,500 natural adversarial example images in 200 classes that are a subset of ImageNet-1K's 1,000 classes. We use the ResNet-50 architecture with 224 × 224 resolution.

Table content: <table><tr><td/><td/><td colspan="3">Adversarial Trained</td><td/><td/><td colspan="3">Standard Trained</td><td/></tr><tr><td>Protection</td><td>Acc</td><td>Pre</td><td>Metric Rec</td><td>Rej</td><td>PR</td><td>Acc</td><td>Pre</td><td>Metric Rec</td><td>Rej</td><td>PR</td></tr><tr><td>without</td><td colspan="10">76.84 39.98 21.78 00.00 00.00 77.69 42.88 20.56 00.00 00.00</td></tr><tr><td>ε = 2</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Results of adversarial trained ResNet-56 against ∞ bounded PGD attack for CIFAR-10.

Table content: <table><tr><td>Attack</td><td>Protection</td><td colspan="5">Metric Acc [%] Pre [%] Rec [%] Rej [%] PR[%]</td></tr><tr><td/><td>without</td><td>98.15</td><td>94.05</td><td>87.00</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>98.71</td><td>94.05</td><td>92.36</td><td>01.58</td><td>36.71</td></tr><tr><td>clean</td><td>ε = 4/255 ε = 8/255</td><td>99.18 99.26</td><td>94.05 94.05</td><td>97.42 99.66</td><td>04.66 22.08</td><td>22.96 5.75</td></tr><tr><td/><td>ε = 16/255</td><td>97.40</td><td>94.05</td><td>100.0</td><td>78.86</td><td>1.65</td></tr><tr><td/><td>ε = 32/255</td><td>94.08</td><td>94.05</td><td>100.0</td><td>90.71</td><td>1.43</td></tr><tr><td/><td>without</td><td>97.16</td><td>91.15</td><td>79.30</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>98.73</td><td>91.15</td><td>94.52</td><td>3.45</td><td>46.67</td></tr><tr><td>ε = 2/255</td><td>ε = 4/255 ε = 8/255</td><td>98.98 98.93</td><td>91.15 91.15</td><td>97.90 99.75</td><td>7.42 25.98</td><td>25.61 7.89</td></tr><tr><td/><td>ε = 16/255</td><td>96.00</td><td>91.15</td><td>100.0</td><td>80.73</td><td>2.56</td></tr><tr><td/><td>ε = 32/255</td><td>91.17</td><td>91.15</td><td>100.0</td><td>91.28</td><td>2.27</td></tr><tr><td/><td>without</td><td>94.80</td><td>80.61</td><td>63.20</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>96.71</td><td>80.61</td><td>79.50</td><td>4.14</td><td>49.52</td></tr><tr><td>ε = 4/255</td><td>ε = 4/255 ε = 8/255</td><td>98.13 97.83</td><td>80.61 80.61</td><td>97.53 100.0</td><td>10.07 29.95</td><td>34.96 12.29</td></tr><tr><td/><td>ε = 16/255</td><td>91.03</td><td>80.61</td><td>100.0</td><td>83.06</td><td>4.43</td></tr><tr><td/><td>ε = 32/255</td><td>80.64</td><td>80.61</td><td>100.0</td><td>92.15</td><td>3.99</td></tr><tr><td/><td>without</td><td>89.22</td><td>44.90</td><td>34.30</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>90.63</td><td>44.90</td><td>41.13</td><td>2.70</td><td>61.48</td></tr><tr><td>ε = 8/255</td><td>ε = 4/255 ε = 8/255</td><td>92.48 93.59</td><td>44.90 44.90</td><td>56.05 100.0</td><td>8.30 34.34</td><td>46.75 19.13</td></tr><tr><td/><td>ε = 16/255</td><td>70.80</td><td>44.90</td><td>100.0</td><td>85.58</td><td>7.68</td></tr><tr><td/><td>ε = 32/255</td><td>45.04</td><td>44.90</td><td>100.0</td><td>92.34</td><td>7.12</td></tr><tr><td/><td>without</td><td>84.94</td><td>12.01</td><td>8.00</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>85.13</td><td>12.01</td><td>8.21</td><td>0.37</td><td>67.57</td></tr><tr><td>ε = 16/255</td><td>ε = 4/255 ε = 8/255</td><td>85.67 87.58</td><td>12.01 12.01</td><td>8.93 20.00</td><td>2.15 27.05</td><td>48.37 22.18</td></tr><tr><td/><td>ε = 16/255</td><td>46.92</td><td>12.01</td><td>100.0</td><td>88.96</td><td>10.34</td></tr><tr><td/><td>ε = 32/255</td><td>12.01</td><td>12.01</td><td>100.0</td><td>93.34</td><td>9.86</td></tr><tr><td/><td>without</td><td>86.11</td><td>10.06</td><td>4.90</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>86.11</td><td>10.06</td><td>4.90</td><td>0.00</td><td>0.00</td></tr><tr><td>ε = 32/255</td><td>ε = 4/255 ε = 8/255</td><td>86.10 85.56</td><td>10.06 10.06</td><td>4.90 4.99</td><td>0.16 5.08</td><td>6.25 3.54</td></tr><tr><td/><td>ε = 16/255</td><td>72.16</td><td>10.06</td><td>24.87</td><td>78.95</td><td>10.17</td></tr><tr><td/><td>ε = 32/255</td><td>10.04</td><td>10.06</td><td>98.00</td><td>95.12</td><td>9.99</td></tr></table>

Results of standard trained ResNet-56 against ∞ bounded PGD attack for CIFAR-10.

Table content: <table><tr><td>Attack</td><td>Protection</td><td colspan="5">Metric Acc [%] Pre [%] Rec [%] Rej [%] PR[%]</td></tr><tr><td/><td>without</td><td>98.78</td><td>94.34</td><td>93.40</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>97.34</td><td>94.34</td><td>100.0</td><td>78.96</td><td>0.84</td></tr><tr><td>clean</td><td>ε = 4/255 ε = 8/255</td><td>94.42 94.36</td><td>94.34 94.34</td><td>100.0 100.0</td><td>89.96 90.07</td><td>0.73 0.73</td></tr><tr><td/><td>ε = 16/255</td><td>94.64</td><td>94.34</td><td>100.0</td><td>89.56</td><td>0.74</td></tr><tr><td/><td>ε = 32/255</td><td>95.17</td><td>94.34</td><td>99.89</td><td>88.20</td><td>0.74</td></tr><tr><td/><td>without</td><td>82.41</td><td>5.61</td><td>4.80</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>62.60</td><td>5.61</td><td>64.86</td><td>77.73</td><td>11.91</td></tr><tr><td>ε = 2/255</td><td>ε = 4/255 ε = 8/255</td><td>32.92 10.36</td><td>5.61 5.61</td><td>96.00 88.89</td><td>87.94 90.93</td><td>10.80 10.40</td></tr><tr><td/><td>ε = 16/255</td><td>13.39</td><td>5.61</td><td>54.55</td><td>90.22</td><td>10.11</td></tr><tr><td/><td>ε = 32/255</td><td>21.23</td><td>5.61</td><td>47.06</td><td>89.07</td><td>10.08</td></tr><tr><td/><td>without</td><td>83.68</td><td>4.86</td><td>3.40</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>79.72</td><td>4.86</td><td>5.30</td><td>37.19</td><td>9.63</td></tr><tr><td>ε = 4/255</td><td>ε = 4/255 ε = 8/255</td><td>78.75 64.85</td><td>4.86 4.86</td><td>8.15 40.96</td><td>50.64 79.66</td><td>11.51 11.51</td></tr><tr><td/><td>ε = 16/255</td><td>7.11</td><td>4.86</td><td>100.0</td><td>92.83</td><td>10.41</td></tr><tr><td/><td>ε = 32/255</td><td>4.86</td><td>4.86</td><td>100.0</td><td>93.00</td><td>10.39</td></tr><tr><td/><td>without</td><td>86.39</td><td>5.65</td><td>2.30</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>85.66</td><td>5.65</td><td>2.35</td><td>6.70</td><td>3.43</td></tr><tr><td>ε = 8/255</td><td>ε = 4/255 ε = 8/255</td><td>85.44 85.15</td><td>5.65 5.65</td><td>2.42 2.96</td><td>9.81 23.36</td><td>4.89 9.55</td></tr><tr><td/><td>ε = 16/255</td><td>71.01</td><td>5.65</td><td>12.99</td><td>81.44</td><td>10.11</td></tr><tr><td/><td>ε = 32/255</td><td>53.43</td><td>5.65</td><td>13.61</td><td>88.62</td><td>9.38</td></tr><tr><td/><td>without</td><td>88.10</td><td>7.59</td><td>1.70</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>87.97</td><td>7.59</td><td>1.71</td><td>1.32</td><td>2.27</td></tr><tr><td>ε = 16/255</td><td>ε = 4/255 ε = 8/255</td><td>87.89 87.84</td><td>7.59 7.59</td><td>1.71 1.72</td><td>2.25 3.09</td><td>2.67 3.88</td></tr><tr><td/><td>ε = 16/255</td><td>86.69</td><td>7.59</td><td>1.92</td><td>19.23</td><td>5.98</td></tr><tr><td/><td>ε = 32/255</td><td>70.54</td><td>7.59</td><td>8.29</td><td>86.59</td><td>9.18</td></tr><tr><td/><td>without</td><td>89.10</td><td>5.88</td><td>0.60</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>89.10</td><td>5.88</td><td>0.60</td><td>0.03</td><td>0.00</td></tr><tr><td>ε = 32/255</td><td>ε = 4/255 ε = 8/255</td><td>89.04 89.02</td><td>5.88 5.88</td><td>0.60 0.60</td><td>0.95 1.11</td><td>4.21 3.60</td></tr><tr><td/><td>ε = 16/255</td><td>88.94</td><td>5.88</td><td>0.60</td><td>1.97</td><td>3.05</td></tr><tr><td/><td>ε = 32/255</td><td>85.72</td><td>5.88</td><td>0.63</td><td>27.37</td><td>1.94</td></tr></table>

Results of adversarial trained ResNet-56 against 2 bounded PGD attack for CIFAR-10.

Table content: <table><tr><td>Attack</td><td>Protection</td><td colspan="5">Metric Acc [%] Pre [%] Rec [%] Rej [%] PR[%]</td></tr><tr><td/><td>without</td><td>98.15</td><td>94.05</td><td>87.00</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>98.71</td><td>94.05</td><td>92.36</td><td>01.58</td><td>36.71</td></tr><tr><td>clean</td><td>ε = 4/255 ε = 8/255</td><td>99.18 99.26</td><td>94.05 94.05</td><td>97.42 99.66</td><td>04.66 22.08</td><td>22.96 5.75</td></tr><tr><td/><td>ε = 16/255</td><td>97.40</td><td>94.05</td><td>100.0</td><td>78.86</td><td>1.65</td></tr><tr><td/><td>ε = 32/255</td><td>94.08</td><td>94.05</td><td>100.0</td><td>90.71</td><td>1.43</td></tr><tr><td/><td>without</td><td>97.38</td><td>91.55</td><td>81.30</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>98.71</td><td>91.55</td><td>94.21</td><td>2.97</td><td>46.13</td></tr><tr><td>ε = 40</td><td>ε = 4/255 ε = 8/2550</td><td>99.01 98.96</td><td>91.55 91.55</td><td>97.95 99.63</td><td>6.79 24.94</td><td>25.04 7.38</td></tr><tr><td/><td>ε = 16/255</td><td>96.20</td><td>91.55</td><td>100.0</td><td>80.26</td><td>2.33</td></tr><tr><td/><td>ε = 32/255</td><td>91.57</td><td>91.55</td><td>100.0</td><td>91.10</td><td>2.05</td></tr><tr><td/><td>without</td><td>95.56</td><td>84.49</td><td>68.10</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>97.59</td><td>84.49</td><td>86.53</td><td>4.19</td><td>50.84</td></tr><tr><td>ε = 80</td><td>ε = 4/255 ε = 8/2550</td><td>98.45 98.26</td><td>84.49 84.49</td><td>97.70 100.0</td><td>9.14 28.12</td><td>33.15 11.34</td></tr><tr><td/><td>ε = 16/255</td><td>93.02</td><td>84.49</td><td>100.0</td><td>82.10</td><td>3.89</td></tr><tr><td/><td>ε = 32/255</td><td>84.55</td><td>84.49</td><td>100.0</td><td>91.91</td><td>3.47</td></tr><tr><td/><td>without</td><td>90.73</td><td>54.75</td><td>42.10</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>92.28</td><td>54.75</td><td>51.53</td><td>3.67</td><td>49.86</td></tr><tr><td>ε = 160</td><td>ε = 4/255 ε = 8/2550</td><td>94.52 94.80</td><td>54.75 54.75</td><td>74.25 98.83</td><td>9.83 32.12</td><td>44.05 17.87</td></tr><tr><td/><td>ε = 16/255</td><td>78.33</td><td>54.75</td><td>100.0</td><td>83.94</td><td>6.90</td></tr><tr><td/><td>ε = 32/255</td><td>54.81</td><td>54.75</td><td>100.0</td><td>92.30</td><td>6.27</td></tr><tr><td/><td>without</td><td>85.10</td><td>16.16</td><td>11.70</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>85.44</td><td>16.16</td><td>12.25</td><td>0.75</td><td>60.00</td></tr><tr><td>ε = 320</td><td>ε = 4/255 ε = 8/2550</td><td>86.27 88.80</td><td>16.16 16.16</td><td>14.10 42.86</td><td>3.84 31.85</td><td>44.27 22.83</td></tr><tr><td/><td>ε = 16/255</td><td>55.24</td><td>16.16</td><td>100.0</td><td>86.44</td><td>10.22</td></tr><tr><td/><td>ε = 32/255</td><td>16.28</td><td>16.16</td><td>100.0</td><td>92.75</td><td>9.52</td></tr><tr><td/><td>without</td><td>84.80</td><td>9.63</td><td>6.20</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>84.80</td><td>9.63</td><td>6.20</td><td>0.01</td><td>0.00</td></tr><tr><td>ε = 640</td><td>ε = 4/255 ε = 8/2550</td><td>84.76 83.63</td><td>9.63 9.63</td><td>6.20 6.60</td><td>0.28 10.90</td><td>0.00 5.60</td></tr><tr><td/><td>ε = 16/255</td><td>60.27</td><td>9.63</td><td>66.67</td><td>84.57</td><td>10.72</td></tr><tr><td/><td>ε = 32/255</td><td>09.63</td><td>9.63</td><td>100.0</td><td>93.56</td><td>10.03</td></tr></table>

Results of standard trained ResNet-56 against 2 bounded PGD attack for CIFAR-10.

Table content: <table><tr><td>Attack</td><td>Protection</td><td colspan="5">Metric Acc [%] Pre [%] Rec [%] Rej [%] PR[%]</td></tr><tr><td/><td>without</td><td>98.78</td><td>94.34</td><td>93.40</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>97.34</td><td>94.34</td><td>100.0</td><td>78.96</td><td>0.84</td></tr><tr><td>clean</td><td>ε = 4/255 ε = 8/255</td><td>94.42 94.36</td><td>94.34 94.34</td><td>100.0 100.0</td><td>89.96 90.07</td><td>0.73 0.73</td></tr><tr><td/><td>ε = 16/255</td><td>94.64</td><td>94.34</td><td>100.0</td><td>89.56</td><td>0.74</td></tr><tr><td/><td>ε = 32/255</td><td>95.17</td><td>94.34</td><td>99.89</td><td>88.20</td><td>0.74</td></tr><tr><td/><td>without</td><td>83.98</td><td>12.09</td><td>09.60</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>54.26</td><td>12.09</td><td>100.0</td><td>84.74</td><td>10.67</td></tr><tr><td>ε = 40</td><td>ε = 4/255 ε = 8/2550</td><td>13.61 12.20</td><td>12.09 12.09</td><td>100.0 100.0</td><td>91.92 92.05</td><td>9.83 9.82</td></tr><tr><td/><td>ε = 16/255</td><td>12.09</td><td>12.09</td><td>100.0</td><td>92.06</td><td>9.82</td></tr><tr><td/><td>ε = 32/255</td><td>12.09</td><td>12.09</td><td>100.0</td><td>92.06</td><td>9.82</td></tr><tr><td/><td>without</td><td>82.74</td><td>5.62</td><td>04.60</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>68.30</td><td>5.62</td><td>42.99</td><td>73.72</td><td>12.11</td></tr><tr><td>ε = 80</td><td>ε = 4/255 ε = 8/2550</td><td>44.40 12.39</td><td>5.62 5.62</td><td>86.79 77.97</td><td>85.99 91.04</td><td>11.01 10.34</td></tr><tr><td/><td>ε = 16/255</td><td>15.12</td><td>5.62</td><td>43.81</td><td>90.21</td><td>9.92</td></tr><tr><td/><td>ε = 32/255</td><td>24.13</td><td>5.62</td><td>38.33</td><td>88.85</td><td>9.90</td></tr><tr><td/><td>without</td><td>84.72</td><td>5.10</td><td>3.00</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>81.63</td><td>5.10</td><td>3.70</td><td>27.12</td><td>6.97</td></tr><tr><td>ε = 160</td><td>ε = 4/255 ε = 8/255</td><td>81.41 74.74</td><td>5.10 5.10</td><td>4.91 20.27</td><td>38.74 73.24</td><td>10.04 11.63</td></tr><tr><td/><td>ε = 16/255</td><td>11.85</td><td>5.10</td><td>66.67</td><td>93.50</td><td>10.21</td></tr><tr><td/><td>ε = 32/255</td><td>8.45</td><td>5.10</td><td>65.22</td><td>93.73</td><td>10.18</td></tr><tr><td/><td>without</td><td>86.90</td><td>6.21</td><td>2.20</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>86.55</td><td>6.21</td><td>2.22</td><td>3.36</td><td>2.98</td></tr><tr><td>ε = 320</td><td>ε = 4/255 ε = 8/255</td><td>86.31 85.87</td><td>6.21 6.21</td><td>2.23 2.37</td><td>5.16 12.40</td><td>2.33 5.81</td></tr><tr><td/><td>ε = 16/255</td><td>79.10</td><td>6.21</td><td>6.77</td><td>69.62</td><td>9.70</td></tr><tr><td/><td>ε = 32/255</td><td>57.63</td><td>6.21</td><td>11.11</td><td>88.01</td><td>9.11</td></tr><tr><td/><td>without</td><td>88.24</td><td>5.10</td><td>1.00</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>88.24</td><td>5.10</td><td>1.01</td><td>0.45</td><td>11.11</td></tr><tr><td>ε = 640</td><td>ε = 4/255 ε = 8/255</td><td>88.16 88.14</td><td>5.10 5.10</td><td>1.01 1.01</td><td>1.29 1.63</td><td>5.43 5.52</td></tr><tr><td/><td>ε = 16/255</td><td>87.62</td><td>5.10</td><td>1.03</td><td>7.67</td><td>4.30</td></tr><tr><td/><td>ε = 32/255</td><td>74.15</td><td>5.10</td><td>4.13</td><td>83.83</td><td>9.04</td></tr></table>

Results of adversarial trained ResNet-50 against ∞ bounded PGD attack for ImageNet-100.

Table content: <table><tr><td>Attack</td><td>Protection</td><td colspan="5">Metric Acc [%] Pre [%] Rec [%] Rej [%] PR[%]</td></tr><tr><td/><td>without</td><td>97.72</td><td>88.14</td><td>89.20</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>98.31</td><td>88.14</td><td>95.50</td><td>3.97</td><td>16.50</td></tr><tr><td>clean</td><td>ε = 4/255 ε = 8/255</td><td>98.49 97.78</td><td>88.14 88.14</td><td>98.67 100.0</td><td>12.30 45.54</td><td>7.74 2.35</td></tr><tr><td/><td>ε = 16/255</td><td>91.50</td><td>88.14</td><td>100.0</td><td>85.20</td><td>1.26</td></tr><tr><td/><td>ε = 32/255</td><td>88.64</td><td>88.14</td><td>100.0</td><td>88.73</td><td>1.21</td></tr><tr><td/><td>without</td><td>96.36</td><td>82.72</td><td>80.40</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>97.89</td><td>82.72</td><td>96.40</td><td>6.07</td><td>27.12</td></tr><tr><td>ε = 2/255</td><td>ε = 4/255 ε = 8/2550</td><td>97.91 96.65</td><td>82.72 82.72</td><td>99.01 100.0</td><td>15.79 49.40</td><td>11.81 3.94</td></tr><tr><td/><td>ε = 16/255</td><td>87.79</td><td>82.72</td><td>100.0</td><td>85.56</td><td>2.27</td></tr><tr><td/><td>ε = 32/255</td><td>83.63</td><td>82.72</td><td>100.0</td><td>89.03</td><td>2.18</td></tr><tr><td/><td>without</td><td>94.16</td><td>71.76</td><td>68.60</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>95.63</td><td>71.76</td><td>82.06</td><td>3.89</td><td>41.84</td></tr><tr><td>ε = 4/255</td><td>ε = 4/255 ε = 8/2550</td><td>96.60 94.26</td><td>71.76 71.76</td><td>98.00 99.71</td><td>16.37 52.18</td><td>18.18 5.93</td></tr><tr><td/><td>ε = 16/255</td><td>79.48</td><td>71.76</td><td>100.0</td><td>86.15</td><td>3.62</td></tr><tr><td/><td>ε = 32/255</td><td>73.53</td><td>71.76</td><td>100.0</td><td>89.09</td><td>3.50</td></tr><tr><td/><td>without</td><td>90.68</td><td>53.76</td><td>48.60</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>91.73</td><td>53.76</td><td>55.10</td><td>1.57</td><td>74.68</td></tr><tr><td>ε = 8/255</td><td>ε = 4/255 ε = 8/255</td><td>93.24 90.91</td><td>53.76 53.76</td><td>70.23 99.59</td><td>7.62 53.39</td><td>40.10 9.51</td></tr><tr><td/><td>ε = 16/255</td><td>64.64</td><td>53.76</td><td>100.0</td><td>87.48</td><td>5.83</td></tr><tr><td/><td>ε = 32/255</td><td>57.26</td><td>53.76</td><td>100.0</td><td>89.50</td><td>5.70</td></tr><tr><td/><td>without</td><td>88.34</td><td>40.10</td><td>33.60</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>88.43</td><td>40.10</td><td>33.94</td><td>0.10</td><td>100.0</td></tr><tr><td>ε = 16/255</td><td>ε = 4/255 ε = 8/255</td><td>88.80 90.00</td><td>40.10 40.10</td><td>35.59 69.42</td><td>0.91 34.72</td><td>60.87 14.74</td></tr><tr><td/><td>ε = 16/255</td><td>61.66</td><td>40.10</td><td>96.00</td><td>85.85</td><td>7.51</td></tr><tr><td/><td>ε = 32/255</td><td>45.84</td><td>40.10</td><td>98.25</td><td>89.90</td><td>7.26</td></tr><tr><td/><td>without</td><td>86.56</td><td>33.14</td><td>33.80</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>86.56</td><td>33.14</td><td>33.80</td><td>0.02</td><td>0.00</td></tr><tr><td>ε = 32/255</td><td>ε = 4/255 ε = 8/255</td><td>86.55 86.57</td><td>33.14 33.14</td><td>33.80 35.58</td><td>0.10 3.63</td><td>0.00 13.66</td></tr><tr><td/><td>ε = 16/255</td><td>85.30</td><td>33.14</td><td>40.62</td><td>19.86</td><td>8.39</td></tr><tr><td/><td>ε = 32/255</td><td>83.76</td><td>33.14</td><td>45.80</td><td>33.12</td><td>7.85</td></tr></table>

Results of adversarial trained ResNet-50 against 2 bounded PGD attack for ImageNet-100.

Table content: <table><tr><td>Attack</td><td>Protection</td><td colspan="5">Metric Acc [%] Pre [%] Rec [%] Rej [%] PR[%]</td></tr><tr><td/><td>without</td><td>97.72</td><td>88.14</td><td>89.20</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>98.31</td><td>88.14</td><td>95.50</td><td>3.97</td><td>16.50</td></tr><tr><td>clean</td><td>ε = 4/255 ε = 8/255</td><td>98.49 97.78</td><td>88.14 88.14</td><td>98.67 100.0</td><td>12.30 45.54</td><td>7.74 2.35</td></tr><tr><td/><td>ε = 16/255</td><td>91.50</td><td>88.14</td><td>100.0</td><td>85.20</td><td>1.26</td></tr><tr><td/><td>ε = 32/255</td><td>88.64</td><td>88.14</td><td>100.0</td><td>88.73</td><td>1.21</td></tr><tr><td/><td>without</td><td>95.84</td><td>79.67</td><td>78.40</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>97.51</td><td>79.67</td><td>95.84</td><td>6.03</td><td>29.93</td></tr><tr><td>ε = 300</td><td>ε = 4/255 ε = 8/2550</td><td>97.48 95.98</td><td>79.67 79.67</td><td>98.74 100.0</td><td>16.41 49.90</td><td>12.45 4.29</td></tr><tr><td/><td>ε = 16/255</td><td>85.55</td><td>79.67</td><td>100.0</td><td>85.48</td><td>2.51</td></tr><tr><td/><td>ε = 32/255</td><td>80.73</td><td>79.67</td><td>100.0</td><td>88.91</td><td>2.41</td></tr><tr><td/><td>without</td><td>93.18</td><td>66.46</td><td>64.20</td><td>0.00</td><td>0.00</td></tr><tr><td/><td>ε = 2/255</td><td>94.55</td><td>66.46</td><td>76.25</td><td>3.89</td><td>40.31</td></tr><tr><td>ε = 600</td><td>ε = 4/255 ε = 8/2550</td><td>95.60 93.10</td><td>66.46 66.46</td><td>93.86 100.0</td><td>16.67 52.62</td><td>18.81 6.75</td></tr><tr><td/><td>ε = 16/255</td><td>75.42</td><td>66.46</td><td>100.0</td><td>86.13</td><td>4.12</td></tr><tr><td/><td>ε = 32/255</td><td>68.54</td><td>66.46</td><td>100.0</td><td>88.99</td><td>3.99</td></tr></table>

Results of adversarial trained ResNet-50 for ImageNet-A.

Table content: <table><tr><td>Protection</td><td colspan="5">Metric Acc [%] Pre [%] Rec [%] Rej [%] PR[%]</td></tr><tr><td>without</td><td>76.84</td><td>39.98</td><td>21.78</td><td>00.00</td><td>00.00</td></tr><tr><td>ε = 2/255</td><td>80.23</td><td>39.98</td><td>39.15</td><td>29.72</td><td>31.18</td></tr><tr><td>ε = 4/255</td><td>78.46</td><td>39.98</td><td>72.40</td><td>60.27</td><td>24.23</td></tr><tr><td>ε = 8/255</td><td>52.41</td><td>39.98</td><td>99.42</td><td>85.60</td><td>19.05</td></tr><tr><td>ε = 16/255</td><td>40.60</td><td>39.98</td><td>100.0</td><td>88.51</td><td>18.45</td></tr><tr><td>ε = 32/255</td><td>39.98</td><td>39.98</td><td>100.0</td><td>88.63</td><td>18.43</td></tr></table>

Results of standard trained ResNet-50 for ImageNet-A.

Table content: <table><tr><td>Protection</td><td colspan="5">Metric Acc [%] Pre [%] Rec [%] Rej [%] PR[%]</td></tr><tr><td>without</td><td>77.69</td><td>42.88</td><td>20.56</td><td>00.00</td><td>00.00</td></tr><tr><td>ε = 2/255</td><td>42.88</td><td>42.88</td><td>100.0</td><td>89.99</td><td>18.43</td></tr><tr><td>ε = 4/255</td><td>42.88</td><td>42.88</td><td>100.0</td><td>89.99</td><td>18.43</td></tr><tr><td>ε = 8/255</td><td>42.88</td><td>42.88</td><td>100.0</td><td>89.99</td><td>18.43</td></tr><tr><td>ε = 16/255</td><td>42.88</td><td>42.88</td><td>100.0</td><td>89.99</td><td>18.43</td></tr><tr><td>ε = 32/255</td><td>42.88</td><td>42.88</td><td>100.0</td><td>89.99</td><td>18.43</td></tr></table>

Peer review (iclr2020):
This paper wants to study the problem of “learning with rejection under adversarial attacks”. It first naively extends the learning with rejection framework for handling adversarial examples. It then considers the classical cost-sensitive learning by transfer the multi-class problem into binary classification problem through one-vs-all and using the technique they proposed to reject predictions on non-important labels, and name such technique as “learning with protection”. Finally, they do some experimental studies. The paper does not show any connection between “learning with rejection” and “adversarial learning”. The method it proposes is also a naïve extension of existing methods. Both the problem setting and the technique does not have novelty. The paper fails to realize that the motivated application is actually called “cost-sensitive learning” and has been studied long time before. The paper also has problems in writing. Finally, there is no comparison with any baseline. Only empirical results of the proposed methods are shown. Due to all these reasons, there is still a long way to go before the paper can be published. I will rate it a clear rejection. More specially, The definition of “suspicious example” in Sec.3.1 has no relationship with adversary examples. Does the paper focus on adversary examples? If the definition has no relationship, it is classical learning with rejection. In the last equation of Page 3, there is no definition of 	ilde L. Actually, according to Figure 1, x’s is more close to the decision boundary, it is an example more hard to classify, which could also be “suspicious”. In the definition of “suspicious example” at the beginning of Sec.3.1, is both x and x’ defined as suspicious examples in this way? In the last equation of page 2, there is a rejection function, so minimizing this loss is a “separation-based approach”. However, at the end of Sec.2 the paper states they “follow a confidence-based approach”. Any comment on the inconsistency? The motivated problem is not new. It is called cost-sensitive learning in machine learning and can date back to 2001: Charles Elkan. The Foundations of Cost-Sensitive Learning. IJCAI 2001: 973-978. Where they study the same problem when misclassifying one class of data may cost a lot than misclassifying another class of data. The current paper has not discussed any related work of cost-sensitive learning although they want to study a problem in its field. The paper should be also improved in writing in the following aspects. There is a lot of inaccurate statements in the paper. For example, “In Sections 3 and 4, we propose and describe our algorithm”, what is the difference between propose and describe? “an estimator hat h might return result that differ greatly from h^* in a case with finite samples”. Actually there are rigorous theoretical results describing how the number of finite samples will impact the estimator hat h on unseen data. For example, Peter L. Bartlett, Shahar Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. JMLR, 2002. So inaccurate/unclear statements that will mislead readers should be avoided. In writing, the paper also lacks the necessary references in many places. For example, “Learning with rejection is a classification scenario where the learner is given the option to reject an instance instead of predicting its label.”, “…classifies adversary attacks to two types of attacks, white-box attack and black-box attack.”, “Methods for protecting against these adversarial examples are also being proposed.”. Necessary references are needed for these places. The organization is also problematic. For example, in the second half of Sec.2 introducing two kinds of learning with rejection models, it should be included in a “related work” part. ------------------------------------------ Thank you for the rebuttal. I raised my score a little bit. But I still think this paper has not been ready to be published yet.