Learning with Protection: Rejection of Suspicious Samples under Adversarial Environment
Authors: Masahiro Kato
Yoshihiro Fukuhara
Hirokatsu Kataoka
Shigeo Morishima
Venue:  (2020)
Keywords: Learning with Rejection
Adversarial Examples

Abstract: We propose a novel framework for avoiding the misclassification of data by using a framework of learning with rejection and adversarial examples. Recent developments in machine learning have opened new opportunities for industrial innovations such as self-driving cars. However, many machine learning models are vulnerable to adversarial attacks and industrial practitioners are concerned about accidents arising from misclassification. To avoid critical misclassifications, we define a sample that is likely to be mislabeled as a suspicious sample. Our main idea is to apply a framework of learning with rejection and adversarial examples to assist in the decision making for such suspicious samples. We propose two frameworks, learning with rejection under adversarial attacks and learning with protection. Learning with rejection under adversarial attacks is a naive extension of the learning with rejection framework for handling adversarial examples. Learning with protection is a practical application of learning with rejection under adversarial attacks. This algorithm transforms the original multi-class classification problem into a binary classification for a specific class, and we reject suspicious samples to protect a specific label. We demonstrate the effectiveness of the proposed method in experiments.

Under review as a conference paper at ICLR 2020
LEARNING WITH PROTECTION: REJECTION OF SUSPICIOUS SAMPLES UNDER ADVERSARIAL ENVIRONMENT
Anonymous authors Paper under double-blind review
ABSTRACT
We propose novel framework for avoiding the misclassification of data by us- ing a framework of learning with rejection and adversarial examples. Recent developments in machine learning have opened new opportunities for industrial innovations such as self-driving cars. However, many machine learning models are vulnerable to adversarial attacks and industrial practitioners are concerned about accidents arising from misclassification. To avoid critical misclassifications, we define a sample that is likely to be mislabeled as a suspicious sample. Our main idea is to apply framework of learning with rejection and adversarial examples to assist in the decision making for such suspicious samples. We propose two frameworks, learning with rejection under adversarial attacks and learning with protection. Learning with rejection under adversarial attacks is naive extension of the learning with rejection framework for handling adversarial examples. Learning with protection is a practical application of learning with rejection under adversarial attacks. This algorithm transforms the original multi-class classification problem into binary classification for specific class, and we reject suspicious samples to protect specific label. We demonstrate the effectiveness of the proposed method in experiments.
INTRODUCTION
Recent machine learning models have achieved great success in applications such as self-driving cars (Bojarski al. 2016) and medical diagnosis (Litjens al., (017). In a classification problem, we sometimes face the situation where the misclassification of a specific label can result in serious accidents. This problem arises in various situations, and below we provide examples.
Example (Object Detection for Self-Driving Cars), For self-driving cars, object detection is crucial. For protecting the life of pedestrians, self-driving cars need to be able to detect human or suspend decision making it detects an object that might be a human.
Example 2 (Medical Diagnosis). Cancer is one of the leading causes of death globally, and machine learning algorithms for detecting cancer from images have been gaining attention. This is done by detecting abnormalities an image that might indicate cancer.
As shown above, we want the labeling of data be conservative for certain situations; for example, in autonomous driving, it is better to misclassify non-human object as human than to risk misclassifying a human. In this paper, we introduce two novel frameworks called learning with rejection under adversarial attacks and learning with protection to protect specific class from critical misclassification. Our algorithm is based on the techniques of learning with rejection and adversarial examples. Learning with rejection under adversarial attacks is naive extension of learning with rejection to hold a decision when we face a suspicious sample that vulnerable to adversarial attack. Learning with protection is application of learning with rejection under adversarial attack. The purpose is to protect a specific class from misclassification. Both frameworks are based on the existing work of learning with rejection and adversarial examples.
Learning with rejection a classification scenario where the learner is given the option to reject an instance instead of predicting its label. The purpose of this framework is to prevent critical

---

Under review as a conference paper at ICLR 2020
misclassification, where rejecting labeling a datum incurs lower cost than misclassification. In this field, considerable theoretical and empirical analysis has been conducted (Bartlett& & Wegkamp 20088)
Cortes et al. 2016a b Grandvalet et al., 2009; Herbei & Wegkamp 2006 Yuan & Wegkamp, 2010). Learning with rejection is one ultimate solution against data with high uncertainty. However, the existing frameworks have not considered adversarial examples, which refers a well-designed attack to mislead people to misclassification.
In the industrial application of machine learning, it has been observed that many machine learning models are vulnerable to adversarial attacks. Threat model of adversarial attacks specifies the capabilities of the adversary and classifies adversary attacks two types of attacks, white-box attack and black-box attack. Adversarial inputs that result in machine learning models returning incorrect outputs are called adversarial examples. Several previous studies have artificially generated adversarial examples by adding small perturbations that are imperceptible to humans (Goodfellow et al. 2015 Gu & Rigazio, 2014| HHuang et al. 2015] ( Wagner 2017). Methods for protecting
against these adversarial examples are also being proposed. Among them, adversarial training is one of the most effective defenses (Szegedy et al.
2014; Goodfellow et al. 2015: (Shaham et al., [2015]
Carlini & Wagner 2016 Papernot et al. 2016 Xu et al.] 2017 Madry al. 20188 Buckman et al.
2018 Kannan et al. 2018 Pang et al. 2018 Wong & Kolter 2018 Tramer et al. 2018).
However, most of the defense method fails to avoid misclassification under adversarial attacks. Carlini & Wagner (2017) defeated representative methods for detection of adversarial examples. Athalye al. (2018) reports that several defense algorithms for white-box setting fail when the attacker uses carefully designed gradient-based method. Besides, SShafhhi et al. (2019); Gilmer et al. (2019) shows besides,,Shafahi that adversarial examples are inevitable in some cases. However, we sometimes face situations in which misclassification can result in fatal accidents. For that purpose, we propose learning with rejection option under adversarial attack. The main idea of our method is to hold a decision when we get suspicious samples. As an application of learning with rejection option, we propose a method for preventing the misclassification of data for a specific label under adversarial examples. This method considers situations where we must protect our model from adversarial examples by any means. In addition, It should be noted that our method can be combined with other existing methods such adversarial training. In the following section, we describe the problem setting. In Sections 3 and 4, we propose and describe our algorithm, and in Section 5, we show the experimental results.
2 PROBLEM SETTING
We use the standard settings for binary classification problems, i.e., training and test data points are ii.d. and sampled from some unknown distribution D over X' x {-1.+1), where X denotes the input space. The goal is to lear a classifier h : X + {-1, +1} that assigns a label j(x) to datum a as ū(æ²) sign(h (x)).. Let & be a loss function L : R x {-1, +1) R. The optimal classifier h² is given by h² = arg mi B[L(h(X), Y)), where F is a set of measurable functions.
When we train classifier with training samples, we can naively replace the expectation with the corresponding sample averages. For hypothesis set H. let define an estimator of the optimal classifier h² as h = arg ml [[C(h(X), Y)], where E denotes the averaging operator over training data. Although an estimator h converges to ht in many cases with infinite samples, an estimator h might return result that differ greatly from h² in a case with finite samples. In addition, when h(x) is not smooth at around x, it might become difficult to estimate the function. These can cause serious problems in real-world applications. such as traffic accidents arising from misclassification by the algorithms used in self-driving cars. To make our algorithm's 's inference more robust, we allow a learner to reject such suspicious samples. Let K denote rejection. For any given instance x E X, the learner has the option of abstaining from assigning a label or rejecting that instance and returning the symbol K, or assigning the label ğ E (-1, +1}. If the learner rejects an instance, then they incur some loss c(x) E R: if it does not reject but assigns an incorrect label, then it incurs a cost of one; otherwise, it suffers no loss. Thus, the learner outputs rejection function r: X >> R. that determines the points x X to be rejected according to r(x: S Let us denote a loss function with rejection option C RR x +11 + R. For example, a loss function can be defined as follows:
(h(x),,) y) 4o + c()
2

---

Under review as a conference paper at ICLR 2020
h(+*(x)) --1
Suspicious sample
Me*(0)) -
Decision boundary
Figure I: suspicious samples: suspicious samples have risk of misclassification against the pertur- bations in Ep ball.
There are two frameworks used in the learning with rejection model: the confidence-based and separation- based approaches Cortes et al.) (2016a). The separation- bbasdd approach is first formulated generalization of confidence-based approach (Bartlet by Cortes al. 6a). This formulation approach, we train a classifier and a rejection function & Wegkamp 2008). In the separation- simultaneously. In confidence- based approach, given the conditional probability p(y +1 (>) and cost function c(æ), we only train a classifier and obtain rejection function automatically after training a classifier. Although the confidence-bate approach, the method of confidence-basee approach is easy to implement. In this paper, we follow the confidence-bald based approach and correspond the cost function to vulnerability for adversarial examples.
approach a special case the separation-baser
Our goal is to obtain the optimal classifier it', which minimizes the classification risk with rejection option, i.e., h* arg min [ (l(X), Y))-
LEARNING WITH REJECTION UNDER ADVERSARIAL ATTACKS
In this section, we describe a method to hold decision against suspicious samples.
3.1 STRATEGY FOR REJECTING SUSPICIOUS SAMPLES
To reject suspicious samples, we define which samples are suspicious. We introduce a concept of p-norm suspicious sample. Let || llp be fo-nomm We refer sample æ as suspicious based on the value of nn-norm if there exits sample æ' such that | I'I and i(æ=') returns different class from h(x). If sample is suspicious, we refrain from assigning a class label.
Suppose that we have a estimator f(x:) of the class conditional probability nıxx) = p(y = (ar) and a classifier (xx) sign
[((x) - 1/ Let BB (<) denote the ép ball centered at a E with radius e, i.c., BB(() (x' X ||x' elIpp S e). We calculate =xxx) e BP (e) as follows:
arg max ((h(z'), x'EB
By defining& x" (xx) as in the above equation, easily calculate it with the techniques used in adversarial examples. If (x)=()))))())) < 0 for a sample æ, we regard the sample as suspicious one. We illustrate the concept in Figure I
3.2 RELATIONSHIP WITH LEARNING WITH REJECTION
To interpret the above strategy, we consider the relationship between the strategy and learning with rejection. First, we consider ideal situation, where we know the true value the conditional class probability TTx)) p④y +1).) In confidence- based approach, we can determine the rejection function r(x) corresponding to the rejection cost as shown by Cortes et al. (0116a).
known optimal. For any E X, the misclassification cost for h" is E y ax] min(n/(=), m)(x)) The optimal rejection
that the classifier ht defined for any E X by h" hxx) <Tx))

---

Under review as a conference paper at ICLR 2020
function (ax) should therefore be defined such that (2)x) < 0 if and only if
1
min()(x); n(x)) 2 c(x) + max(7(a)) ( 2 <x)) |h(x)) )
2 c(z))
Therefore, the optimal rejection function r" (x=) is given as (x) = In(x) ! (5 c(x))) Thus, in confidence-based approach,
determine the optimal rejection function r" (ax) if we know the conditional probability (2x)) and the rejection cost c(x).
To reject suspicious samples, relate the rejection cost c(x) with uncertainty of data æ. One naive idea to set c(ar) as a monotonically decreasing function representing the smoothness of 1)(2). When In(ar) ')(2r)) is zero for æ a' E RRd c(a:) takes its largest value; when max [(x:) [(x')] is large, c(x) is small. This means that, if a sample æ is suspicious, then the cost of rejection is low, and we can reject the sample easily; conversely, if sample is not suspicious, then the cost of rejection is high. When the cost is high, we hesitate to reject the sample. We assumed this condition could be met when designing our algorithm. As we show later, we can easily find such c(æ:) and reject suspicious samples without expending much energy in determining the value of c(x:). In our strategy, we reject a sample æ such that
1 (m(=((a)) - 2 (1())) -
/ s 0.
Therefore, r" (xx) S if and only if
1(2 S 1
and n/x) 2 2) ((>"(x))) 2 and n(x) S 2
))
1
1
and ny(x) 2 5 ((x)) (7(x) + n/ (x))
2 and n(x) - 1
1
(a)= - n/(x) + n| (x)) 2
2
0 s ((x) 2 (mx) ((>((<))
1
()x)) ((=((x)) S ()(x))
2 00
1
(1)
Let us assume that n7(x) n) (x)) 0 when 17( - 2 0 and n)(x) n(== 1(=((x) (x) < 0 when
n())) - 1/2 I O. Then, ① insists that r* (a) < if and only if [\xx) < (n(x) (=== (==()))) Therefore,
the following relationship holds for sample æ such that [n)x)) ()='(()) ) s 1:
1
c(x) 2 [mx) 71 (z)).
Thus, in our strategy, the cost function is a decreasing function of the smoothness of 17xx).
Next, show a relationship between the cost function and the degree of perturbation Let us assume 71x) is Lipschitz continuous, with ly- no-norm, and
|n( - 7>x') s A||z - I'I,, = Aε,
where x E X, x' E Bg (c), and À > is a positive constant. From Lipschitz continuity, c(x) 2 2 d holds. This means that the cost of rejection will be small when is a large value. Therefore, learner can reject sample with low cost when the learner is afraid of the misclassification and set the possible perturbation æ with fn-norm large.
3.3 ALGORITIIM
Based on the above idea, we develop an algorithm for rejecting suspicious samples. To discuss learning the rejection in the framework of confidence-basse approach, we construct an estimator f of rT(z)= p(u +1/z) and classifier h(x)) f()) 2.- Then, we define pseudo loss function e for two samples E X' and x' E Bgg (c) as a binary loss function for pair of (æ', (xx), where ((x) = (h()).. For example, if we assume logistic loss, it can be defined as
[(f(''), (x)) 1 -1 log (f(=)) 1 log (1 f(x))
Using the pseudo loss, we calculate x² 2x:) for sample as follows:
z" (x) arg max (=x)))) (())
(c)(
Because this optimization is hard to calculate, we apply techniques of adversarial examples heuristics. Then, we reject a sample if and only if h1== (x))(()(( < 0.
4

---

Under review as a conference paper at ICLR 2020
Calculation of Adversarial Examples: Here, we describe the two simple first-order methods for calculating adversarial examples used in this paper. Although various types of adversarial examples have been proposed, hereinafter adversarial examples refer to perturbation based methods.
One of the simplest methods for generating adversarial examples is the fast gradient sign method (FGSM) (rGGS))Goodfelloo Goodfellow et al. (2015), which is fast single-step attack that maximizes the loss function in the linear approximation. The perturbation under the FGSM is calculated as follows:
sign( V (..(((())
The projected gradient descent method (PGD) Madry al. (2018) is an iterative variant of the FGSM.
The perturbation under the PGD at time step s + 1 is calculated as follows:
8(++1)_ P (6 + a sign( V<<(( + (,)),)))
where a denotes single step and (-(.) denotes the projection onto the bp-ball with radius &.
LEARNING WITH PROTECTION
In this section, we propose a practical and easily implemented algorithm as application of learning with rejection under adversarial attack. many cases, we want avoid misclassification for specific class. In self-driving cars, we must be able to detect human beings to avoid serious accidents. In medical diagnosis, we should not miss a dangerous disease. We thus propose an algorithm to protect a specific class from misclassification in multi-class classification problem.
We extend the previous problem setting of binary classification to that of multi-class classification. Let X and Y be the feature and label spaces, respectively, and suppose that there is unknown distribution D over X.... We assume that the label space has K 2 labels, i..., V = 2, 3, K}. to avoid misclassification as the defense target class y.
We define the class for which For example, if we want to avoid misclassification of 1, we designate the defense target class as 1. We call the algorithm for protecting defense target class based on the following idea learning with protection.
4.1 PROTECTING A DEFENSE TARGET CLASS
Let F be the of measurable functions from X' y to R. For function f F, we define classifier hy X' x y > TR. as follows:
yy(x) = arg max f(x, k). KE(K]
Let C(f: aff: x.y) v) be the loss function of function f E F for data (x, v). The goal of the learning problem is to find of E F such that the population risk R(f)) := ( [ff; ; x, y)] is minimized. This formulation is the standard setting of multi-class classification problem.
From the original formulation of the multi-class classification problem, we construct a binary classification problem to protect the defense target class t from misclassification. Let define function n X' + {-1, +1) for a function f to transform the prediction of multi-class classification that of binary classification
+1 Vf( =
())))
1-1
yy(m) #
For pair (x. (x)),, we calculate new feature (xx) follows:
2 (x) arg max Z(f; x' ,.t./////// (x)),
where (J;arr,, yzx)) ) is binary loss function for pair (æc, y (x)). This loss function corresponds to the pseudo loss function introduced in the previous section. For example, we can define 0-1 loss as follows:
B(Ja a (x)) = I --)(=)=-11 (|(===+1 - + (0)=+++1 1/0=)===-))"0---111
Then, for this binary classification problem derived from multi-class classification problem, we apply an algorithm for rejection. If y /2(2""0 (z)) t for sample such that (x(x) sample because the sample æ predicted as a class that is not t might be a class t neighborhood of æ with normm We call algorithms based on the abeve idea learning with protection.
then reject the
5

---

Under review as a conference paper at ICLR 2020
Algorithm Learning with Protection
Input:
Trained classier h. defense target class E [K], and test dataset fx.)=.
Construct a pseudo loss function (3) for test dataset {a; )
Compute z" (2r;) for sample my by (2 by method of adversarial example.
Reject sample x; if h(x;)) # t and h (xx))
4.2 ALGORITHM WITH CROSS-ENTROPY Loss
Various algorithms can be used to implement the above framework. this section, we consider a cross-entropy loss function and define algorithm for protecting the defense target class.
Let H. be a hypothesis set. First, we train a model f E H. with the cross-entropy loss by any of the suitable methods for classification problems such adversarial training. After training 9, we obtain f. the function that minimizes the cross-entropy loss. The trained model an estimator of the risk minimizer f population. Second, use the logistic loss for L, i.e.,
exp (f(x', t)) Σne(Ki exp (f(x', = k))
Elf:'' ,f(())())))
exp (x y t))
-1)(00)=+1 log
ExE(K] exp (1x2,, y = k:))
exp(s (=r))=t))
The intuition of this loss function is follows. Let (x', Exci exp() 120 [=k)==k) k|]
Under interpret 9² k) p(y k/æ') with g" Therefore,
cross entropy loss function,
can
apply the log loss function function with 9 (x', = ply = (2) and p(y * l|2) = 1 - ply l|=)
1 1- (xx,, 1). As mentioned above, to find feature z" from (2). we can apply methods of adversarial
examples as heuristics. We show the pseudo algorithm in Algorithm
EXPERIMENTS
To demonstrate the effectiveness of proposed method, we conducted experiments using benchmark data. For the benchmark data, we use the CIFAR- - (Krizhessky 2009) and ImageNet- -00 Kang
et al. 2019) dataset. The CIFAR-10 10 dataset consists of 32 x 32 color images in 10 classes, with 6, 000
images per class. There are 50, 000 training images and 10, 000 test images. The ImageNet- 100
dataset consists of the 100-class subset of ImageNet-IK IK (Deng et al. 2009; Russakovsky et al. 2015)
containing every 10th class by WordNet ID order. As we explained above, learning with protection rejects suspicious samples after we train a model f to protect defense target class. For CIFAR- 10, set airplane class as defense target class. For ImageNet-A, order of WordNet ID as defense target class. Our method allow various training method to estimate
set the first 10 classes in the
ft. When we train the model f. we used both standard training method and adversarial training (Madry et al. 2018). After trainning a model, we adopt 30-step fox bounded PGD attack to find the feature z.. For each experiment, we output the accuracy, precision, recall, rejection rate, and true rejection. Let TP. TN, FP, and FN be the numbers of true positive, true negative, false positive, and false negative of a classification result. Then, accuracy, precision, and recall are defined as TP
accuracy = TF+TN+FP+EX precision TT4FF. and recall = TP-FN- Let true accept (TA) be an outcome where rejection function correctly accepts data such that a sample x does not belong to the defense target class, i.e., h(()) * t. Let true reject (FR) be an outcome where the rejection function correctly rejects data such that a sample x belongs to the defense target class, i.c., h(w) = t. FA and FR can be defined similarly (Ni et al. 2019). Then, rejection rate and precision of rejection are defined as rejection rate
TP
TR TR TA FR PA and precision of rejection TR+PR.
TATER
USec
See htt:://mege-net.orgggee edu www.:e//edddkkrikiiiieeaa http://.aar............

---

Under review as a conference paper at ICLR 2020
Table 1: Results of adversarial trained model against &o bounded PGD attack. Results of standard trained model are shown in Appendix [.I We show the accuracy (Acc), precision (Pre), recall (Rec), rejection rate (Rej), and precision of rejection (PR).
CIFAR-10 10
ImageNat- 100
Attack
Protection
Metric
Attack
Protection
Metric
Rej
Acc
Rej
without
98.15
94,05
57.00
0.00
0.00
without
97.72
88.14
89,20
0.00
0.00
clean
4/255
99.18
94.05
97.42
04.66
22.96
clean
4/255
98.49
88.14
98.67
12.30
7.74
16/255
97.40
94.05
100.0
78.86
1.65
16/255
91.50
88.14
100.0
85.20
1.26
without
$4.80
80.61
63.20
0.00
0.00
without
94,16
71.76
68.00
0.00
0.00
4/255
4/255
98.13
80.61
97.53
10.07
34.96
4/255
4/255
96.00
71.76
98.00
16.37
18.18
255
91.03
80.61
100.0
83.06
4.43
16/255
79.48
71.76
100.0
86,15
3.62
without
81.31
12.01
8.00
0.00
0.00
without
88.31
10.10
33.60
0.000
0..00
16/255
4/255
85.67
12.01
48.37
16/255
4/255
88.80
16/255
46.92
12.01
100.0
88.96
10.34
16/255
61.66
40.10
96.00
85.85
7.51
Neural network model:
For CIFAR-10, use the ResNet-56 (He et al. (2015) architecture. For ImageNet-100, we use the ResNet-50 (He et al. 2013) architecture with 224 x 224 resolution as implemented in torchvision. We describe training hyperparameters in Appendix
Ad Adversarial training: We use lo bounded projected gradient descent (PGD) method generate training samples. We select a attack target class for each image uniformly at random from the set of incorrect classes. For distortion size &, we set as 8/ 8/255 and apply random scaling Uniform(0,, e) to improve performance against smaller distortions. We use 10 optimization steps and step size e/ e/ /steps as described in (Kang al. 2019). We update the model by stochastic gradient descent (SGD) method using only the adversarial images (no clean images).
5.1 PROTECTION AGAINST ARTIFICIAL. ADVERSARIAL EXAMPLES
We evaluate our learning with protection method on the CIFAR-10 10 and ImageNet- 100 validation sets against artificial adversarial examples with the defense target class.
protection against expected attack: We distort the inputs by 50-step foe bounded PGD attack and apply learning with protection method. The single step size of PGD is calculated by e/ /steps. The results are shown in Table T and Appendix B.
Although our learning with protection method, specific adversarial attack method is used to find the feature 2" "(x;)) ) for sample æ; (we adopt for bounded PGD in this experiment), the input might be distorted by different attack methods. order to evaluate the performance against such unexpected adversaries, bounded PGD attack. The results are shown in Table 2 and Appendix [B.
protection against unexpected attack:
conducted experiments against 50-step l2
We can see from the table that our method achieves 100% in the recall, i.e., can reject all samples such that y = t and h(x) * t. As shown Appendix B.II, models with adversarial training outperforms models without adversarial training. It is because negative, and it is difficult to improve the performance of a model when the number of the true positive is small. When we adversarial training, of rejection. For example, in Table [ when attack is l2 bounded PGD (e 80) for CIFAR-10, recall increases about 20% by only reject 9.14% of all samples. On the other hand, On the other hand, when we reject too many samples, the precision of rejection dramatically drops. Therefore, we need control the perturbation not reject samples unnecessarily.
method only reduce the false
increase the recall with the small number
5.2 PROTECTION AGAINST NATURAL ADVERSARIAL EXAMPLES
Natural adversarial examples which are introduced (Hendrycks et al. 2019 are defined as unmodi- fied and naturally occurring examples that cause classifier accuracy degrade drastically. In order to evaluate the proposed method for such real hard samples, we conduct experiments on ImageNet-A
See http:///ggthhboco///////rr/////e -ad--ad-aaaseee adv-examples

---

Under review as a conference paper at ICLR 2020
Table Results of adversarial trained model against Eg bounded PGD attack. Results of standard trained model are shown in Appendix B.I We show the accuracy (Acc), precision (Pre), recall (Rec), rejection rate (Rej), and precision of rejection (PR).
CIFAR-10
ImageNet. 100
Attack
Protection
Metric
Attack
Protection
Metric
Acc
Rej
Rej
PR
without
98.15
91,05
57.00
0.00
0.00
without
97.72
88.14
89.20
0.00
0.00
clean
4/255
99.18
94.05
97.42
04.66
22.96
clean
4/255
98.49
88.14
98.67
12.30
7.74
16/255
97.40
94.05
100.0
78.86
1.65
16/255
91.50
88.14
100.0
85.20
1.26
without
96.36
84,49
68.10
0.00
0.00
without
93.18
06.44
04.20
0.00
0.00
80
4/255
45.45
84.49
97.70
9.14
33.15
600
4/255
95.60
06.16
93.86
16.07
18.81
16/ 2255
93.02
84.49
100.0
82.10
3.89
16/255
75.42
66.46
100.0
86.13
4.12
without
85. 10
16.16
11.70
0.00
0.00
without
8Y.75
37.21
32. 10
0.00
0.00
320
4/255
86.27
16.16
2400
4/255
88,24
37.24
0.65
81.82
16/255
35.24
16.16
100.0
86.44
10.22
16/255
56.47
37.24
96.44
86.49
7.62
Table Results of ImageNet-A. Full experimental results are shown in Appendix B.2
We show the
accuracy (Acc), precision (Pre), recall (Rec), rejection rate (Rej), and precision of rejection (PR).
Adversarial Trained
Standard Trained
Metric
Metric
Protection
Acc
Pre
Rec
Rej
PR
Acc
Pre
Rec
Rej
PR
without
76.84
39.98
21.78
00.00
00.00
77.69
42.88
20.56
00.00
00.00
g
2/255
80.23
39.98
39.15
29.72
31.18
42.88
42.88
100.0
89.99
18.43
8 4/255
78.46
39.98
72.40
60.27
24.23
42.88
42.88
100.0
89.99
18.43
8/255
52.41
39.98
99.42
85.60
19.05
42.88
42.88
100.0
89.99
18.43
dataset. ImageNet-A contains 7,500 natural adversarial example images in 200 classes that are subset of ImageNet-IK's IK's 1,000 classes. We use the ResNet-50 architecture with 224 x 224 resolution. We first trained the model on the training set of ImageNet- -K dataset and evaluate the performance on IeageNet-A with our learning with protection method at test time. Due to the difference in the number classes, only 200 of its 1,000 logits are used in test time. We set the first 20 classes in the order of WordNet ID the defense target class. The results are shown in Table 3
As shown in Table 3 and Appendix B.2 the proposed method works well for natural adversarial samples. For example, when the distortion size is 2/ 2/255, the recall increases about 20% by rejecting only 30% of all samples. On the other hand, the rejection rate of the models without adversarial training is high even when the size of the perturbation is small.
6 CONCLUSION
In this paper, we proposed learning with rejection under adversarial attacks for avoiding the mis- classification. The proposed method suspend decision making when sample is f,-norm suspicious. By extending the strategy, we proposed an easily implementable algorithm called learning with protection. This algorithm prevents a defense target class from misclassification. We showed the performance of learning with protection using benchmark datasets. From the results, we confirmed that our method successfully caught samples that would have misclassified we did not reject it.
REFERENCES
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 284-293 Stockhollmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. J.
Mach. Learn. Res., 1840-1800, June 2008. ISSN 1532-4435.
8

---

Under review as a conference paper at ICLR 2020
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. CoRR, abs/1604.07336, 2016.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Leurning Representations, 2018.
Nicholas Carlini and David Wagner. Defensive distillation robust adversarial examples. 07 2016.
Nicholas Carlini and David Wagner. Adversarial examples detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec '7, pp. 3-14. ACM, 2017. doi: 10. 10.1145//128572.314444
easily detected: Bypassing ten
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. 2016a.
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. In D. D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1660-16600 16600166...00 Curran Associates, Inc., 2016b.
Deng. Dong, R. Socher, L. J. Li, Li, and L.. Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In CVPRO9, 2009.
Justin Gilmer, Nicolas Ford, Nicholas Carlini, and Ekin Cubuk. Adversarial examples are a natural consequence of test eтrоr in noise. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 228002289, Long Beach, California, USA, 09-15 5 Jun 2019. PMLR.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: (tf.::/7060277 2017.
Yves Grandvalet, Alain Rakotomamonjy, Joseph Keshet, and Stéphane Canu. Support vector machines with a reject option. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21, pp. 537-544. Curran Associates, Inc., 2009.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. CoRR, 141.5568, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03333 2015.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. ArXiv, abs/11907077 000774 2019.
Radu Herbei and Marten H. Wegkamp. Classification with reject option. The Canadian Journal of Statistics La Revue Canadienne de Statistique, 34(4):709-721, 2006.
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with a strong adversary. http://arzivoorgaas//511..0
11 2015.
Daniel Kang, Yi Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardt. Testing robustness against unforeseen adversaries, 2019.
Harini Kannan, Alexey Kurakin, and Ian J. Goodfellow. Adversarial logit pairing. ArXiv. abs/1803.00633, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

---

Under review as a conference paper at ICLR 2020
Geert J. S. Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen A. W. M. van der Laak, Bram van Ginneken, and Clara I. Sánchez. A survey deep learning in medical image analysis. Medical Image Analysis, 42:60-88, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On possibility and impossibility multiclass classification with rejection. In Advances in Neural Information Processing Systems 32, 2019.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial examples. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 4579-4589. Curran Associates, Inc., 2018.
N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami. Distillation as a defense to adversarial perturbations against deep neural networks. 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-597, May 2016. doi: (0.1109/SP.001.441
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (0:221-2522 2015.
doi: 10.10///s 1263-015-0816-y;
Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? In International Conference on Learning Representations, 2019.
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of neural nets through robust optimization. CoRR, abs/1511 1.1151111000 1.05432, 2015.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
Florian Tramèr, Alexey Kurakin, Nicolas Pupernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations, 2018.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5286-5295 Stockholmsmääsann Stockholm Sweden, 10-15 Jul 2018. PMLR.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. CoRR, abs//774.001155 2017.
Ming Yuan and Marten Wegkamp. Classification methods with reject option based on convex risk minimization. J. Mach. Learn, Res,, 11:111-130, March 2010. ISSN 1532-4435.
10

---

Under review as a conference paper at ICLR 2020
A TRAINING HYPERPARAMETER
For CIFAR-10, we trained on single NVIDIA V100 GPU for 200 epochs with batch size 32, initial learning rate 0.1, momentum 0.9, and weight decay 10-4, We decayed the learning rate epochs 100 and 150. For ImageNeet1000, we trained on machines with NVIDIA V100 GPUs using standard data augmentation used in (He al., 2013). We ran synchronized SGD (Goyal al., 2017) for 90 epochs with batch size 32 x 8 and a learning rate schedule with "warm-up epochs and decay at epochs 30, 60, and 80 by a factor
B FULL EXPERIMENTAL RESULTS
B. PROTECTION AGAINST ARTIFICIAL ADVERSARIAL EXAMPLES
Table 4: Results of adversarial trained ResNet-56 against foo bounded PGD attack for CIFAR-10.
Metric
Attack
Protection
Ace [%]
Pre [%]
Rec [%
Rej [%]
PR[%]
without
98.15
94.05
87.00
0.00
0.00
2/255
98.71
94.05
92.36
01.58
36.71
E = 4/255
99.18
94.05
97.42
04.66
22.96
clean
8/255
99.26
94.05
99.66
22.08
5.75
E 16/255
97.40
94.05
100.0
78.86
1.65
32/255
94.08
94.05
100.0
90.71
1.43
without
97.16
91.15
79.30
0.00
0.00
e 2/255
98.73
91.15
94.52
3.45
46.67
ê = 4/255
98.98
91.15
97.90
7.42
25.61
E = 2/255
g 8/255
98.93
91.15
99.75
25.98
7.89
16/255
96.00
91.15
100.0
80.73
2.56
& 32/255
91.17
91.15
100.0
91.28
2.27
without
94.80
80.61
63.20
0.00
0.00
g = 2/255
96.71
80.61
79.50
4.14
49.52
4/255
98.13
80.61
97.53
10.07
34.96
4/255
e 8/2255
97.83
80.61
100.0
29.95
12.29
16/255
91.03
80.61
100.0
83.06
4.43
32/255
80.64
80.61
100.0
92.15
3.99
without
89.22
44.90
34.30
0.00
0.00
2/255
90.63
44.90
41.13
2.70
61.48
e = 4/255
92.48
44.90
56.05
8.30
46.75
8/255
8/255
93.59
44.90
100.0
34.34
19.13
Σ 16/255
70.80
44.90
100.0
85.58
7.68
32/255
45.04
44.90
100.0
92.34
7.12
without
84.94
12.01
8.00
0.00
0.00
g 2/255
85.13
12.01
8.21
0.37
67.57
ε 4/255
85.67
12.01
8.93
2.15
48.37
8 = 16/255
ε 8/255
87.58
12.01
20.00
27.05
22.18
16/255
46.92
12.01
100.0
88.96
10.34
E 32/255
12.01
12.01
100.0
93.34
9.86
without
86.11
10.06
4.90
0.00
0.00
e 2/255
86.11
10.06
4.90
0.00
0.00
e 4/255
86.10
10.06
4.90
0.16
6.25
32/ 255
E 8/255
85.56
10.06
4.99
5.08
3.54
16/255
72.16
10.06
24.87
78.95
10.17
32/255
10.04
10.06
98.00
95.12
9.99
11

---

Under review as a conference paper at ICLR 2020
Table Results of standard trained ResNet-56 against &o bounded PGD attack for CIFAR-10.
Metric
Attack
Protection
Acc [%]
Pre [%]
Rec [%]]
Rej [%]]
PR[%]
without
98.78
94.34
93.40
0.00
0.00
e 2/255
97.34
94.34
100.0
78.96
0.84
E = 4/255
94.42
94.34
100.0
89.96
0.73
clean
E 8/255
94.36
94.34
100.0
90.07
0.73
16/255
94.64
94.34
100.0
89.56
0.74
& 32/255
95.17
94.34
99.89
88.20
0.74
without
82.41
5.61
4.80
0.00
0.00
e = 2/255
62.60
5.61
64.86
77.73
11.91
4/255
32.92
5.61
96.00
87.94
10.80
2/255
E 8/2255
10.36
5.61
88.89
90.93
10.40
16/255
13.39
5.61
54.55
90.22
10.11
32/255
21.23
5.61
47.06
89.07
10.08
without
83.68
3.40
0.00
0.00
2/255
79.72
4.86
5.30
37.19
9.63
E = 4/255
78.75
4.86
8.15
50.64
11.51
4/255
8/255
64.85
4.86
40.96
79.66
11.51
ε 16/255
7.11
4.86
100.0
92.83
10.41
32/255
4.86
4.86
100.0
93.00
10.39
without
86.39
5.65
2.30
0.00
0.00
È 2/255
85.66
5.65
2.35
6.70
3.43
e 4/255
85.44
5.65
2.42
9.81
4.89
8 = 8/255
E 8/255
85.15
5.65
2.96
23.36
9.55
16/255
71.01
5.65
12.99
81.44
10.11
E 32/255
53.43
5.65
13.61
88.62
9.38
without
88.10
7.59
1.70
0.00
0.00
è = 2/255
87.97
7.59
1.71
1.32
2.27
g 4/255
87.89
7.59
1.71
2.25
2.67
16/255
E 8/255
87.84
7.59
1.72
3.09
3.88
& 16/255
86.69
7.59
1.92
19.23
5.98
32/255
70.54
7.59
8.29
86.59
9.18
without
89.10
5.88
0.60
0.00
0.00
2/255
89.10
5.88
0.60
0.03
0.00
g = 4/255
89.04
5.88
0.60
0.95
4.21
8 = 32/255
e 8/255
89.02
5.88
0.60
1.11
3.60
E 16/255
88.94
5.88
0.60
1.97
3.05
32/255
85.72
5.88
0.63
27.37
1.94
12

---

Under review as a conference paper at ICLR 2020
Table 6: Results of adversarial trained ResNet--66 against &g bounded PGD attack for CIFAR- 10.
Metric
Attack
Protection
Acc [%]
Pre [%]
Rec [%]
Rej [%]
PR[
without
98.15
94.05
87.00
0.00
0.00
2/255
98.71
94.05
92.36
01.58
36.71
4/255
99.18
94.05
97.42
04.66
22.96
clean
8/255
99.26
94.05
99.66
22.08
5.75
16/255
97.40
94.05
100.0
78.86
1.65
E = 32/255
94.08
94.05
100.0
90.71
1.43
without
97.38
91.55
81.30
0.00
0.00
2/255
98.71
91.55
94.21
2.97
46.13
4/255
99.01
91.55
97.95
6.79
25.04
40
ê 8/2550
98.96
91.55
99.63
24.94
7.38
& 16/255
96.20
91.55
100.0
80.26
2.33
& = 32/255
91.57
91.55
100.0
91.10
2.05
without
95.56
84.49
68.10
0.00
0.00
2/255
97.59
84.49
86.53
4.19
50.84
ε 4/255
98.45
84.49
97.70
9.14
33.15
80
8/2550
98.26
84.49
100.0
28.12
11.34
ε 16/255
93.02
84.49
100.0
82.10
3.89
32/255 255
84.55
84.49
100.0
91.91
3.47
without
90.73
54.75
42.10
0.00
0.00
= 2/255
92.28
54.75
51.53
3.67
49.86
4/255
94.52
54.75
74.25
9.83
44.05
160
ε 8/2550
94.80
54.75
98.83
32.12
17.87
16/255
78.33
54.75
100.0
83.94
6.90
E = 32/255
54.81
34.75
100.0
92.30
6.27
without
85.10
16.16
11.70
0.00
0.00
2/255
85.44
16.16
12.25
0.75
60.00
4/255
86.27
16.16
14.10
3.84
44.27
320
8/2550
88.80
16.16
42.86
31.85
22.83
& 16/255
55.24
16.16
100.0
86.44
10.22
32/255
16.28
16.16
100.0
92.75
9.52
without
84.80
9.63
6.20
0.00
0.00
2/255
84.80
9.63
6.20
0.01
0.00
ε 4/255
84.76
9.63
6.20
0.28
0.00
640
8/2550
83.63
9.63
6.60
10.90
5.60
= = 16/255
60.27
9.63
66.67
84.57
10.72
32/255
09.63
9.63
100.0
93.56
10.03
13

---

Under review as a conference paper at ICLR 2020
Table 7: Results of standard trained ResNet-56 against E bounded PGD attack for CIFAR- 10.
Metric
Attack
Protection
Acc [%]
Pre [%]
Rec [%]
Rej [%]
PR[%
without
98.78
94.34
93.40
0.00
0.00
2/255
97.34
94.34
100.0
78.96
0.84
4/255
94.42
94.34
100.0
89.96
0.73
clean
8/255
94.36
94.34
T00.0
90.07
0.73
16/255
94.64
94.34
100.0
89.56
0.74
E = 32/255
95.17
94.34
99.89
88.20
0.74
without
83.98
12.09
09.60
0.00
0.00
2/255
54.26
12.09
100.0
84.74
10.67
4/255
13.61
12.09
100.0
91.92
9.83
40
ê 8/2550
12.20
12.09
100.0
92.05
9.82
& 16/255
12.09
12.09
100.0
92.06
9.82
& = 32/255
12.09
12.09
100.0
92.06
9.82
without
82.74
5.62
04.60
0.00
0.00
2/255
68.30
5.62
42.99
73.72
12.11
ε 4/255
44.40
5.62
86.79
85.99
11.01
80
8/2550
12.39
5.62
77.97
91.04
10.34
ε 16/255
15.12
5.62
43.81
90.21
9.92
32/255
24.13
5.62
38.33
88.85
9.90
without
84.72
5.10
3.00
0.00
0.00
= 2/255
81.63
5.10
3.70
27.12
6.97
4/255
81.41
5.10
4.91
38.74
10.04
160
8/255
74.74
5.10
20.27
73.24
11.63
16/255
11.85
5.10
66.67
93.50
10.21
E = 32/20 /55
8.45
5.10
65.22
93.73
10.18
without
86.90
6.21
2.20
0.00
0.00
2/255
86.55
6.21
2.22
3.36
2.98
4/255
86.31
6.21
2.23
5.16
2.33
320
8/255
85.87
6.21
2.37
12.40
5.81
& 16/255
79.10
6.21
6.77
69.62
9.70
32/255
57.63
6.21
11.11
88.01
9.11
without
88.24
5.10
1.00
0.00
0.00
2/255
88.24
5.10
1.01
0.45
11.11
ε 4/255
88.16
5.10
1.01
1.29
5.43
640
8/255
88.14
5.10
1.01
1.63
5.52
E = 16/255
87.62
5.10
1.03
7.67
4.30
32/255
74.15
5.10
4.13
83.83
9.04
14

---

Under review as a conference paper at ICLR 2020
Table 8: Results of adversarial trained ResNet-500 against &oo bounded PGD attack for ImageNet-
Metric
Attack
Protection
Acc [%]
Pre [%]
Rec [%
Rej [%)
PR[%]
without
97.72
88.
89.20
0.00
0.00
E 2/255
98.31
88. 14
95.50
3.97
16.50
E 4/255
98.49
88.14
98.67
12.30
7.74
clean
E 8/255
97.78
88.14
100.0
45.54
2.35
16/255
91.50
88.14
100.0
85.20
1.26
E 32/255
88.64
88.14
100.0
88.73
1.21
without
96.36
82.72
80.40
0.00
0.00
è 2/255
97.89
82.72
96.40
6.07
27.12
e 4/255
97.91
82.72
99.01
15.79
11.81
2/255
E 8/2550
96.65
82.72
100.0
49.40
3.94
E 16/255
87.79
82.72
100.0
85.56
2.27
32/255
83.63
82.72
100.0
89.03
2.18
without
94.16
71.76
68.60
0.00
0.00
2/255
95.63
71.76
82.06
3.89
41.84
g = 4/255
96.60
71.76
98.00
16.37
18.18
4/255
8/2550
94.26
71.76
99.71
52.18
5.93
& 16/255
79.48
71.76
100.0
86.15
3.62
32/255
73.53
71.76
100.0
89.09
3.50
without
90.68
53.76
48.60
0.00
0.00
E = 2/255
91.73
53.76
55.10
1.57
74.68
с = 4/255
93.24
53.76
70.23
7.62
40.10
8/255
ş = 8/255
90.91
53.76
99.59
53.39
9.51
16/255
64.64
53.76
100.0
87.48
5.83
Σ 32/255
57.26
53.76
100.0
89.50
5.70
without
88.34
40.10
33.60
0.00
0.00
E = 2/255
88.43
40.10
33.94
0.10
100.0
g 4/255
88.80
40.10
35.59
0.91
60.87
16/255
8/255
90.00
40.10
69.42
34.72
14.74
& 16/255
61.66
40.10
96.00
85.85
7.51
32/255
45.84
40.10
98.25
89.90
7.26
without
86.56
33.14
33.80
0.00
0.00
2/255
86.56
33.14
33.80
0.02
0.00
e = 4/255
86.55
33.14
33.80
0.10
0.00
8 = 32/22/55
8/255
86.57
33.14
35.58
3.63
13.66
g 16/255
85.30
33.14
40.62
19.86
8.39
32/255
83.76
33.14
45.80
33.12
7.85
15

---

Under review as a conference paper at ICLR 2020
Table 9; Results of standard trained ResNet--50 against &o bounded PGD attack for ImageNet-100..
Metric
Attack
Protection
Acc [%]
Pre [%]
Rec [%
Rej [%)
PR[
without
98.80
94.00
94.00
0.00
0.00
E 2/255
94.59
94.00
100.0
88.19
0.67
E 4/255
94.12
94.00
100.0
89.09
0.67
clean
E 8/255
94.09
94.00
100.0
89.13
0.67
16/255
94.04
94.00
100.0
89.23
0.67
E 32/255
94.00
94.00
100.0
89.29
0.67
without
88.88
43.49
37.40
0.00
0.00
è 2/255
86.89
43.49
68.25
49.25
9.11
e 4/255
84.06
43.49
77.27
62.12
8.24
2/255
& 8/2550
66.31
43.49
95.41
84.37
7.15
E 16/255
44.27
43.49
100.0
90.56
6.86
32/255
43.49
43.49
100.0
90.67
6.85
without
88.76
42.51
35.20
0.00
0.00
2/255
88.54
42.51
36.97
6.07
7.84
g = 4/255
88.65
42.51
38.60
8.63
10.11
E 4/255
8/2550
88.46
42.51
46.32
23.19
10.27
& 16/255
71.37
42.51
88.89
81.19
7.38
32/255
42.51
42.51
100.0
90.99
7.06
without
88.28
39.61
32.80
0.00
0.00
E = 2/255
88.25
39.61
32.87
0.44
4.55
с 4/255
88.26
39.61
32.93
0.54
7.41
8/255
ş = 8/255
88.19
39.61
33.00
1.29
4.62
16/255
87.47
39.61
38.23
17.68
7.97
Σ 32/255
49.20
39.61
97.62
89.29
7.38
without
86.92
32.89
29.60
0.00
0.00
E = 2/255
86.91
32.89
29.60
0.08
0.00
g = 4/255
86.91
32.89
29.60
0.08
0.00
16/255
8/255
86.91
32.89
29.60
0.08
0.00
& 16/255
86.81
32.89
29.66
1.96
32/255
82.84
32.89
44.85
43.23
7.80
without
83.34
23.53
29.60
0.00
0.00
2/255
83.33
23.53
29.60
0.06
0.00
e = 4/255
83.33
23.53
29.60
0.08
0.00
8 = 32/22/55
8/255
83.33
23.53
29.60
0.08
0.00
16/255
83.32
23.53
29.60
0.14
0.00
32/255
83.19
23.53
29.96
1.61
7.41
16

---

Under review as a conference paper at ICLR 2020
Table 10: Results of adversarial trained ResNet-50 against &g bounded PGD attack for ImageNee-
Metric
Attack
Protection
Acc [%]
Pre [%]
Rec [%]
Rej [%]
PR[%]
without
97.72
88.14
89.20
0.00
0.00
2/255
98.31
88.14
95.50
3.97
16.50
4/255
98.49
88.14
98.67
12.30
7.74
clean
8/255
97.78
88.14
100.0
45.54
2.35
16/255
91.50
88.14
100.0
85.20
1.26
ε 32/255
88.64
88.14
100.0
88.73
1.21
without
95.84
79.67
78.40
0.00
0.00
2/255
97.51
79.67
95.84
6.03
29.93
4/255
97.48
79.67
98.74
16.41
12.45
300
ε 8/2550
95.98
79.67
100.0
49.90
4.29
16/255
85.55
79.67
100.0
85.48
2.51
32/255
80.73
79.67
100.0
88.91
2.41
without
93.18
66.46
64.20
0.00
0.00
2/255
94.55
66.46
76.25
3.89
40.31
4/255
95.60
66.46
93.86
16.67
18.81
600
8/2550
93.10
66.46
100.0
52.62
6.75
&
16/255
75.42
66.46
100.0
86.13
4.12
32/255
68.54
66.46
100.0
88.99
3.99
without
89.74
48.57
44.20
0.00
0.00
2/255
90.36
48.57
47.73
1.21
60.66
4/255
91.57
48.57
58.62
7.36
33.15
1200
8/255
89.90
48.57
96.09
51.47
10.41
16/255
61.06
48.57
100.0
87.28
6.34
ε 32/255
52.54
48.57
100.0
89.42
6.19
without
87.78
37.24
32.40
0.00
0.00
2/255
87.90
37.24
32.86
0.14
100.0
4/255
88.24
37.24
34.25
0.65
81.82
2400
8/255
88.48
37.24
52.77
27.22
14.07
16/255
56.47
37.24
96.43
86.49
7.62
32/255
41.40
37.24
98.18
89.86
7.40
without
87.10
35.23
34.60
0.00
0.00
2/255
87.10
35.23
34.60
0.00
0.00
4/255
87.10
35.23
34.60
0.00
0.00
4800
8/255
86.91
35.23
35.16
2.62
6.06
16/255
84.77
35.23
48.32
33.69
8.36
32/255
80.51
35.23
58.45
54.31
7.45
17

---

Under review as a conference paper at ICLR 2020
Table 11: Results of standard trained ResNet--0 against bounded PGD attack for ImageNet-1000
Metric
Attack
Protection
Acc [%]
Pre [%]]
Rec [%]
Rej [%]
PR[
without
98.80
94.00
94.00
0.00
0.00
2/255
94.59
94.00
100.0
88.19
0.67
4/255
94.12
94.00
100.0
89.09
0.67
clean
8/255
94.09
94.00
100.0
89.13
0.67
16/255
94.04
94.00
100.0
89.23
0.67
ε 32/255
94.00
94.00
100.0
89.29
0.67
without
88.86
43.23
36.40
0.00
0.00
2/255
72.95
43.23
92.39
80.58
7.46
4/255
60.43
43.23
98.91
87.12
7.20
300
ε 8/2550
46.05
43.23
100.0
90.42
6.98
16/255
43.50
43.23
100.0
90.81
6.95
32/255
43.23
43.23
100.0
90.85
6.94
without
88.80
42.92
36.40
0.00
0.00
2/255
88.59
42.92
50.28
25,81
10.61
4/255
88.07
42.92
55.32
34.52
9.83
600
8/2550
81.49
42.92
77.45
67.58
7.78
&
16/255
45.50
42.92
100.0
90.40
6.98
32/255
42.92
42.92
100.0
90.79
6.95
without
88.36
40.28
34.00
0.00
0.00
2/255
88.22
40.28
34.27
1.88
4.21
4/255
88.15
40.28
34.34
2.62
3.79
1200
8/255
88.00
40.28
35.49
6.43
6.48
16/255
82.23
40.28
61.82
59.35
7.52
ε 32/255
40.57
40.28
100.0
90.79
7.21
without
87.80
37.15
31.80
0.00
0.00
2/255
87.78
37.15
31.80
0.14
0,00
4/255
87.78
37.15
31.80
0.16
0.00
2400
8/255
87.77
37.15
31.80
0.22
0.00
16/255
87.62
37.15
32.45
3.08
6.45
32/255
68.99
37.15
80.71
79.56
7.56
without
85.86
29.42
29.60
0.00
0.00
2/255
85.85
29.42
29.60
0.10
0.00
4/255
85.85
29.42
29.60
0.10
0.00
4800
8/255
85.85
29.42
29.60
0.10
0.00
16/255
85.84
29.42
29.60
0.14
0.00
32/255
85.20
29.42
30.96
7.36
5.93
18

---

Under review as a conference paper at ICLR 2020
B.2 PROTECTION AGAINST NATURAL ADVERSARIAL EXAMPLES
Table 12: Results of adversarial trained ResNet-50 for ImageNet-A.
Metric
Protection
Acc [%]
Pre [%]
Rec [%]
Rej [%]
PR[%]
without
76.84
39.98
21.78
00.00
00.00
2/255
80.23
39.98
39.15
29.72
31.18
4/255
78.46
39.98
72.40
60.27
24.23
8/255
52.41
39.98
99.42
85.60
19.05
16/255
40.60
39.98
100.0
88,51
18.45
32/ 32/255
39.98
39.98
100.0
88.63
18.43
Table 13: Results of standard trained ResNet--50 for ImageNet-A.
Metric
Protection
Ace [%]
Pre [%]
Rec [%]
Rej [%]
PR[%]
without
77.69
42.88
20.56
00.00
00.00
2/255
42.88
42.88
100.0
89.99
18.43
4/255
42.88
42.88
100.0
89.99
18.43
ε 8/255
42.88
42.88
100.0
89.99
18.43
16/255
42.88
42.88
100.0
89.99
18.43
32/255
42.88
42.88
100.0
89.99
18.43
19


Reviews:
[{"review_id": "SylCPVKnFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper wants to study the problem of “learning with rejection under adversarial attacks”. It first naively extends the learning with rejection framework for handling adversarial examples. It then considers the classical cost-sensitive learning by transfer the multi-class problem into binary classification problem through one-vs-all and using the technique they proposed to reject predictions on non-important labels, and name such technique as “learning with protection”. Finally, they do some experimental studies. \n\nThe paper does not show any connection between “learning with rejection” and “adversarial learning”. The method it proposes is also a naïve extension of existing methods. Both the problem setting and the technique does not have novelty. The paper fails to realize that the motivated application is actually called “cost-sensitive learning” and has been studied long time before. The paper also has problems in writing. Finally, there is no comparison with any baseline. Only empirical results of the proposed methods are shown. Due to all these reasons, there is still a long way to go before the paper can be published. I will rate it a clear rejection.\n\nMore specially, \n\nThe definition of “suspicious example” in Sec.3.1 has no relationship with adversary examples. Does the paper focus on adversary examples? If the definition has no relationship, it is classical learning with rejection. \nIn the last equation of Page 3, there is no definition of \\tilde L. Actually, according to Figure 1, x’s is more close to the decision boundary, it is an example more hard to classify, which could also be “suspicious”. \nIn the definition of “suspicious example” at the beginning of Sec.3.1, is both x and x’ defined as suspicious examples in this way?\nIn the last equation of page 2, there is a rejection function, so minimizing this loss is a “separation-based approach”. However, at the end of Sec.2 the paper states they “follow a confidence-based approach”. Any comment on the inconsistency?\n\nThe motivated problem is not new. It is called cost-sensitive learning in machine learning and can date back to 2001:\nCharles Elkan. The Foundations of Cost-Sensitive Learning. IJCAI 2001: 973-978.\nWhere they study the same problem when misclassifying one class of data may cost a lot than misclassifying another class of data. The current paper has not discussed any related work of cost-sensitive learning although they want to study a problem in its field. \n\nThe paper should be also improved in writing in the following aspects. \nThere is a lot of inaccurate statements in the paper. For example,  “In Sections 3 and 4, we propose and describe our algorithm”, what is the difference between propose and describe? “an estimator \\hat h might return result that differ greatly from h^* in a case with finite samples”. Actually there are rigorous theoretical results describing how the number of finite samples will impact the estimator \\hat h on unseen data. For example, \nPeter L. Bartlett, Shahar Mendelson. Rademacher and Gaussian Complexities: Risk Bounds and Structural Results. JMLR, 2002.\nSo inaccurate/unclear statements that will mislead readers should be avoided. \n\nIn writing, the paper also lacks the necessary references in many places. For example, “Learning with rejection is a classification scenario where the learner is given the option to reject an instance instead of predicting its label.”, “…classifies adversary attacks to two types of attacks, white-box attack and black-box attack.”, “Methods for protecting against these adversarial examples are also being proposed.”. Necessary references are needed for these places.\n\nThe organization is also problematic. For example, in the second half of Sec.2 introducing two kinds of learning with rejection models, it should be included in a “related work” part. \n\n------------------------------------------\nThank you for the rebuttal. I raised my score a little bit. But I still think this paper has not been ready to be published yet.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Sye1YXKatH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a framework for learning with rejection using ideas from adversarial examples. The essential idea is, while predicting on a point x, we can reject classifying the point if it has an adversarial example very close to it. So, the algorithm can be simply summarized as,\n1. Learn a classifier function f\n2. On the test set, predict on a point, only if it doesn't have an adversarial example close by.\n\nI am inclined to reject the paper for the following reasons:\n1. The proposed approach is a variation of a fairly well-known heuristic. Having a close adversarial example is same as saying that the current point is very close to the decision boundary. Being close to the decision boundary is a heuristic that has been applied in multiple scenarios in machine learning.\n2. The proposed approach is not novel. For example, [1] uses adversarial example style detection to augment their training data and improve their end-to-end model. \n3. There have been approaches which attempt to learn rejection function [2], so it would have been good to at least do a comparison of the proposed approach with such methods.\n\n[1] Adversarial Examples For Improving End-to-End Attention-based Small-Footprint Keyword Spotting, ICASSP 2019\n[2] SelectiveNet: A Deep Neural Network with an Integrated Reject Option, ICML 2019\n\n---\n\nThanks for the rebuttal. I have raised my scores, but I still believe that this paper falls short of acceptance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xcdBLAtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "There is still no universal method to deal with adversarial examples, and introducing a reject option to flag potential attacks seems a sensitive choice for many applications. The considered problem is well-motivated and introduced, and I’m unaware of prior work studying classification with reject option in the context of adversarial examples. However, I think there are different dimensions along which the paper could be improved:\n\n- My understanding of classification with a reject option is that the rejection cost c(x) is a design choice that can depend on the specific application. While c(x) is introduced as part of the framework it is then derived in a very specific way, removing the design aspect or at least not explaining very clearly how one would design it. Also, Algorithm 1 doesn’t have corresponding parameters.\n\n- The rejection function r(x) relies on z^* which essentially amounts to computing an adversarial perturbation for a given testing point, see (2). The authors state that they use a 30-step of the PGD algorithm to find z^*. The attacks on which the method is tested only uses 10 PGD steps. What if the attacker is stronger in the sense that it runs significantly more PGD iterations than used to compute the rejection function? How sensitive is the rejection function to different initializations?\n\n- Similarly, what happens if the attacker and the rejection function rely on different norms to compute the attack and the rejection score, respectively? I think it is important to investigate this aspect.\n\n- In Tables 1 and 2, I don’t understand why the precision is the same for all rows corresponding to the same attack (strength), while the other metrics vary.\n\nOverall, I think the paper explores an interesting direction, but would greatly benefit from a revision along the lines outlined above.\n\nMinor comments:\n- P3 3.1 2nd paragraph: “Let B^p...” rather than “Let B^\\infty...”\n- P4 bottom: “)” is missing before “..., where”\n- P6 bottom: (FR) should be (TR). The acronyms FA and FR don’t seem to be introduced.\n\n\n\n###\nReply to rebuttal:\n\nI thank the reviewers for their detailed reply. It seems that the authors agree that some of the issues I raised should be addressed and improved. My assessment that the paper should be revised (and accordingly the rating) therefore remains unchanged.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the setting of learning with rejection while incorporating the ideas from learning with adversarial examples to tackle adversarial attacks. While the reviewers acknowledged the importance to study learning with rejection in this setting, they raised several concerns: (1) lack of technical contribution -- see R1’s and R2’s related references, see R3’s suggestion on designing c(x); (2) insufficient empirical evidence -- see R3’s comment about the sensitivity experiment on the strength of the attack, see R1’s suggestion to compare with a baseline that learns the rejection function such as SelectiveNet;  (3) clarity of presentation -- see R2’s suggestions how to improve clarity.
Among these, (3) did not have a substantial impact on the decision, but would be helpful to address in a subsequent revision. However, (1) and (2) make it very difficult to assess the benefits of the proposed approach, and were viewed by AC as critical issues.
AC can confirm that all three reviewers have read the author responses and have revised the final ratings. AC suggests, in its current state the manuscript is not ready for a publication. We hope the reviews are useful for improving and revising the paper.

Decision: Reject