Learning with Protection: Rejection of Suspicious Samples under Adversarial Environment

1 INTRODUCTION . Recent machine learning models have achieved great success in applications such as self-driving cars ( Bojarski et al. , 2016 ) and medical diagnosis ( Litjens et al. , 2017 ) . In a classification problem , we sometimes face the situation where the misclassification of a specific label can result in serious accidents . This problem arises in various situations , and below we provide examples . Example 1 ( Object Detection for Self-Driving Cars ) . For self-driving cars , object detection is crucial . For protecting the life of pedestrians , self-driving cars need to be able to detect human or suspend decision making if it detects an object that might be a human . Example 2 ( Medical Diagnosis ) . Cancer is one of the leading causes of death globally , and machine learning algorithms for detecting cancer from images have been gaining attention . This is done by detecting abnormalities in an image that might indicate cancer . As shown above , we want the labeling of data to be conservative for certain situations ; for example , in autonomous driving , it is better to misclassify a non-human object as a human than to risk misclassifying a human . In this paper , we introduce two novel frameworks called learning with rejection under adversarial attacks and learning with protection to protect a specific class from critical misclassification . Our algorithm is based on the techniques of learning with rejection and adversarial examples . Learning with rejection under adversarial attacks is a naive extension of learning with rejection to hold a decision when we face a suspicious sample that is vulnerable to adversarial attack . Learning with protection is an application of learning with rejection under adversarial attack . The purpose is to protect a specific class from misclassification . Both frameworks are based on the existing work of learning with rejection and adversarial examples . Learning with rejection is a classification scenario where the learner is given the option to reject an instance instead of predicting its label . The purpose of this framework is to prevent critical misclassification , where rejecting labeling a datum incurs a lower cost than misclassification . In this field , considerable theoretical and empirical analysis has been conducted ( Bartlett & Wegkamp , 2008 ; Cortes et al. , 2016a ; b ; Grandvalet et al. , 2009 ; Herbei & Wegkamp , 2006 ; Yuan & Wegkamp , 2010 ) . Learning with rejection is one ultimate solution against data with high uncertainty . However , the existing frameworks have not considered adversarial examples , which refers to a well-designed attack to mislead people to misclassification . In the industrial application of machine learning , it has been observed that many machine learning models are vulnerable to adversarial attacks . Threat model of adversarial attacks specifies the capabilities of the adversary and classifies adversary attacks to two types of attacks , white-box attack and black-box attack . Adversarial inputs that result in machine learning models returning incorrect outputs are called adversarial examples . Several previous studies have artificially generated adversarial examples by adding small perturbations that are imperceptible to humans ( Goodfellow et al. , 2015 ; Gu & Rigazio , 2014 ; Huang et al. , 2015 ; Carlini & Wagner , 2017 ) . Methods for protecting against these adversarial examples are also being proposed . Among them , adversarial training is one of the most effective defenses ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Shaham et al. , 2015 ; Carlini & Wagner , 2016 ; Papernot et al. , 2016 ; Xu et al. , 2017 ; Madry et al. , 2018 ; Buckman et al. , 2018 ; Kannan et al. , 2018 ; Pang et al. , 2018 ; Wong & Kolter , 2018 ; Tramèr et al. , 2018 ) . However , most of the defense method fails to avoid misclassification under adversarial attacks . Carlini & Wagner ( 2017 ) defeated representative methods for detection of adversarial examples . Athalye et al . ( 2018 ) reports that several defense algorithms for white-box setting fail when the attacker uses a carefully designed gradient-based method . Besides , Shafahi et al . ( 2019 ) ; Gilmer et al . ( 2019 ) shows that adversarial examples are inevitable in some cases . However , we sometimes face situations in which misclassification can result in fatal accidents . For that purpose , we propose learning with rejection option under adversarial attack . The main idea of our method is to hold a decision when we get suspicious samples . As an application of learning with a rejection option , we propose a method for preventing the misclassification of data for a specific label under adversarial examples . This method considers situations where we must protect our model from adversarial examples by any means . In addition , It should be noted that our method can be combined with other existing methods such as adversarial training . In the following section , we describe the problem setting . In Sections 3 and 4 , we propose and describe our algorithm , and in Section 5 , we show the experimental results . 2 PROBLEM SETTING . We use the standard settings for binary classification problems , i.e. , training and test data points are i.i.d . and sampled from some unknown distribution D over X × { −1 , +1 } , where X denotes the input space . The goal is to learn a classifier h : X → { −1 , +1 } that assigns a label ŷ ( x ) to a datum x as ŷ ( x ) = sign ( h ( x ) ) . Let L be a loss function L : R× { −1 , +1 } → R. The optimal classifier h∗ is given by h∗ = argminh∈F E [ L ( h ( X ) , Y ) ] , where F is a set of measurable functions . When we train a classifier with training samples , we can naively replace the expectation with the corresponding sample averages . For a hypothesis set H , let us define an estimator of the optimal classifier h∗ as ĥ = argminh∈H Ê [ L ( h ( X ) , Y ) ] , where Ê denotes the averaging operator over training data . Although an estimator ĥ converges to h∗ in many cases with infinite samples , an estimator ĥ might return result that differ greatly from h∗ in a case with finite samples . In addition , when h ( x ) is not smooth at around x , it might become difficult to estimate the function . These can cause serious problems in real-world applications , such as traffic accidents arising from misclassification by the algorithms used in self-driving cars . To make our algorithm ’ s inference more robust , we allow a learner to reject such suspicious samples . Let R© denote rejection . For any given instance x ∈ X , the learner has the option of abstaining from assigning a label or rejecting that instance and returning the symbol R© , or assigning the label ŷ ∈ { −1 , +1 } . If the learner rejects an instance , then they incur some loss c ( x ) ∈ R ; if it does not reject but assigns an incorrect label , then it incurs a cost of one ; otherwise , it suffers no loss . Thus , the learner outputs a rejection function r : X → R , that determines the points x ∈ X to be rejected according to r ( x ) ≤ 0 . Let us denote a loss function with rejection option L R© : R× { −1 , +1 } → R. For example , a loss function can be defined as follows : L R© ( h ( x ) , y ) = 1yh ( x ) ≤01r ( x ) ≥0 + c ( x ) 1r ( x ) ≤0 . There are two frameworks used in the learning with rejection model : the confidence-based and separation-based approaches Cortes et al . ( 2016a ) . The separation-based approach is first formulated by Cortes et al . ( 2016a ) . This formulation is a generalization of confidence-based approach ( Bartlett & Wegkamp , 2008 ) . In the separation-based approach , we train a classifier and a rejection function simultaneously . In confidence-based approach , given the conditional probability p ( y = +1|x ) and a cost function c ( x ) , we only train a classifier and obtain a rejection function automatically after training a classifier . Although the confidence-based approach is a special case of the separation-based approach , the method of confidence-based approach is easy to implement . In this paper , we follow the confidence-based approach and correspond the cost function to vulnerability for adversarial examples . Our goal is to obtain the optimal classifier h∗ , which minimizes the classification risk with rejection option , i.e. , h∗ = argminh∈H E [ L R© ( h ( X ) , Y ) ] . 3 LEARNING WITH REJECTION UNDER ADVERSARIAL ATTACKS . In this section , we describe a method to hold a decision against suspicious samples . 3.1 STRATEGY FOR REJECTING SUSPICIOUS SAMPLES . To reject suspicious samples , we define which samples are suspicious . We introduce a concept of a p-norm suspicious sample . Let ‖ · ‖p be ` p-norm . We refer a sample x as suspicious based on the value of ` p-norm if there exits a sample x′ such that ‖x − x′‖p ≤ ε and ĥ ( x′ ) returns a different class from ĥ ( x ) . If a sample is suspicious , we refrain from assigning a class label . Suppose that we have a estimator f̂ ( x ) of the class conditional probability η ( x ) = p ( y = +1|x ) and a classifier ĥ ( x ) = sign ( f̂ ( x ) − 12 ) . Let B∞x ( ) denote the ` p ball centered at x ∈ X with radius , i.e. , Bpx ( ) = { x′ ∈ X : ‖x′ − x‖p ≤ } . We calculate z∗ ( x ) ∈ Bpx ( ) as follows : z∗ ( x ) = argmax x′∈Bpx ( ) L̃ ( ĥ ( x′ ) , x ) . By defining z∗ ( x ) as in the above equation , we can easily calculate it with the techniques used in adversarial examples . If ĥ ( z∗ ( x ) ) ĥ ( x ) ≤ 0 for a sample x , we regard the sample as suspicious one . We illustrate the concept in Figure 1 . 3.2 RELATIONSHIP WITH LEARNING WITH REJECTION . To interpret the above strategy , we consider the relationship between the strategy and learning with rejection . First , we consider the ideal situation , where we know the true value of the conditional class probability η ( x ) = p ( y = +1|x ) . In confidence-based approach , we can determine the rejection function r ( x ) corresponding to the rejection cost as shown by Cortes et al . ( 2016a ) . It is known that the classifier h∗ defined for any x ∈ X by h∗ ( x ) = η ( x ) − 12 is optimal . For any x ∈ X , the misclassification cost for h∗ is E [ 1yh ( x ) ≤0|x ] = min { η ( x ) , 1− η ( x ) } . The optimal rejection function r∗ ( x ) should therefore be defined such that r∗ ( x ) ≤ 0 if and only if min { η ( x ) , 1− η ( x ) } ≥ c ( x ) ⇔ 1−max { η ( x ) , 1− η ( x ) } ≥ c ( x ) ⇔ |h ( x ) | ≤ 1 2 − c ( x ) . Therefore , the optimal rejection function r∗ ( x ) is given as r∗ ( x ) = |η ( x ) − 12 | − ( 1 2 − c ( x ) ) . Thus , in confidence-based approach , we can determine the optimal rejection function r∗ ( x ) if we know the conditional probability η ( x ) and the rejection cost c ( x ) . To reject suspicious samples , we relate the rejection cost c ( x ) with uncertainty of data x . One naive idea is to set c ( x ) as a monotonically decreasing function representing the smoothness of η ( x ) . When |η ( x ) − η ( x′ ) | is zero for x , x′ ∈ Rd , c ( x ) takes its largest value ; when max |f ( x ) − f ( x′ ) | is large , c ( x ) is small . This means that , if a sample x is suspicious , then the cost of rejection is low , and we can reject the sample easily ; conversely , if a sample x is not suspicious , then the cost of rejection is high . When the cost is high , we hesitate to reject the sample . We assumed this condition could be met when designing our algorithm . As we show later , we can easily find such c ( x ) and reject suspicious samples without expending much energy in determining the value of c ( x ) . In our strategy , we reject a sample x such that ( η ( z∗ ( x ) ) − 1 2 ) ( η ( x ) − 1 2 ) ≤ 0 . Therefore , r∗ ( x ) ≤ 0 if and only if ( η ( z∗ ( x ) ) ≤ 1 2 and η ( x ) ≥ 1 2 ) or ( η ( z∗ ( x ) ) ≥ 1 2 and η ( x ) ≤ 1 2 ) ⇔ ( η ( x ) − η ( x ) + η ( z∗ ( x ) ) ≤ 1 2 and η ( x ) ≥ 1 2 ) or ( η ( x ) − η ( x ) + η ( z∗ ( x ) ) ≥ 1 2 and η ( x ) ≤ 1 2 ) ⇔ ( 0 ≤ η ( x ) − 1 2 ≤ η ( x ) − η ( z∗ ( x ) ) ) or ( η ( x ) − η ( z∗ ( x ) ) ≤ η ( x ) − 1 2 ≤ 0 ) ( 1 ) Let us assume that η ( x ) − η ( z∗ ( x ) ) ≥ 0 when η ( x ) − 12 ≥ 0 and η ( x ) − η ( z ∗ ( x ) ) ≤ 0 when η ( x ) − 12 ≤ 0 . Then , ( 1 ) insists that r ∗ ( x ) ≤ 0 if and only if |h ( x ) | ≤ |η ( x ) −η ( z∗ ( x ) ) | . Therefore , the following relationship holds for a sample x such that |η ( x ) − η ( z∗ ( x ) ) | ≤ 12 : c ( x ) = 1 2 − |η ( x ) − η ( z∗ ( x ) ) | . Thus , in our strategy , the cost function is a decreasing function of the smoothness of η ( x ) . Next , we show a relationship between the cost function and the degree of perturbation ε . Let us assume η ( x ) is Lipschitz continuous , with ` p-norm , and |η ( x ) − η ( x′ ) | ≤ λ‖x− x′‖p = λε , wherex ∈ X , x′ ∈ Bpx ( ) , and λ > 0 is a positive constant . From Lipschitz continuity , c ( x ) ≥ 12−λε holds . This means that the cost of rejection will be small when ε is a large value . Therefore , a learner can reject a sample with low cost when the learner is afraid of the misclassification and set the possible perturbation of x with ` p-norm large .

Summary: This paper wants to study the problem of “learning with rejection under adversarial attacks”. It first naively extends the learning with rejection framework for handling adversarial examples. It then considers the classical cost-sensitive learning by transfer the multi-class problem into binary classification problem through one-vs-all and using the technique they proposed to reject predictions on non-important labels, and name such technique as “learning with protection”. Finally, they do some experimental studies.