Towards Certified Defense for Unrestricted Adversarial Attacks

1 INTRODUCTION . Classifiers trained using machine learning can often achieve high accuracy on test data coming from the same distribution as the training one . Unfortunately , for a class of inputs called adversarial examples ( Szegedy et al. , 2014 ) , many classifiers can be fooled to give wrong predictions with high confidence . This poses a significant security risk for real world deployment of machine learning models . To mitigate this threat , many methods have been proposed to defend against adversarial examples ( see , e.g. , Papernot et al . ( 2016 ) ; Song et al . ( 2018a ) ; Ma et al . ( 2018 ) ; Buckman et al . ( 2018 ) ) . However , empirically grounded methods are often found to be ineffective using newer attack methods that they had originally not considered ( Carlini & Wagner , 2017 ; Athalye et al. , 2018 ) . To end this arms race , defenses with certified theoretical guarantees against a threat model have recently gained more traction ( see , e.g. , Raghunathan et al . ( 2018 ) ; Wong & Kolter ( 2017 ) ; Dvijotham et al . ( 2018 ) ; Cohen et al . ( 2019 ) ) . Existing certified defense methods are all restricted to perturbation-based attacks ( Song et al. , 2018a ) , a threat model where the attacker is only allowed to generate an adversarial example by perturbing existing data points using small adversarial noise . In addition , a certified defense for one type of perturbation ( e.g. , based on l2 norm ) can still be vulnerable to a different type of perturbation ( e.g . spatial transform ( Xiao et al. , 2018 ) ) . Following Song et al . ( 2018b ) and Brown et al . ( 2018 ) , we consider the much more general threat model of unrestricted adversarial attacks , where the small perturbation restriction is removed : any input is considered a valid adversarial example as long as it induces the classifier to predict a different label than an oracle classifier . As shown in Song et al . ( 2018b ) , existing certified defenses are not robust to unrestricted adversarial attacks . In this paper , we provide a defense strategy with ( asymptotically ) certified guarantees against unrestricted adversarial attacks . The attacker can use any strategy to choose any input , and the attack is successful as long as the input is confidently classified into the wrong class by the target classifier . Note that in the white-box setup , where the attacker has access to all parameters of the classifier , defending against unrestricted adversarial examples is essentially impossible unless the classifier is perfect , because the attacker can simply enumerate all possible inputs to find adversarial ones ( we do not assume any computational restriction of the attacker ) . As a result , we focus our discussion on black-box unrestricted attacks , where the attacker does not know the exact parameters of the model nor the exact training data ( otherise , the attacker could recover the parameters by simulating the learning algorithm ) , but can query the model for a limited number of times before making unrestricted adversarial examples . For example , for online services , it is easy to limit the number of queries allowed per user . Consider a classifier learned using empirical risk minimization . Because the training data are randomly drawn from the data-generating distribution , the parameters of the classifier will be random even if the training algorithm is deterministic . Since we assume a black-box setting where the attacker does not know the learned parameters or the training dataset , the attacker has to obtain extra information by querying the model , even though the exact training algorithm and even the data-generating distribution might be known to the attacker . As a result , we can adopt methods similar to those used in differential privacy ( Dwork et al. , 2014 ) to reduce the amount of information about the parameters leaked from each query issued by the attacker . We give concrete lower bounds on the minimum number of queries an attacker needs in order to successfully generate adversarial examples . For most classifiers the guarantees are asymptotic , meaning that the guarantees are valid if the number of training examples is sufficiently large . For very simple classifiers we prove finite sample guarantees . Our bound is also asymptotically minimax optimal with respect to important parameters of the problem — the dimension of the input and the size of the training set . We prove this by exhibiting a class of classifiers and a concrete attacking algorithm that successfully generates adversarial examples while querying no more than the minimum necessary according to the lower bounds ( up to a constant ) . Experimentally , we show that the defense strategy is effective against strong black-box unrestricted attacks , even when we use complex classifiers for which no theoretical guarantee with finite sample size is available . Compared to models without any defense , the number of queries needed to identify vulnerabilities often increases by an order of magnitude using our defense strategy , with negligible effect on the accuracy on “ clean ” inputs . Providing finite sample guarantees for complex models such as deep neural networks is an open problem for future work . 2 PROBLEM DEFINITION . 2.1 CLASSIFICATION . Given a training dataset { ( xi , yi ) ∈ X × Y } ni=1 i.i.d.∼ p∗ ( x ) p∗ ( y | x ) , we consider the task of learning a classifier to predict the label y for an input x . To simplify our discussion , we assume that X = [ −1 , 1 ] d , and Y = { −1 , 1 } . Let { g ( w , · ) : X → R } be a class of score functions with parameter w , whose signs determine the class labels , i.e. , y = sign ( g ( w , x ) ) , where we define sign ( x ) = 1 when x ≥ 0 , and sign ( x ) = −1 otherwise . The model family is assumed to be well-specified , i.e. , there exists a ground-truth parameter w∗ such that the true labeling function is given by y = sign ( g ( w∗ , x ) ) . We can use any learning algorithm to obtain w based on a training dataset . Since the training data are randomly generated from the underlying distribution p∗ ( x ) p∗ ( y | x ) , w is also a random vector and we denote its distribution as plearn ( w ) . With a reasonable learning algorithm , plearn ( w ) should be increasingly concentrated around w∗ while the size of the training dataset n grows . When deploying the classifier to real world applications , we might want to use a ( possibly stochastic ) surrogate score function f ( w , · ) : X → R which depends on the learned parameter w of our model . Usually , it is just g ( w , · ) , but to mitigate adversarial attacks we might choose f ( w , · ) to be some special classifier with a defense strategy . To avoid low confidence incorrect predictions , we give no response when the confidence is low . Specifically , we allow a threshold α ∈ R+ such that given an input query x , the predicted label is y = 1 , if f ( w , x ) > α −1 , if f ( w , x ) < −α No response , otherwise . ( 1 ) 2.2 PERFORMANCE EVALUATION . We evaluate the expected performance of the classifier f ( w , x ) on a test distribution q ( x ) p∗ ( y | x ) . The traditional setting in statistical machine learning assumes that q ( x ) = p∗ ( x ) . In adversarial settings , however , q ( x ) is the distribution of adversarial examples and is very different from p∗ ( x ) . Since we consider unrestricted attacks , we assume no relationship between q ( x ) and p∗ ( x ) . The classifier f ( w , x ) can make two kinds of errors for a given input x . First , f ( w , x ) might give no response . We call this the no-response error , which is denoted as NR ( x ; α ) , I ( |f ( w , x ) | ≤ α ) , where I [ P ] is the indicator function whose value is 1 when the property P holds ; other it produces 0 . Secondly , f ( w , x ) can produce a wrong prediction with high confidence . We call this the margin error , which is defined as ME ( x ; α ) = I ( g ( w∗ , x ) < 0 ∩ f ( w , x ) > α ) + I ( g ( w∗ , x ) > 0 ∩ f ( w , x ) < −α ) . An ideal classifier f ( w , x ) should minimize the expected no-response error Eq ( x ) [ NR ( x ; α ) ] and margin error Eq ( x ) [ ME ( x ; α ) ] simultaneously . When defending against an adversary , the margin errors are more important because a confident but wrong prediction is arguably more harmful than giving no prediction at all . Consequently , we only focus on bounding the margin errors achievable by attackers in the sequel . We will only consider no response error for clean data x ∼ p∗ ( x ) . 2.3 THE ATTACK-DEFENSE PROTOCOL . In Figure 1 , we provide a specific protocol of attack and defense to capture our assumptions of the threat model . Since we consider the threat model of unrestricted adversarial attacks , we assume the attacker to be fully general : The attacker is allowed to use any adaptive set of intermediate distributions { qt ( x | o1 : t−1 ) } Tt=1 to issue queries based on all previous observations . For the final attack distribution q , the attacker is allowed to sample it from any distribution of distributions Qattack ( q | o1 : T ) . We call a specific choice of { qt ( x | o1 : t−1 ) } Tt=1 andQattack an attack strategy , and a specific choice of plearn ( w ) and f ( · , · ) a defense strategy . We additionally allow the attacker to know the exact defense strategy . Note that when the attack and defense strategies are fixed beforehand , the attack-defense protocol defines a joint probability over w , o1 : T and q . The protocol can be modified in the defender ’ s favor if defender only sends yt according to Eq . ( 1 ) instead of zt . However , using zt allows normal users ( sampling from p∗ ( x ) ) to get a confidence score , and any certified defense results are more general ( they are still true if attacker only has access to yt ) . Intuitively , the attack strategy is successful under the attack-defense protocol , if with high probability the final attack distribution q incurs large expected margin errors for f ( w , x ) . To formalize this intuition , we define a winning strategy of the attacker as follows . Definition 1 . For a given defense strategy , an attack strategy in the attack-defense protocol is an ( α , γ , δ ) -winning strategy if with probability at least γ , the attacker outputs a q such that Prq ( x ) [ ME ( x ; α ) ] ≥ δ . In the following part , we will propose a concrete defense strategy and prove that no winning strategies can exist under some conditions .

Summary: The authors propose a new certified defense strategy that considers unrestricted black box attacks.  The paper provides bounds for the minimum number of queries needed for the attacker to attack the classifier successfully and then the authors prove that they can devise a defender to be robust against that attack. Here are a few points to consider: