Thwarting finite difference adversarial attacks with output randomization

1 INTRODUCTION . The success of deep neural networks has led to scrutiny of the security vulnerabilities in deep neural network based models . One particular area of concern is weakness to adversarial input : carefully crafted inputs that resist detection and can cause arbitrary errors in the model ( 1 ; 2 ) . This is especially highlighted in the domain of image classification , where an adversary creates an image that resembles a natural image to a human observer but easily fools deep neural network based image classifiers ( 3 ) . Different types of adversarial attacks exists throughout the lifecycle of a deep neural network model . For example , adversaries can attack a model during training by injecting corrupting data into the training set or by modifying data in the training set . However , inference time attacks are more worrisome as they represent the bulk of realistic attack surfaces ( 4 ; 5 ; 6 ) . The input created under an inference time attack is known as an adversarial example and methods for generating such examples have recently attracted much attention . In most cases , the adversarial example is created by perturbing a ( correctly classified ) input such that the model commits an error . The perturbation applied to the `` clean '' input is typically constrained to be undetectably small ( 7 ; 8 ; 9 ) . Defending againsts adversarial attacks on deep neural networks is crucial and has seen relatively slow progress compared to the sophistication and progress of adversarial attacks ( 10 ; 11 ; 12 ; 13 ; 14 ; 15 ) . This is because it is difficult to prove a defense can withstand the types of attacks it is designed for especially when the defense must prove capable of withstanding `` adaptive '' adversaries that have knowledge of the details of the defense ( 16 ) . A defense against adversarial attacks is defined by the threat model it is designed to defend against ( 16 ) . The most permissive threat model makes the weakest assumptions about the adversary . One such threat model can be assuming the adversary has complete knowledge of the model , including architecture and parameters of the underlying network . This is known as the `` white-box '' setting . Other threat models can also be useful , such as assuming the adversary has knowledge of the network architecture but not the parameters of the model . More restrictive threat models allow only so called “ black-box ” attacks , attacks that can create adversarial examples without having access to the model architecture or weights and only accessing some form of output of the model . The rest of the paper is organized as follows . In Section 2 , we discuss the threat model considered . In Section 3 , we describe valid black box attacks under the threat model . Output randomization as a defense is described in Section 4 . We show empirical results in Section 5 , cover related approaches in Section 6 , and conclude in Section 7 . 2 THREAT MODEL . Adversary goals : The goal of the adversary is to force the classifier to commit an error within a distortion limit , such that the example crafted by the adversary is similar to the original example . The adversary induces such an error in an untargeted or targeted attack . The goal of an untargeted attack is misclassification of an input , whereas the goal of a targeted attack is misclassification of an input as a class specified by the adversary . Adversary knowledge : The adversary has access to the model only at the input and output level , and has no knowledge of its architecture or parameters . This black-box adversary is aware of the details of the defenses protecting the model and the type of randomness associated with any defense but not the exact random numbers generated . Adversary capability : The adversary only has access to the model by providing examples as input and observing the output probability vector generated by the model as output . The adversary can modify aspects of the input in any way as long as it remains similar to the original input . This similarity is controlled by an lp distortion penalty on the adversarial image . Common choices for the distortion penalty include l1 , l2 and l∞ . We use the l2 perturbation penalty as this type of attack results in the strongest attacks ( 17 ) . Attackers are allowed to query the model up to a maximum limit , which we increase to our computational limit to strengthen attacks . In addition to a maximum query limit , attackers often use an early stopping parameter to avoid wasting computation on unpromising direction during optimization . We also increase this parameter to test our defense against stronger attacks . 3 EXISTING BLACK BOX ATTACKS . Black box attacks are called “ gradient-free ” attacks since they do not involve computing gradients of the input by backpropagation on the model under attack . Instead the gradients of the input are estimated by using the finite difference estimate for each input feature . Black box attacks can be categorized by the type of output the adversary is allowed to observe : logit layer , class probabilities/softmax layer , or top k class probabilities/labels . In general , designing successful black box attacks in the label only setting is much harder than the setting where the logit layer or softmax layer is available to the attacker . We consider the easiest setting for black box attacks , where all the class probabilities are available . 3.1 ZOO BLACK BOX ATTACK . The Zeroth Order Optimization based black-box ( ZOO ) attack ( 18 ) is a method for creating adversarial examples for deep neural networks that only requires input and output access to the model . ZOO adopts a similar iterative optimization based approach to adversarial example generation as other successful attacks , such as the Carlini & Wagner ( C & W ) attack ( 17 ) . The attack begins with a correctly classified input image x , defines an adversarial loss function that scores perturbations δ applied to the input , and optimizes the adversarial loss function using gradient descent to find δ∗ that creates a succesful adversarial example . Specifically , gradient descent is used to find δ∗ such that : f ( x+ δ∗ ) = ya ‖x− ( x+ δ∗ ) ‖ ≤ Namely , that the perturbed input x + δ∗ successfully fools the classifier f to predict the incorrect class ya and that the perturbed input is similar to the original input up to some distortion limit . The primary ( and strongest ) adversarial loss used by the ZOO attack for targeted attacks is given by : L ( x , t ) = max { max i 6=t { log f ( x ) i − log f ( x ) t } , −κ } ( 1 ) Where x is an input image , t is a target class , and κ is a tuning parameter . Minimizing this loss function over the input x causes the classifier to predict class t for the optimized input . For untargeted attacks , a similar loss function is used : L ( x ) = max { log f ( x ) i −max j 6=i { log f ( x ) j } , −κ } ( 2 ) where i is the original label for the input x . This loss function simply pushes x to enter a region of misclassification for the classifier f . In order to limit distortion of the original input , the adversarial loss function is combined with a distortion penalty in the full optimization problem . This is given by : min x ‖x− x0‖22 + c · L ( x , t ) subject to x ∈ [ 0 , 1 ] n In the white box setting , attackers can take advantage of the backprogation algorithm to calculate the gradient of the adversarial loss function with respect to the input coordinates ( δLδxi ) and solve the optimization problem using gradient descent . In lieu of this , ZOO uses `` zeroth order stochastic coordinate descent '' to optimize input on the adversarial loss directly . This is most easily understood as a finite difference estimate of the gradient of the input with the symmetric difference quotient ( 18 ) : δL δxi ≈ gi : = L ( x+ hei ) − L ( x− hei ) 2h with ei as the basis vector for coordinate/pixel i and h set to a small constant . The ZOO attack uses this approximation to the gradients to create an adversarial example from the given input . Note that for an image with n pixels , computing an estimate of the gradient with respect to each pixel requires 2n queries to the model . Since this is usually prohibitive , the ZOO attack circumvents this by only estimating the gradients for a subset of coordinates at each step . ZOO also uses dimensionality reduction and a hierarchical approach to further increase the efficiency of the attack and show empirically that these methods are effective ( 18 ) . 3.2 QUERY LIMITED ( QL ) BLACK BOX ATTACK . A similar approach to ZOO is adopted by ( 19 ) in a query limited setting . Like ZOO , the QL attack estimates the gradients of the adversarial loss using a finite difference based approach . However , the QL attack reduces the number of queries required to estimate the gradients by employing a search distribution . Natural Evolutionary Strategies ( NES ) ( 20 ) is used as a black box to estimate gradients from a limited number of model evaluations . Projected Gradient Descent ( PGD ) ( 12 ) is used to update the adversarial example using the estimated gradients . PGD uses the sign of the estimated gradients to perform an update : xt = xt−1−η · sign ( gt ) , with a step size η and the estimated gradient gt . The estimated gradient for the QL attack using NES is given by : gt = m∑ i=1 L ( x+ σ · ui ) · ui − L ( x− σ · ui ) · ui 2mσ where ui is sampled from a standard normal distribution with the same dimension as the input x , σ is the search variance , and m is the number of samples used to estimate the gradient . The difference between this approach and ZOO is that ZOO attempts to estimate the gradient with respect to one coordinate at a time while this approach averages over perturbations to many coordinates to estimate the entire gradient directly . In the next section , we show that applying a simple randomization function ( that does not affect model accuracy ) to the output of a model causes these types of attacks to fail even if the attack is adapted to the specific randomization function . 4 THWARTING BLACK BOX ATTACKS . The intuition behind output randomization is that a model may deliberately make errors in its predictions in order to thwart a potential attacker . This simple idea introduces a tradeoff between accurate predictions and the effectiveness of finite difference based black box adversarial attacks . Output randomization for a model that produces a probability distribution over class labels replaces the output of the model p by a stochastic function d ( p ) . The function d must satisfy two conditions : 1 . The probability of misclassifying an input due to applying d is bounded by K 2 . The vector d ( p ) prevents adversaries under the given threat model from generating adversar- ial examples . The first condition ensures that the applied defense minimally impacts non-adversarial users of the model , such as users of an online image classification service . The effectiveness of the defense comes from satisfying the second condition as the introduced randomness must prevent an adversary ( in the appropiate setting ) from producing an adversarial example . In the following two sections , we consider a simple noise-inducing function d ( p ) = p+ where is a random variable .

Summary: This paper proposes applying randomization to the output layer of a DNN to defend against query-based attacks based on finite difference estimates. Then some theoretical analysis is provided, showing that with perturbation of a suitable scale, the randomization layer will not affect the accuracy of the model, while causing a large estimation error of finite difference methods that prevents finite-difference based attacks. Empirical results verify that the proposed defense is still effective against adaptive attacks where the randomness is averaged.