Certified Defenses for Adversarial Patches

1 INTRODUCTION . Despite the great success of neural networks for vision problems , they are easily fooled by adversarial attacks in which the input to a machine learning model is modified with the goal of manipulating its output . Research in this area is largely focused on norm-bounded attack ( Madry et al. , 2017 ; Tramèr & Boneh , 2019 ; Shafahi et al. , 2019 ) , where the adversary is allowed to perturb all pixels in an image provided that the ` p-norm of the perturbation is within prescribed bounds . Other adversarial models were also proposed , such as functional ( Laidlaw & Feizi , 2019 ) , rotation/translation ( Engstrom et al. , 2017 ) , and Wasserstein ( Wong et al. , 2019 ) , all of which allow modification to all pixels . Whole-image perturbations are unrealistic for modeling ” physical-world ” attacks , in which a realworld object is modified to evade detection . A physical adversary usually modifies an object using stickers or paint . Because this object may only occupy a small portion of an image , the adversary can only manipulate a limited number of pixels . As such , the more practical patch attack model was proposed ( Brown et al. , 2017 ) . In a patch attack , the adversary may only change the pixels in a confined region , but is otherwise free to choose the values yielding the strongest attack . The threat to real-world computer vision systems is well-demonstrated in recent literature where carefully crafted patches can fool a classifier with high reliability ( Brown et al. , 2017 ; Karmon et al. , 2018 ) , make objects invisible to an object detector ( Wu et al. , 2019 ; Lee & Kolter , 2019 ) , or fool a face recognition system ( Sharif et al. , 2017 ) . In light of such effective physical-world patch attacks , very few defenses are known to date . In this paper , we study principled defenses against patch attacks . We begin by looking at existing defenses in the literature that claim to be effective against patch attacks , including Local Gradient Smoothing ( LGS ) ( Naseer et al. , 2019 ) and Digital Watermarking ( DW ) ( Hayes , 2018 ) . Similar to what has been observed for whole-image attacks by ( Athalye et al. , 2018 ) , we show that these patch defenses are easily broken by stronger adversaries . Concretely , we demonstrate successful white-box attacks , where the adversary designs an attack against a known model , including any pre-processing ∗This work was supported by the DARPA GARD , DARPA QED4RML programs , and National Science Foundation DMS division . †equal contribution steps . To cope with such potentially stronger adversaries , we train a robust model that produces a lower-bound on adversarial accuracy . In particular , we propose the first certifiable defense against patch attacks by extending interval bound propagation ( IBP ) defenses ( Gowal et al. , 2018 ; Mirman et al. , 2018 ) . We also propose modifications to IBP training to make it faster in the patch setting . Furthermore , we study the generalization of certified patch defenses to patches of different shapes , and observe that robustness transfers well across different patch types . We also present preliminary results on certified defense against the stronger sparse attack model , where a fixed number of possibly non-adjacent pixels can be arbitrarily modified ( Modas et al. , 2019 ) . 2 PROBLEM SETUP . We consider a white-box adversary that is allowed to choose the location of the patch ( chosen from a set L of possible locations ) and can modify pixels within the particular patch ( chosen from the set P ) similar to ( Karmon et al. , 2018 ) . An attack is successful if the adversary changes the classification of the network to a wrong label . In this paper , we are primarily interested in the patch attack robust accuracy ( adversarial accuracy for short ) as defined by E x∼X min p∈P , l∈L X [ f ( A ( x , p , l ) ; θ ) = y ] , ( 1 ) where the operator A places the adversarial patch p on a given image x at location l , f is a neural network with parameter θ , X is a distribution of images , and X is a characteristic function that takes value 1 if its argument is true , and 0 otherwise . In this model , the strength of the adversary can vary depending on the set of possible patches allowed , and the type of perturbation allowed within the patch . In what follows , we assume the standard setup in which the adversary is allowed any perturbation that maintains pixel intensities in the range [ 0 , 1 ] . Unless otherwise noted , we also assume the patch is restricted to a square of prescribed size . We consider two different options for the set L of possible patch locations . First , we consider a weak adversary that can only place patches at the corner of an image . We find that even this weak model is enough to break existing patch defenses . Then , we consider a stronger adversary with no restrictions on patch location , and use this model to evaluate our proposed defenses . Note that an adversary , when restricted to modify only a square patch at location l in the image , has the freedom to modify any non-square subset of these pixels . In other words , a certified defense against square patch attacks also provably subverts any non-square patch attack that fits inside a small enough square . In general , calculating the adversarial accuracy ( 1 ) is intractable due to non-convexity . Common approaches try to approximate it by solving the inner minimization using a gradient-based method . However , in Section 3 , we show that depending on how the minimization is solved , the upper bound could be very loose : a model may appear to be very robust , but fail when faced with a stronger attack . To side-step the arms race between attacks and defenses , in Section 4 , we extend the work of ( Gowal et al. , 2018 ) and ( Mirman et al. , 2018 ) to train a network that produces a lower bound on adversarial accuracy . We will refer to approximations of the upper bound as empirical adversarial accuracy and the lower bound as certified accuracy . 3 VULNERABILITY OF EXISTING DEFENSES . We start by examining existing defense strategies that claim to be effective against patch attacks . Similar to what has been observed for whole-image attacks by Athalye et al . ( 2018 ) , we show that these patch defenses can easily be broken by white-box attacks , where the adversary optimizes against a given model including any pre-processing steps . 3.1 EXISTING DEFENSES . Under our threat model , two defenses have been proposed that each use input transformations to detect and remove adversarial patches . The first defense is based on the observation that the gradient of the loss with respect to the input image often exhibits large values near the perturbed pixels . In ( Hayes , 2018 ) , the proposed digital watermarking ( DW ) approach exploits this behavior to detect unusually dense regions of large gradient entries using saliency maps , before masking them out in the image . Despite a 12 % drop in accuracy on clean ( non-adversarial ) images , this defense method supposedly achieves an empirical adversarial accuracy of 63 % for non-targeted patch attacks of size 42× 42 ( 2 % of the image pixels ) , using 400 randomly picked images from ImageNet ( Deng et al. , 2009 ) on VGG19 ( Simonyan & Zisserman , 2014 ) . The second defense , Local Gradient Smoothing ( LGS ) by Naseer et al . ( 2019 ) is based on the empirical observation that pixel values tend to change sharply within these adversarial patches . In other words , the image gradients tend to be large within these adversarial patches . Note that the image gradient here differs from the gradient in Hayes ( 2018 ) , the former is with respect the changes of adjacent pixel values and the later is with respect to the classification loss . Naseer et al . ( 2019 ) propose suppressing this adversarial noise by multiplying each pixel with one minus its image gradient as in ( 2 ) . To make their methods more effective , Naseer et al . ( 2019 ) also pre-process the image gradient with a normalization and a thresholding step . x̂ = x ( 1− λg ( x ) ) . ( 2 ) The λ here is a smoothing hyper-parameter . Naseer et al . ( 2019 ) claim the best adversarial accuracy on ImageNet with respect to patch attacks among all of the defenses we studied . They also claim that their defense is resilient to Backward Pass Differential Approximation ( BPDA ) from Athalye et al . ( 2018 ) , one of the most effective methods to attack models that include a non-differentiable operator as a pre-processing step . 3.2 BREAKING EXISTING DEFENSES . Using a similar setup as in ( Hayes , 2018 ; Naseer et al. , 2019 ) , we are able to mostly replicate the reported empirical adversarial accuracy for Iterative Fast Gradient Sign Method ( IFGSM ) , a common gradient based attack , but we show that when the pre-processing step is taken into account , the empirical adversarial accuracy on ImageNet quickly drops from ∼ 70 % ( ∼ 50 % ) for LGS ( DW ) to levels around ∼ 10 % as shown in Table 1 . Specifically , we break DW ( Hayes , 2018 ) by applying BPDA , in which the non-differentiable operator is approximated with an identity mapping during the backward pass . We break LGS ( Naseer et al. , 2019 ) by directly incorporating the smoothing step during backpropagation . Even though the windowing and thresholding steps are non-differentiable , the smoothing operator provides enough gradient information for the attack to be effective . To make sure that our evaluation is fair , we used the exact same models as Hayes ( 2018 ) ( VGG19 ) and Szegedy et al . ( 2016 ) ( Inception V3 ) . We also consider a weaker set of attackers that can only attack the corners , the same as their setting . Further , we ensure that we were able to replicate their reported result under similar setting . 4 CERTIFIED DEFENSES . Given the ease with which these supposedly strong defenses are broken , it is natural to seek methods that can rigorously guarantee robustness of a given model to patch attacks . With such certifiable guarantees in hand , we need not worry about an adversary with a stronger optimizer , or a more clever algorithm for choosing patch locations . 4.1 BACKGROUND ON CERTIFIED DEFENSES . Certified defenses have been intensely studied with respect to norm-bounded attacks ( Cohen et al. , 2019 ; Wong & Kolter , 2017 ; Gowal et al. , 2018 ; Mirman et al. , 2018 ; Zhang et al. , 2019b ) . In all of these methods , in addition to the prediction model , there is also a verifier . Given a model and an input , the verifier outputs a certificate if it is guaranteed that the image can not be adversarially perturbed . This is done by checking whether there exists any nearby image ( within a prescribed ` p distance ) with a different label than the image being classified . While theoretical bounds exist on the size of this distance that hold for any classifier ( Shafahi et al. , 2018 ) , exactly computing bounds for a specific classifier and test image is hard . Alternatively , the verifier may output a lower bound on the distance to the nearest image of a different label . This latter distance is referred to as the certifiable radius . Most of these verifiers provide a rather loose bound on the certifiable radius . However , if the verifier is differentiable , then the network can be trained with a loss that promotes tightness of this bound . We use the term certificate training to refer to the process of training with a loss that promotes strong certificates . Interval bound propagation ( IBP ) ( Mirman et al. , 2018 ; Gowal et al. , 2018 ) is a very simple verifier that uses layer-wise interval arithmetic to produce a certificate . Even though the IBP certificate is generally loose , after certificate training , it yields state-of-the-art certifiably-robust models for l∞-norm bounded attacks ( Gowal et al. , 2018 ; Zhang et al. , 2019b ) . In this paper , we extend IBP to train certifiably-robust networks resilient to patch attacks . We first introduce some notation and basic algorithms for IBP training . Notation We represent a neural network with a series of transformations h ( k ) for each of its k layers . We use z ( k ) ∈ Rnk to denote the output of layer k , where nk is the number of units in the kth layer and z ( 0 ) stands for the input . Specifically , the network computes z ( k ) = h ( k−1 ) ( z ( k−1 ) ) ∀k = 1 , . . . , K. Certification Problem To produce a certificate for an input x0 , we want to verify that the following condition is true with respect to all possible labels y : ( eytrue − ey ) T z ( K ) = my ≥ 0 ∀z ( 0 ) ∈ B ( x0 ) ∀y . ( 3 ) Here , ei is the ith basis vector , and my is called the margin following Wong & Kolter ( 2017 ) . Note that mytrue is always equal to 0 . The vector m contains all margins corresponding to all labels . B ( x0 ) is the constraint set over which the adversarial input image may range . In a conventional setting , this is an ` ∞ ball around x0 . In the case of patch attack , the constraint set contains all images formed by applying a patch to x0 ; B ( x0 ) = { A ( x0 , p , l ) |p ∈ P and l ∈ L } . ( 4 ) The Basics of Interval Bound Propagation ( IBP ) We now describe how to produce certificates using interval bound propagation as in ( Gowal et al. , 2018 ) . Suppose that for each component in z ( k−1 ) we have an interval containing all the values which this component reaches as z ( 0 ) ranges over the ball B ( x0 ) . If z ( k ) = h ( k ) ( z ( k−1 ) ) is a linear ( or convolutional ) layer of the form z ( k ) = W ( k ) z ( k−1 ) + b ( k ) , then we can get an outer approximation of the reachable interval range of activations by the next layer z ( k ) using the formulas below z ( k ) = W ( k ) z ( k−1 ) + z ( k−1 ) 2 + |W ( k ) |z ( k−1 ) − z ( k−1 ) 2 + b ( k ) , ( 5 ) z ( k ) = W ( k ) z ( k−1 ) + z ( k−1 ) 2 − |W ( k ) |z ( k−1 ) − z ( k−1 ) 2 + b ( k ) . ( 6 ) Here z ( k−1 ) denotes the upper bound of each interval , z ( k−1 ) the lower bound , and |W ( k ) | the element-wise absolute value . Alternatively , if h ( k ) ( z ( k−1 ) ) is an element-wise monotonic activation ( e.g. , a ReLU ) , then we can calculate the outer approximation of the reachable interval range of the next layer using the formulas below . z ( k ) = h ( k ) ( z ( k−1 ) ) ( 7 ) z ( k ) = h ( k ) ( z ( k−1 ) ) . ( 8 ) When the feasible set B ( x0 ) represents a simple ` ∞ attack , the range of possible z ( 0 ) values is trivially characterized by an interval bound z ( 0 ) and z ( 0 ) . Then , by iteratively applying the above rules , we can propagate intervals through the network and eventually get z ( K ) and z ( K ) . A certificate can then be given if we can show that ( 3 ) is always true for outputs in the range z ( K ) and z ( K ) with respect to all possible labels . More specifically , we can check that the following holds for all y my = e T ytruez ( K ) − eTy z ( K ) = z ( K ) ytrue − z ( K ) y ≥ 0 ∀y . ( 9 ) Training for Interval Bound Propagation To train a network to produce accurate interval bounds , we simply replace standard logits with the −m vector in ( 3 ) . Note that all elements of m need to be larger than zero to satisfy the conditions in ( 3 ) , and mytrue is always equal to zero . Put simply , we would like mytrue = 0 to be the least of all margins . We can promote this condition by training with the loss function Certificate Loss = Cross Entropy Loss ( −m , y ) . ( 10 ) Unlike regular neural network training , stochastic gradient descent for minimizing equation 10 is unstable , and a range of tricks are necessary to stabilize IBP training ( Gowal et al. , 2018 ) . The first trick is merging the last linear weight matrix with ( ey − eytrue ) before calculating −my . This allows a tighter characterization of the interval bound that noticeably improves results . The second trick uses an “ epsilon schedule ” in which training begins with a perturbation radius of zero , and this radius is slowly increased over time until a sentinel value is reached . Finally , a mixed loss function containing both a standard natural loss and an IBP loss is used . In all of our experiments , we use the merging technique and the epsilon schedule , but we do not use a mixed loss function containing a natural loss as it does not increase our certificate performance .

Summary: This paper attempts to extend the Interval Bound Propagation algorithm from (Gowal et al. 2018) to defend against adversarial patch-based attacks. In order to defend against patches which could appear at any location, all the patches need to be considered. This is too computationally expensive, hence they proposed to use a random subset of patches, or a U-net to predict the locations of the patches and then use those patches to train. The algorithm is tested on the MNIST and CIFAR-10 datasets and it was shown that sometimes the IBP approach is useful for defense, although often with a significant loss on accuracy on clean data (e.g. on CIFAR the loss on clean accuracy is an astounding 300% -- from 66.5% - 35.7%).