On the Power of Abstention and Data-Driven Decision Making for Adversarial Robustness

1 INTRODUCTION . A substantial body of work has shown that deep networks can be highly susceptible to adversarial attacks , in which minor changes to the input lead to incorrect , even bizarre classifications ( Nguyen et al. , 2015 ; Moosavi-Dezfooli et al. , 2016 ; Su et al. , 2019 ; Brendel et al. , 2018 ; Shamir et al. , 2019 ) . Much of this work has considered ` p-norm adversarial examples , but there has also been recent interest in exploring adversarial models beyond bounded ` p-norm ( Brown et al. , 2018 ; Engstrom et al. , 2017 ; Gilmer et al. , 2018 ; Xiao et al. , 2018 ; Alaifari et al. , 2019 ) . What these results have in common is that changes that either are imperceptible or should be irrelevant to the classification task can lead to drastically different network behavior . One reason for this vulnerability to adversarial attack is the non-Lipschitzness property of typical neural networks : small but adversarial movements in the input space can often produce large perturbations in the feature space . In this work , we consider the question of whether non-Lipschitz networks are intrinsically vulnerable , or if they could still be made robust to adversarial attack , in an abstract but ( we believe ) instructive adversarial model . In particular , suppose an adversary , by making an imperceptible change to an input x , can cause its representation F ( x ) in feature space ( the penultimate layer of the network ) to move by an arbitrary amount : will such an adversary always win ? Clearly if the adversary can modify F ( x ) by an arbitrary amount in an arbitrary direction , then yes . But what if the adversary can modify F ( x ) by an arbitrary amount but only in a random direction ( which it can not control ) ? In this case , we show an interesting dichotomy : if the classifier must output a classification on any input it is given , then yes the adversary will still win , no matter how well-separated the classes are in feature space and no matter what decision surface the classifier uses . However , if the classifier is allowed to abstain , then it can defeat such an adversary so long as natural data of different classes are reasonably well-separated in feature space . Our results hold for generalizations of these models as well , such as adversaries that can modify feature representations in random low-dimensional subspaces , or directions that are not completely random . More broadly , our results provide a theoretical explanation for the importance of allowing abstaining , or selective classification , in the presence of adversarial attack . Apart from providing a useful abstraction for non-Lipschitz feature embeddings , our model may be viewed as capturing an interesting class of real attacks . There are various global properties of an image , such as brightness , contrast , or rotation angle whose change might be “ perceptible but not relevant ” to classification tasks . Our model could also be viewed as an abstraction of attacks of that nature . Feature space attacks of other forms , where one can perturb abstract features denoting styles , including interpretable styles such as vivid colors and sharp outlines and uninterpretable ones , have also been empirically studied in ( Xu et al. , 2020 ; Ganeshan & Babu , 2019 ) . An interesting property of our model is that it is critical to be able to refuse to predict : any algorithm which always predicts a class label—therefore without an ability to abstain—is guaranteed to perform poorly . This provides a first formal hardness result about abstention in adversarial defense , and also a first provable negative result in feature-space attacks . We therefore allow the algorithm to output “ don ’ t know ” for some examples , which , as a by-product of our algorithm , serves as a detection mechanism for adversarial examples . It also results in an interesting trade-off between robustness and accuracy : by controlling how frequently we refuse to predict , we are able to trade ( robust ) precision off against recall . We also provide results for how to provably optimize for such a trade-off using a data-driven algorithm . Our strong theoretical advances are backed by empirical evidence in the context of contrastive learning ( He et al. , 2020 ; Chen et al. , 2020 ; Khosla et al. , 2020 ) . 1.1 OUR CONTRIBUTIONS . Our work tackles the problem of defending against adversarial perturbations in a random feature subspace , and advances the theory and practice of robust machine learning in multiple ways . • We introduce a formal model that captures feature-space attacks and the effect of nonLipschitzness of deep networks which can magnify input perturbations . • We begin our analysis with a hardness result concerning defending against adversary without the option of “ don ’ t know ” . We show that all classifiers that partition the feature space into two or more classes—thus without an ability to abstain—are provably vulnerable to adversarial examples for at least one class of examples with nearly half probability . • We explore the power of abstention option : a variant of nearest-neighbor classifier with the ability to abstain is provably robust against adversarial attacks , even in the presence of outliers in the training data set . We characterize the conditions under which the algorithm does not output “ don ’ t know ” too often . • We leverage and extend dispersion techniques from data-driven decision making , and present a novel data-driven method for learning data-specific optimal hyperparameters in our defense algorithms to simultaneously obtain high robust accuracy and low abstention rates . Unlike typical hyperparameter tuning , our approach provably converges to a global optimum . • Experimentally , we show that our proposed algorithm achieves certified adversarial robustness on representations learned by supervised and self-supervised contrastive learning . Our method significantly outperforms algorithms without the ability to abstain . 2 RELATED WORK . Adversarial robustness with abstention options . Classification with abstention option ( a.k.a . selective classification ( Geifman & El-Yaniv , 2017 ) ) is a relatively less explored direction in the adversarial machine learning . Hosseini et al . ( 2017 ) augmented the output class set with a NULL label and trained the classifier to reject the adversarial examples by classifying them as NULL ; Stutz et al . ( 2020 ) and Laidlaw & Feizi ( 2019 ) obtained robustness by rejecting low-confidence adversarial examples according to confidence thresholding or predictions on the perturbations of adversarial examples . Another related line of research to our method is the detection of adversarial examples ( Grosse et al. , 2017 ; Li & Li , 2017 ; Carlini & Wagner , 2017 ; Ma et al. , 2018 ; Meng & Chen , 2017 ; Metzen et al. , 2017 ; Bhagoji et al. , 2018 ; Xu et al. , 2017 ; Hu et al. , 2019 ) . However , theoretical understanding behind the empirical success of adversarial defenses with an abstention option remains elusive . Data-driven decision making . Data-driven algorithm selection refers to choosing a good algorithm from a parameterized family of algorithms for given data . It is known as “ hyperparameter tuning ” to machine learning practitioners and typically involves a “ grid search ” , “ random search ” ( Bergstra & Bengio ( 2012 ) ) or gradient-based search , with no guarantees of convergence to a global optimum . It was formally introduced to the theory of computing community by Gupta & Roughgarden ( 2017 ) as a learning paradigm , and was further extended in ( Balcan et al. , 2017 ) . The key idea is to model the problem of identifying a good algorithm from data as a statistical learning problem . The technique has found useful application in providing provably better algorithms for several domains including clustering , mechanism design , and mixed integer programs , and providing guarantees like differential privacy and adaptive online learning ( Balcan et al. , 2018a ; b ; 2020 ) . For learning in an adversarial setting , we provide the first demonstration of the effectiveness of data-driven algorithm selection in a defense method to optimize over the accuracy-abstention trade-off with strong theoretical guarantees . 3 PRELIMINARIES . Notation . We will use bold lower-case letters such as x and y to represent vectors , lower-case letters such as x and y to represent scalars , and calligraphy capital letters such as X , Y and D to represent distributions . Specifically , we denote by x 2 X the sample instance , and by y 2 Y the label , where X ✓ Rn1 and Y indicate the image and label spaces , respectively . Denote by F : X ! Rn2 the feature embedding which maps an instance to a high-dimensional vector in the latent space F ( X ) . It can be parameterized , e.g. , by deep neural networks . We will frequently use v 2 Rn2 to represent an adversarial perturbation in the feature space . Denote by dist ( · , · ) the distance between any two vectors in the image or feature space . Examples of distances include dist ( x1 , x2 ) = kx1 x2k—the one induced by vector norm . We use B ( x , ⌧ ) to represent a neighborhood of x : { x0 : dist ( x , x0 ) ⌧ } in the image or feature space . We will frequently denote by DX the distribution of instances in the input space , by DX|y the distribution of instances in the input space conditioned on the class y , by DF ( X ) the distribution of features , and by DF ( X ) |y the distribution of features conditioned on the class y . 3.1 RANDOM FEATURE SUBSPACE THREAT MODEL . In principle , the adversarial example for a given labeled data ( x , y ) is a data point x0 that causes a classifier to output a different label on x0 than the true label y . Probably one of the most popular adversarial examples is the norm-bounded perturbation in the input space . Despite a large literature devoted to defending against norm-bounded adversary by improving the Lipschitzness of neural network as a function mapping from input space to feature space ( Zhang et al. , 2019 ; Yang et al. , 2020 ) , it is typically not true that small perturbation in the input space necessarily implies small modification in the feature space . In this paper , we study a threat model where an adversary can modify the data by a large amount in the feature space . Note that because this large modification in feature space is assumed to come from a small perturbation in input space , we always assume that the true correct label y is the same for x0 as for x . Our model highlights the power of abstention in the adversarial learning : there is a provable separation when we have and do not have an abstention option under our threat model . Our threat model . In the setting of ( robust ) representation learning , we are given a set of training instances x1 , ... , xm 2 X . Let x be an n1-dimensional test input for classification . The input is embedded into a high n2-dimensional feature space using a deep neural network F . We predict the class of x by a prediction function on F ( x ) which can potentially output “ don ’ t know ” . The adversary may corrupt F ( x ) such that the modified feature vector is restricted in a random n3-dimensional affine subspace denoted by S + { F ( x ) } , while the perturbation magnitude might be arbitrarily large . The adversary is given access to everything including F , x , S and the true label of x . Throughout the paper , we will refer adversary and adversarial example to this threat model . Algorithm 1 ROBUSTCLASSIFIER ( ⌧ , ) 1 : Input : A test feature F ( x ) ( potentially an adversarial example ) , a set of training features F ( xi ) and their labels yi , i 2 [ m ] , a threshold parameter ⌧ , a separation parameter . 2 : Preprocessing : Delete training examples F ( xi ) if minj2 [ m ] , yi 6=yj dist ( F ( xi ) , F ( xj ) ) < 3 : Output : A predicted label of F ( x ) , or “ don ’ t know ” . 4 : if mini2 [ m ] dist ( F ( x ) , F ( xi ) ) < ⌧ then 5 : Return yarg mini2 [ m ] dist ( F ( x ) , F ( xi ) ) 6 : else 7 : Return “ don ’ t know ”

Summary: This paper proves some fundamental facts about classifiers that can't abstain (provide a non-classification) and their robustness to adversarial perturbations. In Sec. 4, they provide a result that such classifiers are always vulnerable to adversarial perturbations in a technical sense. In particular, there will always be a class in which most training examples can be randomly perturbed in a way that an incorrect label will result nearly half the time. In Sec 5, they propose a modified nearest-neighbor classification algorithm, with two parameters that control abstention and "noise removal". They provide upper bounds on error in a random subspace attack scheme, and refine/loosen these results in several more specific/general scenarios. In Secs. 6 & 7, they discuss methods to tune the two parameters and provide experimental evidence of their theoretical results.