Consistent Non-Parametric Methods for Maximizing Robustness

1 Introduction . Adversarially robust classification , that has been of much recent interest , is typically formulated as follows . We are given data drawn from an underlying distribution D , a metric d , as well as a pre-specified robustness radius r. We say that a classifier c is r-robust at an input x if it predicts the same label on a ball of radius r around x . Our goal in robust classification is to find a classifier c that maximizes astuteness , which is defined as accuracy on those examples where c is also r-robust . While this formulation has inspired a great deal of recent work , both theoretical and empirical [ 5 , 17 , 19 , 20 , 26 , 15 , 18 , 21 , 22 , 23 , 30 ] , a major limitation is that enforcing a pre-specified robustness radius r may lead to sub-optimal accuracy and robustness . To see this , consider what would be an ideally robust classifier the example in Figure 1 . For simplicity , suppose that we know the data distribution . In this case , a classifier that has an uniformly large robustness radius r will misclassify some points from the blue cluster on the left , leading to lower accuracy . This is illustrated in panel ( a ) , in which large robustness radius leads to intersecting robustness regions . On the other hand , in panel ( b ) , the blue cluster on the right is highly separated from the red cluster , and could be accurately classified with a high margin . But this will not happen if the robustness radius is set small enough to avoid the problems posed in panel ( a ) . Thus , enforcing a fixed robustness radius that applies to the entire dataset may lead to lower accuracy and lower robustness . In this work , we propose an alternative formulation of robust classification that ensures that in the large sample limit , there is no robustness-accuracy trade off , and that regions of space with higher separation are classified more robustly . An extra advantage is that our formulation is achievable by existing methods . In particular , we show that two very common non-parametric algorithms – nearest neighbors and kernel classifiers – achieve these properties in the large sample limit . Our formulation is built on the notion of a new large-sample limit . In the standard statistical learning framework , the large-sample ideal is the Bayes optimal classifier that maximizes accuracy on the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data distribution , and is undefined outside . Since this is not always robust with radius r , prior work introduces the notion of an r-optimal classifier [ 31 ] that maximizes accuracy on points where it is also r-robust . However , this classifier also suffers from the same challenges as the example in Figure 1 . We depart from both by introducing a new limit that we call the neighborhood preserving Bayes optimal classifier , described as follows . Given an input x that lies in the support of the data distribution D , it predicts the same label as the Bayes optimal . On an x outside the support , it outputs the prediction of the Bayes Optimal on the nearest neighbor of x within the support of D. The first property ensures that there is no loss of accuracy – since it always agrees with the Bayes Optimal within the data distribution . The second ensures higher robustness in regions that are better separated . Our goal is now to design classifiers that converge to the neighborhood preserving Bayes optimal in the large sample limit ; this ensures that with enough data , the classifier will have accuracy approaching that of the Bayes optimal , as well as higher robustness where possible without sacrificing accuracy . We next investigate how to design classifiers with this convergence property . Our starting point is classical statistical theory [ 25 ] that shows that a class of methods known as weight functions will converge to a Bayes optimal in the large sample limit provided certain conditions hold ; these include k-nearest neighbors under certain conditions on k and n , certain kinds of decision trees as well as kernel classifiers . Through an analysis of weight functions , we next establish precise conditions under which they converge to the neighborhood preserving Bayes optimal in the large sample limit . As expected , these are stronger than standard convergence to the Bayes optimal . In the large sample limit , we show that kn-nearest neighbors converge to the neighborhood preserving Bayes optimal provided kn = ω ( log n ) , and kernel classifiers converge to the neighborhood preserving Bayes optimal provided certain technical conditions ( such as the bandwidth shrinking sufficiently slowly ) . By contrast , certain types of histograms do not converge to the neighborhood preserving Bayes optimal , even if they do converge to the Bayes optimal . We round these off with a lower bound that shows that for nearest neighbor , the condition that kn = ω ( log n ) is tight . In particular , for kn = O ( log n ) , there exist distributions for which kn-nearest neighbors provably fails to converge towards the neighborhood preserving Bayes optimal ( despite converging towards the standard Bayes optimal ) . In summary , the contributions of the paper are as follows . First , we propose a new large sample limit the neighborhood preserving Bayes optimal and a new formulation for robust classification . We then establish conditions under which weight functions , a class of non-parametric methods , converge to the neighborhood preserving Bayes optimal in the large sample limit . Using these conditions , we show that kn-nearest neighbors satisfy these conditions when kn = ω ( log n ) , and kernel classifiers satisfy these conditions provided the kernel function K has faster than polynomial decay , and the bandwidth parameter hn decreases sufficiently slowly . To complement these results , we also include negative examples of non-parametric classifiers that do not converge . We provide an example where histograms do not converge to the neighborhood preserving Bayes optimal with increasing n. We also show a lower bound for nearest neighbors , indi- cating that kn = ω ( log n ) is both necessary and sufficient for convergence towards the neighborhood preserving Bayes optimal . Our results indicate that the neighborhood preserving Bayes optimal formulation shows promise and has some interesting theoretical properties . We leave open the question of coming up with other alternative formulations that can better balance both robustness and accuracy for all kinds of data distributions , as well as are achievable algorithmically . We believe that addressing this would greatly help address the challenges in adversarial robustness . 2 Preliminaries . We consider binary classification over Rd × { ±1 } , and let ρ denote any distance metric on Rd . We let µ denote the measure over Rd corresponding to the probability distribution over which instances x ∈ Rd are drawn . Each instance x is then labeled as +1 with probability η ( x ) and −1 with probability 1− η ( x ) . Together , µ and η comprise our data distribution D = ( µ , η ) over Rd × { ±1 } . For comparison to the robust case , for a classifier f : Rd → { ±1 } and a distribution D over Rd × { ±1 } , it will be instructive to consider its accuracy , denoted A ( f , D ) , which is defined as the fraction of examples from D that f labels correctly . Accuracy is maximized by the Bayes Optimal classifier : which we denote by g. It can be shown that for any x ∈ supp ( µ ) , g ( x ) = 1 if η ( x ) ≥ 12 , and g ( x ) = −1 otherwise . Our goal is to build classifiers Rd → { ±1 } that are both accurate and robust to small perturbations . For any example x , perturbations to it are constrained to taking place in the robustness region of x , denoted Ux . We will let U = { Ux : x ∈ Rd } denote the collections of all robustness regions . We say that a classifier f : Rd → { ±1 } is robust at x if for all x′ ∈ Ux , f ( x′ ) = f ( x ) . Combining robustness and accuracy , we say that classifier is astute at a point x if it is both accurate and robust . Formally , we have the following definition . Definition 1 . A classifier f : Rd → { ±1 } is said to be astute at ( x , y ) with respect to robustness collection U if f ( x ) = y and f is robust at x with respect to U . If D is a data distribution over Rd × { ±1 } , the astuteness of f over D with respect to U , denoted AU ( f , D ) , is the fraction of examples ( x , y ) ∼ D for which f is astute at ( x , y ) with respect to U . Thus AU ( f , D ) = P ( x , y ) ∼D [ f ( x′ ) = y , ∀x′ ∈ Ux ] . Non-parametric Classifiers We now briefly review several kinds of non-parametric classifiers that we will consider throughout this paper . We begin with weight functions , which are a general class of non-parametric algorithms that encompass many classic algorithms , including nearest neighbors and kernel classifiers . Weight functions are built from training sets , S = { ( x1 , y1 ) , ( x2 , y2 , ) , . . . , ( xn , yn ) } by assigning a function wSi : Rd → [ 0 , 1 ] that essentially scores how relevant the training point ( xi , yi ) is to the example being classified . The functions wSi are allowed to depend on x1 , . . . , xn but must be independent of the labels y1 , . . . , yn . Given these functions , a point x is classified by just checking whether ∑ yiw S i ( x ) ≥ 0 or not . If it is nonnegative , we output +1 and otherwise −1 . A complete description of weight functions is included in the appendix . Next , we enumerate several common Non-parametric classifiers that can be construed as weight functions . Details can be found in the appendix . Histogram classifiers partition the domain Rd into cells recursively by splitting cells that contain a sufficiently large number of points xi . This corresponds to a weight function in which wSi ( x ) = 1 kx if xi is in the same cell as x , where kx denotes the number of points in the cell containing x. kn-nearest neighbors corresponds to a weight function in which wSi ( x ) = 1 kn if xi is one of the kn nearest neighbors of x , and wSi ( x ) = 0 otherwise . Kernel-Similarity classifiers are weight functions built from a kernel function K : R≥0 → R≥0 and a window size ( hn ) ∞1 such that w S i ( x ) ∝ K ( ρ ( x , xi ) /hn ) ( we normalize by dividing by∑n 1 K ( ( ρ ( x , xi ) /hn ) ) ) .

Summary: This paper studies classification rules that are "astute", that is, those that are accurate and robust. Importantly, the authors note that common definitions of robust classifiers, which are given in terms of a fixed parameter (typically a size of a perturbation set) are not astute, as they do not fully exploit the fact that margins are not uniform across the data distribution. This paper first studies conditions under their so-called "neighborhood optimal classifier" maximizes robustness while retaining the Bayes classifier accuracy in the unperturbed distribution. Furthermore, the authors show that several popular non-parametric classifier provide consistent estimates for this classifier. Finally, this is demonstrated numerically in a simple experimental setting.