Consistent Non-Parametric Methods for Maximizing Robustness

1 Introduction . Adversarially robust classification , that has been of much recent interest , is typically formulated as follows . We are given data drawn from an underlying distribution D , a metric d , as well as a pre-specified robustness radius r. We say that a classifier c is r-robust at an input x if it predicts the same label on a ball of radius r around x . Our goal in robust classification is to find a classifier c that maximizes astuteness , which is defined as accuracy on those examples where c is also r-robust . While this formulation has inspired a great deal of recent work , both theoretical and empirical [ 5 , 17 , 19 , 20 , 26 , 15 , 18 , 21 , 22 , 23 , 30 ] , a major limitation is that enforcing a pre-specified robustness radius r may lead to sub-optimal accuracy and robustness . To see this , consider what would be an ideally robust classifier the example in Figure 1 . For simplicity , suppose that we know the data distribution . In this case , a classifier that has an uniformly large robustness radius r will misclassify some points from the blue cluster on the left , leading to lower accuracy . This is illustrated in panel ( a ) , in which large robustness radius leads to intersecting robustness regions . On the other hand , in panel ( b ) , the blue cluster on the right is highly separated from the red cluster , and could be accurately classified with a high margin . But this will not happen if the robustness radius is set small enough to avoid the problems posed in panel ( a ) . Thus , enforcing a fixed robustness radius that applies to the entire dataset may lead to lower accuracy and lower robustness . In this work , we propose an alternative formulation of robust classification that ensures that in the large sample limit , there is no robustness-accuracy trade off , and that regions of space with higher separation are classified more robustly . An extra advantage is that our formulation is achievable by existing methods . In particular , we show that two very common non-parametric algorithms – nearest neighbors and kernel classifiers – achieve these properties in the large sample limit . Our formulation is built on the notion of a new large-sample limit . In the standard statistical learning framework , the large-sample ideal is the Bayes optimal classifier that maximizes accuracy on the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data distribution , and is undefined outside . Since this is not always robust with radius r , prior work introduces the notion of an r-optimal classifier [ 31 ] that maximizes accuracy on points where it is also r-robust . However , this classifier also suffers from the same challenges as the example in Figure 1 . We depart from both by introducing a new limit that we call the neighborhood preserving Bayes optimal classifier , described as follows . Given an input x that lies in the support of the data distribution D , it predicts the same label as the Bayes optimal . On an x outside the support , it outputs the prediction of the Bayes Optimal on the nearest neighbor of x within the support of D. The first property ensures that there is no loss of accuracy – since it always agrees with the Bayes Optimal within the data distribution . The second ensures higher robustness in regions that are better separated . Our goal is now to design classifiers that converge to the neighborhood preserving Bayes optimal in the large sample limit ; this ensures that with enough data , the classifier will have accuracy approaching that of the Bayes optimal , as well as higher robustness where possible without sacrificing accuracy . We next investigate how to design classifiers with this convergence property . Our starting point is classical statistical theory [ 25 ] that shows that a class of methods known as weight functions will converge to a Bayes optimal in the large sample limit provided certain conditions hold ; these include k-nearest neighbors under certain conditions on k and n , certain kinds of decision trees as well as kernel classifiers . Through an analysis of weight functions , we next establish precise conditions under which they converge to the neighborhood preserving Bayes optimal in the large sample limit . As expected , these are stronger than standard convergence to the Bayes optimal . In the large sample limit , we show that kn-nearest neighbors converge to the neighborhood preserving Bayes optimal provided kn = ω ( log n ) , and kernel classifiers converge to the neighborhood preserving Bayes optimal provided certain technical conditions ( such as the bandwidth shrinking sufficiently slowly ) . By contrast , certain types of histograms do not converge to the neighborhood preserving Bayes optimal , even if they do converge to the Bayes optimal . We round these off with a lower bound that shows that for nearest neighbor , the condition that kn = ω ( log n ) is tight . In particular , for kn = O ( log n ) , there exist distributions for which kn-nearest neighbors provably fails to converge towards the neighborhood preserving Bayes optimal ( despite converging towards the standard Bayes optimal ) . In summary , the contributions of the paper are as follows . First , we propose a new large sample limit the neighborhood preserving Bayes optimal and a new formulation for robust classification . We then establish conditions under which weight functions , a class of non-parametric methods , converge to the neighborhood preserving Bayes optimal in the large sample limit . Using these conditions , we show that kn-nearest neighbors satisfy these conditions when kn = ω ( log n ) , and kernel classifiers satisfy these conditions provided the kernel function K has faster than polynomial decay , and the bandwidth parameter hn decreases sufficiently slowly . To complement these results , we also include negative examples of non-parametric classifiers that do not converge . We provide an example where histograms do not converge to the neighborhood preserving Bayes optimal with increasing n. We also show a lower bound for nearest neighbors , indi- cating that kn = ω ( log n ) is both necessary and sufficient for convergence towards the neighborhood preserving Bayes optimal . Our results indicate that the neighborhood preserving Bayes optimal formulation shows promise and has some interesting theoretical properties . We leave open the question of coming up with other alternative formulations that can better balance both robustness and accuracy for all kinds of data distributions , as well as are achievable algorithmically . We believe that addressing this would greatly help address the challenges in adversarial robustness . 2 Preliminaries . We consider binary classification over Rd × { ±1 } , and let ρ denote any distance metric on Rd . We let µ denote the measure over Rd corresponding to the probability distribution over which instances x ∈ Rd are drawn . Each instance x is then labeled as +1 with probability η ( x ) and −1 with probability 1− η ( x ) . Together , µ and η comprise our data distribution D = ( µ , η ) over Rd × { ±1 } . For comparison to the robust case , for a classifier f : Rd → { ±1 } and a distribution D over Rd × { ±1 } , it will be instructive to consider its accuracy , denoted A ( f , D ) , which is defined as the fraction of examples from D that f labels correctly . Accuracy is maximized by the Bayes Optimal classifier : which we denote by g. It can be shown that for any x ∈ supp ( µ ) , g ( x ) = 1 if η ( x ) ≥ 12 , and g ( x ) = −1 otherwise . Our goal is to build classifiers Rd → { ±1 } that are both accurate and robust to small perturbations . For any example x , perturbations to it are constrained to taking place in the robustness region of x , denoted Ux . We will let U = { Ux : x ∈ Rd } denote the collections of all robustness regions . We say that a classifier f : Rd → { ±1 } is robust at x if for all x′ ∈ Ux , f ( x′ ) = f ( x ) . Combining robustness and accuracy , we say that classifier is astute at a point x if it is both accurate and robust . Formally , we have the following definition . Definition 1 . A classifier f : Rd → { ±1 } is said to be astute at ( x , y ) with respect to robustness collection U if f ( x ) = y and f is robust at x with respect to U . If D is a data distribution over Rd × { ±1 } , the astuteness of f over D with respect to U , denoted AU ( f , D ) , is the fraction of examples ( x , y ) ∼ D for which f is astute at ( x , y ) with respect to U . Thus AU ( f , D ) = P ( x , y ) ∼D [ f ( x′ ) = y , ∀x′ ∈ Ux ] . Non-parametric Classifiers We now briefly review several kinds of non-parametric classifiers that we will consider throughout this paper . We begin with weight functions , which are a general class of non-parametric algorithms that encompass many classic algorithms , including nearest neighbors and kernel classifiers . Weight functions are built from training sets , S = { ( x1 , y1 ) , ( x2 , y2 , ) , . . . , ( xn , yn ) } by assigning a function wSi : Rd → [ 0 , 1 ] that essentially scores how relevant the training point ( xi , yi ) is to the example being classified . The functions wSi are allowed to depend on x1 , . . . , xn but must be independent of the labels y1 , . . . , yn . Given these functions , a point x is classified by just checking whether ∑ yiw S i ( x ) ≥ 0 or not . If it is nonnegative , we output +1 and otherwise −1 . A complete description of weight functions is included in the appendix . Next , we enumerate several common Non-parametric classifiers that can be construed as weight functions . Details can be found in the appendix . Histogram classifiers partition the domain Rd into cells recursively by splitting cells that contain a sufficiently large number of points xi . This corresponds to a weight function in which wSi ( x ) = 1 kx if xi is in the same cell as x , where kx denotes the number of points in the cell containing x. kn-nearest neighbors corresponds to a weight function in which wSi ( x ) = 1 kn if xi is one of the kn nearest neighbors of x , and wSi ( x ) = 0 otherwise . Kernel-Similarity classifiers are weight functions built from a kernel function K : R≥0 → R≥0 and a window size ( hn ) ∞1 such that w S i ( x ) ∝ K ( ρ ( x , xi ) /hn ) ( we normalize by dividing by∑n 1 K ( ( ρ ( x , xi ) /hn ) ) ) .

Summary: This paper argues that the standard robust learning method ignores the data heterogeneity: the robustness radius is fixed as some artificial value r for all the examples. The authors then propose a new neighborhood optimal classifier to address this problem, and theoretically studies the convergence condition for a class of non-parametric methods, including histogram classifiers, nearest neighbors, and kernel classifiers. Synthetic experiments are provided to verify their theoretical results.