ANCER: Anisotropic Certification  via Sample-wise Volume Maximization

1 INTRODUCTION . The well-studied fact that Deep Neural Networks ( DNNs ) are vulnerable to additive imperceptible noise perturbations has led to a growing interest in developing robust classifiers ( Goodfellow et al. , 2015 ; Szegedy et al. , 2014 ) . A recent promising approach to achieve state-of-the-art provable robustness ( i.e . a theoretical bound on the output around every input ) at the scale of ImageNet ( Deng et al. , 2009 ) is randomized smoothing ( Lecuyer et al. , 2019 ; Cohen et al. , 2019 ) . Given an input x and a network f , randomized smoothing constructs g ( x ) = E ∼D [ f ( x + ) ] such that g ( x ) = g ( x + δ ) ∀δ ∈ R , where the certification region R is characterized by x , f , and the smoothing distribution D. For instance , Cohen et al . ( 2019 ) showed that if D = N ( 0 , σ2I ) , thenR is an ` 2-ball whose radius is determined by x , f and σ . Since then , there has been significant progress towards the design of D leading to the largestR for all inputs x . The interplay betweenR characterized by ` 1 , ` 2 and ` ∞-balls , and a notion of optimal distribution D has been previously studied Yang et al . ( 2020 ) . Despite this progress , current randomized smoothing approaches provide certification regions that are isotropic in nature , limiting their capacity to certifying smaller and worst-case regions . We provide an intuitive example of this behavior in Figure 1 . The isotropic nature ofR in prior art is due to the common assumption that the smoothing distribution D is identically distributed ( Yang et al. , 2020 ; Kumar et al. , 2020 ; Levine & Feizi , 2021 ) . Moreover , comparisons between various randomized smoothing approaches were limited to methods that produce the same ` p certificate , with no clear metrics for comparing with other certificates . In this paper , we address both concerns and present new state-of-the-art certified accuracy results on both CIFAR-10 and ImageNet datasets . Our contributions are threefold . ( i ) We provide a general and simpler analysis compared to prior art ( Cohen et al. , 2019 ; Yang et al. , 2020 ) that paves the way for the certification of anisotropic regions characterized by any norm , holding prior art as special cases . We then specialize our result to regions that , for a positive definite A , are ellipsoids , i.e . ‖Aδ‖2 ≤ c , and generalized cross-polytopes , i.e . ‖Aδ‖1 ≤ c , generalizing both ` 2 ( Cohen et al. , 2019 ) and ` 1 ( Lecuyer et al. , 2019 ; Yang et al. , 2020 ) certification ( Section 4 ) . ( ii ) We introduce a new evaluation framework to compare methods that certify general ( isotropic or anisotropic ) regions . We compare two general certificates by defining that a method certifyingR1 is superior to another certifyingR2 , ifR1 is a strict superset to R2 . Further , we define a standalone quantitative metric as the volume of the certified region , and specialize it for the cases of ellipsoids and generalized cross-polytopes ( Section 5 ) . ( iii ) We propose ANCER , an anisotropic certification method that performs sample-wise ( i.e . per sample in the test set ) region volume maximization ( Section 6 ) , generalizing the data-dependent , memory-based solution from Alfarra et al . ( 2020 ) . Through experiments on CIFAR-10 ( Krizhevsky , 2009 ) and ImageNet ( Deng et al. , 2009 ) , we show that restricting ANCER ’ s certification region to ` 1 and ` 2-balls outperforms state-of-the-art ` 1 and ` 2 results from previous works ( Yang et al. , 2020 ; Alfarra et al. , 2020 ) . Further , we show that the volume of the certified regions are significantly larger than all existing methods , thus setting a new state-of-the-art in certified accuracy . We highlight that while we effectively achieve state-of-the-art performance , it comes at a high cost given the data-dependency requirements . A discussion of the limitations of the solution is presented in Section 6 . Notation . We consider a base classifier f : Rn → P ( K ) , where P ( K ) is a probability simplex over K classes , i.e . f i ≥ 0 and 1 > f = 1 , for i ∈ { 1 , . . . , K } . Further , we use ( x , y ) to be a sample input x and its corresponding true label y drawn from a test set Dt , and fy to be the output of f at the correct class . We use ` p to be the typically defined ‖ · ‖p norm ( p ≥ 1 ) , and ` Ap or ‖ · ‖A , p for p = { 1 , 2 } to be a composite norm defined with respect to a positive definite matrix A as ‖A−1/pv‖p . 2 RELATED WORK . Verified Defenses . Since the discovery that DNNs are vulnerable against input perturbations ( Goodfellow et al. , 2015 ; Szegedy et al. , 2014 ) , a range of methods have been proposed to build classifiers that are verifiably robust ( Huang et al. , 2017 ; Gowal et al. , 2019 ; Bunel et al. , 2018 ; Salman et al. , 2019b ) . Despite this progress , these methods do not yet scale to the networks the community is interested in certifying ( Tjeng et al. , 2019 ; Weng et al. , 2018 ) . Randomized Smoothing . The first works on randomized smoothing used Laplacian ( Lecuyer et al. , 2019 ; Li et al. , 2019 ) and Gaussian Cohen et al . ( 2019 ) distributions to obtain ` 1 and ` 2-ball certificates , respectively . Several subsequent works improved the performance of smooth classifiers by training the base classifier using adversarial augmentation ( Salman et al. , 2019a ) , regularization ( Zhai et al. , 2019 ) , or general adjustments to training routines ( Jeong & Shin , 2020 ) . Recent work derived ` p-norm certificates for other isotropic smoothing distributions ( Yang et al. , 2020 ; Levine & Feizi , 2020 ; Zhang et al. , 2019 ) . Concurrently , Dvijotham et al . ( 2020 ) developed a framework to handle arbitrary smoothing measures in any ` p-norm ; however , the certification process requires significant hyperparameter tuning . Similarly , Mohapatra et al . ( 2020 ) introduces larger certificates that require higher-order information , yet do not provide a closed-form solution . This was followed by a complementary data-dependent smoothing approach , where the parameters of the smoothing distribution were optimized per test set sample to maximize the certified radius at an individual input ( Alfarra et al. , 2020 ) . All prior works considered smoothing with isotropic distributions and hence certified isotropic ` p-ball regions . In this paper , we extend randomized smoothing to certify anisotropic regions , by pairing it with a generalization of the data-dependent framework ( Alfarra et al. , 2020 ) to maximize the certified region at each input point . 3 MOTIVATING ANISOTROPIC CERTIFICATES . Certification approaches aim to find the safe region R , where arg maxi f i ( x ) = arg maxi f i ( x + δ ) ∀δ ∈ R. Recent randomized smoothing techniques perform this certification by explicitly optimizing the isotropic ` p certified region around each input ( Alfarra et al. , 2020 ) , obtaining state-ofthe-art performance as a result . Despite this ` p optimality , we note that any ` p-norm certificate is worst-case from the perspective of that norm , as it avoids adversary regions by limiting its certificate to the ` p-closest adversary . This means that it can only enjoy a radius that is at most equal to the distance to the closest decision boundary . However , decision boundaries of general classifiers are complex , non-linear , and non-radially distributed with respect to a generic input sample ( Karimi et al. , 2019 ) . This is evidenced by the fact that , within a reasonably small ` p-ball around an input , there are often only a small set of adversary directions ( Tramèr et al. , 2017 ; 2018 ) ( e.g . see the decision boundaries in Figure 1 ) . As such , while ` p-norm certificates are useful to reason about worst-case performance and are simple to obtain given previous works ( Cohen et al. , 2019 ; Yang et al. , 2020 ; Lee et al. , 2019 ) , they are otherwise uninformative in terms of the shape of decision boundaries , i.e . which regions around the input are safe . To visualize these concepts , we illustrate the decision boundaries of a base classifier f trained on a toy 2-dimensional , radially separable ( with respect to the origin ) binary classification dataset , and consider two different input test samples ( see Figure 1 ) . We compare the optimal isotropic and anisotropic certified regions of different shapes at these points . In Figures 1a and 1b , we compare an isotropic cross-polytope ( of the form ‖δ‖1 ≤ r ) with an anisotropic generalized cross-polytope ( of the form ‖Aδ‖1 ≤ r ) , while in Figures 1c and 1d we compare an isotropic ` 2 ball ( of the form ‖δ‖2 ≤ r ) with an anisotropic ellipsoid ( of the form ‖Aδ‖2 ≤ r ) . Notice that in Figures 1a and 1c , due to the curvature of the classification boundary ( shown in white ) , the optimal certification region is isotropic in nature , which is evidenced by the similarities of the optimal isotropic and anisotropic certificates . On the other hand , in Figures 1b and 1d , the location of the decision boundary allows for the anisotropic certified regions to be considerably larger than their isotropic counterparts , as they are not as constrained by the closest decision boundary , i.e . the worst-case performance . We note that these differences are further highlighted in higher dimensions , and we study them for a single CIFAR-10 test set sample in Appendix A.1 . Further , we also showcase how anisotropic certification allows for further insights into constant prediction ( safe ) directions in Appendix A.2 . 4 ANISOTROPIC CERTIFICATION . One of the main obstacles in enabling anisotropic certification is the complexity of the analysis required . To alleviate this , we follow a Lipschitz argument first observed by Salman et al . ( 2019a ) and Jordan & Dimakis ( 2020 ) and propose a simple and general certification analysis . We start with the following two observations . All proofs are in Appendix B . Proposition 1 . Consider a differentiable function g : Rn → R. If supx‖∇g ( x ) ‖∗ ≤ L where ‖ · ‖∗ has a dual norm ‖z‖ = maxx z > x s.t . ‖x‖∗ ≤ 1 , then g is L-Lipschitz under norm ‖ · ‖∗ , that is |g ( x ) − g ( y ) | ≤ L‖x− y‖ . Given the previous proposition , we formalize ‖ · ‖ certification as follows : Theorem 1 . Let g : Rn → RK , gi be L-Lipschitz continuous under norm ‖ · ‖∗ ∀i ∈ { 1 , . . . , K } , and cA = arg maxi gi ( x ) . Then , we have arg maxi gi ( x+ δ ) = cA for all δ satisfying : ‖δ‖ ≤ 1 2L ( gcA ( x ) −max c gc 6=cA ( x ) ) . Theorem 1 provides an ‖ · ‖ norm robustness certificate for any L-Lipschitz classifier g under ‖ · ‖∗ . The certificate is only informative when one can attain a tight non-trivial estimate of L , ideally supx‖∇g ( x ) ‖∗ , which is generally difficult when g is an arbitrary neural network . Framework Recipe . In light of Theorem 1 , randomized smoothing can be viewed differently as an instance of Theorem 1 with the favorable property that the constructed smooth classifier g enjoys an analytical form for L = supx‖∇g ( x ) ‖∗ by design . As such , to obtain an informative ‖ · ‖ certificate , one must , for an arbitrary choice of a smoothing distribution , compute the analytic Lipschitz constant L under ‖ · ‖∗ for the smooth g. While there can exist a notion of “ optimal ” smoothing distribution for a given choice of ‖ · ‖ certificate , as in part addressed earlier for the isotropic ` 1 , ` 2 and ` ∞ certificates ( Yang et al. , 2020 ) , this is not the focus of this paper . The choice of the smoothing distribution in later sections is inspired by previous work for the purpose of granting anisotropic certificates . This recipe complements randomized smoothing works based on Neyman-Pearson ’ s lemma ( Cohen et al. , 2019 ) or the Level-Set and Differential Method ( Yang et al. , 2020 ) . We will deploy this framework recipe to show two specializations for anisotropic certification , namely ellipsoids ( Section 4.1 ) and generalized cross-polytopes ( Section 4.2 ) .1 .

Summary: In this paper, the authors discuss the extension of $\ell_{p}$-randomized smoothing to anisotropic counterparts. In particular, they consider the extension of $\ell_{2}$-certificates from (hyper)spheres to (hyper)ellipsoids by sampling anisotropic rather than isotropic Gaussian noise, as well as the extension of $\ell_{1}$-certificates to cross-polytopes by sampling scaled Uniform noise rather than unscaled noise. Further, the authors discuss how these extended certificates can be compared to their base-counter parts to establish superiority (inclusion). The introduced certification algorithm, ANCER, utilizes the idea of data-dependent randomized smoothing to find the anisotropic shape with maximal certification volume. In experimental evaluation on Cifar-10 and Imagenet the authors show that the obtained certificates permit higher isotropic certificating radii than other methods in the per-sample optimization setting.