ANCER: Anisotropic Certification  via Sample-wise Volume Maximization

1 INTRODUCTION . The well-studied fact that Deep Neural Networks ( DNNs ) are vulnerable to additive imperceptible noise perturbations has led to a growing interest in developing robust classifiers ( Goodfellow et al. , 2015 ; Szegedy et al. , 2014 ) . A recent promising approach to achieve state-of-the-art provable robustness ( i.e . a theoretical bound on the output around every input ) at the scale of ImageNet ( Deng et al. , 2009 ) is randomized smoothing ( Lecuyer et al. , 2019 ; Cohen et al. , 2019 ) . Given an input x and a network f , randomized smoothing constructs g ( x ) = E ∼D [ f ( x + ) ] such that g ( x ) = g ( x + δ ) ∀δ ∈ R , where the certification region R is characterized by x , f , and the smoothing distribution D. For instance , Cohen et al . ( 2019 ) showed that if D = N ( 0 , σ2I ) , thenR is an ` 2-ball whose radius is determined by x , f and σ . Since then , there has been significant progress towards the design of D leading to the largestR for all inputs x . The interplay betweenR characterized by ` 1 , ` 2 and ` ∞-balls , and a notion of optimal distribution D has been previously studied Yang et al . ( 2020 ) . Despite this progress , current randomized smoothing approaches provide certification regions that are isotropic in nature , limiting their capacity to certifying smaller and worst-case regions . We provide an intuitive example of this behavior in Figure 1 . The isotropic nature ofR in prior art is due to the common assumption that the smoothing distribution D is identically distributed ( Yang et al. , 2020 ; Kumar et al. , 2020 ; Levine & Feizi , 2021 ) . Moreover , comparisons between various randomized smoothing approaches were limited to methods that produce the same ` p certificate , with no clear metrics for comparing with other certificates . In this paper , we address both concerns and present new state-of-the-art certified accuracy results on both CIFAR-10 and ImageNet datasets . Our contributions are threefold . ( i ) We provide a general and simpler analysis compared to prior art ( Cohen et al. , 2019 ; Yang et al. , 2020 ) that paves the way for the certification of anisotropic regions characterized by any norm , holding prior art as special cases . We then specialize our result to regions that , for a positive definite A , are ellipsoids , i.e . ‖Aδ‖2 ≤ c , and generalized cross-polytopes , i.e . ‖Aδ‖1 ≤ c , generalizing both ` 2 ( Cohen et al. , 2019 ) and ` 1 ( Lecuyer et al. , 2019 ; Yang et al. , 2020 ) certification ( Section 4 ) . ( ii ) We introduce a new evaluation framework to compare methods that certify general ( isotropic or anisotropic ) regions . We compare two general certificates by defining that a method certifyingR1 is superior to another certifyingR2 , ifR1 is a strict superset to R2 . Further , we define a standalone quantitative metric as the volume of the certified region , and specialize it for the cases of ellipsoids and generalized cross-polytopes ( Section 5 ) . ( iii ) We propose ANCER , an anisotropic certification method that performs sample-wise ( i.e . per sample in the test set ) region volume maximization ( Section 6 ) , generalizing the data-dependent , memory-based solution from Alfarra et al . ( 2020 ) . Through experiments on CIFAR-10 ( Krizhevsky , 2009 ) and ImageNet ( Deng et al. , 2009 ) , we show that restricting ANCER ’ s certification region to ` 1 and ` 2-balls outperforms state-of-the-art ` 1 and ` 2 results from previous works ( Yang et al. , 2020 ; Alfarra et al. , 2020 ) . Further , we show that the volume of the certified regions are significantly larger than all existing methods , thus setting a new state-of-the-art in certified accuracy . We highlight that while we effectively achieve state-of-the-art performance , it comes at a high cost given the data-dependency requirements . A discussion of the limitations of the solution is presented in Section 6 . Notation . We consider a base classifier f : Rn → P ( K ) , where P ( K ) is a probability simplex over K classes , i.e . f i ≥ 0 and 1 > f = 1 , for i ∈ { 1 , . . . , K } . Further , we use ( x , y ) to be a sample input x and its corresponding true label y drawn from a test set Dt , and fy to be the output of f at the correct class . We use ` p to be the typically defined ‖ · ‖p norm ( p ≥ 1 ) , and ` Ap or ‖ · ‖A , p for p = { 1 , 2 } to be a composite norm defined with respect to a positive definite matrix A as ‖A−1/pv‖p . 2 RELATED WORK . Verified Defenses . Since the discovery that DNNs are vulnerable against input perturbations ( Goodfellow et al. , 2015 ; Szegedy et al. , 2014 ) , a range of methods have been proposed to build classifiers that are verifiably robust ( Huang et al. , 2017 ; Gowal et al. , 2019 ; Bunel et al. , 2018 ; Salman et al. , 2019b ) . Despite this progress , these methods do not yet scale to the networks the community is interested in certifying ( Tjeng et al. , 2019 ; Weng et al. , 2018 ) . Randomized Smoothing . The first works on randomized smoothing used Laplacian ( Lecuyer et al. , 2019 ; Li et al. , 2019 ) and Gaussian Cohen et al . ( 2019 ) distributions to obtain ` 1 and ` 2-ball certificates , respectively . Several subsequent works improved the performance of smooth classifiers by training the base classifier using adversarial augmentation ( Salman et al. , 2019a ) , regularization ( Zhai et al. , 2019 ) , or general adjustments to training routines ( Jeong & Shin , 2020 ) . Recent work derived ` p-norm certificates for other isotropic smoothing distributions ( Yang et al. , 2020 ; Levine & Feizi , 2020 ; Zhang et al. , 2019 ) . Concurrently , Dvijotham et al . ( 2020 ) developed a framework to handle arbitrary smoothing measures in any ` p-norm ; however , the certification process requires significant hyperparameter tuning . Similarly , Mohapatra et al . ( 2020 ) introduces larger certificates that require higher-order information , yet do not provide a closed-form solution . This was followed by a complementary data-dependent smoothing approach , where the parameters of the smoothing distribution were optimized per test set sample to maximize the certified radius at an individual input ( Alfarra et al. , 2020 ) . All prior works considered smoothing with isotropic distributions and hence certified isotropic ` p-ball regions . In this paper , we extend randomized smoothing to certify anisotropic regions , by pairing it with a generalization of the data-dependent framework ( Alfarra et al. , 2020 ) to maximize the certified region at each input point . 3 MOTIVATING ANISOTROPIC CERTIFICATES . Certification approaches aim to find the safe region R , where arg maxi f i ( x ) = arg maxi f i ( x + δ ) ∀δ ∈ R. Recent randomized smoothing techniques perform this certification by explicitly optimizing the isotropic ` p certified region around each input ( Alfarra et al. , 2020 ) , obtaining state-ofthe-art performance as a result . Despite this ` p optimality , we note that any ` p-norm certificate is worst-case from the perspective of that norm , as it avoids adversary regions by limiting its certificate to the ` p-closest adversary . This means that it can only enjoy a radius that is at most equal to the distance to the closest decision boundary . However , decision boundaries of general classifiers are complex , non-linear , and non-radially distributed with respect to a generic input sample ( Karimi et al. , 2019 ) . This is evidenced by the fact that , within a reasonably small ` p-ball around an input , there are often only a small set of adversary directions ( Tramèr et al. , 2017 ; 2018 ) ( e.g . see the decision boundaries in Figure 1 ) . As such , while ` p-norm certificates are useful to reason about worst-case performance and are simple to obtain given previous works ( Cohen et al. , 2019 ; Yang et al. , 2020 ; Lee et al. , 2019 ) , they are otherwise uninformative in terms of the shape of decision boundaries , i.e . which regions around the input are safe . To visualize these concepts , we illustrate the decision boundaries of a base classifier f trained on a toy 2-dimensional , radially separable ( with respect to the origin ) binary classification dataset , and consider two different input test samples ( see Figure 1 ) . We compare the optimal isotropic and anisotropic certified regions of different shapes at these points . In Figures 1a and 1b , we compare an isotropic cross-polytope ( of the form ‖δ‖1 ≤ r ) with an anisotropic generalized cross-polytope ( of the form ‖Aδ‖1 ≤ r ) , while in Figures 1c and 1d we compare an isotropic ` 2 ball ( of the form ‖δ‖2 ≤ r ) with an anisotropic ellipsoid ( of the form ‖Aδ‖2 ≤ r ) . Notice that in Figures 1a and 1c , due to the curvature of the classification boundary ( shown in white ) , the optimal certification region is isotropic in nature , which is evidenced by the similarities of the optimal isotropic and anisotropic certificates . On the other hand , in Figures 1b and 1d , the location of the decision boundary allows for the anisotropic certified regions to be considerably larger than their isotropic counterparts , as they are not as constrained by the closest decision boundary , i.e . the worst-case performance . We note that these differences are further highlighted in higher dimensions , and we study them for a single CIFAR-10 test set sample in Appendix A.1 . Further , we also showcase how anisotropic certification allows for further insights into constant prediction ( safe ) directions in Appendix A.2 . 4 ANISOTROPIC CERTIFICATION . One of the main obstacles in enabling anisotropic certification is the complexity of the analysis required . To alleviate this , we follow a Lipschitz argument first observed by Salman et al . ( 2019a ) and Jordan & Dimakis ( 2020 ) and propose a simple and general certification analysis . We start with the following two observations . All proofs are in Appendix B . Proposition 1 . Consider a differentiable function g : Rn → R. If supx‖∇g ( x ) ‖∗ ≤ L where ‖ · ‖∗ has a dual norm ‖z‖ = maxx z > x s.t . ‖x‖∗ ≤ 1 , then g is L-Lipschitz under norm ‖ · ‖∗ , that is |g ( x ) − g ( y ) | ≤ L‖x− y‖ . Given the previous proposition , we formalize ‖ · ‖ certification as follows : Theorem 1 . Let g : Rn → RK , gi be L-Lipschitz continuous under norm ‖ · ‖∗ ∀i ∈ { 1 , . . . , K } , and cA = arg maxi gi ( x ) . Then , we have arg maxi gi ( x+ δ ) = cA for all δ satisfying : ‖δ‖ ≤ 1 2L ( gcA ( x ) −max c gc 6=cA ( x ) ) . Theorem 1 provides an ‖ · ‖ norm robustness certificate for any L-Lipschitz classifier g under ‖ · ‖∗ . The certificate is only informative when one can attain a tight non-trivial estimate of L , ideally supx‖∇g ( x ) ‖∗ , which is generally difficult when g is an arbitrary neural network . Framework Recipe . In light of Theorem 1 , randomized smoothing can be viewed differently as an instance of Theorem 1 with the favorable property that the constructed smooth classifier g enjoys an analytical form for L = supx‖∇g ( x ) ‖∗ by design . As such , to obtain an informative ‖ · ‖ certificate , one must , for an arbitrary choice of a smoothing distribution , compute the analytic Lipschitz constant L under ‖ · ‖∗ for the smooth g. While there can exist a notion of “ optimal ” smoothing distribution for a given choice of ‖ · ‖ certificate , as in part addressed earlier for the isotropic ` 1 , ` 2 and ` ∞ certificates ( Yang et al. , 2020 ) , this is not the focus of this paper . The choice of the smoothing distribution in later sections is inspired by previous work for the purpose of granting anisotropic certificates . This recipe complements randomized smoothing works based on Neyman-Pearson ’ s lemma ( Cohen et al. , 2019 ) or the Level-Set and Differential Method ( Yang et al. , 2020 ) . We will deploy this framework recipe to show two specializations for anisotropic certification , namely ellipsoids ( Section 4.1 ) and generalized cross-polytopes ( Section 4.2 ) .1 .

Summary: The paper proposes the anisotropic version of randomized smoothing. Evaluation metrics based on the volume of the certified region are proposed, allowing comparisons with the certified regions provided from isotropic randomized smoothing. Experimental results show the usefulness of introducing anisotropic randomized smoothing as it certifies larger regions.