Tight Second-Order Certificates for Randomized Smoothing

1 INTRODUCTION . A topic of much recent interest in machine learning has been the design of deep classifiers with provable robustness guarantees . In particular , for an m-class classifier h : Rd → [ m ] , the L2 certification problem for an input x is to find a radius ρ such that , for all δ with ‖δ‖2 < ρ , h ( x ) = h ( x + δ ) . This robustness certificate serves as a lower bound on the magnitude of any adversarial perturbation of the input that can change the classification : therefore , the certificate is a security guarantee against adversarial attacks . There are many approaches to the certification problem , including exact methods , which compute the precise norm to the decision boundary ( Tjeng et al. , 2019 ; Carlini et al. , 2017 ; Huang et al. , 2017 ) as well as methods for which the certificate ρ is merely a lower bound on the distance to the decision boundary ( Wong & Kolter , 2018 ; Gowal et al. , 2018 ; Raghunathan et al. , 2018 ) . One approach that belongs to the latter category is Lipschitz function approximation . Recall that a function f : Rd → R is L-Lipschitz if , for all x , x′ , |f ( x ) − f ( x′ ) | ≤ L‖x − x′‖2 . If a classifier is known to be a Lipschitz function , this immediately implies a robustness certificate . In particular , consider a binary classification for simplicity , where we use an L-Lipschitz function f as a classifier , using the sign of f ( x ) as the classification . Then for any input x , we are assured that the classification ( i.e , the sign ) will remain constant for all x′ within a radius |f ( x ) |/L of x . Numerous methods for training Lipschitz neural networks with small , known Lipschitz constants have been proposed . ( Fazlyab et al. , 2019 ; Zhang et al. , 2019 ; Anil et al. , 2019 ; Li et al. , 2019b ) It is desirable that the network be as expressive as possible , while still maintaining the desired Lipschitz property . Anil et al . ( 2019 ) in particular demonstrates that their proposed method can universally approximate Lipschitz functions , given sufficient network complexity . However , in practice , for the robust certification problem on large-scale input , randomized smoothing ( Cohen et al. , 2019 ) is the current state-of-the-art method . The key observation of randomized smoothing ( as formalized by ( Salman et al. , 2019 ; Levine et al. , 2019 ) ) is that , for any arbitrary base classifier function f : Rd → [ 0 , 1 ] , the function x→ Φ−1 ( pa ) where pa ( x ) : = E ∼N ( 0 , σ2I ) f ( x + ) ( 1 ) is ( 1/σ ) -Lipschitz , where N ( 0 , σ2I ) is a d-dimensional isometric Gaussian distribution with variance σ2 and Φ−1 is the inverse normal CDF function . As a result , given the smoothed classifier value pa ( x ) at x , one can calculate the certified radius ρ ( x ) = σΦ−1 ( pa ( x ) ) in which pa ( x ) ≥ 0.5 ( i.e. , Φ−1 ( pa ( x ) ) ≥ 0 ) . This means that we can use pa ( x ) ∈ Rd → [ 0 , 1 ] as a robust binary classifier ( with one class assignment if pa ( x ) ≥ 0.5 , and the other if pa ( x ) < 0.5 ) . Cohen et al . ( 2019 ) shows that this is a tight certificate result for a classifier smoothed with Gaussian noise : given the value of pa ( x ) , there exists a base classifier function f such that , if pa is the Gaussian-smoothed version of f , then there exists an x′ with ‖x− x′‖2 = ρ such that pa ( x′ ) = 0.5 . In other words , the certificate provided by ( Cohen et al. , 2019 ) is the largest possible certificate for Gaussian smoothing , given only the value of pa ( x ) . Previous results ( Li et al. , 2019a ; Lecuyer et al. , 2019 ) provided looser bounds for Gaussian smoothing . Singla & Feizi ( 2020 ) have recently shown , for shallow neural networks , that , rather than globally bounding the ( first-order ) Lipschitz constant of the network , it is possible to achieve larger robustness certificates by instead globally bounding the Lipschitz constant of the gradient of the network . This second-order , curvature-based method takes advantage of the fact that the gradient at x can be computed easily via back-propagation , so certificates can make use of both f ( x ) and∇xf ( x ) . This leads to a question : can we also use the gradient of a smoothed classifier ∇xpa ( x ) to improve smoothing-based certificates ? In this work , we show that there is a universal curvature-like bound for all randomly-smoothed classifiers . Therefore , given pa ( x ) and∇xpa ( x ) , we can compute larger certificates than is possible using the value of pa ( x ) alone . Moreover , our bound is tight in that , given only the pair ( pa ( x ) , ∇xpa ( x ) ) , the certificate we provide is the largest possible certificate for Gaussian smoothing . We call our certificates “ Second-order Smoothing ” ( SoS ) certificates . As shown in Figure 1 , the smoothing-based certificates which we can achieve using second-order smoothing represent relatively modest improvements compared to the first-order bounds . This is a meaningful negative result , given the tightness of our bounds , and is therefore useful in guiding ( or limiting ) future research into higher-order smoothing certificates . Additionally , this result shows that randomized smoothing ( or , specifically , functions in the form of Equation 1 ) can not be used to universally approximate Lipschitz functions : all randomly smoothed functions will have the additional curvature constraint described in this work . If the base classifier f is a neural network , computing the expectation in Equation 1 analytically is not tractable . Therefore it is standard ( Lecuyer et al. , 2019 ; Cohen et al. , 2019 ; Salman et al. , 2019 ) to estimate this expectation using N random samples , and bound the expectation probabilistically . The certificate is then as a high-probability , rather than exact , result , using the estimated lower bound of pa ( x ) . In Section 3.1 , we discuss empirical estimation of the gradient norm of a smoothed classifier for second-order certification , and develop an estimator for this quantity , in which the number of samples required to estimate the gradient scales linearly with the dimensionality d of the input.1 In order to overcome this , in Section 4 , we develop a modified form of Gaussian randomized smoothing , Gausian Dipole Smoothing , which allows for a dipole certificate , related to the secondorder certificate , to be computed . Unlike the second-order certificate , however , the dipole certificate has no explicit dependence of dimensionality in its estimation , and therefore can practically scale to real-world high-dimensional datasets . 2 PRELIMINARIES , ASSUMPTIONS AND NOTATION . We use f ( x ) to represent a generic scalar-valued “ base ” function to be smoothed . In general , we assume f ∈ Rd → [ 0 , 1 ] . However , for empirical estimation results ( Theorem 3 ) , we assume that f is a “ hard ” base classifier : f ∈ Rd → { 0 , 1 } . This will be made clear in context . The smoothed version of f is notated as pa ∈ Rd → [ 0 , 1 ] , defined as in equation 1 . Recall that Φ is the normal CDF function and Φ′ is the normal PDF function . In randomized smoothing for multi-class problems , the base classifier is typically a vector-valued function f ∈ Rd → { 0 , 1 } m , ∑ c fc ( x ) = 1 , where m is the number of classes . The final classification returned by the smoothed classifier is then given by a : = arg maxc E fc ( x + ) . However , in most prominent implementations ( Cohen et al. , 2019 ; Salman et al. , 2019 ) , certificates are computed using only the smoothed value for the estimated top class a , where a is estimated using a small number N0 of initial random samples , before the final value of pa ( x ) is computed using N samples . The certificate then determines the radius in which pa ( x′ ) will remain above 0.5 : this guarantees that a will remain the top class , regardless of the other logits . While some works ( Lecuyer et al. , 2019 ; Feng et al. , 2020 ) independently estimate each smoothed logit , this incurs additional estimation error as the number of classes increases . In this work , we assume that only estimates for the top-class smoothed logit pa ( x ) and its gradient ∇xpa ( x ) are available ( although we briefly discuss the case with more estimated logits in Section 3.2 ) . When discussing empirical estimation , we use η as the accepted probability of failure of an estimation method . 3 SECOND-ORDER SMOOTHING CERTIFICATE . We now state our main second-order robustness certificate result : Theorem 1 . For all x , x′ with ‖x− x′‖2 < ρ , and for all f : Rd → [ 0 , 1 ] , pa ( x ′ ) ≥ Φ ( Φ−1 ( a′ + pa ( x ) ) − ρ σ ) − Φ ( Φ−1 ( a′ ) − ρ σ ) ( 2 ) where a′ is the ( unique ) solution to Φ′ ( Φ−1 ( a′ ) ) − Φ′ ( Φ−1 ( a′ + pa ( x ) ) ) = −σ‖∇xpa ( x ) ‖2 . ( 3 ) Further , for all pairs ( pa ( x ) , ‖∇xpa ( x ) ‖2 ) which are possible , there exists a base classifier f and an adversarial point x′ such that Equation 2 is an equality . This implies that our certificate is realizable , and therefore tight . Note that the right-hand side of Equation 2 is monotonically decreasing with ρ : we can then compute a robustness certificate by simply setting pa ( x′ ) = 0.5 and solving for the certified radius ρ . Also , 1In a concurrent work initially distributed after the submission of this work , Mohapatra et al . ( 2020 ) have proposed an identical second-order smoothing certificate , along with a tighter empirical estimator for the gradient norm . In this estimator , the number of samples required scales with √ d. a′ can be computed easily , because the left-hand side of Equation 3 is monotonic in a′ . Evaluated certificate values are shown in Figure 1-b , and compared with first-order certificates . All proofs are presented in Appendix A . Like in Cohen et al . ( 2019 ) , we proceed by constructing the worst-case base classifier f given pa ( x ) and ‖∇xpa ( x ) ‖2 . This is the base classifier f which creates an adversarial point to the smoothed classifier as close as possible to x , given the constraints that pa ( x ) and ‖∇pa ( x ) ‖2 are equal to their reported values . In Cohen et al . ( 2019 ) , given only pa ( x ) , this is simply a linear classifier . With the gradient norm , the worst case is that x lies in a region with class a which is a slice between two linear decision boundaries , both perpendicular to ∇pa ( x ) . See Figure 3 . Note that , by isometry and because ∇pa ( x ) is the only vector information we have , there is no benefit in certified radius to having the direction of ∇pa ( x ) : the norm is sufficient . In the case of a linear classifier the gradient takes its maximum possible value : ‖∇xpa ( x ) ‖2 = σ−1Φ′ ( Φ−1 ( pa ( x ) ) . This case is shown in Figure 3-a : if the gradient norm is equal to this value , the second-order certificate is identical to the first-order certificate ( Cohen et al. , 2019 ) . However , if the gradient norm is smaller , then we can not be in this worst-case linear-classifier scenario . Instead , the new “ worst case ” is constructed by introducing a second “ wrong class ” region opposite to the direction of the adversarial point ( Figure 3-b ) . In the extreme case ( Figure 3-c ) where the gradient norm is zero , this is accomplished by balancing two adversarial regions in a “ sandwich ” around x . This “ sandwich ” configuration reveals the relative weakness of gradient information in improving robustness certificates : having zero gradient does not require that the adversarial regions be evenly distributed around x . Rather , it is sufficient to distribute the adversarial probability mass 1− pa ( x ) into just two adversarial regions . Therefore , the certified radius , even in this most extreme case , is similar to the Cohen et al . ( 2019 ) certificate in the case with half as much adversarial probability mass ( the first-order certificate for pa ( x ) : = ( 1 + pa ( x ) ) /2 ) . This can be seen in Figure 1-b : note that at pa ( x ) = 0.6 , if the gradient norm is known to be zero , the certificate is slightly below the certificate for pa ( x ) = 0.8 with no gradient information . The second-order certificate when ( pa ( x ) = 0.6 , ‖∇xpa ( x ) |‖2 = 0 ) is in fact slightly below the first-order certificate for pa ( x ) = 0.8 , because the Gaussian noise samples throughout all of space , so the smoothed classifier decision boundary is slightly affected by the adversarial region in the opposite direction of x . Because we can explicitly construct “ worst-case ” classifiers which represent the equality case of Equation 2 , our certificates are known to be tight : the reported certified radii are the largest possible certificates , if only pa ( x ) and ‖∇pa ( x ) ‖2 are known . In Figure 2 , we show how our second-order certificate behaves on a simple , two-dimensional , nonlinearly separable dataset , the classic Swiss Roll . The increases are marginal , mostly because the certificates using standard randomized smoothing are already fairly tight . On these data , the certified radii for the two classes are nearly touching in many places along the decision boundary . However , for the blue class , which is surrounded on multiple sides by the red class , there are noticeable increases in the certified radius . This is especially true for points near the center of the blue class , which are at the “ top of the hill ” of the blue class probability , and therefore have smaller gradient .

Summary: This paper presents a randomized second-order smoothing certificate for providing robustness guarantees against adversarial attacks. By additionally using the gradient estimation of smoothed classifier, the proposed method has been shown to outperform the existing randomized smoothing certificate in practice. A variant of the method without explicitly estimating gradient vector has also been proposed to avoid the dependence of feature dimension in concentration analysis.