Tight Second-Order Certificates for Randomized Smoothing
Authors: Alexander Levine
Aounon Kumar
Tom Goldstein
Soheil Feizi
Venue:  (2021)
Keywords: certificates
adversarial
robustness
defenses
smoothing
curvature

Abstract: Randomized smoothing is a popular way of providing robustness guarantees against adversarial attacks: randomly-smoothed functions have a universal Lipschitz-like bound, allowing for robustness certificates to be easily computed. In this work, we show that there also exists a universal curvature-like bound for Gaussian random smoothing: given the exact value and gradient of a smoothed function, we compute a lower bound on the distance of a point to its closest adversarial example, called the Second-order  Smoothing (SoS) robustness certificate. In addition to proving the correctness of this novel certificate, we show that SoS certificates are realizable and therefore tight. Interestingly, we show that the maximum achievable benefits, in terms of certified robustness, from using the additional information of the gradient norm are relatively small: because our bounds are tight, this is a fundamental negative result. The gain of SoS certificates further diminishes if we consider the estimation error of the gradient norms, for which we have developed an estimator. We therefore additionally develop a variant of Gaussian smoothing, called Gaussian dipole smoothing, which provides similar bounds to randomized smoothing with gradient information, but with much-improved sample efficiency. This allows us to achieve (marginally) improved robustness certificates on high-dimensional datasets such as CIFAR-10 and ImageNet.

Under review as a conference paper at ICLR 2021
TIGHT SECOND-ORDER CERTIFICATES FOR RANDOMIZED SMOOTHING
Anonymous authors Paper under double-blind review
ABSTRACT
Randomized smoothing is popular way of providing robustness guarantees against adversarial attacks: randomly-smoothed functions have universal Lipschitz-like bound, allowing for robustness certificates be easily computed. In this work, we show that there also exists universal curvature-like bound for Gaussian random smoothing: given the exact value and gradient of smoothed function, we compute lower bound on the distance of point to its closest adver- sarial example, called the Second-order Smoothing (SoS) robustness certificate. In addition to proving the correctness of this novel certificate, we show that SoS certificates are realizable and therefore tight. Interestingly, we show that the max- imum achievable benefits, in terms of certified robustness, from using the addi- tional information of the gradient norm are relatively small: because our bounds are tight, this is fundamental negative result. The gain of SoS certificates further diminishes if we consider the estimation error of the gradient norms, for which we have developed an estimator. We therefore additionally develop variant of Gaus- sian smoothing, called Gaussian dipole smoothing, which provides similar bounds randomized smoothing with gradient information, but with much-improved sample efficiency. This allows us to achieve (marginally) improved robustness certificates on high-dimensional datasets such as CIFAR- 10 and ImageNet. Code is available at httpt:ts:///// /tgs::////ttte hhub, com wwwoo//aevinnnnl /tt./tmoottttioo _eccond_eddd_ ordex
INTRODUCTION
A topic of much recent interest in machine learning has been the design of deep classifiers with provable robustness guarantees. In particular, for an m-class classifier h : IRd + [m]] the Le certification problem for input is find radius such that, for all 8 with ||ö]|2 < p. h(x) h(x + 8). This robustness certificate serves as a lower bound on the magnitude of any adversarial perturbation of the input that can change the classification: therefore, the certificate security guarantee against adversarial attacks.
There are many approaches the certification problem, including exact methods, which compute the precise norm to the decision boundary (Tjeng et al., 2019 Carlini et al. 2017 Huang et al., 2017) as well as methods for which the certificate P merely a lower bound on the distance the
decision boundary (Wong& & Kolter 2018 Gowal et al. 2018 Raghunathan et al. 2018).
One approach that belongs the latter category is function approximation. Recall that function f Rd - IR is L-Lipschitz if, for all x', [x(x) fxx)) < L||x x'122. If a classifier known to be a Lipschitz function, this immediately implies robustness certificate. In particular, consider a binary classification for simplicity, where we use an I.-Lipschitz function f as a classifier, using the sign of / (x) as the classification. Then for any input x, we are assured that the classification (i.e, the sign) will remain constant for all x' within a radius /x xf
Numerous methods for training Lipschitz neural networks with small, known Lipschitz constants have been proposed. (Fazlyab et al. 2019 Zhang al. 2019 Anil et al. 2019 Li et al., 2019b) It is desirable that the network be as expressive as possible, while still maintaining the desired Lipschitz property. Anil al. (2019) in particular demonstrates that their proposed method can universally approximate Lipschitz functions, given sufficient network complexity. However, in practice, for the robust certification problem large-scale input, randomized smoothing (Cohen et al. 2019 is the

---

Under review as a conference paper at ICLR 2021
(a)
(b)
0
1.75
Secoedoonoo
catilizne
pv00 0.8
1.50-
J-00 0.
Certitee Rtdtuss
1.25
Nntilione
Coben
(p.(0))
04-
Coben al setifeaee
1.00
Secced onler catificat:
-04
0.00
0.0
0.1
033
0.5
Figure 1: (a) Tight lower bound on the value of smoothed function at x' (i.e. Pa (x()) as a function of ||x x-12- In this example, (x) = 0.8 and the smoothing standard deviation o 1. The red line shows the lower bound for the function, with no information about the gradient given. The blue line incorporates the additional information that [x)alx) O. Note that the axis at @ -- (p) = corresponds to Po (x) = 0.5, the decision boundary for a binary classifier. (b) Tight robustness certificates for a randomized-smoolke
classifier, given the top-class value (x and the gradient norm V (x) 22 The dashed lines show the certificates given Pa (x(x) alone. Note that the maximum possible gradient for smoothed classifier depends Pa (x) (see Equation]] Ip.
current state-of-the-art method. The key observation of randomized smoothing (as formalized by Salman et al. 2019 Levine et al. 2019)) is that, for any arbitrary base classifier function f Rd 1]. the function
x x 1 x-10)pp)) (po) where Pa (a(x): =
E f(x+0)
(I)
c=1(0.0021)
is (1/s)-Lipschitz, where 020 is d-dimensional isometric Gaussian distribution with vari- ance o² and & is the inverse normal CDF function. As a result, given the smoothed classifier value (x) at one can calculate the certified radius p(x) σ (P)(x)) in which Px(x) 2 0.5 (i.e., $ (p- (x)) 2 0). This means that we can use (x) de
[0, 1 as a robust binary classifier (with one class assignment if Pa (x) 2 0.5, and the other if Pa (x) < Cohen et al. (2019) shows Cohen
that this a tight certificate result for classifier smoothed with Gaussian noise: given the value of Pa (x), there exists a base classifier function f such that, if pa is the Gaussian-smoothed version of f. then there exists an x' with
x' ||| = P such that (x') 0.5. In other words, the certificate provided by (Cohen et al. 2019) is the largest possible certificate for Gaussian smoothing, given only the value of pa (x). Previous results (Li et al. 2019a Lecuyer et al., 2019) provided looser
bounds for Gaussian smoothing.
Singla & Feizi (020) have recently shown, for shallow neural networks, that, rather than globally bounding the (first-order) Lipschitz constant of the network, is possible to achieve larger robust- ness certificates by instead globally bounding the Lipschitz constant of the gradient of the network. This second-order, curvature-based method takes advantage of the fact that the gradient at x can be computed easily via back- proppaaapaat,onn certificates can make use of both f(x) and vxf(x).
This leads question: can we also use the gradient of a smoothed classifier VxPa (x) im- prove smoothing- certificates? In this work, we show that there is a universal curvature- bound for all aandomly-smootiee larger certificates than is possible using the value of Pa (x) alone. Moreover, our bound is tight in that, given only the pair (p. (x), VxPax)), the certificate we provide is the largest possible certifi- cate for Gaussian smoothing. We call our certificates "Second-order Smoothing" (SoS) certificates. As shown in Figure I the smoothing based certificates which we can achieve using second-ord -rddr smoothing represent relatively modest improvements compared to the first-order bounds. This is meaningful negative result, given the tightness of our bounds, and therefore useful in guiding (or limiting) future research into higher-order smoothing certificates. Additionally, this result shows that
classifiers. Therefore, given (x) and VxPa(x), we can compute
2

---

Under review as a conference paper at ICLR 2021
randomized smoothing (or, specifically. functions in the form of Equation 1 can not be used to uni- versally approximate Lipschitz functions: all randomly smoothed functions will have the additional curvature constraint described in this work.
If the base classifier f is a neural network, computing the expectation in Equation D analytically is not tractable. Therefore it is standard (Lecuyer al. 2019 Cohen et al. 2019 Salman et al. 2019) to estimate this expectation using N random samples, and bound the expectation probabilistically.
The certificate is then as high-probability, rather than exact, result, using the estimated lower bound of Pa (x). In Section we discuss empirical estimation of the gradient norm of a smoothed classifier for second-order certification, and develop an estimator for this quantity, in which the number of samples required to estimate the gradient scales linearly with the dimensionality d of the
input!] In order to overcome this, in Section (4] we develop a modified form of Gaussian randomized smoothing, Gausian Dipole Smoothing, which allows for dipole certificate, related to the second- order certificate, to be computed. Unlike the second-order certificate, however, the dipole certificate has no explicit dependence of dimensionality in its estimation, and therefore can practically scale to real-world high-dimensional datasets.
2 PRELIMINARIES, ASSUMPTIONS AND NOTATION
We use f(x) to represent generic scalar-valued "base" function to be smoothed. In general, we assume f E Rd - 10, 1]. However, for empirical estimation results (Theorem 3), we assume that f is a "hard" base classifier: I R2d + {0. 1}. This will be made clear context. The smoothed version of I is notated Pa E RRd + [0, 1), defined as in equation
Recall that & is the normal CDF function and क' is the normal PDF function. In random- ized smoothing for multi-class problems, the base classifier is typically a vector-valued function E Rd + {0, 11mh, >. fx(x) = 1, where m is the number of classes. The final classification returned by the smoothed classifier is then given by prominent implementations (Cohen et al.] 2019 Salman et al.) 2211), certificates are computed us- ing only the smoothed value for the esti top class a, where is estimated using small number No of initial random samples, before the final value of Pa Px(x) is computed using N samples. The certificate then determines the radius in which Pa (x') will remain above 0.5: this guarantees that will remain the top class, regardless of the other logits. While some works (Lecuyer et al.) 22119; Feng et al. 2020) independently estimate each smoothed logit, this incurs additional estimation er- ror as the number of classes increases. In this work, we assume that only estimates for the top-class smoothed logit (x) and its gradient VxPa (x) are available (although we briefly discuss the case with more estimated logits in Section 3.2). When discussing empirical estimation, we use v as the accepted probability of failure an estimation method.
arg max, E. fx + c). However, in most
SECOND-ORDER SMOOTHING CERTIFICATE
We now state our main second-order robustness certificate result:
Theorem 1. For all x' with ||x x'112 p. and for all f Rd - 10. 1].
(x('') 2 +b (a' + Pa(x)) - () +b ( (a') 2)
(2)
where a' is the (unique) solution to
Φ' (a')) - 0' + +a((x)) =
(3)
Further, for all pairs (pa (x), xx) (2) which are possible, there exists base classifier f and an adversarial point x' such that Equation 2 is an equality. This implies that our certificate is realizable, and therefore tight.
Note that the right-hand side of Equation 2 is monotonically decreasing with p: we can then compute a robustness certificate by simply setting Pa (x') = 0.5 and solving for the certified radius p. Also,
II concurrent work initially distributed after the submission of this work, MMoapptra et al. (020) have proposed an identical second-order smoothing certificate, along with a tighter empirical estimator for the gra- dient norm. In this estimator, the number of samples required scales with vd.
3

---

Under review as a conference paper at ICLR 2021
(a)
[6]
(c)
Figure 3: Worst case base classifiers for second-order smoothing for the same value of (x(x) at different values of Vxp (x) The base classifier is f in the blue regions and f 0 in the red regions. The point x is shown as a blue dot, with the Gaussian sampled region used for calculating Pa (x) approximately shown as dashed blue cirele. VxPa(x) is shown as a blue arrow. (a) The gradient takes its maximum possible value: V = o "'(""'(((>>>>" (b) The gradient has an intermediate value: x/Vxp. (x) < -l' 1000-11p (0=+((x(x)) (c) The gradient is zero: || Vx)a(x)ll = 0.
a' can be computed easily, because the left-hand side of Equation 3 monotonic in a'. Evaluated certificate values are shown in Figure b and compared with first-order certificates.
All proofs are presented in Appendix A Like in Cohen et al.] (2019). we proceed by constructing the worst-case base classi- fier / given Pa (x) and (x))2x))22 This is the base classifier f which creates an adversarial point to the smoothed classi- fier as close as possible to x, given the constraints that Pa (x) and [/Vpa())) are equal to their reported values. In Cohen al. (2019), given only pa (x). this is simply a linear classifier. With the gra- dient norm, the worst case that x lies in a region with class which is a slice between two linear decision boundaries, both perpendicular to (x). See Fig- ure 3 Note that, by isometry and be- cause (x) is the only vector infor- mation we have, there no benefit certified radius to having the direction of Vpa(x); the norm is sufficient. the case of a linear classifier the gradient takes its maximum possible value: VxPa (x)]l2 ~''''0 ( -1 (x)). This case shown Figure if the gradient norm is equal to this value, the second-order certificate is identical to the first-order certificate (Co- hen et al. 2019). However, if the gradient norm is smaller, then we cannot be in this worst-case linear-classifier scenario. Instead, the new "worst case" constructed by introducing a second "wrong class" region opposite to the direction of the adversarial point (Figure Bb) In the extreme case (Figure 3cc) where the gradient norm is zero, this is accomplished by balancing two adversarial regions in "sandwich" around x.
Figure 2: Comparison of second-order smoothing cer- tificates to standard Gaussian smoothing certificates on selection of points from the Swiss Roll dataset. Cor- rectly labeled points with (second-order certificates are shown in light red and blue, and points with in- correct label or no certificate are in black. For a selec- tion of points, shown in red/blue, the first-order certi- fied radii shown are as red/blue rings. Increases to cer- tiñied radii due to second-order smoothing shown are as light blue (light red, absent) rings around certificate radii. For both experiments, N 10 and v = 0.001.
This "sandwich" configuration reveals the relative weakness gradient information in improving robustness certificates: having gradient does not require that the adversarial regions be evenly distributed around x. Rather, is sufficient to distribute the adversarial probability mass 1 1- (x.(() into just two adversarial regions. Therefore, the certified radius, in this most extreme case, is
4

---

Under review as a conference paper at ICLR 2021
similar to the Cohen al. (2019) certificate in the case with half much adversarial probability mass (the first-order certificate for (x) := (1 +1+)) Pa (x)))/2)). This can be seen in Figure b note 0.6, if the gradient norm is known to be zero, the certificate is slighily below the certificate for Pa (x) = 0.8 with no gradient information. The second-orde certificate when (x)(x) 0.6, VPPa (x)]ll2 = 0) is in fact slightly below the first-order certificate for Pa Pa(x) 0.8, because the Gaussian noise samples throughout all of space, so the smoothed classifier decision boundary slightly affected by the adversarial region in the opposite direction of x.
that at (x)
Because we can explicitly construct "worst-case" classifiers which represent the equality case of Equation 2 our certificates are known be tight: the reported certified radii are the largest possible certificates, if only Pa (x) and [x) are known.
In Figure 2 we show how our second-order certificate behaves on simple, two-dimensional, non- linearly separable dataset, the classic Swiss Roll. The increases are marginal, mostly because the certificates using standard randomized smoothing are already fairly tight. On these data, the certified radii for the two classes are nearly touching in many places along the decision boundary. However, for the blue class, which is surrounded on multiple sides by the red class, there are noticeable increases in the certified radius. This is especially true for points near the center of the blue class, which are at the "top of the hill" of the blue class probability, and therefore have smaller gradient.
3.1 GRADIENT NORM ESTIMATION
In order to the second-order certificate in practice, must first bound, with sigh-probbbilitt, the gradient norm VVV (x)]l2 using samples from the base classifier f. Because Theorem pro- vides certificates that are strictly decreasing with (x)axx)]]. it is only necessary to lower bound ||VxPa (x))a(x)]2 with high probability.
Salman al. 2019) suggest ways of approximate the gradient vector VxPa(x) itself, both based on the following important observation:
V (x)
E [Vx[(x + c]]=
E
[cf(x c))/o²
(4)
c-N10,021)
c-N(00,021)
These two methods are:
At each sampled point, one can measure the gradient of using back-propagation, and take the mean vector of these estimates.
2. At each sampled point, one can multiply f(x + c) by the noise vector e, and take the mean vector of these estimates.
Note, however, that Salman et al. (2199) does not provide statistical bounds on these estimates; for our certificate application, we must do so. While we ultimately use an approach based on method 2, we will first briefly discuss method 1. The major obstacle to using method is that it requires that the base classifier f itself to be Lipschitz function, with a small Lipschitz. constant. This can be understood from Markov's inequality. For example, consider the value of some component 2(x) == u fx)), where u is an arbitrary vector. Suppose samples are taken, but that z is distributed such that:
0 with probability 1 2N
x + e)
(5)
2N with probability 2N
This would be the case if is a function that approximates step function from 0 to 1, with a small buffer region of very high slope, for example. Note that the probability that any of the N samples measures the nonzero gradient component is < 0.5, but the expected value of this component is in fact 1.0. This example shows that, in order to accurately estimate the gradient with high probability, the number of samples used must at least scale linearly with the maximum possible value of the gradient norm for f. For un-restricted deep neural networks, Lipschitz constants are NP-hard to compute, and upper bounds on them are typically very large (Virmaux & Scaman 2018). Of course, we could use Lipschitz-constrainee networks as described in Section for the base classifier, but this would defeat the purpose of using randomized smoothing in the first place. Moreover, in standard "hard" randomized smoothing as typically implemented (Cohen et al. 2019 Salman et al. 2019), the range of f is {0, 1), so f is non-differentiable: therefore, this back Boock-ropagatton method can not be used at all.
5

---

Under review as a conference paper at ICLR 2021
(a)
(b)
Population second order certificate
14:00 U.
Empinical second order cotiifizate
0,00
0.5
180
14000
1.25
Empiriall Cohen et cottificttc
0.50
80
6.1
0.5
11.
655
Figure 4: (a) Empirical second-order smoothing certificates, with d 49 (corresponding 7 MNIST experiments), N = 10 and .001 (b) Worst case classifier for dipole smoothing.
We therefore method 2. In particular, reject the naive approach of estimating each component independently. taking union bound, and the taking the norm: not only would the error in the norm- squared scale with as the error from each component accumulates, but there would be an additional dependence on d from the union bound: each component would have to be bounded with failure probability 7/dd, where 1) the total failure probability for measuring the gradient norm. Note that this issue will also be encountered in method above, but in that case, a loose upper bound could at least be achieved without this dependency using Jensen's 's inequality (the mean of the norms of gradient is larger than the norm of the mean).
Instead, we estimate the norm-squared of the mean using a single, unbiased estimator. Note that:
[fxx, + e) = o - 'E lef(x + - [ef(xx +c]
(6)
BR. [[e][[[++x]]]]]]][( (e
In other words, we can estimate the norm coun-e of the mean by taking pairs of smoothing samples, and taking the dot product of the noise vectors times the product of the sampled values. We show that this is subexponential random variable (see Appendix), which gives an asymptotically linear scaling of with
(e and V be its empirical estimate. If n pairs Theorem 2. Let V == Ee,e ((e(x + +))
of samples (= N/2) are used to estimate V, then, with probability most E[V] - V 2 t, where:
40
In(n))
if 2222n(1)) s dn
t
4w20² In(
if 2f > dn
Note that in practice, (x). However, this requires reducing the failure probability each estimate to 1/' = n/2, in order use union bound. This means that, if N is small (or d large), second-order smoothing can in fact give worse certificates than standard smoothing. because the benefit of a (loose, for N small) estimate of the gradient is less significant than the negative effect of reducing the estimate of Pa (x). As shown in Figure 4aa even for very large N and relatively small dimension, the empirical estimation significantly reduces the radii of certificates which be calculated. See Section 5 for experimental results.
the same samples to estimate V/V
used to estimate
3.2 UPPER-BOUND AND MULTI-CLASS CERTIFICATES
We can easily convert Theorem I into a tight upper bound on Pa (x') by simply evaluating it for f' (and therefore P/2 P.). If eestimatte and gradients available for multiple classes,

---

Under review as a conference paper at ICLR 2021
it would then be possible to achieve even larger certificate, by setting the lower bound of the top logit equal to the upper bounds of each of the other logits. Note, however, that unlike first-order smoothing works (Lecuyer et al. 2019 Feng et al. 2020) which use this approach, it is not suffi- cient to compare against just the "runner-up" class, because other logits may have less restrictive upper-bounds due to having larger gradients. As discussed above, gradient norm estimation can be computationally expensive, so gradient estimation for many classes may not be feasible. Also, note that while this approach would produce larger, correct certificates, we do not claim that these would be right certificates given the value and gradient information for all classes: the "worst case" constructions we deseribe above for single logit might not be simultaneously construct-able for multiple logits.
4 DIPOLE SMOOTHING
For large-scale image datasets, the dependence on d in Theorem 2 can create statistical barriers. However, the general approach of second-order smoothing, especially using the discrete estimation method (method 2) described above, has an interesting interpretation: rather than using simply the mean of f(x + <), we are also using the geometrical distribution the values of f(x + e) in space compute a larger certified bound. In particular, if we show that points which are adversarial for the base classifier (points with f(x + c) 0) are dispersed, then this will imply larger certificates, because it makes it impossible for perturbation in single direction move towards the adversarial region. Second-order smoothing, above, is merely an example of this.
We therefore introduce Gaussian Dipole smoothing. This is method which, like second-order smoothing, also harnesses the geometrical distribution the values of f(x) to improve certificates. However, unlike second-order smoothing, there is explicit dependence dipole smoothing bound. In this method, when we sample f(x - E) when estimating Pa (x), we also sample f(x -- e). This allows us compute two quantities:
d in the empirical
C = =)([((++))((x C.V - ((
(8)
The certificate we can calculate is then as follows:
Theorem 3. For all x, xf' with || x 2'12 < p. and for all f I2d +10.11
10,1].
Pa (x)) 2 (4 (CM) 2) + + (1
1 S
0
x-11
P
(9)
2
2
We also compute this bound by constructing the worst possible classifier. In this case, the trick is that, if two adversarial sampled points are opposite one another (i.c., f(x+e) = fx--)) 0) then they cannot both contribute the same adversarial "direction".. the worst case, the "reflected" adversarial points form a plane opposite the base classifier decision boundary (See Figure b). In the extreme case where C 0. the "worst case" classifier is the same as for second-order smoothing.
Experimentally, we simply need lower-hound both CS and C from samples. This reduces the precision of our estimates, for two reasons: we have half as many independent samples for the same number of evaluations we must perform, and we are bounding two quantities, which requires halving the error probability for each. However, unlike second-order smoothing, there is no dependence on d: this allows for practical certificates of worl-world datasets.
5 EXPERIMENTS
Experimental results are presented in Figures 5 and 6 with further results in Appendix B Be-
cause both dipole and second-order certificates reduce the precision with which empirical quantities needed for certification can estimated, but both provide strictly larger certificates at the popula- tion level, the key question becomes what number of samples N does each higher-order method

---

Under review as a conference paper at ICLR 2021
(b)
Upplle MMSSs. embuzh, N--g*
802
703
703
603
601
592
502
11% C00
Chreett oet
15%
(c)
(d)
Ni-10
Dipote. MNSTT. g.0.25, No10*
300
700
200-
440
500
3000
(e)
N-10"
Uipule, MMtill ambuut N-jo!
703
503
603
592
302
150 200
Coreet oet
(g)
(h)
N-11P
Diplle, 757 MNST, 0-0.25 H=10"
700
500
400
500
1nh-
10.37%
%%
2000
20064
Figure 5: Experiments on 7 7 MNIST. Reported the distribution of the improvement (or reduc- tion) of bigher-order certificates from certificates computed using standard (first-order) randomized smoothing, for each tested image. For all, a 0.25. For (a, c, e, g), Second-order Smoothing used. For (b, d, f, h), Gaussian dipole smoothing is used. For (a, b), N 10 For (c, d), N 10 For (c, f), N 10". For (g. h), 10a.
8

---

Under review as a conference paper at ICLR 2021
(a)
(b)
Opplle CIFAR, c-022.N-10
Dlogre. OFFRR ow0.55, N-1225.N-100
400
355
.00
250
550 10039
(c)
(d)
N=10
Dipele,
100
10% 200
20
Figure 6: Dipole smoothing experiments, with o 0.25, on CIFAR-I0 (a.b) and ImageNet (c,d).
For (a,c), 10 For (b,d), N 10
become beneficial. Note that in the figures, we are comparing the new methods to standard smooth- ing, using the same N for standard smoothing for the new method. Due to the poor scaling of second-order certificates with dimension, we tested second-order smoothing on a low-dimensional dataset, 7 x 7 MNIST. However, significant increases to certificates were not seen until N 107 even on this dataset. By contrast, dipole smoothing is beneficial for many images even when smaller numbers of smoothing samples are used. Because it scales to higher-dimensionall data, also tested Gaussian dipole smoothing on CIFAR-10 and ImageNet. where led to modest improve- ments in certificates, in particular N 10 In Appendix C show the absolute, rather than relative, certified accuracy curves for the experiments shown in Figures 5 and 6 These plot show that higher-order smoothing techniques (SoS and Gaussian Dipole smoothing) for increasing the certificates of images with small certified radii. In cases where certificates are already large, increased estimation eиrоr can lead a decrease certificates, but this effect small relative to the magnitudes of these certificates (typically < 1%).
mostly beneficial
6 CONCLUSION
In this work, we explored the limits of using gradient information to improve randomized smooth- ing certificates. In particular, introduced second-order smoothing certificates and showed tight and realizable upper bounds their maximum achievable benefits. We also proposed Gaussian dipole smoothing, a novel method for robustness certification, which can improve smoothing based robustness certificates even large-scale data sets. This introduces broader question for future work: what other information about the spacial distribution classes randomized smoothing can be efficiently used to improve robustness certificates?
REFERENCES
Cem Anil, James Lucas, and Roger Grosse. Sorting out Lipschitz function approximation. vol- ume 97 of Proceedings of Machine Learning Research, pp. 291-301, Long Beach, Califor- nia, USA, 09-15 Jun 2019. PMLR. URL http::/ppocceeaii
mlr press/99///97
ani Unil19an htmll
Nicholas Carlini, Guy Katz, Clark Barrett, and David L Dill. Provably minimally-distorte adver- sarial examples. arXiv preprint arXiv:1709, 10207, 2017.

---

Under review as a conference paper at ICLR 2021
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Research, pp. 1310-1320, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http: //proceed ngg ml press c 9c. html
Machine Learning, volume 97 of Proceedings of Machine Learning
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. In Advances in Neural Information Processing Systems, pp. 1127--14338, 2019.
Huijie Feng, Chunpeng Wu, Guoyang Chen, Weifeng Zhang, and Yang Ning. Regularized training and tight certification for randomized smoothed classifier with provable robustness. In The Thirty- Artificial Intelligence, AAAI 2020, The Thirty-Seoond Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 3858-3865, AAAI Press, 2020. URL httph: http://aaa.i.oo wwt.ooggg///oooooo /itpxxinxxxxx//oo php/
Fourth AAAI Conference
AAAI/article/vieeeee..... 5798
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Ue- sato, Relja Arandjelovie, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv: 11.8000.17115 2018.
Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neural networks. In International Conference on Computer Aided Verification, pp. 3-29. Springer, 2017.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 656-672. IEEE, 2019.
Alexander Levine, Sahil Singla, and Soheil Feizi. Certifiably robust interpretation in deep learning. CoRR, abs/1905.12105. 2019.
wwtprg//bs//ie 1905 12105
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. In Advances in Neural Information Processing Systems, pp. 9464-9474, 2019a.
Qiyang Li, Saminul Haque, Cem Anil, James Lucas, Roger B Grosse, and Hönn-k Jacobsen. Preventing gradient attenuation in lipschitz constrained convolutional networks. In Advances in neural information processing systems, pp. 15390-15402, 2019b.
Jeet Mohapatra, Ching-Yun Yun Ko, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, and Luca Daniel. Higher- order certification for randomized smoothing. Advances Neural Information Processing Sys- tems, 33, 2020.
Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. Semidefinite relaxations for certifying robustness to adversarial examples. In Advances in Neural Information Processing Systems, pp- 10877-10887, 2018
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. In Advances in Neural Information Processing Systems, pp. 11222-11303, 2019.
Sahil Singla and Soheil Feizi. Second-order provable defenses against adversarial at- tacks. In Proceedings of the 37th International Conference on Machine Learning (pre- proceedings), 2020. URL https: tttt:p::s/proooeeii /ppppppeednngs..... cml wwc.lcc//ee wamm.ec//taaic/e. wtss//taaac/ppppr_fiiii icml 00000/29333222
pdf
Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2019.
URL BokL/nttppps//eeeeeveee wwt.:e/eooouuniddddyiic.cd..aa..
Roman Vershynin. digeediimesional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
10

---

Under review as a conference paper at ICLR 2021
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, pp. 2018.
Martin Wainwright. Hligh-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cam- bridge University Press, 2019.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 9286652955. PMLR, 2018.
Huan Zhang, Pengchuan Zhang, and Cho-Jui Hsieh. Recurjac: An efficient recursive algorithm for bounding jacobian matrix neural networks and its applications. Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 5757-5764, 2019.
A PROOFS
A.
SIMPLE PROOF OF THEOREM 1 FROM COIIEN ET AL. (2019)
We first provide novel, simple, and intuitive proof for first-order randomized smoothing: this will allow us to develop methods and notations for later proofs.
Theorem. Let c No0,, o² For all x, x' with ||x
x' x'11 < p. and all f R [0, 1]:
Ef + 1 ④ (E.If(x +)) ))
(10)
Where t is the normal cdf function and +-1 is its inverse.
Proof. Let ||x x' 112- Choose our basis that and x' [R, 0, o........0) 0J" (Note that by isometry, we still have (O, o² f)). Then define g -R [0, 1]:
g(2) Ee en [f E2, ..... cn 1T)
(11)
Note that:
E,, [a(())) [x +)]
(12)
E,, (R a)] = E.If(x + c)]
Now, in one dimension, 61 (/0, o), and so has a pdf of 2 - o 1-1 (E), where E' is the normal pdf function. By the definition of expected value:
Ee, /
g(E1)o- ( dei
(13)
IlR 11e0--lo ' E1 dei = - g(E1)o - (
R
E, (g +
de
du We perform a change of integration variables, using y 1 (5) (and noting that
des
a- 1 (('()))) (a)):
/' g(00 - 110
dei
/ g00
[ggg(())
1000
dy
(2))dd
dy
0
Es, [9(R E1)] / g(04 (v)) 100
E1
R
deg dy
(14)
0
dy
(v)) T ( (gy
ft
glo
dy
(+-1(w))
Note that:
Φ ④ (v)
e
- (v) 5)
4 [(()
R2
3
(15)
4' ( (y))
1 (w))
11

---

Under review as a conference paper at ICLR 2021
Also, to simplify notation, define Ie
[0, 1] [0, 1] go(v) g (v))
Then have
(combining Equations 14 and 15):
E. [g((e))] = S/" go go(v)dy
(16)
E, (g R + +1) = S. go (o(u)e "0)))))>) (x)- 20 dy
Fix the expectation x, E,, [[(e))] at constant C, let consider the function go which minimizes the expectation at x':
min e p.1] l (x)e *00)) ((0)- 20 dy
E, (g( 18 + +1)l 2
(17)
Sg so (v)dy=C
n2 However, note that ex (())
20
increases monotonically with y. Then the minimum is achieved at:
(v) {
1 if S C
(18)
if C
In terms of the function g(2), this is:
1 2 1
(=)
0 if > o (C))
(19)
Then we can evaluate the minimum, using the form of the integral given Equation
E,, [g( EI)] >
g ge(E1oo-
w''
61
R
dei
o (
b p
(1
R
de1
(20)
t (C)
R
R
+
$
0
0
R
+ Φ (C)
By the definition of C and Equation 12 this is:
Ex[f(x + o) 2
[E,f(x + =])
R
2 ক
[E,[(x(( + +])
()
(21)
which was to be proven.
A.2 SECOND ORDER SMOOTHING
Theorem 1. Lete ~ o2). For all x, x' with ||x x'12 12 < p. and all f Rdd - [0, 1].
Ef + +c] 2 &b ④ 2a' + E. [f + c))) -) + ① (a)) ")
(22)
Where b is the normal cdf function. --1 is its inverse, and a' is the (unique) solution
Φ' (a')) 5' (a' + F. (x + ))) - R. [f + +)lll2
(23)
Further, for all pairs (E, [f c))), [f + l)lll which are possible, there exists base classifier f and an adversarial point such that Equation 4 an equality.
As show by Salman et al. 2019), we have, for all x" Rd:
V Ex c) o 2E (x" + +c]
(24)
Under the choice of basis of the above proof (in particular, x 0), when evaluated at x this becomes:
V. <)] - (R
(25)
12

---

Under review as a conference paper at ICLR 2021
Let [1,0.0,.... ... 0J", and define 9 as in the above proof. Note that:
-I c)lll2 Su I +c]
(e((((()
=o [c
(26)
-o -Ber (el (Brr EE,
By the definition of expectation, and again using the change of integration variables y 1 (+).
Rg [ag(() = o (v))((() (1)) t (
de1 dy
dy
(27)
o$ (v)g(oo (v)g(o+-'
Define:
C = E,,((())
(= E= +c)
(28)
C² E, (((())
(2 E,
Then, by Equations 16 and and defining Gø, as above,
[ go (y)e
n² ((0)
E, ( R + +1)] >
min s0
20 dy
(29)
L (w) fa o (y)dy("
Note that our constraints are linear the space of functions; we can then introduce Lagrange mul- tipliers:
[ go(y)e**'(0) (v)) (w)-
= dy >1 (S, go -c)
min
20=
04 [0. (0. 10
(30)
A2 (f)) o4 (x)))) -Cy)
l' so(y)e 0o(y)c-'(u00- (x) 20 dy \ggo(v) \20 (1))~(())00 constants
min go 10 [0.11
(31)
R2 (' (v( (c (v) 20
min |0. (.
-2 (2) dy + constants
This is simply the inner product between Ie and function: the inner product is minimized by setting go where the expression is negative, and go where the expression is positive:
1 if (v)
2 A1 + - (1()
2²
(v
(32)
0 if + (v)
A 22000 (w))
Sign changes occur at:
R2 2
X1
① --1 = -W
(33)
X2
A2
Where W is the product-log (Lambert W) function. This returns zero, one, or two possible values, depending on the argument (zero values the argument < -e two values on [-e-l, 0), and one value on non-negative arguments). Therefore there are at most two sign changes. Also, note that as y + -y) + oo, taking the limit in Equation 32 we know that
gā =
Therefore, taking into account the constraint for go (y)dy C, we know that d is either:
13

---

Under review as a conference paper at ICLR 2021
0 everywhere (and C 0), if no sign changes
at < C, otherwise, if one sign change
on the interval [a, CJ. O otherwise, for some E [0, C7. if two sign changes
In fact, the final case includes the first two, so that we need do now is find a to satisfy the Cf constraint. This constraint (Fquation 27 becomes:
C²
(y)dy σ
(34)
Because & -x)(w) is monotone increasing, the LHS of Equation 34 is monotone increasing function of a. Using the indefinite integral 4> o-1;
/ ④ (v)dy = / erf 1)dy
e (ed (22-11) + +C -0 CC
v 22x
Where e is the inverse error function. Then the constraint becomes:
C"
4'(2 (a)) - 4' ( + C))
(a
(35)
We can now evaluate the value of the smoothed function at x', again using the form of the integral given Equation 13
E,,
('(
E1
R
dep
0
r (a-C)
(1
R
deg
(36)
(a)
0
R
④ 1 (a
+
(a)
If we consider the form of this integral in Equation 29 which reduces to simply:
(2)
dy
(37)
we see that is is monotonically increasing function in Furthermore, we have that the LHS of Equation 35 is monotonic a. Therefore, we define a' as the solution to:
② (a")) - Φ 7 + C)) = - V + c)]l2
(38)
Then by Equation 26
we know a' S a. Then because the RHS of Equation 36 is also monotonic in
R
Ex, [g)
④ (a' + +
Φ ④ (a-)(a))
(39)
By the definitions of g and C, and because the RHS is monotonically decreasing in (See Equation
37 we can conclude the theorem as stated.
Further, we can conclude that an equality case is possible by noting that it achieved by the function g" (2) as described above: the minimal (z() can then be constructed as f= "...) (z).
Note that we also need Equation 26 to be tight: this is achieved where the adversarial direction x is parallel the gradient of the smoothed function.
A.3 PRACTICAL CERTIFICATION ALGORITIIM
Define
C = lower bound on
(40)
C = upper bound on C"
Note that
Vx e)]ll²
(41)
E.l
o (err(lelte)
14

---

Under review as a conference paper at ICLR 2021
Theorem 2. Let V == (efe e)) (e'((x and V be its empirical estimate. If n pairs of samples (= N/2) are used to estimate V, then, with probability at most 9, [E[] V where:
40 In(n)
if - S dn
t
(42)
4v2g In(n))
2f > dn
Wainwright (2019) gives the following condition for any centered (mean-zero) sub-exponential ran- dom variable X: Definition 1. A centered R.V. X is (a.b)/subexponential if:
B AX < e²2
VAEl-b-166-1
(43)
First, we establish bounds for e (This can be considered simplified case of Gaussian chaos of the second order, see Vershynin (018)). For each i E [d] and ef are independent Gaussian random variables. Recall the nooment-geneaaing function for a Gaussian:
[ 1A
(44)
Then for the product GC, we have that:
Blederd]- = E., E.g [[] = E., (122
(45)
Note that this has a similar form to the moment generating function of the Chi-squared distribution for k = 1:
1
E2, (e
(46)
v1 202t
Then:
Bledeidij Ee,
VT- Xa S Seti0 4X2 S
(47)
Where the final inequality can be shown by observing that, if X2 $12
X²011 X2o4 S 2X S 22200
X204 =1
(48)
and taking square roots. Because E is centered, this implies that EØę is (V20², v20²)- subexponential. Now, e is simply the sum of such identical, independent, centered subexponen- tial variables: by (Wainwright (2019) Equation 2.18), we conclude that e e' is (V202 Vd, v20²). subexponential. This implies:
Ble I 20
A A AE (
(49)
Recall that the quantity which we are measuring is (ef(e)) (E' (é'(())). For notation convenience, let u(e,e'): R2d x IRd {0, 1] be defined as (E)), that the quantity of interest is
V = 6 e')(,,))
(50)
We further define centered version of this quantity
V = E
(51)
We now introduce an important lemma:
Lemma 1. V' is (2/20² Vd, 2 "sbexxp)nnnoooettiii
Proof. Define
p:= p: ::=Pr(u(e,e' 1] cc
(52)
Then:
V = = (1-r(c, c'))c
(53)
E [
Elc e'] E[(1 ())e < -F((1 - -((
(54)
15

---

Under review as a conference paper at ICLR 2021
E Ble. cb(c,))) = pble -llceelleleee)== - 1(1-1((1)))) c'lu(e,ee) ())(,e) 0l
(55)
Therefore
V - c'(c,c))
(56)
=c-év(E,e) - elle -1- [|
V' = =' - ( =
(57)
+ (1 -1- (t
0]
Define:
- A = (1
=0
(58)
B = = DBle e'u((, [[, = - p u(e, eB) e'/u 0]
(59)
V = A
(60)
Trivially, we have:
1 ] pElede ( = 1] + (1 - pillpppBeeeee/eeee (u(E,))
() = 0]
(61)
However, note that also:
Blead] pB[eAA/o(ee =1 + (1 ((( =0 B[e pBleaeed (((,,()== 1] + (1 -)edB(cclllled))
(62)
Then by Jensen's 's inequality, we have:
Biea1 S E[eedrre) KA
(63)
Similarly:
E E = p
1] + (1 - [(c, 0)
(64)
E AB = pe pA + (1 - p)e pAl,ecc(o(e.e)==)
Again, by Jensen's 's inequality:
B < -peedd 4A
(65)
B[eAB] S Ble-n) S E 20 22 < 20222212
4 pd (-((200)- ( (66)
Because p I L, we then have:
B[e>B] S 20
A \E (
(67)
In other words, have shown that both A and B are both (v2a² /2 ( "uuex-ssbbxoonn Then, by Cauchy-Schwarte
B[e
=BleAA (BB
(68)
S v [EE22AIBE(E2AB
S v (2 V2A E 1-((203)- (V202)-1
(
AV AV < 80
e 22 (2V202)
(69)
In other words, V is (2 202 /d, (tuo:
Finally, using the form of the one-sided Bernstein tail bound for subexponential random variables given in Wainwright (2019), we have, given n mcasurements and an empirical mean estimate of V as V:
if 2d
Pr(E[V] V
2
(70)
ift> 2/20²d
Then, given a failure rate we can compute the minimum deviation such that the failure proba- bility is less than ny
402
In(n)
if -f-2 < dn if 2n))n(()) > dn
n
(71)
2v
In(n)
16

---

Under review as a conference paper at ICLR 2021
A.4 DIPOLE SMOOTHING
Theorem 3. Let o2). For all x, with ||x - x'12 1x'xl2
p. and for all f R 10, 1].
define:
c = [
(72)
CN f
Then:
[/ + +e]] 2 1
x ) L
0
+ 4
1(
0
(73)
2
0
2
Where is the normal cdf function and 4-1 is its inverse.
Proof. As in the proof of Theorem A.1 let R |x x' x'112. and choose our basis so that 0 and
x' RR 0, O,.....0 0J"
First, for f Rd + [0, 1). define decomposition into symmetric and non-symmetric compo-
nents, f fN Rd + [0, 1]:
)())=f((f((+
(74)
Note that f(e) = (c) + p (e) and also that ($(() = (-c)) Define go (2), g0.g((
NN (2) [0.1]
by analogy to Equation 12 By linearity of expectation, note that g(z) = (2()) + (2). Also note that:
g² (-2) = Be in 18 (1-2, E2, Enl 1n
E _
[ $11 - E2 .... -En] [T]
Begenea
1 $11 -2 -2, -en] [1
(75)
Beg
[ Cn. 1T]
(2)
Similarly, define Oet and -h We still have:
go (v) g -1 (v)) 9 - 1g0
(oo =g(()) +
(76)
Also (using db (y) = - --11011 (1 y)):
gh و" (aΦ (g)) = 8 (-o v) = ((a)))=g(( 8)) = -
(77)
Note that all of the mechanics of the proof of Theorem can be applied to f, S and f Following Equation 13 have:
[( ge (y)
cs E.
Ex (x+
NN E.J (x + c)] = [' g (y)dy
(78)
C: = B=]sx + { (o)dd = [ 9o(ف) + 1 = C + N
17

---

Under review as a conference paper at ICLR 2021
We may then write the minimization in Equation 17
fixing C.V and C$ as constants separately ?
Be, Lo( + e1) >
min
Io (o(y)e 4 '0)
(1) 26 dy
96. [ -0. 1 Io (y)
Jo 96 (y)dy=CN
(o)(i)) + ot (v))e* '(x) n2 dy
min
86 86.96 (0.1 Jg (g) Jo 96 (z)dg=C
(79)
A² (v)
min -0.
20 dy
Ju nu (y)
AN L. Fe (w)e
x (v))
12 20 dy
min [011 Jd 9 (v)dy=C
The second minimum can be computed in the proof of Theorem 1, it is simply ① ( (C*) - #). For the first minimum, we consider the additional constraint, that (v) 5 - y). Then we can rewrite the integral as:
(y)
8² 20 dy
(v)
R² 20 dy +
8² ge - y)e 000001000 01)())-
20 dy
S.
90 (v)
(v) dy
D0 0 )e x -1100) E2 (- 1)dy
(80)
(v) B2 dy So g0(1/ ) -0'1 22 dy
To (v)e
e S 2 (x) 2 (v0) (v) + e [00 dy
22 2e
L/² F4
(y) (y) (v))dd
So the minimization becomes:
2 cosh(
n²
min of 10.
2e
20
(81)
1/2 (
Note that cosh(# ④ (x)) is a monotonically decreasing function of y on the range [0, 2). Then the minimum is achieved by the function:
{
0
if < 1- if --0 v 2
2
(82)
5 !
Where the value in the domain [ 1] can be computed using ght 5s(11 y) = (5) terms of the function (2)),, this is:
if < σΦ |2|
1( 2 o - 1(1-08
(2)
(83)
Note that we are not considering all applicable constraints here: in particular we are not restricting the range of go (2) to j0. 1] explicitly. However, the lower bound presented here must be at least as low as the lower bound with this additional constraint, the inequality is still valid. Also, this constraint does in fact hold in the final construction.
18

---

Under review as a conference paper at ICLR 2021
We can evaluate the integral, again using the form of the integral given in Equation B
Ee, (R
St 61
(1)0
de
σ
(110))))
El
R
dei
(84)
1 C
R
1 1+
CS
R
④
Φ
2
0
2
1
C
R
C
R
④
2
2
So, combining the 9 and n terms, we have:
R
Ee jo + E1)]
2 ②
(C
1 C
R
(85)
2
0
CS
R
④
2
From Equation 12 and noting that the last two terms together are monotonically decreasing with R, we complete the proof.
B ADDITIONAL EXPERIMENTS
Here, we present experiments at wider range of parameters. For all figures, images misclassified or certified for both the baseline and the tested method are not counted: the total test set size is 1000 for MNIST, 500 for CIFAR and ImageNet with N 10 and 100 for CIFAR and ImageNet with N = 10 For all experiments, No = 100. Also, note that we test independently for the baseline and higher-order methods (i.e., fairly to dipole smoothing, where the sampling method different; however, it does lead to some noise, especially at N = 10S.
different smoothing samples). This is necessary to compare
B. NOISE LEVEL
We see (Figures and 8 that at a smaller level of noise (a = 0.12), the effect of higher-order smoothing is diminished. This can be understood in terms of the curves Figure [ lower noise leads to more inputs with higher Pa. which reduces the benefit of the higher-order certificate. Con- versely, higher noise increases the effects of the higher-order certificates, although it also leads to decreased total accuracy. The dipole certificate underperforms at N = 10 a = 0.5: this is likely due to the increase estimation error, which becomes significant near pa = 0.5.
B.2 DIMENSIONALITY d
To test second-order smoothing lower-dimensional dataset, performed PCA the 7 7 MNIST images, and classified using the top 10 principal components. (d 10). Results are shown in Figures 9 10 and 11 We see that, at N = 10%, second-order smoothing has marginal positive impact at this smaller scale.
B.3 DIPOLE SMOOTHING ON CIFAR-10
In Figure 12 we see experiments on CIFAR-10 using dipole smoothing, for range of a {0.12, 0.25, 0.50, 1.00) and N E {10", 10%). Note that dipole smoothing appears to be beneficial even N 10 on CIFAR- 10, all noise levels 0.25.
"To this, note the form of the integral of gs equal these terms given in Equation 80
19

---

Under review as a conference paper at ICLR 2021
B.4 DIPOLE SMOOTHING ON IMAGENET
In Figure 13 we see experiments on ImageNet using dipole smoothing, for a range of a E (0.25, 0.50, I.00} and N E {10%, 10%). There is anomalous result for 0.50, N 109, in that this is the only case where dipole smoothing appears to perform worse than standard smooth- ing. However, this turms out to be computational artifact. At both
0.50 and 0.25, there are a large number of images where every smoothing sample is correctly classified, so Pa is as close to as the measurement bounds allow. Note that if Pa truly equals 1, the certified radius is infinite, so in this domain, the reported certificate is entirely a function of the estimation error. Because dipole smoothing reduces measurement precision, these samples have somewhat smaller certified radii un- der dipole smoothing, especially at small N. However, this gap should be exactly proportional to a. The cause of the anomaly the fact that our code (adapted from (Cohen et al.] 2019)) records each radius three significant figures. At 0.5, for image where all noise samples are correctly classified, the ratio the dipole smoothing radius to the standard smoothing radius is reported as 9991191 98.95%, while for 0.25 it is reported 0.947 0.947/0.953 /0.93 99.37%. This explains the large number of samples with reported >> 1% decrease in certificates for a 0.50, N = 10".
C ABSOLUTE CERTIFICATES FOR MAIN-TEXT EXPERIMENTS
In Figures 14 and 13] we show the absolute, rather than relative, values of the certificates reported in the main text, compared the baseline first-order randomized smoothing. We see that the benefit of the proposed techniques is greatest for images with small absolute certificates, and that, on CIFAR- 10 and ImageNet, there is some disadvantage to dipole smoothing on the largest possible certificates, where all smoothing samples are classified correctly. This is because, for these images, the certificate depends entirely on estimation error.
20

---

Under review as a conference paper at ICLR 2021
So5,
D pall, 7X7 MNIII, awo aw0.22, Ne1p"
160
700
200
230
200
200
100
20504
20%
So5. 7X7 NNST, 0-0.12, N-10"
400
100
000
200
126
300
.0%
Coreect copt
for
Seh, CXL NNIST, mm0.12, X-16"
100
900
300
Canse set
.2%
cart
de
for Dracke
Dipole.
120
000
500
230
330
22 Poree cot
2438 50V
URL 1000
200
Connectet
Figure 7: 7 X MNIST, 0.12
21

---

Under review as a conference paper at ICLR 2021
Se5. 7X7 MNIST, 0-0.50 N-107
Dipcle, 787 MUNST 0-0.50 N=16
900
400-
901
200
200
1001
200
Sal,, 7X7 SINIST, BHO 50, N=110
Dipple,
301
600
100
500
20
12%
7X7 N-20"
N=10"
800
100
400
300
200
100
200
11% 10% SOA 1E00
to
5O5. 7X7 MNIST. 0-069
Dipcie, 7X7 MNIST, 0-0.50 N=10"
800
00
300
200
100-
100
for
Figure 8: x MNIST, 0.50
22

---

Under review as a conference paper at ICLR 2021
Se5, 10 PCs MNST, C-0.12, N-10P
Vipol, MCs Mhis amo 12, N=10"
200
200
5% 10
12000
1000
So5, 10 PCS NNIST, c-0.12 N-10"
Dioole, 10 PCs MNIST. No
300
100
700
500
405
5004
Se5, PCs NINIST 0.0.12, No110
N=10"
100
100
300
20N
TO
So5, 10 PCs MNIST C-0.12, K-10P
Dápsle, 10 PCs NNST. a-0.12, N-10'
300
800
201-
20%
30% Mec
Figure 9: 10 PC MNIST, = 0.12
23

---

Under review as a conference paper at ICLR 2021
So5, N=163
400
20
So5. 10P MNIT. 9-0.55 N-20*
Dppic, 10 NNST. c-0.25, 25-0
000
700
CorNec
ConFect
S05. 10 9-0.25, N=20"
100
500
300
100
Contec LOC
250
MNST. dw0.25, N=10"
50%, PUPUS MNIST aw0.25, No
700
100
200
Comen oet
18079000
220000
dectee
to
Figure 10: 10 PC MNIST, 0.25
24

---

Under review as a conference paper at ICLR 2021
So5, 10 PCS MNST. dw0.50, N=
Dipule, 10 PCs MNIST, awd -00,00 N=13P
100
100
Dipele,
100
500
100
Eurao cark
250
So6, 10 PCs MNST. c-0.50, N-10"
Dipari, 10 PCs MNIST, e- 150 N-100
000
000
100
100
600
200
So5, 10 PCS MUSIT Cw0.50 1=
Dipnle, MNIST mun No:10
100
200
100
2014
Figure 11: 10 PC MNIST, = 0.50
25

---

Under review as a conference paper at ICLR 2021
Oboole CIFAR 0-0112, N-10"
Dipole, CIFAR, 9-0.12 N-10"
z00
200
Dipole, am0.25, N-10P
300
2o.
Dipare, CIFR, a-0.50, N-16"
Dipole 0-0.50
400-
200
Coneect
CIRAR, 1000, N=10"
N-10"
10
100
12
200
100
52%0
2004
CONMET
Con
Figure 12: CIFAR-10
26

---

Under review as a conference paper at ICLR 2021
pole. Imageket, 0w0.25, No30!
Dipplee mageliet. a-0.55 N=10*
350
15% 200
THE
Dpplle, Imagelell, p-0.30), N=jo²
Dipole, Imapehell p-1.00 N-10"
350
391
2004
15.5%
JPC 27
Centet cart For
1000
CATE
CATTE
ImageeNll,
200
200
Dic 200
10% 2000
Figure 13: ImageNet
27

---

Under review as a conference paper at ICLR 2021
(a)
7X7 MNIST, SoS, a 0.25, M 10"
(b)
7X7 MNIST, Dipole, - 0.25, N 10
1.0
Baselina
Beseline Dipole
Ceniifee accuraEy
6.8
Cortifeed accurnyy
0.8
0.6
0.8
0.44
0.4
0.0
0.2
0.4
0.6
08
redius
radius
(c)
7X7 MNIST, SoS, 0= 0.25, N 10
(d)
7X7 MNIST, Dipole, - 0.25, 10
1.0
Baseline
Baseline
acquracy
0.0
ceriilee eccurscy
0.8
0.6
0.4
0.4
0.0
0.0
0.2
0.4
0.5
0.8 radius
1.0
12
14
0.0
0.8
radius
(e)
7X7 MNIST, Dipole, a 0.25, N 10'
7X7 MNIST, Dipole, 0.25, 10²
1.0
3.0
Baseline
Baseline
Cerified accuracy
0.8
ceriiied eECUParC
0.8
6.6
6.4
0.0
02
04
0.6
0.8
1.
12
1.4
0.0
0.2
0.6
0.6
0.8 radius
1.0
3.2
1.4
redius
(g)
(h)
MNIST, SoS, 0.25,
MNIST, Dipole, 0.25,
1.0
3.0
Baseline
Raseline
accuracy
0.8
Secund Order
certilee SCCUPIICS
0.6
0.8
0.4
0.0
0.2
0.4
0.6
088
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.5
1.0
3.2
1.4
tadius
radhus
Figure 14: Experiments on 7 x 7 MNIST. For all, = 0.25. For (a, c, g). Second-order Smoothing used. For (b, d, h), Gaussian dipole smoothing used. For (a, b), 100. For (c, d), N 10a_ For (c, f), N 10a. For (g. h). N 10a.
28

---

Under review as a conference paper at ICLR 2021
(a)
CIFAR, dipole, a 0.25, M= 10
(b)
CIFAR, dipole, o= 0.25, N 10
1.0
1.0
Baseline Dipole
Baseline
Cetiiied accuracy
certiied accurazy
0.8
0.6
0.00
0.25 3.50
0:75
1.00 reclus
1.25
1.50
1.75 2.00
0.00
0.25 0.50
0.75
1.00 125 1.50 rodlus
1.75 2.00
(c)
(d)
ImageNet, Dipole. 0.25, 10a
ImageNet, Dipole, a 0.25, N 10
Dipolc
Dipcle
Centifed acturncy
cortfed accural
0.6
0.4
0.2
02
0.6
0.
redius
Figure 15: Dipole smoothing experiments, with a 0.25, on CIFAR-10 (a, and ImageNet (c, d). For (a, c), N 105. For (b, d), N = 10
29


Reviews:
[{"review_id": "hU5zF68a6hG", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  This paper presents a randomized second-order smoothing certificate for providing robustness guarantees against adversarial attacks. By additionally using the gradient estimation of smoothed classifier, the proposed method has been shown to outperform the existing randomized smoothing certificate in practice. A variant of the method without explicitly estimating gradient vector has also been proposed to avoid the dependence of feature dimension in concentration analysis.\n\nStrong points:\n\n-S1. The addressed topic of randomized smoothing certificate is of significant importance and interest to the society of adversarial learning.\n\n-S2. The certificate radius of the proposed method is novel as far as the reviewer knows about. \n \nWeak points:\n\n-W1. The advantage of the certificate radius in Theorem 1 over the existing ones is not clearly justified. Unlike the original randomized smoothing classifiers, there seems no explicit expression available for the certificate radius in Theorem 1. Based on the current bound in Equation 2, it is hard to evaluate the theoretical gain of the proposed method in robustness. \n\n-W2. The proof of Theorem in Appendix Section A.1 looks not correct in general. The current proof argument is only customized for $x=0$ and $x’=[R,0,…,0]^\\top$. It is not clear if the same claim and technique extend to arbitrary $x$ and $x’$ satisfying $\\|x – x’\\|\\le R$.\n\n-W3. The paper is poorly organized and presented. The main results in Section 3 are somewhat hard to follow for non-expert audiences, mainly due to the lack of a clear statement of method before indulging into theoretical analysis. Also, much of the space was allocated for elaborating a gradient estimation method which in my opinion is mostly incremental as a side contribution. Such a practice of material organization makes the paper unclear and perhaps pointless in presentation. \n\n-W4. As an adversarial learning paper that introduces a new alternative method for robustness certificate, it is desirable to provide a sufficiently detailed experimental study in the main paper (rather than in the appendix) to more convincingly justify the real benefit of method. \n\n-W5. There are many typos in the manuscript. Here are a few examples: \n\n(1) Equation 4 -> Equation 2 in Theorem 1; \n\n(2) Page 2: it possible -> it is possible; \n\n(3) Page 5: \\|nabla_x p_a(x)\\| -> $\\|\\nabla_x p_a(x)\\|$; \n\n(4) Page 5: Salman et al. (2019) suggests -> suggest; \n\n(5) Theorem 2: $n$ -> $N$. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ayGHDsDvMIN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "My main concern is that the improvement of the proposed method over standard randomized smoothing is marginal.  \nIn addition, the evaluation metric used for comparison is not standard.  It would be better to use certified accuracy as the evaluation metric used by the related works.  \n\nPros:\n\n1. The studied problem is important.  In particular, it's important to study the certified robustness of the classifier against adversarial perturbations. \n\n2. The proposed second-order smoothing is novel. This paper aims to improve randomized smoothing by incorporating the gradient information of the smoothed classifier, which is novel.  \n\n\nCons:\n1. The evaluation metric is not standard.  Standard randomized smoothing and the follow-up work use certified accuracy as the metric to evaluate the certified robustness.  It's better for the authors to also use certified accuracy as the metric for a fair comparison. \n\nIt's not clear how different parameters (e.g., c, lambda, eta) impact the certified accuracy on CIFAR10 and ImageNet dataset. \n\n2. The improvement of the proposed method (Gaussian dipole smoothing) over the standard randomized smoothing is marginal (most 1% in Figure 5). \n\n\nQuestions during the rebuttal period:\n\nWhy not using certified accuracy? \n\nWhat's the impact of the parameters on the results? \n\nIs it possible to address the issue of marginal improvement? \n\n\nI would be happy to increase my score if the authors can also show results on certified accuracy and improve their results?  \n\nTypos:\n\n1. In Theorem 1, \"there exists a base classifier ... such that Equation 4 is an equality\". Is it Equation 2?\n\n2. $nabla_x$ --> $\\nabla_x$. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HLqDr-jbY2n", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work extends previous results on certified robustness guarantees via randomised smoothing by incorporating gradient information of the smoothed function into the final certificate (referred to as SoS). The authors show a number of interesting properties such as (1) that the certificate is tight, by showing, for a linear classifier, the Cohen et al. [1] certificate and the SoS certificate are identical, and (2) certificates are slightly improved if gradient norms are small, but due to poor estimation of these value, sometimes the gains provided by \"second-order information\" are eliminated. The authors then show estimation of the gradient norm is dependent on the dimensionality of the data, and provide alternative methods that do not have a strict dependency. Although, as far as I understand, this alternative method requires computing two new lower bounds and so requires more samples to reach the same level of precision as the Cohen et al. certificate. Experiments on CIFAR-10 and ImageNet show in some cases SoS finds larger certificates.\n\nStrength:\n\n[+] Detailed analysis and theorem.\n\n[+] Interesting use of geometrical information to make gradient norm estimation efficient.\n\n[+] Experiments on large scale datasets.\n\nWeaknesses / Questions:\n\n1. I  am confused about the positioning of this work. The authors mention a few times that this work could be interpreted as a negative result, but the authors write in a more positive manner in other parts. Should the work be viewed as an improvement over recent work, or as a negative result that incorporating gradients doesn't improve certificates by much? If it is the latter, it isn't clear which dimension of the work is responsible for the limitations. Is it simply that, in practice, the gradient norms are substantially larger than zero, and so this new certificate more or less matches the Cohen et al. certificate? Or is it that difficulty in estimation of the gradient norms wipes away any gains? Is it both? A more substantial study of these effects would be most welcome.\n\n2. I found it difficult to interpret the experimental results. How many of the +/- 1% changes are, in fact, decreases? It would be useful to plot  these results in terms of certified accuracy vs epsilon, as has become standard in this line of work. My concern is that if the majority of SoS certificates are marginally smaller than the Cohen et al. certificates, and a few SoS certificates are substantially higher, the overall certified accuracy curve will look worse for some epsilon values in comparison to Cohen et al..\n\n3. Analysis does not easily extend to multi-class setting.\n\n[1] Cohen, Jeremy M., Elan Rosenfeld, and J. Zico Kolter. \"Certified adversarial robustness via randomized smoothing.\" arXiv preprint arXiv:1902.02918 (2019).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors develop a novel robustness certificate based on randomized smoothing that accounts for second-order smoothness of functions smoothed with Gaussian noise. They develop a variant of Gaussian smoothing based on these insights that improves sample-efficiency of randomized smoothing using gradient information.

While the ideas presented were interesting, reviewers were concerned about the quality of presentation of the paper (confused positioning of results relative to prior work) as well as the lack of significant improvements upon existing methods in the experimental section. Overall, the paper is borderline based on the reviewers' comments and ratings - however, there is not sufficient evidence to justify acceptance.

I would encourage the authors to consider a significant revision to improve the clarity of contributions made and strengthen experimental results to demonstrate significant improvements, which would validate the power of the theoretical ideas presented.
Decision: Reject