Year: 2021

Abstract:
Deep neural networks (DNNs) have revealed severe vulnerability to adversarial perturbations, besides empirical adversarial training for robustness, the design of provably robust classifiers attracts more and more attention. Randomized smoothing method provides the certified robustness with agnostic architecture, which is further extended to a provable robustness framework using f -divergence. While these methods cannot be applied to smoothing measures with bounded support sets such as uniform probability measures due to the use of likelihood ratio in their certification methods. In this paper, we introduce a framework that is able to deal with robustness properties of arbitrary smoothing measures including those with bounded support set, by using Wasserstein distance as well as total variation distance. By applying our methodology to uniform probability measures with support set B2(O, r), we obtain certified robustness properties concerning lp-perturbations. And by applying to uniform probability measures with support set B∞(O, r), we obtain certified robustness properties with respect to l1, l2, l∞perturbations. We present experimental results on CIFAR-10 dataset with ResNet to validate our theory. It is worth mentioning that our certification procedure only costs constant computation time, which is an improvement upon the state-of-theart methods in terms of the computation time.

1 INTRODUCTION
Vulnerability to adversarial samples is a major obstacle that various classifiers obtained by machine learning algorithms, especially deep neural networks (DNNs), need to overcome (Szegedy et al., 2013; Nguyen et al., 2015). For instance, in computer vision applications, deliberately adding some subtle perturbation δ that humans cannot perceive to the input image x will cause DNNs to give a wrong classification output with high probability. Many empirical adversarial defenses have been proposed, among which adversarial training (Madry et al., 2018) is the most effective one (Athalye et al., 2018), however, it still faces stronger or adaptive attacks to decrease its effectiveness to a certain degree (Croce & Hein, 2020). This motivates research on certified robustness: algorithms that are provably robust to the worst-case attacks.
Some works propose algorithms to learn DNNs that are provably robust against norm-bounded adversarial perturbations by using some convex relaxation methods (Wong & Kolter, 2018; Weng et al., 2018; Xiao et al., 2018; Zhang et al., 2019). However, these approaches are usually computationally expensive and require extensive knowledge of classifier architecture. Besides, randomized smoothing (first introduced in Cohen et al. (2019)) has received significant attention in recent years for verifying the robustness of classifiers. Based on this method, several papers have studied which smoothing strategies perform better for specific lp perturbations. Cohen et al. (2019) concludes that randomized smoothing can be well understood for the l2 case by using Gaussian probability measure for smoothing. It follows that several special cases of the conjecture have been proven for p < 2. Li et al. (2018) show that l1 robustness can be achieved with the Laplacian distribution, and Lee et al. (2019) show that l0 robustness can be achieved with a discrete distribution. Other papers start from the opposite perspective and focus on studying under specific assumptions which perturbation is provably difficult to handle and which smoothing methods are ineffective for particular disturbance. As for the existence of a noise distribution that works for the case of p > 2, Blum et al. (2020) and Kumar et al. (2020) show hardness results for random smoothing to achieve certified adversarial robustness against attacks in the lp ball of radius . Nevertheless, since these works provide hardness results for every possible base classifier f : Rd → Y including those unusual and even bizarre ones, hardness results given by these papers might be attributed to taking into account classifiers that
will never appear in real-world applications. From this perspective, the order of difficulty restricted within the common classifiers subset still remains unresolved.
Notably, based on randomized smoothing strategy, Dvijotham et al. (2020) introduce a provable robustness framework using f -divergence as their convex relaxation technique. However, due to the use of likelihood ratio in their certification methods, the framework cannot be applied to smoothing measures with bounded support sets such as uniform probability measures. In this paper, we introduce a framework that is able to deal with robustness properties of arbitrary smoothing measures, including those with bounded support set, by using Wasserstein distance as well as total variation distance. Our contributions are summarized as follows:
• By applying our methodology to uniform probability measures with support set B2(O, r), we obtain certified robustness properties with respect to lp-perturbations. By applying our methodology to uniform probability measures with support set B∞(O, r), we obtain certified robustness properties with respect to l1, l2, l∞-perturbations.
• Furthermore, we analyze the cases when smoothing measure is taken as uniform probability measure with more general support set Bp(O, r) and show the unavoidable curse of dimension for the usage of bounded support set smoothing measures.
• We present experimental results on CIFAR-10 dataset with ResNet to validate our theory. It is worth mentioning that our certification procedure only costs constant computation time, which is an improvement upon the state-of-the-art methods in terms of the computation time.
2 PROBLEM SETTING
Given a binary base classifier h : Rd → Y = {±1} and smoothing probability measure µ, the randomly smoothed classifier hµ(x) is defined as follows. Definition 1 (smoothed classifier, smoothing measure). The smoothed version of a base binary classifier h producing labels in set Y = {±1} is defined as
hµ(x) = arg max y∈Y
PX∼x+µ[h(X) = y], (1)
where µ ∈ P(X ) is called smoothing measure.
Another way to understand this definition is to say that the smoothed classifier first scores point x as hµ,y(x) = PX∼x+µ[h(X) = y] for each specific class y ∈ Y and then outputs the class y∗ with the highest score. We want to study the robustness of the smoothed classifier hµ against adversarial perturbations of size at most with respect to a given norm || · ||p. The question that whether a bounded lp norm adversarial attack on a fixed input x satisfying hµ(x) = +1 is successful or not can be formulated as solving the optimization problem below:
min ||x′−x||p≤
PX∼x′+µ[h(X) = +1]. (2)
The attack is successful if and only if the minimum value is smaller than 12 . Since we know little about the information of the black-box classifier h, we follow the approach introduced in Dvijotham et al. (2020): rather than studying the adversarial attack in the input spaceX , we study it in the space of probability measures defined on input space P(X ),
min ||x′−x||≤ PX∼x′+µ[h(X) = +1] = min ν∈Dx, ,q PX∼ν [h(X) = +1], (3)
where Dx, ,q := {x′+µ : ||x−x′||q ≤ } represents an lq-norm-based constraint set of radius for smoothing measure µ centered at a particular sample point x. Then, we follow the full-information robust certification framework established in Dvijotham et al. (2020) and analyze the generalization of binary classifier h, which they called specification and denote it as φ : X ⊆ Rd → Z ⊆ R. Besides, define reference measure ρ as x + µ and a collection of perturbed probability measures D ⊆ P(X ). Checking whether a given specification φ is robustly certified at ρ with respect to D or not is equivalent to estimating the optimal value of following optimization problem is non-negative or not:
OPT (φ, ρ,D) := min ν∈D EX∼ν [φ(X)]. (4)
And certifying lp robustness on input x with output of smoothed classifier hµ(x) = +1 is equivalent to verify whether OPT (h, x+ µ,Dx, ,q) ≥ 0 or not.
3 OUR CERTIFICATION PROCEDURES
Notice that the certification in Dvijotham et al. (2020) uses the likelihood ratio r(X) = ν(X)ρ(X) , while r(X) is well-defined only when the support set of ρ contains the support set of ν. Thus, when the support set of reference measure ρ is bounded, and ν takes even a small translation of ρ, the support set of ν will cross over the boundary of support set of ρ. Their certification is invalid in this case. In this paper, by using Wasserstein distance as well as total variance distance, we provide analytical techniques able to analyze bounded support set which is not covered by Dvijotham et al. (2020).
Since the set of measures Dx, ,q constraint in optimization problem OPT (h, x + µ,Dx, ,q) is intractable to deal with, we consider relaxations of this by using Wasserstein distance as well as total variance distance constraints between ν and x+ µ, i.e. Dx, ,q ⊆ {ν : Wp(x+ µ, ν) ≤ δ} := Dx,δ,p which represents Wp-distance-based constraint set of radius δ for smoothing measure µ centered at sample point x and D ⊆ {ν : TV (x + µ, ν) ≤ ξ} := Dx,ξ which represents TV-distance-based constraint set of radius ξ for smoothing measure µ centered at sample point x. Combining the two relaxations, we know Dx, ,q ⊆ Dx,δ,p ∩ Dx,ξ and therefore
OPT (h, x+ µ,Dx, ,q) ≥ OPT (h, x+ µ,Dx,δ,p ∩ Dx,ξ). (5)
And we can obtain tighter relaxation by combining multiple Wasserstein distance relaxation with different p, i.e. Dx, ,q ⊆ (⋂ i∈I Dx,δi,pi ) ∩ Dx,ξ where I ⊆ R+ and therefore
OPT (h, x+ µ,Dx, ,q) ≥ OPT ( h, x+ µ, ( ⋂ i∈I Dx,δi,pi ) ∩ Dx,ξ ) . (6)
Thus, for a fixed input x, it suffices to consider the Wasserstein distance and total variance distance relaxed problem and verify whether OPT (h, x+ µ,Dx,δ,p ∩Dx,ξ) ≥ 0 or not. The analysis of this problem can be divided into three parts: computing the Wasserstein distance relaxation measure set, computing the total variance distance relaxation measure set, and computing the Lagrange function as well as dual problem of the relaxed optimization problem OPT (h, x + µ,Dx,δ,p ∩ Dx,ξ). The details are discussed in the following three sections.
3.1 RELAXATION USING WASSERSTEIN DISTANCE
In this section, we show the following relaxation from norm-based constraint sets into W-distancebased constraint sets for general smoothing measures as well as Gaussian smoothing measure.
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } ⊆ {ν : Wp(x+ µ, ν) ≤ δ} = Dx,δ,p. (7)
3.1.1 GENERAL PROBABILITY MEASURE
Here, we want to find a δq( ) such that
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } ⊆ {ν : Wp(x+ µ, ν) ≤ δq( )} = Dx,δq( ),p for all µ ∈ P(X ). (8)
Theorem 3.1. For all x ∈ Rd, > 0, q > 0, norm-based constraint set Dx, ,q can be relaxed into W-distance-based constraint set Dx,δq( ),p with radius δq( ) = max{ , d 1 2− 1 q } which can be formulated as
Dx, ,q ⊆ { ν : Wp(x+ µ, ν) ≤ max{ , d 1 2− 1 q } }
:= D x,max{ , d 1 2 − 1 q },p
(9)
And this relaxation radius max{ , d 1 2− 1 q } works for any Wasserstein distance parameter p > 0 as well as any smoothing measure µ.
Note that for lq(q ≤ 2) adversarial perturbations, the relaxed radius avoids the influence of dimension d, whereas for lq(q > 2) adversarial perturbations, as q increases, 12 − 1 q increases from 0 to 1 2 correspondingly. The fact that the radius of Wq-distance constraint set grows with order Θ(d 1 2− 1 q ) provides us with an intuition that it is increasingly harder to bound Dx, ,q with larger q, therefore, W-distance-relaxation works better for lq(q ≤ 2) norm perturbation. And this relaxation radius is tight for W2 distance and Gaussian smoothing measures which is proved in the appendix D and therefore shows that W2-distance-relaxation works well for Gaussian smoothing measure.
3.2 RELAXATION USING TOTAL VARIANCE DISTANCE
In this section, we show the following relaxation from norm-based constraint sets into TV-distancebased constraint sets for Gaussian and uniform smoothing measures.
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } ⊆ {ν : TV (x+ µ, ν) ≤ ξ} = Dx,ξ (10)
3.2.1 GAUSSIAN PROBABILITY MEASURE
Here, we want to find a ξ( ) for Gaussian measure µ = N (0, σ2I) such that Dx, ,q = {x′ + µ : ||x− x′||q ≤ } ⊆ {ν : TV (x+ µ, ν) ≤ ξ( )} = Dx,ξ( ). (11)
The magnitude of ξ( ) is given by the following theorem. Theorem 3.2. For Gaussian probability measure µ = N (0, σ2I) on Euclidean space Rd and for all x ∈ R, > 0, q > 0, norm-based constraint set Dx, ,q can be relaxed into TV-distance-based con-
straint setDx,ξ( ) with radius ξ( ) = 2G (max{ , d 12− 1q }
2σ
) −1 whereG is the cumulative distribution
function for standard normal distribution N (0, 1) which can be formulated as Dx, ,q ⊆ { ν : TV (x+ µ, ν) ≤ 2G ( max { 2σ , d 1 2− 1 q 2σ }) − 1 } . (12)
This theorem theoretically shows that TV distance relaxation works effectively for lq(q ≤ 2) perturbation due to the irrelevance of the radius to dimension d and increasingly bad for lq(q > 2) perturbation because of the dependence of the radius to dimension d as order Θ(d 1 2− 1 q ).
3.2.2 UNIFORM PROBABILITY MEASURE
Here, we want to find a ξ( ) for uniform measure µ = U(K), where K is a specific convex compact set, with density function fK(x) = 1Vol(K)Ix∈K such that
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } ⊆ {ν : TV (x+ µ, ν) ≤ ξ( )} = Dx,ξ( ). (13)
In this paper, we mainly focus on the case when K is lp-norm ball centered at original point O with radius r, i.e., K = Bp(O, r). We give following theorems about special cases when p = 1, 2,∞. Theorem 3.3. When K is an l1 norm ball centered at O with radius r, for uniform probability measure U(K) on Euclidean space Rd, we have
Dx, ,q \ { ν : TV (x+ µ, ν) ≤ 1− δ } 6= ∅ for all q > 1 and arbitrarily small δ > 0, (14)
when ≥ 2rd 1 q−1. Note that ≥ 2r√ d which decays with order Θ(d− 1 2 ) for q = 2 and ≥ 2rd which decays with order Θ(d−1) for q =∞.
This theorem theoretically shows that for uniform smoothing measures with l1 ball support set, and total variance distance failed to relax measure set Dx, ,q effectively when q = 2,∞. And this will consequently lead to bad performance for l2 and l∞ robustness certification task, which can be seen from the following section discussing the importance of TV-distance-based relaxation radius. Theorem 3.4. When K is an l2 (Euclidean) ball centered at O with radius r, for uniform probability measure U(K) on Euclidean space Rd and for all x ∈ R, > 0, q > 0, when > min{2r, 2rd 1 q− 1 2 }, norm-based constraint set Dx, ,q failed to be relaxed into TV-distancebased constraint set which can be formulated as
Dx, ,q \ { ν : TV (x+ µ, ν) ≤ 1− δ } 6= ∅ for all q > 1 and arbitrarily small δ > 0. (15)
And when ≤ min{2r, 2rd 1 q− 1 2 }, norm-based constraint set Dx, ,q can be relaxed into valid TVdistance-based constraint set Dx,ξ( ) with radius ξ( ) = 1− ∫ arccos( max{ , d 12− 1q } 2r ) 0 sin n(t)dt∫ π
2 0 sin
n(t)dt which can
be formulated as
Dx, ,q ⊆ { ν : TV (x+ µ, ν) ≤ 1− ∫ arccos( max{ , d 12− 1q }2r ) 0 sinn(t)dt∫ π 2
0 sinn(t)dt
} . (16)
This theorem shows that for uniform smoothing measures with l2 ball support set, when q ≤ 2, relaxation radius is independent of dimension d, whereas when q > 2 relaxation radius starts to be bound up with dimension d and the impact of d grows as q increases. To put it another way, total variance distance relaxation performs well for uniform smoothing measures with l2 ball support set when q ≤ 2 and increasingly poor when q > 2. Theorem 3.5. When K is an l∞ cube centered at O with radius r, for uniform probability measure U(K) on Euclidean space Rd and for all x ∈ R, > 0, q > 0, when > 2r, norm-based constraint set Dx, ,q failed to be relaxed into TV-distance-based constraint set which can be formulated as
Dx, ,q \ { ν : TV (x+ µ, ν) ≤ 1− δ } 6= ∅ for all q > 0 and arbitrarily small δ > 0. (17)
And when ≤ 2r, norm-based constraint set Dx, ,q can be relaxed into valid TV-distance-based constraint set Dx,ξ( ). When q = 1, ξ( ) can be taken as 2r , which can be formulated as
Dx, ,1 ⊆ { ν : TV (x+ µ, ν) ≤
2r
} . (18)
When q = 2, ξ( ) can be taken as 1− ( 1−
2d 1 2 r
)d when 0 < ≤ 2tnr where √ n−1 n ≤ tn < 1 and
tn approaches 1 at an exponential rate, which can be formulated as Dx, ,2 ⊆ { ν : TV (x+ µ, ν) ≤ 1− ( 1−
2d 1 2 r )d} ≈ {ν : TV (x+ µ, ν) ≤ 1− e− 2r d 12 }, (19) ξ( ) can be taken as 1 − ( d−1+ √ d( 2r )
2−d+1 d )d−1( 1−√d( 2r )2−d+1 d ) when 2tnr < < 2r, which
can be formulated as Dx, ,2 ⊆ { ν : TV (x+ µ, ν) ≤ 1− (d− 1 +√d( 2r )2 − d+ 1 d )d−1(1−√d( 2r )2 − d+ 1 d )} . (20) When q =∞, ξ( ) can be taken as 1− ( 1− 2r )d , which can be formulated as
Dx, ,∞ ⊆ { ν : TV (x+ µ, ν) ≤ 1− ( 1−
2r
)d} . (21)
As for uniform smoothing measures with l∞ cube support set, this theorem shows that the performance towards l1 perturbation turns out to be fine since TV distance relaxation radius 2r has nothing to do with dimension d and the dimensional curse is avoided. However, in this case, TV distance relaxation shows incapability to cope with l2 and l∞ perturbation in some extent due to the rate of increasing radius tending to 1 as Θ(ed 1 2 ) and Θ(ed).
After discussing the special cases when K is an l∞ cube or an l2 Euclidean ball, we then consider the general case when K is an lp ball centered at the original point with radius r and give a lower bound for TV distance relaxation radius in the following theorem.
Theorem 3.6. When K is an lp ball centered at O with radius r, for uniform probability measure U(K) on Euclidean space Rd and assume norm-based constraint set Dx, ,q can be relaxed into TV-distance-based constraint set Dx,ξ( ), then
ξ( ) ≥ 2 ∫ d 1p 4r(pe) 1 p Γ(1+ 1 p )
0
exp (1 p − e(2xΓ(1 + 1 p ))p ) dx
for all perturbation norm parameter q > 0 with high probability when d is sufficiently large, which can be formulated as
Dx, ,q \ { ν : TV (x+ µ, ν) ≤ 2 ∫ d 1p 4r(pe) 1 p Γ(1+ 1 p )
0
exp (1 p − e(2xΓ(1 + 1 p ))p ) dx− δ } 6= ∅ (22)
for arbitrarily small δ > 0.
Figure 1: Graph of density function f(x) = exp
(
1 p − e( 2 pΓ( 1 p ))
pxp )
when p = 2, 4, 6 from left to
right
A way to interpret this theorem is that as p increases and K correspondingly translates from l1 norm cross-polytope into l∞ norm cube, the dependence of integral upper limit d 1 p
4r(pe) 1 p Γ(1+ 1p )
on
dimension d is gradually reduced, which theoretically shows by taking all kinds of lq perturbations into consideration, in average scale U ( Bp(O, r) ) tends to perform better than U ( Bq(O, r) ) where p > q. Nevertheless, note that the curse of dimension is unavoidable when we use uniform smoothing measure U(K) with bounded support set. Note the graphs of density function shown in Figure 1 when p = 2, 4, 6, we know that as p increases, the density curve is increasingly short, fat, and light-tailed. From another perspective, we consider a ball with a fixed radius r. As the dimension d of the base Euclidean space increases, fixed proportion of mass concentrates within a slab of width Θ(d− 1 p ). Thus, intuitively, it is increasingly difficult to bound the perturbed measure set Dx, ,q by using TV distance and certify as dimension d enlarge due to the existence of TV-relaxation upper bound in the dual optimization problem.
3.3 VERIFYING FULL-INFORMATION ROBUST CERTIFICATION
Based on the above analysis, in this section, we are now prepared to compute the Lagrange function and dual problem of the relaxed optimization problem OPT (φ, x + µ,Dx,δ,p ∩ Dx,ξ). Here we mainly focus on the case when reference measure ρ = x+µ and perturbed probability measure ν are absolutely continuous w.r.t. Lebesgue measure λ on Rd, i.e., ρ, ν λ and discard uncommon cases when ρ, ν are discrete, single or mixed w.r.t. λ. Since ρ, ν λ, assume the density function of ρ and ν w.r.t. Lebesgue measure λ are f(x) and g(x), x ∈ Rd respectively. Instead of using likelihood ratio r(x), we consider the difference between g(x) and f(x) and define it as q(x) := g(x)− f(x). The objective function EX∼ν [φ(X)] of optimization problem OPT (φ, ρ,D) can be rewritten in terms of difference function q(x). And we give the theorems below. Theorem 3.7 (Wp distance relaxation with 0 < p ≤ 1). The relaxed optimization problem OPT (φ, x+ µ,Dx,δ,p ∩Dx,ξ) is equivalent to the convex optimization problem with only one functional variable as below
inf q∈L1(X ) ∫ X φ(x)q(x)dx+ EX∼x+µ[φ(X)], s.t. sup ||f ||L,p≤1 ∫ f(x)q(x)dx ≤ δ, ∫ |q(x)|dx ≤ 2ξ
(23) where ||f ||L,p := supx,y∈Rd,x 6=y
|f(x)−f(y)| ||x−y||p2 .
Theorem 3.8 (Wp distance relaxation with p > 1). When smoothing measure µ possesses a convex compact support setK andR := supy∈K ||y||2, R∗ := ||x||2+R+max{ , d 1 2− 1 q }, the relaxed optimization problemOPT (φ, x+µ,Dx,δ,p∩Dx,ξ) can be further relaxed into the convex optimization problem with only one functional variable below
inf q∈L1(X ) ∫ X φ(x)q(x)dx+ EX∼x+µ[φ(X)], (24)
s.t. sup ||f ||L≤p(2R∗)p−1
∫ f(x)q(x)dx ≤ δp + (p− 1)(2R∗)p−1, ∫ |q(x)|dx ≤ 2ξ,
where ||f ||L := supx,y∈Rd,x 6=y |f(x)−f(y)| ||x−y||2 .
Theorem 3.9. The Lagrange function of optimization problem in (23) and (24) is L(λ) = EX∼x+µ[φ(X)]− 2ξ − λC, (25)
Smoothing Measure Perturbation Certification Objective Prerequisite
U(B2(O, r)) lq(q ≤ 2) EX∼x+µ[φ(X)]− 2 ( 1− ∫ arccos( 2r ) 0 sin n(t)dt∫ π
2 0 sin n(t)dt
) ≤ 2r
lq(q > 2) EX∼x+µ[φ(X)]− 2 ( 1− ∫ arccos( d 12− 1q 2r ) 0 sin n(t)dt∫ π
2 0 sin n(t)dt
) ≤ 2rd 1 q− 1 2
U(B∞(O, r)) l1 EX∼x+µ[φ(X)]− r ≤ 2r l2 EX∼x+µ[φ(X)]− 2 ( 1− ( 1−
2d 1 2 r
)d) ≤ 2tnr
l∞ EX∼x+µ[φ(X)]− 2 ( 1− ( 1− 2r )d) ≤ 2r
N (0, σ2I) lq(q ≤ 2) EX∼x+µ[φ(X)]− 2 ( 2G( 2σ )− 1 ) -
lq(q > 2) EX∼x+µ[φ(X)]− 2 ( 2G( d 1 2 − 1 q 2σ )− 1 )
-
Table 1: Certification objectives and prerequisites.
where λ ≥ 0 is the dual variable w.r.t. constraint sup||f ||L≤1 ∫ f(x)q(x)dx ≤ δ or constraint
sup||f ||L≤p(2R∗)p−1 ∫ f(x)q(x)dx ≤ δp + (p− 1)(2R∗)p−1 and C := δ when 0 < p ≤ 1 whereas C := δp + (p− 1)(2R∗)p−1 when p > 1.
Using the duality result, we know the optimal value in (23) can be obtained by computing
max λ≥0
EX∼x+µ[φ(X)]− ξ − λC = EX∼x+µ[φ(X)]− ξ, (26)
which is only related to the radius ξ of TV distance relaxation set. We can see from this formula the significance of TV distance relaxation radius. By plugging the TV distance relaxation radius given in theorem 3.4, 3.5 and 3.2 in dual optimization problem, we obtain the certification objective in Table 1 and we return certified for lp norm perturbation with magnitude if the objective function has non-negative value.
3.4 RELATIONSHIP WITH PREVIOUS WORK
By applying our methodology to Gaussian probability measure, we miraculously obtain the same certified robustness properties provided in Dvijotham et al. (2020) using as Hockey-stick divergence with β = 1. Theorem 3.10. When smoothing measure is taken as Gaussian probability measure, the certificate EX∼x+µ[φ(X)]−2 ( 2G( 2σ )−1 ) given in our paper is equivalent to the certificate HS,1 ≤ [ θa−θb2 ]+ given in paper Dvijotham et al. (2020).
Therefore, when applying both methodologies to Gaussian measure, the formulas obtained are theoretically equivalent. Despite the similarity in analyzing Gaussian measure, our work covers cases with bounded support sets, which is our main contribution.
4 EXPERIMENTS
For adversarial robustness certification, we choose the test set certified accuracy as our metric of interest, which is defined as the percentage of data points in the test set that can be correctly classified and can also pass the robustness certification within an l2 ball of an assigned radius r. To pass the robustness certification at data point x, the classification results of all points within an l2 distance to the original point x must be consistent. For a model using the smoothing method, the classification result of a data point is the class with the highest score in the smoothed data distribution, not to be confused with the direct output of the base classifier at that data point.
In all experiments, the certification process on the test set with assigned perturbation l2 radius is shown in the following Algorithm 1. Note that the cert(scorea, 1− scorea, r) function returns true if the certification objective is non-negative, otherwise it returns false. Such objective is calculated
using formulas in Table 1 with l2 perturbation and corresponding smoothing distribution. Since our method using Wasserstein distance does not involve iterations, our certification procedure only costs constant computation time, which is much faster than Dvijotham et al. (2020).
Algorithm 1 Certification process Input: T : test set, gt(x): true class of image x, f(x): base classifier,D(x): smoothing distribution,
n: sample amount, r: perturbation radius Output: acc: test set certified accuracy
1: certCount← 0, allCount← 0 2: for all x ∈ T do 3: S ← {n samples from D(x)} 4: countc ← 0 for every class c 5: for all x′ ∈ S do 6: countf(x′) ← countf(x′) + 1 7: end for 8: scorec ← countccard(S) for every class c 9: a← arg maxc{scorec}
10: if a = gt(x) ∧ cert(scorea, 1− scorea, r) then 11: certCount← certCount+ 1 12: end if 13: allCount← allCount+ 1 14: end for 15: return acc← certCountallCount
We achieve identical results when comparing our W-distance method with the F-divergence method in Dvijotham et al. (2020) using Gaussian distribution and with specific metric parameter settings, which is proved possible in Section 3.4. However, there is no previous work done yet to examine the usage of uniform distribution when smoothing, so we mainly focus on comparing Gaussian, l2 and l∞ uniform distribution all using our W-distance method.
4.1 SETUPS
We choose CIFAR-10 as our dataset and ResNet-110 as our base classifier. We firstly train the base classifier on the 50000 image training set without smoothing and achieve 89.6% prediction accuracy on the 10000 image test set. Then we run the certification process on the test set with incremental perturbation radius r. We test out different smoothing distributions as mentioned above, and we change the parameters of such distributions to illustrate the effect of different distributions further. We also try increasing the smoothing sample amount to examine the trade-off between performance and accuracy improvement. All training, testing, and certification are run on an NVIDIA RTX 3090.
4.2 W-DISTANCE METHOD WITH DIFFERENT SMOOTHING DISTRIBUTIONS
We firstly implement our W-distance method with N(x, 0.05) as smoothing distribution and sample amount n=100, and then we change the variance of the Gaussian distribution to 0.025 and 0.1. As shown in Figure 2, there is a neat cut-off for each setting where the perturbation gets too big, and no data point can pass the certification at this point. By changing the variance of the smoothing distribution, we observe a clear trend that the increase of variance leads to a drop of initial certification
accuracy but also stronger robustness that can endure more significant perturbation, and the decrease of the variance leads to the opposite change accordingly.
Next, for smoothing process, we substitute Gaussian distribution with l2 or l∞ uniform distribution, with the norm range set to 0.025, 0.05 and 0.1. In Figure 2, both experiment results show identical characteristics as with Gaussian distribution, but they bring along a critical issue, the mismatch of the perturbation radius’s magnitude. Comparing the perturbation radius at the cut-off point, we find that the radius of Gaussian distribution is about 50 times larger than that of two uniform distributions.
We assume that this phenomenon is caused by the lack of intersection of the smoothing distributions before and after perturbation. For Gaussian distribution, there is always an intersection no matter how big the perturbation radius gets, but two uniform distributions will separate quickly and become disjoint under perturbation. Furthermore, the dimension of a 32× 32× 3 image is 3072, the square root of which is around 55.4, very close to the cut-off radius’s 50 times ratio difference. Such correlation may trace to the involvement of dimension when calculating the finite support set volume of l2 and l∞ uniform distribution, while the support set volume of Gaussian distribution is infinitely large. We conjecture that such deficiency is inherent when using the uniform distribution, which can hardly be further improved.
4.3 W-DISTANCE METHOD WITH DIFFERENT SAMPLING AMOUNTS
When calculating scores for each class in the smoothing process, as we cannot classify all possible data points, we shall only acquire approximate scores by sampling from the smoothed data distribution. Thus such scores may differ in multiple runs due to the randomness of sampling. However, through our experiments, we find that with a certain amount of samples, we can already obtain sufficiently accurate scores, which cannot be significantly improved by increasing the sample amount.
We set the sample amount n to 100, 1000, and 10000 with three different smoothing distributions, and they all obtain similar results: it takes only 10 minutes to run through the 10000 images test set with 100 samples for each image, 30 minutes with 1000 samples and 3 hours with excessive 10000 samples. It is ten times faster than the 2 hours running time with the iteration-based method in Dvijotham et al. (2020) using just 100 samples. It is also worth noting in Figure 3 that by increasing the sample amount, no significant improvement is observed with Gaussian distribution. However, there is minor progress made with both uniform distributions when the samples are getting overly abundant. We assume that the extra samples make up for the lack of intersections of smoothing uniform distributions before and after the perturbation, while Gaussian distribution has no such issues.
5 CONCLUSION
We have introduced a framework based on Wasserstein distance and total variance distance relaxation as well as Lagrange duality. This methodology is able to deal with the analysis of bounded support set smoothing measures, which is not covered by previous work. Moreover, we have analyzed the performance of specific smoothing measures, including Gaussian probability measure and uniform probability measures with support set B2(O, r), B∞(O, r) theoretically and experimentally, which shows the relative incapability of bounded support set smoothing measures compared with Gaussian smoothing measure.
REFERENCES Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of
security: Circumventing defenses to adversarial examples. In ICML, 2018.
Avrim Blum, Travis Dick, Naren Manoj, and Hongyang Zhang. Random smoothing might be unable to certify l∞ robustness for high-dimensional images. Journal Of Machine Learning Research, 2020.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In ICML, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
D.C Dowson and B.V Landau. The fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis, 1982.
Krishnamurthy Dj Dvijotham, Jamie Hayes, Borja Balle, Zico Kolter, Chongli Qin, Andras Gyorgy, Kai Xiao, Sven Gowal, and Pushmeet Kohli. A framework for robustness certification of smoothed classifiers using f-divergences. In ICLR, 2020.
Aounon Kumar, Alexander Levine, Tom Goldstein, and Soheil Feizi. Curse of dimensionality on randomized smoothing for certifiable robustness. In ICML, 2020.
Guang-He Lee, Yang Yuan, Shiyu Chang, and Tommi S Jaakkola. Tight certificates of adversarial robustness for randomly smoothed classifiers. arXiv preprint arXiv:1906.04948, 2019.
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Certified adversarial robustness with additive noise. arXiv preprint arXiv:1809.03113, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Matthew Thorpe. Introduction to optimal transport, 2018.
Lily Weng, Huan Zhang, Hongge Chen, Zhao Song, Cho-Jui Hsieh, Luca Daniel, Duane Boning, and Inderjit Dhillon. Towards fast computation of certified robustness for relu networks. In ICML, 2018.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, 2018.
Kai Y Xiao, Vincent Tjeng, Nur Muhammad Shafiullah, and Aleksander Madry. Training for faster adversarial robustness verification via inducing relu stability. arXiv preprint arXiv:1809.03008, 2018.
Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint arXiv:1906.06316, 2019.
A OPTIMAL TRANSPORT THEORY
Assume µ, ν ∈ P(Rd). Besides, assume µ, ν are absolutely continuous w.r.t. Lebesgue measure λ and let density functions be f and g. Definition 2 (Push Forward). If T : Rd → Rd, then the distribution of T (X) is called the pushforward of P , denoted by T#P . In other words,
T#P (A) = P (T (x) ∈ A) = P (T−1(A)) Definition 3 (Optimal Distance, Optimal Transport Map). The Monge version of the optimal transport distance is
inf T :T#P=Q
∫ ||x− T (x)||pdP (x) (27)
A minimizer T ∗, if one exists, is called the optimal transport map. Definition 4 (Wasserstein Distance, Earth Mover Distance, Optimal Transport Plan). Let Γ(µ, ν) denote all joint distributions γ for (X,Y ) that have marginals µ and ν. Then the Wasserstein distance is
Wp(µ, ν) =
( inf
γ∈Γ(µ,ν)
∫ ||x− y||p2dγ(x, y) ) 1 p where p ≥ 1 (28)
When p = 1, this is also called the Earth Mover distance. The minimizer γ∗ (which does exist) is called the optimal transport plan. Lemma A.1 (Dual Formulation of Wasserstein Distance When p ≤ 1). It can be shown that
W pp (µ, ν) = sup ψ,φ
∫ ψ(y)dν(y)− ∫ φ(x)dµ(x) (29)
where ψ(y)− φ(x) ≤ ||x− y||p. In the special case when p = 1, we have the very simple representation
W1(µ, ν) = sup ϕ∈F1
∫ ϕ(x)dµ(x)− ∫ ϕ(x)dν(x) = sup
ϕ∈F1
∫ ϕ(x)d(µ−ν)(x) = sup
ϕ∈F1
∫ ϕ(x)(f−g)(x)dx
(30) where F1 denotes all maps from Rd to R such that |f(x)− f(y)| ≤ ||x− y|| for all x, y. In the case when 0 < p < 1, we have similar simple representation
Wp(µ, ν) = sup ϕ∈Fp
∫ ϕ(x)dµ(x)− ∫ ϕ(x)dν(x) = sup
ϕ∈Fp
∫ ϕ(x)d(µ−ν)(x) = sup
ϕ∈Fp
∫ ϕ(x)(f−g)(x)dx
(31) where Fp denotes all maps from Rd to R such that |f(x)− f(y)| ≤ ||x− y||p for all x, y. Lemma A.2 (Dual Formulation of Wasserstein Distance When 1 < p < ∞). In the case when 1 < p < ∞ and the support sets of measure µ and ν are included in a convex compact set K. Define R = supx∈K ||x||2, then we have slightly different dual formulation
Wp(µ, ν) ≥ (
sup ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)d(ν − µ)(y)− (p− 1)(2R)p−1 ) 1 p
= ( sup
ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)(g − f)(y)dy − (p− 1)(2R)p−1 ) 1 p
(32)
where Lip(p(2R)p−1) denotes all maps f from Rd to R such that |f(x)−f(y)| ≤ p(2R)p−1||x−y|| for all x, y ∈ K. Definition 5 (Total Variation Distance). The total variation distance between two probability distribution µ and ν on Rd is defined by
||µ− ν||TV = max { |µ(A)− ν(A)| : A ⊆ Rd } (33)
whereRd is the set of all Borel subsets. Lemma A.3. Let µ and ν be two probability distributions on Rd and absolutely continuous w.r.t. Lebesgue measure λ. Assume the density function of measure µ and ν w.r.t. λ are f(x) and g(x). Then,
||µ− ν||TV = 1
2 ∫ Rd ∣∣f(x)− g(x)∣∣dx (34)
B PROOF OF LEMMA A.2
Recall the dual form of Wasserstein distance
W pp (µ, ν) = sup ψ,φ∈C(Rd)
∫ ψ(y)dν(y)− ∫ φ(x)dµ(x) (35)
where ψ(y)− φ(x) ≤ ||x− y||p. For simplicity of the proof, consider equivalent form
W pp (µ, ν) = sup ψ,φ∈C(Rd)
∫ ψ(y)dν(y) + ∫ φ(x)dµ(x) (36)
where ψ(y)− φ(x) ≤ ||x− y||p. First, we introduce a theorem in Thorpe (2018) Theorem B.1 (Existence of a Maximiser to the Dual Problem). Let µ ∈ P(X), ν ∈ P(Y ), where X and Y are polish, and c : X × Y → [0,+∞). Assume that there exists cX ∈ L1(µ), cY ∈ L1(ν) such that c(x, y) ≤ cX(x) + cY (y) for µ-almost every x ∈ X and ν-almost every y ∈ Y . In addition, assume that
M := ∫ X cX(x)dµ(x) + ∫ Y cY (y)dν(y) <∞ (37)
Then there exists (ϕ,ψ) ∈ Φc = {(ϕ,ψ) ∈ L1(µ) × L1(ν) : ϕ(x) + ψ(y) ≤ c(x, y)} where the inequality is understood to hold for µ-almost every x ∈ X and ν-almost every y ∈ Y such that
sup Φc
J = J(ϕ,ψ) (38)
where J is defined by J : L1(µ) × L1(ν) → R, J(ϕ,ψ) = ∫ X ϕdµ + ∫ Y ψdν. Futhermore we can choose (ϕ,ψ) = (ηcc, ηc) for some η ∈ L1(µ). For η : X → R̄, the c-transforms ηc, ηcc are defined by
ηc : Y → R̄, ηc(y) = inf x∈X (c(x, y)− η(x)) (39)
ηcc : Y → R̄, ηcc(y) = inf x∈X (c(x, y)− ηc(x)) (40)
Lemma B.1. For a, b ∈ R and 1 ≤ p <∞,
|a+ b|p ≤ 2p−1(|a|p + |b|p) (41)
Proof. First, it’s easy to verify the cases when either of a = 0, b = 0, p = 1 holds. Then, Wlog, assume a, b ∈ R+
|a+ b|p ≤ 2p−1(|a|p + |b|p) ⇐⇒ (a+ b)p ≤ 2p−1(ap + bp)
⇐⇒ 2p−1 (( a
a+ b )p + ( b a+ b )p) ≥ 1
⇐⇒ 2p−1[xp + (1− x)p] ≥ 1,∀x ∈ (0, 1)
where the last inequality is easy to verify.
In our case, c(x, y) = ||x− y||p ≤ (||x||+ ||y||)p ≤ 2p−1(||x||p + ||y||p) and the requirement that M < ∞ is exactly the condition that µ and ν have finite pth moments which is easy to verify by noting that supp(µ) = supp(ν) = K is compact set in Rd. Then, according to the theorem, there exists η ∈ L1(µ) such that
W pp (µ, ν) = sup η∈L1(µ)
∫ ηc(y)dν(y) + ∫ ηcc(x)dµ(x) (42)
Note that ηc possesses Lipschitz continuous property stated below
Lemma B.2. For η ∈ L1(K) where K ⊆ Rd is a convex compact set, then ηcp is a p(2R)p−1Lipschitz function where R := supx∈K ||x|| and cp(x, y) = ||x− y||p, i.e.,
||ηcp(x)− ηcp(y)|| ≤ p(2R)p−1||x− y||, x, y ∈ K (43)
Proof. |ηcp(x)− ηcp(y)| = ∣∣∣ inf z1∈K ( ||x− z1||p − η(z1) ) − inf z2∈K ( ||y − z2||p − η(z2) )∣∣∣ (44) = ∣∣∣ inf z1∈K sup z2∈K (( ||x− z1||p − ||y − z2||p ) − ( η(z1)− η(z2)
))∣∣∣ ≤ sup z∈K
∣∣∣(||x− z||p − η(z))− (||y − z||p − η(z))∣∣∣ (45) = sup z∈K
∣∣||x− z||p − ||y − z||p∣∣ where 44 is due to the definition of c-transform; 45 is obtained by taking a specific value of z1 as z2. Note that K is a compact set and
∣∣||x− z||p− ||y− z||p∣∣ is a continuous function w.r.t. z, then there exists a point z∗ such that
∣∣||x− z∗||p − ||y− z∗||p∣∣ = supz∈K ∣∣||x− z||p − ||y− z||p∣∣. According to the first order condition, z∗ satisfies the equation below
∇z(||x− z||p − ||y − z||p) = ∇z||x− z||p −∇z||y − z||p = ∇z||z − x||p −∇z||z − x||p
= p||z − x|| p 2−1(z − x)> − p||z − y|| p 2−1(z − y)> = 0 (46)
=⇒||z − x|| p 2−1(z − x)> = ||z − y|| p 2−1(z − y)>
=⇒(||z − x|| p 2−1 − ||z − y|| p 2−1)z> = ||z − x|| p 2−1x> − ||z − y|| p 2−1y>
=⇒z = ||z − x|| p 2−1 ||z − x|| p2−1 − ||z − y|| p2−1 x− ||z − y|| p 2−1 ||z − x|| p2−1 − ||z − y|| p2−1 y
where 46 is due to ∇x||x||p = ∇x(x>x) p 2 = p(x>x) p 2−1x> = p||x|| p 2−1x>. And this equation shows that z∗ lie on the line determined by x and y but does not lies on the part between x and y, which can be formulated as z∗ = λx+ (1− λ)y, λ ∈ R \ (0, 1). Note that
sup λ∈R\(0,1) ∣∣∣||x− (λx+ (1− λ)y)||p − ||y − (λx+ (1− λ)y)||p∣∣∣ = sup λ∈R\(0,1)
∣∣∣||(1− λ)(x− y)||p − ||λ(y − x)||p∣∣∣ = sup λ∈R\(0,1) ∣∣∣|1− λ|p − |λ|p∣∣∣ · ||x− y||p = ( sup λ∈R\(0,1)
∣∣∣|1− λ||p − |λ|p∣∣∣) · ||x− y||p Then, we just need to optimize
sup λ∈R\(0,1) ∣∣∣|1− λ|p − |λ|p∣∣∣ s.t. λx+ (1− λ)y ∈ K
Note that we can relax the constraint as below
λx+ (1− λ)y ∈ K ⇐⇒ λ(x− y) + y = (1− λ)(y − x) + x ∈ K =⇒||λ(x− y) + y|| = ||(1− λ)(y − x) + x|| ≤ R (47) =⇒||λ(x− y)|| ≤ R+ ||y||, ||(1− λ)(y − x)|| ≤ R+ ||x|| (48) =⇒|λ| · ||x− y|| ≤ 2R, |1− λ| · ||x− y|| ≤ 2R (49)
=⇒1− 2R ||x− y|| ≤ λ ≤ 2R ||x− y||
where 47 and 49 is due to the definition of R as supx∈K ||x||; 48 is due to triangular inequality.
Using the relaxed constraint, we can show that when λ ≥ 1, ∣∣∣|1−λ|p− |λ|p∣∣∣ = λp− (λ− 1)p is an increasing function w.r.t. λ as p ≥ 1, then∣∣∣|1− λ|p − |λ|p∣∣∣ = λp − (λ− 1)p ≤ ( 2R||x− y|| )p − ( 2R ||x− y|| − 1 )p (50)
And when λ ≤ 0, ∣∣∣|1 − λ|p − |λ|p∣∣∣ = (1 − λ)p − (−λ)p is a decreasing function w.r.t λ as p ≥ 1, then ∣∣∣|1− λ|p − |λ|p∣∣∣ = (1− λ)p − (−λ)p ≤ ( 2R||x− y|| )p − ( 2R ||x− y|| − 1 )p (51) Note that( 2R
||x− y||
)p − ( 2R ||x− y|| − 1 )p = p ( k · ( 2R ||x− y|| ) + (1− k) · ( 2R ||x− y|| − 1 ))p−1 (52)
= p
( 2R
||x− y|| + (k − 1)
)p−1 ≤ p ( 2R
||x− y|| )p−1 where 52 is due to the Differential Mean Value Theorem where k ∈ (0, 1). Thus, we have
|ηcp(x)− ηcp(y)| ≤ p ( 2R
||x− y||
)p−1 · ||x− y||p = p(2R)p−1||x− y|| (53)
i.e. ηcp(x) is a p(2R)p−1-Lipschitz function.
Using Lipschitz continuous property of ηc, we get
W pp (µ, ν) = sup η∈L1(µ)
∫ ηc(y)dν(y) + ∫ ηcc(x)dµ(x) ≤ sup
ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)dν(y) + ∫ ϕc(x)dµ(x)
= sup ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)dν(y) + ∫ ϕc(y)dµ(y)
(54)
where Lip(p(2R)p−1) denotes the set of p(2R)p−1-Lipschitz functions. On the other hand, recall that
W pp (µ, ν) = sup ψ,φ∈C(Rd)
∫ ψ(y)dν(y) + ∫ φ(x)dµ(x) (55)
where ψ(y) + φ(x) ≤ ||x − y||p. Keeping ψ(x) fixed and optimizing w.r.t. φ(y), then we just need to optimize ∫ φ(y)dµ(y) under constraint φ(y) ≤ ||x − y||p − ψ(x). Then obviously we
have φ∗(y) = infx∈K ( ||x − y||p − ψ(x) ) = ψcp(y) where cp(x, y) = ||x − y||p. The map (φ, ψ) ∈ C(K)2 7→ (ψcp , ψ) ∈ C(K)2 replaces dual potentials by ”better” ones improving the dual objective W pp (µ, ν).
Using c-transform, we can reformulate constrained problem into unconstrained convex problem over a single potential
W pp (µ, ν) = sup ψ∈C(Rd)
∫ ψ(y)dν(y) + ∫ ψcp(x)dµ(x) = sup
ψ∈C(Rd)
∫ ψ(y)dν(y) + ∫ ψcp(y)dµ(y)
(56) Combining 54 and 56, we know that when the support set of measure µ and ν supp(µ) = supp(ν) = K where K is a convex compact set, we have
sup ψ∈C(K)
∫ ψ(y)dν(y)+ ∫ ψcp(y)dµ(y) = W pp (µ, ν) ≤ sup
ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)dν(y)+ ∫ ϕc(y)dµ(y)
(57) Note that Lipschitz function must be continuous and therefore Lip(p(2R)p−1) ⊆ C(K). Then, we have
sup ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)dν(y) + ∫ ϕc(y)dµ(y) ≤ sup
φ∈C(K)
∫ ψ(y)dν(y) + ∫ ψcp(y)dµ(y) (58)
Combining 57 and 58, we know the inequality in 57 changes into equality
W pp (µ, ν) = sup ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)dν(y) + ∫ ϕc(y)dµ(y) (59)
Note that for f(x) = xp − p(2R)p−1x, x ∈ R+ achieves its minimum when f ′(x) = pxp−1 − p(2R)p−1 = 0, i.e. x = 2R and the minimum is f(2R) = −(p− 1)(2R)p−1. Then,
ϕcp(y) = inf x∈K
( ||x−y||p−ϕ(x) ) ≥ inf x∈K ( ||x−y||p−ϕ(y)−p(2R)p−1||x−y|| ) = −ϕ(y)−(p−1)(2R)p−1
(60) Thus, we attain a lower bound of W pp (µ, ν)
W pp (µ, ν) = sup ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)dν(y) + ∫ ϕc(y)dµ(y) (61)
≥ sup ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)dν(y)− ∫ ( ϕ(y) + (p− 1)(2R)p−1 ) dµ(y) (62)
= sup ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)dν(y)− ∫ ϕ(y)dµ(y)− (p− 1)(2R)p−1
= sup ϕ∈Lip(p(2R)p−1)
∫ ϕ(y)d(ν − µ)(y)− (p− 1)(2R)p−1
where 61 is due to 59 and 62 is due to 60.
C PROOF OF THEOREM 3.1
Proof. Dx, ,q = {x′ + µ : ||x− x′||q ≤ } (63)
Note that
sup ν∈Dx, ,q Wp(µ, ν) = sup ||x−x′||q≤
Wp(x+ µ, x ′ + µ) = sup
||z||q≤ Wp(µ, z + µ) (64)
where the first equality is due to the definition of Dx, ,q and the second equality is due to the translation invariance property of Wasserstein distance.
Then recall the Monge version of Wasserstein distance Wp(µ, ν) ≤ (
inf T :T#µ=ν
∫ ||x− T (x)||pdµ(x) ) 1 p
(65)
Noticing the inf operator in the Monge version definition of Wp, we can get an upper bound for Wp(µ, ν) by specializing a transport map T̃ satisfying T̃ µ = ν. In our case, we take T̃ : Rd → Rd, T̃ : x 7→ x+ z, and it’s easy to verify that T̃#µ = z + µ. Then we get the upper bound below
Wp(µ, z + µ) ≤ (
inf T :T#µ=z+µ
∫ ||x− T (x)||pdµ(x) ) 1 p ≤ (∫ ||x− T̃ (x)||pdµ(x) ) 1 p = ||z||
(66) where the last equality is due to µ is a probability measure. This provides us with an intuition that the upper bound of Wp(µ, z + µ) is determined by the Euclidean norm of displacement z. Using this upper bound,
sup ||z||q≤ Wp(µ, z + µ) ≤ sup ||z||q≤ ||z||2 (67)
When 0 < q ≤ 2, using the lemma that when 0 < p < q < ∞, ||z||q ≤ ||z||p,∀z ∈ Rd holds, we have sup||z||q≤ ||z||2 ≤ sup||z||q≤ ||z||q = . On the other hand, note that || e1||2 = || e1||q = , we have sup||z||q≤ ||z||2 = . And when q > 2, recall Holder’s Inequality below
Lemma C.1 (Holder’s Inequality for Rn). For {ai}1≤i≤n, {bi}1≤i≤n ⊆ R, r > 1, we have n∑ i=1 |ai||bi| ≤ ( n∑ i=1 |ai|r ) 1 r ( n∑ i=1 |ai| r r−1 ) r−1 r (68)
Apply it to the case n = d, |ai| = |xi|2, |bi| = 1 and r = q2 > 1, d∑ i=1 |xi|2 = d∑ i=1 |xi|2 · 1 ≤ ( d∑ i=1 (|xi|2) q 2 ) 2 q ( d∑ i=1 1 q q−2 ) q−2 q = ( d∑ i=1 |xi|q ) 2 q d1− 2 q (69)
||x||2 = ( d∑ i=1 |xi|2 ) 1 2 ≤ ( d∑ i=1 |xi|q ) 1 q d 1 2− 1 q = ||x||qd 1 2− 1 q (70)
Thus, sup||z||q≤ ||z||2 ≤ sup||z||q≤ ||x||qd 1 2− 1 q = d 1 2− 1 q . On the other hand, note that
|| n 1 q
∑d i=1 ei||q = , ||
n 1 q
∑d i=1 ei||2 = d 1 2− 1 q , we have sup||z||q≤ ||z||2 = d 1 2− 1 q . Combin-
ing the case when 0 < q ≤ 2 and q > 2, we have
sup ||x−x′||q≤
Wp(x+ µ, x ′ + µ) = sup ||z||q≤ Wp(µ, z + µ) ≤
{ when 0 < q ≤ 2
d 1 2− 1 q when q > 2
= max{ , d 1 2− 1 q }
(71)
D W2 DISTANCE RELAXATION IS TIGHT FOR GAUSSIAN PROBABILITY MEASURE
Here, we show that W2 distance relaxation for Gaussian probability measure is tight. Theorem D.1. When µ = N (0, σ2I) and p = 2, the relaxation in 9 is tight. In other words, Dx, ,q ⊆ D
x,max{ , d 1 2 − 1 q },2 but Dx, ,q \ D x,max{ , d 1 2 − 1 q }−δ,2 6= ∅ for any sufficiently small δ > 0. (72)
Proof. Note that Dowson & Landau (1982) established the formula of Wasserstein distance between two Gaussian measures.
Theorem D.2. For Gaussian probability measures µ = N (µ1,Σ1) and ν = N (µ2,Σ2), W2distance between µ and ν have closed form formula
W2(µ, ν) 2 = ||µ1 − µ2||2 + tr ( Σ1 + Σ2 − 2(Σ1Σ2) 1 2 ) (73)
Using above theorem, we yield following tight relaxation between norm-based constraint set Dx, ,q and W2-distance based constraint sets Dx,δ,2 for Gaussian smoothing measures centered at origin, i.e. µ = N (0, σ2I)
sup ν∈Dx, ,q W2(µ, ν) = sup ||x−x′||q≤
W2(x+ µ, x ′ + µ) = sup
||z||q≤ W2(µ, z + µ)
= sup ||z||q≤
||z||2 = max{ , d 1 2− 1 q } (74)
where 74 is due to theorem D.2 and equality 71. And generalization of above theorem when µ = N (0,Σ) can be proved in the same way.
E PROOF OF THEOREM 3.2
Proof. First, we introduce the lemma below.
Lemma E.1. Let X be a random variable that follows d-dimensional Gaussian distribution with density function
f(x;µ,Σ) = 1
(2π) d 2 |Σ| 12
e− 1 2 (x−µ) TΣ−1(x−µ) (75)
where x, µ ∈ Rd and Σ ∈ Sd++. Let H : xTw + b = 0 be a hyperplane in the d-dimensional Euclidean space Rd, where w ∈ Rd and b ∈ R. The hyperplane H defines two half-spaces:
Ω+ = {x ∈ Rd|xTw + b ≥ 0}, Ω− = {x ∈ Rd|xTw + b < 0} (76)
Define the integral over half-space Ω+ as
P = ∫ Ω+ f(x;µ,Σ)dx = ∫ Ω+
1
(2π) d 2 |Σ| 12
e− 1 2 (x−µ) TΣ−1(x−µ)dx
Since Σ is positive definite symmetric, there exist an orthogonal matrix U and a diagonal matrix D with positive diagonal elements such that Σ = UTDU . Let x0 = − µ >w+b
|| √ DUw||2 and hence P =∫∞ x0 1√ 2π e− 1 2x 2 dx.
(The proof of this lemma is credit to https://math.stackexchange.com/questions/ 556977/gaussian-integrals-over-a-half-space.)
Recall the definition of lp-norm constraint set of probability measures
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } (77) Note that
sup ν∈Dx, ,q TV (µ, ν) = sup ||x−x′||q≤ TV (x+ µ, x′ + µ) = sup ||z||q≤ TV (µ, z + µ) (78)
where the first equality is due to the definition of Dx, ,q and the second equality is due to the translation invariance property of total variance distance.
Define hyperplane H1 : xT z − ||z|| 2 2
2 = 0 and H 2 : xT z + ||z||22 2 = 0. The hyperplane H 1 defines
two half-spaces: Ω1+ = {x ∈ Rd|xT z − ||z||22 2 ≥ 0} and Ω 1 − = {x ∈ Rd|xT z − ||z||22 2 < 0}. And the hyperplane H2 defines two half-spaces: Ω2+ = {x ∈ Rd|xT z + ||z||22 2 ≥ 0} and Ω 2 − = {x ∈ Rd|xT z + ||z|| 2 2
2 < 0}. Applying lemma E.1 and lemma A.3, we know that sup ||z||q≤ TV (µ, z + µ)
= sup ||z||q≤
1
2 ∫ ∣∣∣ 1 (2π) d 2 σd e− xT x 2σ2 − 1 (2π) d 2 σd e− (x−z)T (x−z) 2σ2 ∣∣∣dx (79) = 1
2 sup ||z||q≤ ∫ Ω1+
1
(2π) d 2 σd ∣∣∣e− xT x2σ2 − e− (x−z)T (x−z)2σ2 ∣∣∣dx+ ∫ Ω1−
1
(2π) d 2 σd ∣∣∣e− xT x2σ2 − e− (x−z)T (x−z)2σ2 ∣∣∣dx = 1
2 sup ||z||q≤ ∫ Ω1+
1
(2π) d 2 σd
( e− (x−z)T (x−z) 2σ2 − e− xT x 2σ2 ) dx+ ∫ Ω1−
1
(2π) d 2 σd
( e− xT x 2σ2 − e− (x−z)T (x−z) 2σ2 ) dx
(80)
= 1
2 sup ||z||q≤ ∫ Ω1+ d(z + µ)− ∫ Ω1+ dµ+ ∫ Ω1− dµ− ∫ Ω1− d(z + µ)
= 1
2 sup ||z||q≤ ∫ Ω2+ dµ− ∫ Ω1+ dµ+ ∫ Ω1− dµ− ∫ Ω2− dµ (81)
= 1
2 sup ||z||q≤
∫ ∞ −||z||2
2σ
1√ 2π e− 1 2x 2 dx− ∫ ∞ ||z||2
2σ
1√ 2π e− 1 2x 2 dx+
∫ ||z||2 2σ
−∞
1√ 2π e− 1 2x 2 dx− ∫ − ||z||22σ −∞ 1√ 2π e− 1 2x 2 dx
(82)
= 1
2 sup ||z||q≤
( G ( ||z||2
2σ
) −G ( − ||z||2
2σ
) +G ( ||z||2 2σ ) −G ( − ||z||2 2σ )) (83)
= 1
2 sup ||z||q≤ 2
( 2G ( ||z||2
2σ
) − 1 )
(84)
= 2G (max{ , d 12− 1q }
2σ
) − 1 (85)
where 79 is due to lemma A.3; 80 is due to the consistency of sign of integrand function on Ω1+ and Ω1−; 81 is due to the transformation formula of space coordinates; 82 is due to lemma E.1; 83 and 85 is due to the definition and central symmetry property of G as the cumulative density function of standard normal distribution; 85 is due to 71.
F PROOF OF THEOREM 3.3
Proof. Recall the definition of lp-norm constraint set of probability measures
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } (86) Note that
sup ν∈Dx, ,q TV (µ, ν) = sup ||x−x′||q≤ TV (x+ µ, x′ + µ) = sup ||z||q≤ TV (µ, z + µ) (87)
where the first equality is due to the definition of Dx, ,q and the second equality is due to the translation invariance property of total variance distance. Next, compute the value of TV (µ, z + µ).
Lemma F.1. K is a l1 norm ball centered at original point of radius r, then K ∩ (z + K) = ∅ if and only if ||z||1 > 2r.
Proof. First, we prove the if part and assume ||z||1 > 2r. Consider arbitrarily taken x ∈ (z + K), i.e. ||x− z||1 ≤ r. According to the triangular inequality with respect to l1 norm, we have
||x||1 = ||z − (x− z)||1 ≥ ||z||1 − ||x− z||1 > 2r − r = r (88) which shows that x /∈ K and therefore K ∩ (z +K) = ∅. Then we prove the only if part by using reduction to absurdity and assume ||z||1 ≤ 2r. Take y = 12z, then ||y||1 = 12 ||z||1 ≤ r and ||y − z||1 = 1 2 ||z||1 ≤ r which shows that y ∈ K ∩ (z + K) and therefore K ∩ (z +K) 6= ∅ which leads to a contradiction. According to lemma F.1, we know that when { z ∣∣||z||q ≤ , ||z||1 ≥ 2r} 6= ∅, we have
sup ν∈Dx, ,q TV (µ, ν) = sup ||z||q≤ TV (µ, z + µ) = 1 (89)
Define z̄ = 2rd ∑d i=1 ei, and it’s easy to verify that ||z̄||1 = 2r and ||z̄||q = 2rd 1 q−1 for q > 1. Thus, when > 2rd 1 q−1, we have
sup ν∈Dx, ,q TV (µ, ν) = sup ||z||q≤ TV (µ, z + µ) = 1 (90)
G PROOF OF THEOREM 3.4
Proof. First, we introduce the lemmas below for the convenience of later proof.
Lemma G.1 (Volume Formula of d-dimensional spherical cap). The volume of a d-dimensional hyperspherical cap of height h and radius r is given by:
V = π d−1 2 rd
Γ(d+12 ) ∫ arccos( r−hr ) 0 sind(t)dt (91)
where we define h as the value shown in figure 4 and Γ (the gamma function) is given by Γ(z) =∫∞ 0 tz−1e−tdt.
Lemma G.2 (Volume formula of d-dimensional Euclidean ball). The volume of d-dimensional Euclidean ball of radius r is given by
V = π d 2 rd
Γ(d2 + 1) (92)
Recall the definition of lp-norm constraint set of probability measures
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } (93) Note that
sup ν∈Dx, ,q TV (µ, ν) = sup ||x−x′||q≤ TV (x+ µ, x′ + µ) = sup ||z||q≤ TV (µ, z + µ) (94)
where the first equality is due to the definition of Dx, ,q and the second equality is due to the translation invariance property of total variance distance.
Lemma G.3. K is a l2 norm ball centered at original point of radius r, then K ∩ (z + K) = ∅ if and only if ||z||2 > 2r.
According to this lemma, we know that when q ≤ 2 and > 2r, we have
1 ≥ sup ||z||q≤
TV (µ, z + µ) ≥ TV (µ, e1 + µ) = Vol(K∆( e1 +K))
2Vol(K) = 1 (95)
where the last equality is due to || e1||2 = > 2r and applying lemma G.3. And when q > 2 and > 2rd 1 q− 1 2 , we have
1 ≥ sup ||z||q≤ TV (µ, z + µ) ≥ TV (µ, d 1 q d∑ i=1 ei + µ) = Vol(K∆( d 1 q
∑d i=1 ei +K))
2Vol(K) = 1 (96)
where the last equality is due to || d 1 q
∑d i=1 ei||2 = d 1 2− 1 q > 2r and applying lemma G.3. Combin-
ing the results for q ≤ 2 and q > 2, we have
sup ||z||q≤
TV (µ, z + µ) = 1 when > min{2r, 2rd 1 q− 1 2 } (97)
Next, consider the case when ≤ min{2r, 2rd 1 q− 1 2 }. Applying G.1, lemma G.2 and lemma A.3, we have
sup ||z||q≤ TV (µ, z + µ)
= sup ||z||q≤
1
2 ∫ ∣∣∣ 1 Vol(K) Ix∈K − 1 Vol(K) Ix∈z+K ∣∣∣dx (98) = sup ||z||q≤ 1 2Vol(K) ∫ Ix∈K∆(z+K)dx = sup ||z||q≤ Vol(K∆(z +K)) 2Vol(K)
= sup ||z||q≤
Vol(K)− 2π d−1 2 rd
Γ( d+12 )
∫ arccos( ||z||22r ) 0 sind(t)dt
Vol(K) = sup ||z||q≤
1− 2π
d−1 2 rd
Γ( d+12 )
∫ arccos( ||z||22r ) 0 sind(t)dt
Vol(K) (99)
= sup ||z||q≤
1− 2π
d−1 2 rd
Γ( d+12 )
∫ arccos( ||z||22r ) 0 sind(t)dt
π d 2
Γ( d2 +1) rd
= sup ||z||q≤
1− 2Γ(d2 + 1)
π 1 2 Γ(d+12 )
∫ arccos( ||z||22r ) 0 sind(t)dt
(100)
= 1− 2Γ(d2 + 1)
π 1 2 Γ(d+12 )
∫ arccos( max{ , d 12− 1q }2r ) 0 sind(t)dt (101)
where 114 is due to lemma A.3; 99 is due to lemma G.1; 100 is due to lemma G.2; 101 is due to 71. Because of the computation difficulty (overflow), we have to simplify the term Γ( d 2 +1)
Γ( d+12 ) .
When d is even, assume d = 2k, k ∈ N and note that Γ(1) = 1,Γ( 12 ) = π 1 2 , then
Γ(d2 + 1)
Γ(d+12 ) =
Γ(k + 1) Γ(k + 12 ) =
k!Γ(1)
Πki=1(i− 12 )Γ( 1 2 )
= k!
π 1 2 Πki=1(i− 12 )
= (2k)!!
π 1 2 (2k − 1)!!
(102)
Recall the Wallis integral lemma that when d is even∫ π 2
0
sind(t)dt = ∫ π 2
0
cosd(t)dt = π 2 · (d− 1)!! d!! , d = 2k ∈ N (103)
Thus,
Γ(d2 + 1)
Γ(d+12 ) =
(2k)!!
π 1 2 (2k − 1)!!
= 1
2π− 1 2 · (π2 · (2k−1)!! (2k)!! )
= 1
2π− 1 2
∫ π 2
0 sin2k(t)dt
= 1
2π− 1 2
∫ π 2
0 sind(t)dt
(104) When d is odd, assume d = 2k + 1, k ∈ N and note that Γ(1) = 1,Γ( 12 ) = π 1 2 , then
Γ(d2 + 1)
Γ(d+12 ) =
Γ(k + 32 ) Γ(k + 1) = Πki=0(i+ 1 2 )Γ( 1 2 ) k!Γ(1) = π 1 2 Πki=0(i+ 1 2 ) k! = π 1 2 (2k + 1)!! 2 · (2k)!! (105)
Recall the Wallis integral lemma that when d is odd∫ π 2
0
sind(t)dt = ∫ π 2
0
cosd(t)dt = (d− 1)!! d!! , d = 2k + 1 ∈ N (106)
Thus,
Γ(d2 + 1)
Γ(d+12 ) = π
1 2 (2k + 1)!! 2 · (2k)!! =
1
2π− 1 2 · ( (2k)!!(2k+1)!! )
= 1
2π− 1 2
∫ π 2
0 sin2k+1(t)dt
= 1
2π− 1 2
∫ π 2
0 sind(t)dt
(107) To sum up, for all d ∈ N, we have
Γ(d2 + 1)
Γ(d+12 ) =
1
2π− 1 2
∫ π 2
0 sind(t)dt
(108)
Then we avoid the computation of Γ(d2 + 1),Γ( d+1 2 ) and transfer it into the computation of an integral. Applying formula 108, we have
sup ||z||q≤
TV (µ, z + µ) = 1− 1∫ π 2
0 sind(t)dt
∫ arccos( max{ , d 12− 1q }2r ) 0 sind(t)dt (109)
H PROOF OF THEOREM 3.5
Proof. Recall the definition of lp-norm constraint set of probability measures
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } (110)
Note that
sup ν∈Dx, ,q TV (µ, ν) = sup ||x−x′||q≤ TV (x+ µ, x′ + µ) = sup ||z||q≤ TV (µ, z + µ) (111)
where the first equality is due to the definition of Dx, ,q and the second equality is due to the translation invariance property of total variance distance.
When ≥ 2r, 1 ≥ sup
||z||q≤ TV (µ, z + µ) ≥ TV (µ, e1 + µ) = 1 (112)
where the first inequality is due to the fact that µ and z + µ are probability measures; the second inequality is due to supp(µ) ∩ supp( e1 + µ) = ∅. Thus, in this case,
sup ||z||q≤ TV (µ, z + µ) = 1 (113)
When < 2r,
sup ||z||q≤ TV (µ, z + µ)
= sup ||z||q≤
1
2 ∫ ∣∣∣ 1 Vol(K) Ix∈K − 1 2Vol(K) Ix∈z+K ∣∣∣dx (114) = sup ||z||q≤ 1 2Vol(K) ∫ Ix∈K∆(z+K)dx = sup ||z||q≤ Vol(K∆(z +K)) 2Vol(K) = sup ||z||q≤ 1− Vol(K ∩ (z +K)) Vol(K)
= sup ||z||q≤
1− Π d i=1(2r − |zi|)
(2r)d = sup ||z||q≤
1−Πdi=1 (
1− |zi| 2r
) (115)
First, we study typical cases when q = 1, 2,∞. When q = 1, we need to solve the following optimization problem
inf ||z||1≤
Πdi=1(2r − |zi|) (116)
Here we use mathematical induction to prove that
inf ||z||1≤
Πdi=1(2r − |zi|) = (2r)d−1(2r − ) (117)
When d = 2,
inf ||z||1≤ Πdi=1(2r − |zi|) = inf|z1|+|z2|≤ (2r − |z1|)(2r − |z2|) = inf |z2|≤ (2r − + |z2|)(2r − |z2|)
= inf 0≤z2≤ (2r − + z2)(2r − z2) = inf 0≤z2≤ z2( − z2) + 2r(2r − ) = 2r(2r − )
Thus, induction hypothesis holds for d = 2. Then, assume induction hypothesis holds for d = n. When d = n+ 1,
inf ||z||1≤ Πn+1i=1 (2r − |zi|) = inf∑n+1 i=1 |zi|≤ Πn+1i=1 (2r − |zi|) = inf∑n i=1 |zi|≤ −|zn+1|
( Πni=1(2r − |zi|) ) (2r − |zn+1|)
= inf |zn+1|≤
(2r)n−1(2r − + |zn+1|)(2r − |zn+1|) (118)
= (2r)n(2r − ) = (2r)d−1(2r − ) (119)
where 118 is due to the induction hypothesis when d = n; 119 is due to the induction hypothesis when d = 2. Therefore, we have already proved that
inf ||z||1≤
Πdi=1(2r − |zi|) = (2r)d−1(2r − ),∀d ∈ N (120)
Plugging in this result, it follows that
sup ||z||1≤ TV (µ, z + µ) = sup ||z||1≤
1− Π d i=1(2r − |zi|) (2r)d = 1− (2r) d−1(2r − ) (2r)d = 2r (121)
When q = 2, we need to solve the following optimization problem
inf ||z||2≤
Πdi=1(2r − |zi|) (122)
When d = 2,
inf ||z||2≤ Πdi=1(2r − |zi|) = inf|z1|2+|z2|2≤ 2 (2r − |z1|)(2r − |z2|) = inf |z2|≤
( 2r − ( 2 − z22) 1 2 ) (2r − |z2|)
= inf 0≤z2≤
( 2r − ( 2 − z22) 1 2 ) (2r − z2)
Define f(z2) = ln ( 2r − ( 2 − z22) 1 2 ) + ln (2r − z2), then
f ′(z2) = z2(
2 − z22)− 1 2
2r − ( 2 − z22) 1 2
− 1 2r − z2 = 2rz2(
2 − z22)− 1 2 − z22( 2 − z22)− 1 2 − 2r + ( 2 − z22) 1 2(
2r − ( 2 − z22) 1 2 ) (2r − z2)
(123) Define g(z2) = 2rz2( 2 − z22)− 1 2 − z22( 2 − z22)− 1 2 − 2r + ( 2 − z22) 1 2 , then
g′(z2) = (2z 3 2 − 3 2z2 + 2r 2)( 2 − z22)− 3 2 (124)
Define h(z2) = 2z32 − 3 2z2 + 2r 2, then h′(z2) = 6z22 − 3 2 = 6(z2 − √2 )(z2 + √ 2 ). Thus, when 0 ≤ z2 ≤ √2 , h ′(x) ≤ 0; when √ 2 < z2 ≤ , h′(x) > 0. Thus, the minimum value of h(x)
on interval [0, ] is h( √ 2 ) = √ 2 2( √
2r − ). Therefore, function f(z2) behaves differently when 0 < ≤ √ 2r and when √ 2r < < 2r. When 0 < ≤ √
2r, h(z2) ≥ h( √2 ) = √ 2 2( √
2r − ) ≥ 0 on interval [0, ] and therefore g′(z2) = h(z2)( 2 − z22)− 3 2 ≥ 0. Note that g(0) = − 2r < 0, g( √ 2 ) = 0, g( −) = ∞ and therefore f ′(z2) ≤ 0 when 0 ≤ z2 ≤ √2 while f ′(z2) > 0 when √2 < z2 ≤ . Thus, f(z2) takes its minimum when z2 = √2 . In this case,
inf 0≤z2≤
( 2r − ( 2 − z22) 1 2 ) (2r − z2) = (2r −
√ 2 )2 (125)
When √
2r < < 2r, we have h(0) = 2r 2 > 0, h( √ 2 ) =
√ 2 2( √
2r − ) < 0, h( ) = 2(2r − ) > 0. Assume h(t1) = h(t2) = 0, 0 < t1 < √2 < t2 < , then when 0 ≤ z2 ≤ t1 or t2 ≤ z2 ≤ , h(z2) ≥ 0 and when t1 < z2 < t2, h(z2) < 0. Therefore, g′(z2) ≥ 0 when 0 ≤ z2 ≤ t1 or t2 ≤ z2 ≤ ; g′(z2) < 0 when t1 < z2 < t2. Note that g(z2) = 0 ⇐⇒ (2z22 − 2) ( 2(z2−r)2+2r2− 2 ) = 0, therefore when 0 ≤ z2 ≤ r− √ 2
2 − r2 or √ 2 ≤ z2 ≤ r+
√ 2
2 − r2, g(z2) ≤ 0; when r − √ 2
2 − r2 < z2 < √ 2
or r + √ 2
2 − r2 < z2 < , g(z2) > 0. Thus, when 0 ≤ z2 ≤ r− √ 2
2 − r2 or √ 2 ≤ z2 ≤ r+
√ 2
2 − r2, f ′(x) ≤ 0; when r−
√ 2
2 − r2 < z2 < √ 2 or r + √ 2
2 − r2 < z2 < , f ′(x) > 0. Thus, f(z2) takes its minimum when z2 = r −
√ 2
2 − r2 or z2 = r + √ 2 2 − r2. In this case,
inf 0≤z2≤
( 2r − ( 2 − z22) 1 2 ) (2r − z2) = ( r −
√ 2
2 − r2
)( r +
√ 2
2 − r2
) = 2r2 − 2
2 (126)
inf ||z||2≤ Πdi=1(2r−|zi|) = inf∑n+1 i=1 z 2 i≤ 2 Πn+1i=1 (2r−|zi|) = inf∑n i=1 z 2 i≤ 2−z2n+1
( Πni=1(2r−|zi|) ) (2r−|zn+1|)
(127) By then, we have understand clearly the optimization problem when d = 2.
Then, consider the case when d = 3. When d = 3,
inf ||z||2≤ Πdi=1(2r − |zi|) = inf z21+z 2 2+z 2 3≤ 2 (2r − |z1|)(2r − |z2|)(2r − |z3|) (128)
When 0 < ≤ √
2r, assume the optimal point is z∗. We will prove that each coordinate of z∗ has the same value. Here we use reduction to absurdity, and wlog assume z∗1 6= z∗2 . By fixing the value of z∗3 , the optimization problem 122 is equivalent to
inf z21+z 2 2≤ 2−(z∗3 )2
(2r − |z1|)(2r − |z2|) (129)
And (z∗1 , z ∗ 2) should be an optimal point of above problem. Note that 2 − (z∗3)2 ≤ 2 ≤ 2r2 and applying 125, we know that z∗1 = z ∗ 2 which is a contradiction. Thus, z ∗ 1 = z ∗ 2 = z ∗ 3 = c. And
inf ||z||2≤ Πni=1(2r − |zi|) = inf c≤ √
3
(2r − c)3 = (
2r − √ 3
)3 (130)
When √ 2r < ≤ √
3r, it’s obvious that the optimal point z∗ of optimization problem 128 must lie on the boundary of feasible region, i.e. (z∗1) 3 + (z∗2) 3 + (z∗3) 3 = 2. Wlog, assume (z∗3) 3 ≥ 2
3 and (z∗1) 2 + (z∗2) 2 ≤ 2 2 3 ≤ 2r 3. By fixing the value of z∗3 and following similar deduction procedure as above we know that z∗1 = z ∗ 2 = c ∗, where c∗ is the optimal point of following optimization problem.
inf 0≤c≤ √
3
(2r − c)2(2r − √ 2 − 2c2) (131)
Define f(x) = 2 ln (2r − x) + ln (2r − √ 2 − 2x2) where 0 ≤ x ≤ √
3 , then
f ′(x) = 2(3x2 − 2rx− 2 + 2r
√ 2 − 2x2)
(x− 2r)(2r − √ 2 − 2x2) √ 2 − 2x2
(132)
It’s obvious that the denominator of f ′(x) is negative. As for the numerator, define g(x) = 3x2 − 2rx− 2 where 0 ≤ x ≤ √
3 . Note that g(x) ≤ max { g(0), g ( √ 3 )} = max { − 2,−2r √ 3 } ≤ 0 (133)
Thus, we have the following equivalent relationship 3x2 − 2rx− 2 + 2r √ 2 − 2x2 ≤ 0
⇐⇒ 3x2 − 2rx− 2 ≤ −2r √ 2 − 2x2 ≤ 0
⇐⇒ (3x2 − 2rx− 2)2 ≥ ( − 2r √ 2 − 2x2 )2 ≥ 0 ⇐⇒ (3x2 − 2)(3x2 − 4rx+ 4r2 − 2) ≥ 0 ⇐⇒ 3x2 − 4rx+ 4r2 − 2 ≤ 0
⇐⇒  ∅ when √ 2r < ≤ 2 √ 2 3 r 2r − √
3 2 − 8r2 3
≤ x ≤ 2r + √
3 2 − 8r2 3 when 2
√ 2
3 r < ≤
√ 3r
where the last equivalent relationship is due to the discriminant of the quadratic equation 3x2 − 4rx+4r2− 2 is ∆ = 4(3 2−8r2). Therefore, when √ 2r < ≤ 2 √ 2 3r, f ′(x) ≤ 0,∀0 ≤ x ≤ √ 3 and hence the optimal point c∗ in the optimization problem 131 takes value √ 3 , whereas when
2 √
2 3r < ≤
√ 3r, f ′(x) ≤ 0 for 0 ≤ x ≤ 2r− √ 3 2−8r2 3 , 2r+ √ 3 2−8r2 3 ≤ x ≤ √ 3 and f ′(x) > 0
for 2r− √
3 2−8r2 3 ≤ x ≤
2r+ √
3 2−8r2 3 and note that f(
2r− √
3 2−8r2 3 ) < f( √ 3 ) hence the optimal
point c∗ in the optimization problem takes value 2r− √
3 2−8r2 3 . To sum up, when
√ 2r < ≤ 2 √ 2 3r,
the optimal point z∗ of optimization problem 128 satisfies z∗1 = z ∗ 2 = z ∗ 3 = √ 3
. And when 2 √
2 3r <
≤ √
3r, the optimal point z∗ of optimization problem 128 satisfies z∗1 = z ∗ 2 =
2r− √
3 2−8r2 3 , z ∗ 3 =
4r+ √
3 2−8r2 3 or one of its permutations.
When √
3r < < 2r, similarly we have (z∗1) 3 + (z∗2) 3 + (z∗3) 3 = 2. If there exists 1 ≤ i ≤ 3 such
that (z∗i ) 2 ≥ 2 − 2r2, wlog assume (z∗3)2 ≥ 2 − 2r2. By substituting the value range of x from [0, √ 3 ] into [0, r], following similar deduction procedure and noticing that f( 2r− √ 3 2−8r2 3 ) < f(r), we know that the optimal point z∗ in this case satisfies z∗1 = z ∗ 2 = 2r− √ 3 2−8r2 3 , z ∗ 3 = 4r+ √ 3 2−8r2 3 or one of its permutations. On the other hand, if (z∗i ) 2 < 2 − 2r2 for all 1 ≤ i ≤ 3, then
(z∗1) 2 + (z∗2) 2 = 2 − (z∗3)2 > 2r2. Applying 126 and taking z∗1 = r − √ 2−(z∗3 )2 2 − r2, z ∗ 2 =
r +
√ 2−(z∗3 )2
2 − r2, we know the optimization problem is equivalent to
inf 0≤z3< √ 2−2r2
( 2r2 −
2 − z23 2
) (2r − z3) (134)
According to monotonicity analysis of the cubic function above, the optimal point z∗3 is either 2r− √
3 2−8r2 3 or
√ 2 − 2r2. And it’s easy to verify that f( 2r− √ 3 2−8r2 3 ) < f( √ 2 − 2r2) and
therefore z∗3 = 2r− √ 3 2−8r2 3 . However, (z ∗ 2) 2 > 2 − 2r2 which leads to a contradiction.
In summary, considering the case d = 3, when 0 < ≤ 2 √
2 3r, the optimal point z ∗ of original
optimization problem satisfies z∗1 = z ∗ 2 = z ∗ 3 = √ 3 and the optimal value is (2r − √ 3 )3 and when
√ 3r < < 2r, the optimal point z∗, the optimal point z∗ of original optimization problem satisfies
z∗1 = z ∗ 2 =
2r− √
3 2−8r2 3 , z ∗ 3 =
4r+ √
3 2−8r2 3 or one of its permutations.
Next, consider the general case when d = n ≥ 4. In the first place, we point out and prove two useful properties of the optimal point z∗ which help simplify our later discussion a lot.
• All coordinates of optimal point z∗ takes at most two different values.
• If the coordinates of an optimal point z∗ takes exactly two different values c1 and c2, then the number of coordinates equal to c1 must be n− 1 or 1.
Proof. On one hand, by using reduction to absurdity, wlog assume z∗1 , z ∗ 2 , z ∗ 3 take three different values. Fixing the value of the other n − 3 coordinates, we know that (z∗1 , z∗2 , z∗3) is the optimal point of a special case of original problem when d = 3. And note that for all the optimal points of d = 3, there must exist two coordinates taking the same value, which leads to a contradiction. Thus, the first property is satisfied. On the other hand, similarly, by applying reduction to absurdity, wlog assume z∗1 = z ∗ 2 = c1 and z∗3 = z ∗ 4 = c2 where c1 6= c2. Fixing z∗2 , z∗4 and the value of the other n−4 coordinates and aware of the fact that (z∗1) 2 + (z∗3) 2 ≤ 2 2 < 2r 2, we know that (z∗1 , z ∗ 3) is the optimal point of a special case
of original problem when d = 2, < √
2r and therefore z∗1 = z ∗ 3 , which leads to a contradiction.
Thus, the second property is satisfied.
Using the two properties above, we know that the optimal point z∗ has only two possible forms: z∗ = ( √ n , · · · , √ n ) and z∗ = ( c, · · · , c, √ 2 − (n− 1)c2 ) or one of its permutations where
0 ≤ c ≤ √ n−1 , c 6= √ n
, which can be unified into one form: z∗ = ( c, · · · , c, √ 2 − (n− 1)c2 ) or one of its permutations where 0 ≤ c ≤ √
n−1 . Thus, the original problem can be simplified into following optimization problem with one degree of freedom:
inf 0≤c≤ √
n−1
(2r − c)n−1 ( 2r − √ 2 − (n− 1)c2 ) (135)
Define f(x) = (n− 1) ln (2r − x) + ln (2r − √ 2 − (n− 1)x2) where 0 ≤ x ≤ √
n−1 , then
f ′(x) = (n− 1)
( nx2 − 2rx− 2 + 2r √ 2 − (n− 1)x2 ) (x− 2r) ( 2r − √ 2 − (n− 1)x2 )√ 2 − (n− 1)x2 , where 0 ≤ x < √ n− 1 (136)
It’s obvious that the denominator of f ′(x) is negative. As for the numerator, define g(x) = nx2 − 2rx− 2 where 0 ≤ x ≤ √
n−1 . Note that
g(x) ≤ min { g(0), g ( √ n− 1 )} = max { 0, ( − 2 √ n− 1r) n− 1 } ≤ 0 (137)
where the last inequality is due to < 2r < 2 √ n− 1r. Thus, when 0 ≤ x ≤ √
n−1 ,
nx2 − 2rx− 2 + 2r √ 2 − (n− 1)x2 ≤ 0
⇐⇒ nx2 − 2rx− 2 ≤ −2r √ 2 − (n− 1)x2 ≤ 0
⇐⇒ (nx2 − 2rx− 2)2 ≥ ( − 2r √ 2 − (n− 1)x2 )2 ⇐⇒ (nx2 − 2)(nx2 − 4rx+ 4r2 − 2) ≥ 0
⇐⇒ ( x− √
n
) (nx2 − 4rx+ 4r2 − 2) ≥ 0
⇐⇒  x ≥ √ n when 0 < < 2 √ n− 1 n r
x ≥ √ n
or 2r −
√ n 2 − 4(n− 1)r2
n ≤ x ≤
2r + √ n 2 − 4(n− 1)r2
n when 2 √ n− 1 n r ≤ < 2r
where the last equivalence relationship is due to the discriminant of the quadratic equation nx2 − 4rx+ 4r2 − 2 = 0 is
∆ = 4 ( n 2 − 4(n− 1)r2 ) < 0 ⇐⇒ 0 < < 2 √ n− 1 n r (138)
Thus, if 0 < < 2 √
n−1 n r, then f ′(x) ≥ 0 when √ n ≤ x ≤ √ n−1 and f ′(x) < 0 when
0 ≤ x < √ n . Thus, f(x) takes its minimum when x = √ n and therefore c∗ = √ n .
If 2 √
n−1 n r ≤ < 2r, then f
′(x) ≥ 0 when 2r− √ n 2−4(n−1)r2 n ≤ x ≤ 2r+ √ n 2−4(n−1)r2 n or
√ n ≤ x ≤ √ n−1 and f ′(x) < 0 when 0 ≤ x < 2r−
√ n 2−4(n−1)r2 n or 2r+ √ n 2−4(n−1)r2 n <
x < √ n
. In this case, f(x) takes its minimum when x = 2r− √ n 2−4(n−1)r2
n or x = √ n . For the convenience of analysis, assume t = 2r , √ n−1 n ≤ t < 1 and it follows that
e f( √ n ) = (2r)n (
1− t√ n
)n (139)
ef ( 2r− √ n 2−4(n−1)r2 n ) = (2r)n ( (n− 1) + √ nt2 − (n− 1) n )n−1( 1− √ nt2 − (n− 1) n ) (140)
We can prove that there exists tn ∈ [√ n−1 n , 1 ) such that c∗ = √ n when 2 √ n−1 n r ≤ ≤ 2tnr
and c∗ = 2r− √ n 2−4(n−1)r2
n when 2tnr < < 2r while tn converge to 1 at an exponential rate as shown in figure 5.
In conclusion, for the case d = n ≥ 4, when 0 < ≤ 2tnr, c∗ = √n and therefore
inf ||z||2≤
Πni=1(2r−|zi|) = (2r−c∗)n−1 ( 2r− √ 2 − (n− 1)(c∗)2 ) = (
2r− n 1 2
)n = (
2r− d 1 2 )d (141)
Plugging in this result, it follows that
sup ||z||2≤ TV (µ, z+µ) = sup ||z||2≤
1−Π d i=1(2r − |zi|)
(2r)d = 1−
( 2r −
d 1 2 )d (2r)d = 1− ( 1− 2d 1 2 r )d (142)
Figure 5: Graphs of functions f1(t) =
( 1 − t√
n )n
, f2(t) =(
(n−1)+
√
nt2−(n−1) n
)n−1( 1−√nt2−(n−1)
n
)
when n = 4, 16, 64 from left to right. Accord-
ing to the figure, on interval [√
n−1 n , 1 ] , f2(t) is greater than f1(t) at first and then f2(t) exceeds
f1(t). Furthermore, as n increases, the horizontal coordinate of the intersection point converge to 1, which can be seen intuitively from the figure above.
and when 2tnr < < 2r, c∗ = 2r− √ n 2−4(n−1)r2
n and therefore
inf ||z||2≤
Πni=1(2r − |zi|) = (2r − c∗)n−1 ( 2r − √ 2 − (n− 1)(c∗)2 ) = ( 2(n− 1)r + √ n 2 − 4(n− 1)r2 n )n−1( 2r − √ n 2 − 4(n− 1)r2 n
) = ( 2(d− 1)r + √ d 2 − 4(d− 1)r2 n )n−1( 2r − √ d 2 − 4(d− 1)r2 d
) Plugging in this result, it follows that
sup ||z||2≤ TV (µ, z + µ) = sup ||z||2≤
1− Π d i=1(2r − |zi|)
(2r)d
= 1−
( 2(d−1)r+
√ d 2−4(d−1)r2 d )d−1( 2r−√d 2−4(d−1)r2 d ) (2r)d
= 1− (d− 1 +√d( 2r )2 − d+ 1
d
)d−1(1−√d( 2r )2 − d+ 1 d ) When q =∞, it’s easy to verify that
inf ||z||∞≤
Πdi=1(2r − |zi|) = (2r − )d (143)
Plugging in the formula above, it follows
sup ||z||∞≤ TV (µ, z + µ) = sup ||z||∞≤
1− Π d i=1(2r − |zi|) (2r)d = 1− (2r − ) d (2r)d = 1−
( 1−
2r
)d (144)
I PROOF OF THEOREM 3.6
Proof. Recall the definition of lp-norm constraint set of probability measures
Dx, ,q = {x′ + µ : ||x− x′||q ≤ } (145) Assume Dx, ,q ⊆ Dx,ξ( ), then
ξ( ) ≥ sup ν∈Dx, ,q TV (µ, ν) = sup ||x−x′||q≤ TV (x+µ, x′+µ) = sup ||z||q≤ TV (µ, z+µ) ≥ TV (µ, e1+µ)
(146)
which indicates that TV (µ, e1+µ) provides a lower bound for ξ( ). Thus, we only need to estimate the value of TV (µ, e1 + µ). According to lemma A.3, we have
TV (µ, e1 + µ) = Vol(K∆( e1 +K)) 2Vol(K) = 1− Vol(K ∩ ( e1 +K))
Vol(K) Note that K ∩ ( e1 +K)
= { x ∈ Rd ∣∣∣|x1|p + · · ·+ |xd|p ≤ rp, |x1 − |p + |x2|p + · · ·+ |xd|p ≤ rp} = { x ∈ Rd
∣∣∣ − (rp − (|x2|p + · · ·+ |xd|p|)) 1p ≤ x1 ≤ (rp − (|x2|p + · · ·+ |xd|p|)) 1p} = { x ∈ Rd ∣∣∣ − (rp − (|x2|p + · · ·+ |xd|p)) 1p ≤ x1 ≤ 2 } ∪ { x ∣∣∣ 2 ≤ x1 ≤ (rp − (|x2|p + · · ·+ |xd|p|)) 1 p
} := Ω1 ∪ Ω2 where Ω1 ∩ Ω2 = ∅
It’s easy to verify that Vol(Ω1) = Vol(Ω2) according to integration by substitution and therefore Vol(K ∩ ( e1 +K)) = 2Vol(Ω2). To estimate the volume of Ω2, we first introduce several lemmas below for the convenience of later discussion.
Lemma I.1 (Volume formula of d-dimensional lp norm ball). The volume of d-dimensional lp ball of radius r is given by
V (d)p = (2r) d
Γ(1 + 1p ) d
Γ(1 + dp ) (147)
Lemma I.2. The d-dimensional lp ball of volume 1 has radius about d 1 p
2(pe) 1 p Γ(1+ 1p )
.
Proof. When dimension d is big enough, we can obtain an asymptotic volume estimation of lp norm ball with radius r.
V (d)p = (2r) d
Γ(1 + 1p ) d
Γ(1 + dp ) ≈ (2r)d
Γ(1 + 1p ) d√
2π dp ( d pe ) d p =
√ p
2πd
( 2r(pe)
1 pΓ(1 + 1p ) d 1 p
)d (148)
where the first equality is due to lemma I.1 and the approximate equality is due to Stirling’s formula about the estimation of gamma function that Γ(z + 1) ≈ √ 2πz ( z e )z . Thus, when V (d)p = 1, we have
r ≈ d 1 p
2(pe) 1 pΓ(1 + 1p )
(149)
Then we estimate the volume of lp norm ball cap by studying the asymptotic property of the mass distribution of lp norm ball. To begin with, let’s estimate the (d− 1)-dimensional volume of a slice through the center of the lp ball of volume 1. Note that the ball has radius r = (V (d) p )− 1 d . The slice is an (d− 1)-dimensional ball of this radius, so its volume is
V (d−1)p r d−1 = V (d−1)p (V (d) p )
− d−1d = 2d−1 Γ(1 + 1p )
d−1
Γ(1 + d−1p )
( 2d Γ(1 + 1p ) d
Γ(1 + dp )
)− d−1d (150)
Using Stirling’s formula again, when d is sufficiently large, we have
V (d−1)p r d−1
= 2d−1 Γ(1 + 1p )
d−1
Γ(1 + d−1p )
( 2d Γ(1 + 1p ) d
Γ(1 + dp )
)− d−1d = 2d−1 Γ(1 + 1p ) d−1√
2π d−1p ( d−1 pe ) d−1 p
( 2d Γ(1 + 1p ) d√
2π dp ( d pe ) d p
)− d−1d
= 1√
2π d−1p ( d−1 pe ) d−1 p · 1(√ 2π dp ( d pe ) d p )− d−1d = 1( 2πp ) 12d (d− 1) d−1p + 12 d− d−1p − d−12d =
(1 + 1d−1 ) d−1 p + 1 2
( 2πdp ) 1 2d
≈ e 1 p
where the second equality is due to Stirling’s formula for Γ(1+ d−1p ) and Γ(1+ d p ); the third equality just eliminate the exponential of 2 and Γ(1 + 1p ). Thus, we conclude that the slice has volume about e 1 p when d is large.
Then, consider the (d− 1)-dimensional volumes of parallel slices. The slice at distance x from the center is an (d− 1)-dimensional ball whose radius is (rp − xp) 1 p , so the volume of the smaller slice is about
e 1 p
( (rp − xp) 1 p
r
)d−1 = e 1 p ( 1− (x r )p) d−1p Since r is roughly d 1 p
2(pe) 1 p Γ(1+ 1p )
, this is about
e 1 p ( 1− (2x(pe) 1pΓ(1 + 1p )
d 1 p
)p) d−1p = e 1 p ( 1− pe(2xΓ(1 + 1p )) p
d
) d−1 p ≈ exp (1 p −e(2xΓ(1+1 p ))p )
Thus, if we project the mass distribution of the lp ball of volume 1 onto a single coordinate direction, we get a distribution with density function f(x) = exp( 1p − e(2xΓ(1 + 1 p )) p) =
exp (
1 p − e( 2 pΓ( 1 p ))
pxp ) .
Thus, for an lp ball centered at original pointO with volume 1 and approximate radius d 1 p
2(pe) 1 p Γ(1+ 1p )
,
then we can use the integral 2 ∫ s
0 exp ( 1 p − e(2xΓ(1 + 1 p )) p ) dx to estimate the volume between two
parallel slices at the same distance s from the center. Then the volume of lp ball cap corresponding to the slice at distance s from the center can be approximated by 12 − ∫ s 0 exp ( 1 p −e(2xΓ(1+ 1 p )) p ) dx.
Note that the ratio k of slice’s distance d from center to radius r is about s /
d 1 p
2(pe) 1 p Γ(1+ 1p )
=
2s(pe) 1 p Γ(1+ 1p )
d 1 p
, i.e. s = kd 1 p
2(pe) 1 p Γ(1+ 1p )
. Thus, the volume of cap can be represented as
1 2 − ∫ kd 1p 2(pe) 1 p Γ(1+ 1 p )
0
exp (1 p − e(2xΓ(1 + 1 p ))p ) dx
which is only related to the ratio k. Then, we can conclude that for a lp ball with radius r, when dimension d is large enough and its cap corresponding to the slice at distance h form the center, then the volume ratio of cap to ball is approximately
1 2 − ∫ sd 1p 2r(pe) 1 p Γ(1+ 1 p )
0
exp (1 p − e(2xΓ(1 + 1 p ))p ) dx (151)
Thus,
Vol(Ω2) Vol(K) = 1 2 − ∫ d 1p 4r(pe) 1 p Γ(1+ 1 p )
0
exp (1 p − e(2xΓ(1 + 1 p ))p ) dx (152)
and therefore
TV (µ, e1 + µ) = 1− Vol(K ∩ ( e1 +K)) Vol(K) = 1− 2Vol(Ω2) Vol(K)
= 2
∫ d 1p 4r(pe) 1 p Γ(1+ 1
p )
0
exp (1 p − e(2xΓ(1 + 1 p ))p ) dx
J PROOF OF THEOREM 3.7
Proof. Note that f(x) and g(x) are respectively density functions of reference measure ρ = x + µ and perturbed measure ν and q(x) is defined as g(x)− f(x). Therefore
EX∼ν [φ(X)] = ∫ φ(x)g(x)dx = ∫ φ(x) ( g(x)− f(x) ) dx+ ∫ φ(x)f(x)dx = ∫ φ(x)q(x)dx+ ∫ φ(x)f(x)dx
= ∫ φ(x)q(x)dx+ EX∼x+µ[φ(X)]
where the first term contains all the uncertainty in one functional variable q(x) and the second term is a constant when sample point x, smoothing measure µ and specification φ are fixed. And when ν ∈ Dx,δ,p or equivalently Wp(ν, x + µ) ≤ δ, applying the dual form of Wp distance given in formula 30 and 31, we have
W1(ν, x+ µ) = sup ϕ∈F1
∫ ϕ(x)(f − g)(x)dx = sup
ϕ∈F1
∫ ϕ(x)(g − f)(x)dx = sup
ϕ∈F1
∫ ϕ(x)q(x)dx
= sup ||f ||L≤1
∫ f(x)q(x)dx ≤ δ (153)
And when ν ∈ Dx,ξ or equivalently TV (ν, x+ µ) ≤ ξ, applying lemma A.3 for absolutely continuous measure, we have
TV (ν, x+ µ) = 1
2
∫ |f(x)− g(x)|dx = 1
2
∫ |q(x)|dx ≤ ξ (154)
It follows that OPT (φ, x+ µ,Dx,δ,p ∩Dx,ξ) is equivalent to minν∈Dx,δ,p∩Dx,ξ E[φ(X)] according to the definition and therefore equivalent to optimization problem 23 which is obviously convex according to 153 and 154.
K PROOF OF THEOREM 3.8
Recall the following result proved in the section before EX∼ν [φ(X)] = ∫ φ(x)q(x)dx+ EX∼x+µ[φ(X)] (155)
When ν ∈ Dx,δ,p or equivalently Wp(ν, x + µ) ≤ δ, applying the dual form of Wp distance given in formula 32 and noticing that supy∈spt(ν)∪spt(x+µ) ||y||2 = ||x||2 + R + max{ , d 1 2− 1 q } := R∗, we have( sup
ϕ∈Lip(p(2R∗)p−1)
∫ ϕ(y)(g − f)(y)dy − (p− 1)(2R∗)p−1 ) 1 p ≤Wp(ν, x+ µ) ≤ δ (156)
or equivalently
sup ϕ∈Lip(p(2R∗)p−1)
∫ ϕ(y)(g − f)(y)dy = sup
||f ||L≤p(2R∗)p−1
∫ f(x)q(x)dx ≤ δp + (p− 1)(2R∗)p−1
(157) where Lip ( p(2R∗)p−1 ) denotes all maps f from Rd to R such that |f(x)−f(y)| ≤ p(2R∗)p−1||x−
y|| for all x, y ∈ K. Note OPT (φ, x+µ,Dx,δ,p∩Dx,ξ) is equivalent to minν∈Dx,δ,p∩Dx,ξ E[φ(X)] according to the definition and therefore can be relaxed into optimization problem which is obviously convex according to 157 and 154.
L PROOF OF THEOREM 3.9
Proof. For 0 < p ≤ 1, the optimization over q can be solved using Lagrangian duality as follows: we dualize the constraints on q and obtain
L(λ) = inf ||q||1≤2ξ
(∫ φ(x)q(x)dx+ EX∼x+µ[φ(X)] + λ ( sup
||f ||L,p≤1
∫ f(x)q(x)dx− δ )) = EX∼x+µ[φ(X)] + inf
||q||1≤2ξ sup
||f ||L,p≤1
(∫ φ(x)q(x)dx+ λ ( sup ||f ||L≤1 ∫ f(x)q(x)dx− δ )) = EX∼x+µ[φ(X)] + inf
||q||1≤2ξ sup
||f ||L,p≤1
∫ ( φ(x) + f(x) ) q(x)dx− λδ
= EX∼x+µ[φ(X)] + sup ||f ||L,p≤1 inf ||q||1≤2ξ
∫ ( φ(x) + f(x) ) q(x)dx− λδ
= EX∼x+µ[φ(X)] + sup ||f ||L,p≤1 inf ||q||1≤2ξ
− ∫ ∣∣(φ(x) + f(x))q(x)∣∣dx− λδ (158)
= EX∼x+µ[φ(X)]− inf ||f ||L,p≤1 sup ||q||1≤2ξ ∫ ∣∣(φ(x) + f(x))q(x)∣∣dx− λδ = EX∼x+µ[φ(X)]− 2ξ inf
||f ||L,p≤1 ∣∣∣∣φ(x) + f(x)∣∣∣∣∞ − λδ (159) = EX∼x+µ[φ(X)]− 2ξ − λδ (160)
where 158 is due to the choice of q(x) such that sgn(q(x)) = sgn(φ(x) + f(x)); 159 is due to Holder inequality when q = 1, p =∞; 160 is due to the fact that inf ||f ||L≤1 ∣∣∣∣φ(x) + f(x)∣∣∣∣∞ = 1 since the range of φ(x) is {±1} in applications and f cannot change suddenly when crossing the decision region boundary of φ due to the Lipschitz constant constraint.
Similarly, for p > 1, we have
L(λ)
= inf ||q||1≤2ξ
(∫ φ(x)q(x)dx+ EX∼x+µ[φ(X)] + λ ( sup
||f ||L≤p(2R∗)p−1
∫ f(x)q(x)dx− ( δp + (p− 1)(2R∗)p−1 ))) = EX∼x+µ[φ(X)]
+ inf ||q||1≤2ξ sup ||f ||L≤p(2R∗)p−1
(∫ φ(x)q(x)dx+ λ ( sup
||f ||L≤p(2R∗)p−1
∫ f(x)q(x)dx− ( δp + (p− 1)(2R∗)p−1 ))) = EX∼x+µ[φ(X)] + inf
||q||1≤2ξ sup
||f ||L≤p(2R∗)p−1
∫ ( φ(x) + f(x) ) q(x)dx− λ ( δp + (p− 1)(2R∗)p−1 ) = EX∼x+µ[φ(X)] + sup
||f ||L≤p(2R∗)p−1 inf ||q||1≤2ξ
∫ ( φ(x) + f(x) ) q(x)dx− λ ( δp + (p− 1)(2R∗)p−1 ) = EX∼x+µ[φ(X)] + sup
||f ||L≤p(2R∗)p−1 inf ||q||1≤2ξ − ∫ ∣∣(φ(x) + f(x))q(x)∣∣dx− λ(δp + (p− 1)(2R∗)p−1)
= EX∼x+µ[φ(X)]− inf ||f ||L≤p(2R∗)p−1 sup ||q||1≤2ξ ∫ ∣∣(φ(x) + f(x))q(x)∣∣dx− λ(δp + (p− 1)(2R∗)p−1) = EX∼x+µ[φ(X)]− 2ξ inf
||f ||L≤p(2R∗)p−1 ∣∣∣∣φ(x) + f(x)∣∣∣∣∞ − λ(δp + (p− 1)(2R∗)p−1) = EX∼x+µ[φ(X)]− 2ξ − λ ( δp + (p− 1)(2R∗)p−1 )
M PROOF OF THEOREM 3.10
Recall the certificate by using Hockey-stick divergence provided in table 4 in Dvijotham et al. (2020) as below HS,β ≤ [β(θa − θb)− |β − 1|
2
] +
When β = 1, it follows that
HS,1 ≤ [θa − θb
2 ] +
Besides, recall the relaxation radius using Hockey-stick divergence as below HS,1 = G (
2σ
) −G ( −
2σ
) = 2G ( 2σ ) − 1
And plug it in above inequality, we have
HS,1 = 2G( 2σ )− 1 ≤ [θa − θb 2 ] +
And recall the definition of θa and θb, we have
EX∼x+µ[φ(X)] = θa − θb Thus, our certificate EX∼x+µ[φ(X)]− 2 ( 2G( 2σ )− 1 ) ≥ 0 is equivalent to
2G( 2σ )− 1 ≤ θa − θb 2
Thus, the equivalence relation holds.

Review summaries:
The authors provide robustness certification results for smoothed classifiers. Specifically, the considered setting follows that of Dvijotham (2020) except the base smoothing distribution is specified to have bounded support, and the relaxation follows from an intersection of Wasserstein and TV balls. Under this setting, much of the technical machinery describes which Wasserstein and TV balls are needed to cover these bounded-support smoothing distributions.