Sliced Wasserstein Estimation with Control Variates
Authors: Khai Nguyen
Nhat Ho
Venue: ICLR 2024 poster (2024)
Keywords: Sliced Wasserstein
Monte Carlo Methods
Point-Cloud
Generative Models
Optimal Transport
Primary area: general machine learning (i.e., none of the above)
TL;DR: We propose computational efficient control variates to reduce the variance and approximation error of estimating Sliced Wasserstein distance with Monte Carlo samples.

Abstract: The sliced Wasserstein (SW) distances between two probability measures are defined as the expectation of the Wasserstein distance between two one-dimensional projections of the two measures. The randomness comes from a projecting direction that is used to project the two input measures to one dimension. Due to the intractability of the expectation, Monte Carlo integration is performed to estimate the value of the SW distance. Despite having various variants, there has been no prior work that improves the Monte Carlo estimation scheme for the SW distance in terms of controlling its variance. To bridge the literature on variance reduction and the literature on the SW distance, we propose computationally efficient control variates to reduce the variance of the empirical estimation of the SW distance. The key idea is to first find Gaussian approximations of projected one-dimensional measures, then we utilize the closed-form of the Wasserstein-2 distance between two Gaussian distributions to design the control variates. In particular, we propose using a lower bound and an upper bound of the Wasserstein-2 distance between two fitted Gaussians as two computationally efficient control variates. We empirically show that the proposed control variate estimators can help to reduce the variance considerably when comparing measures over images and point-clouds. Finally, we demonstrate the favorable performance of the proposed control variate estimators in gradient flows to interpolate between two point-clouds and in deep generative modeling on standard image datasets, such as CIFAR10 and CelebA.

Published as a conference paper at ICLR 2024
SLICED WASSERSTEIN ESTIMATION WITH CONTROL VARIATES
Khai Nguyen & Nhat Ho Department of Statistics and Data Sciences
The University of Texas Austin Austin, TX 78712, USA (khainb, m1 nhnhat) @utexas. edu
ABSTRACT
The sliced Wasserstein (SW) distances between two probability measures are defined as the expectation of the Wasserstein distance between two one-dimensional projections of the two measures. The randomness comes from a projecting direction that is used to project the two input measures to one dimension. Due to the intractability of the expectation, Monte Carlo integration is performed to estimate the value of the SW distance. Despite having various variants, there has been no prior work that improves the Monte Carlo estimation scheme for the SW distance in terms of controlling its variance. To bridge the literature variance reduction and the literature on the SW distance, we propose computationally efficient control variates to reduce the variance of the empirical estimation of the SW distance. The key idea is to first find Gaussian approximations of projected -inddimensional measures, then we utilize the closed-form of the Wasserstein-22 distance between two Gaussian distributions to design the control variates. In particular, we propose using lower bound and an upper bound of the Wasserstein-2 distance between fitted Gaussians as two computationally efficient control variates. We empirically show that the proposed control variate estimators can help reduce the variance considerably when comparing measures over images and point-clouds. Finally, we demonstrate the favorable performance of the proposed control variate estimators in gradient flows to interpolate hetween two point-clouds and in deep generative modeling on standard image datasets, such as CIFAR 10 and CelebA
INTRODUCTION
Recent machine learning applications have recognized the Wasserstein distance (Villani 2008b
Peyré & Cuturi 2019), a universal effective tool. In particular, there are various applications that achieve notable performance by using the Wasserstein distance as a component, for example. (hierarchical) clustering (Ho et al. 2017). domain adaptation (Courty et 2016 Damodaran et al., 2018), generative modeling (Arjovsky et al., 2017)] Tolstikhin et al.) 2018). and so on. Despite its appealing performance, the Wasserstein distance has two major weaknesses. The first weakness is that it has high computational complexities. As discussed in (Peyré & Cuturi, (019). the time complexity of computing the Wasserstein distance between two discrete measures that have at most n supports is (n log r1). Additionally, the space complexity required for the pair-wis, transportation cost matrix is (n²)) As a result, computing the Wasserstein distance on a large-scale discrete distribution is expensive. The second weakness is that the Wasserstein distance suffers from the curse of dimensionality. More specifically, the sample complexity of the Wasserstein distance is On Therefore, using the Wasserstein distance in high-dimmnssonal statistical inference may not be stable.
The Wasserstein distance has famous alternative called the sliced Wasserstein (SW) distance, which is derived from the one-dimensional Wasserstein distance as its base metric. The one-dimensional Wasserstein distance has closed-form solution, which can be computed in O(n log n) time complex- ity and (n) space complexity in the discrete case. Here, "supports" refers to the discrete probability measures being compared. To apply this closed-form solution in high-dimension setting, random
Code for the paper is published at /t/p:t/pss//ggttt..ccc///

---

Published as a conference paper at ICLR 2024
projections are employed, transforming two measures into infinite pairs of one-dimensional measures using a projecting function with infinite projecting directions that belong to the unit unit-hyperspheen The closed-form of the one-dimensional Wasserstein distance is then applied to pairs of one-dimensional measures to obtain projections, and the SW distance is computed by aggregating the values of projections. The most commonly used method of aggregation is averaging. resulting in expectation of the projected one-dimensional Wasserstein distance with respect to the uniform distribution over projecting directions. Since the space of projecting directions is the hiiisshpersppere, the SW distance does not suffer from the curse of dimensionality, and its sample complexity is O(n Due its scalability, the SW distance has been successfully applied in various applications such as point-cloud reconstruction (Nguyen al., 2023, Savkin al.] (2022), generative models (Deshpande et al. 2018 domain adaptation (Lee et al. [2019]. 1.199........ shape matching (Le et al. 2024b), clustering (Kolouri et al. 2018 variational inference (Yi & Liu 202T). and many other tasks.
The SW distance is typically estimated using the Monte Carlo integration, as the intractable expec- tation requires approximation in practice. The number of Monte Carlo samples drawn from the unit-form distribution over the unit-hypersphere is referred to as the number of projections. Increasing the number of projections improves the accuracy and stability of the estimation. However, to the best of our knowledge, there is no prior work on improving the Monte Carlo approximation scheme of the SW distance, especially in terms of variance reduction. In this work, we propose a novel approach that combines the SW distance literature with the variance reduction literature by introducing the very first control variates, which are a technique for reducing the variance of Monte Carlo estimates.
Contribution. In summary, our contributions are two-fold: 1. To address the issue of variance in estimating the SW distance using Monte Carlo samples, we propose a novel approach based on control variates. Specifically, we first identify two Gaussian distributions that closely approximate the two projected one-dimensional measures by minimizing Kullback-Leibler divergence. Next, we construct control variates using the closed-form solution of the Wasserstein distance between two Gaussian distributions. We propose two control variates, an upper bound and lower bound, for the Wasserstein distance between the fitted Gaussian distributions. By using these control variates, we show that the computation is only linear in terms of the number of supports and the number of dimensions, even when dealing with discrete probability measures. This means that the proposed control variates estimators have the same computational complexity as the conventional estimator of the SW distance. Overall, our proposed approach provides a practical and efficient method for estimating the SW distance with reduced variance.
2. We empirically show that the proposed control variate estimators yield considerably smaller variances than the conventional computational estimator of SW distance, using finite number of projections, when comparing empirical distributions over images and point clouds. Moreover, we illustrate that the computation for control variates is negligible compared to the computation of the one-dimensional Wasserstein distance. Finally, we further demonstrate the favorable performance of the control variate approach in gradient flows between two point clouds, and in learning deep generative models on the CIFAR10 10 (Krizhevsky et al. 2009) and CelebA (Liu et al., 20155) datasets. From our experiments, we observe that using the proposed control variate estimators considerably improves performance, while their computation time is approximately the same as that of the conventional estimator.
Organization. The remainder of the paper is organized as follows. First, we review the background on the Wasserstein distance, including its special cases, such as the one-dimensional case and the Gaussian case, as well as the sliced Wasserstein distance and the Monte Carlo estimation scheme, in Section 2 Next, we discuss the construction of control variates and their resulting estimator of the SW distance in Section 3 We then present experimental results in Section [ to demonstrate the benefits of the proposed control variate estimators. Finally, we conclude in Scction 5 and provide proof's of key results, related works, and additional materials in the Appendices.
Notations. For any d Z 2, we denote Sd-1 := {0 E Rd ||0| = 1} as the unit hyper-sphere and U($ as its corresponding uniform distribution. We denote P(S the set of all probability measures on gd-1 For p 2 1. Pp (R²) is the set of all probability measures on Id that have finite p-moments. For any two sequences a, and b,, the notation a. Obb,,) means that a, < Cb, for all n 2 1, where C is some universal constant. We denote 0łμ as the push-forward measure of through the function ʃ Rdd - RR where f(x) 0 J.
0T.. We denote Px as the empirical measure
1 Eis dz,, where X == (I1, Im) ) E idd is a vector. We denote Tr(A) (A) as the trace operator
2

---

Published as a conference paper at ICLR 2024
of the matrix A and A is the square root of matrix A... A = B such that BB = A. Given probability measure H4 we denote Fp. is the cumulative distribution function (CDF) of μ.
BACKGROUND
We now review some essential materials for the paper.
Wasserstein distance. Given p 2 1, two probability measures μ E (RR(R)) and v E (
the Wasserstein distance (Villani 2008a Peyré & Cuturi 2019) between μ and v is Wg(,,v) inf rnf())) x || y plldd (z. y), where H(,,, v) is set of all couplings that have marginals and respectively. The computational complexity and memory complexity of Wasserstein distance are O(n log n) and Onn²) in turn when μ and y/ have at most n supports. However, there are some special cases where the Wasserstein distance can be computed efficiently.
Special Cases. When = 2, we have Gaussian distributions y. :- Σ1) and v == (m2,, E2), the Wasserstein distance between μ and y has the following closed-form:
W P) ||m m2l + Tr(E; + Σg - -( -/2//3///
When d = the Wasserstein distance between μ E (R)) and E (R) can also be computed
with a closed form: Wp(u, = fo (z) F (=) When μ and are one-dimension discrete probability measures that have a most n supports, the computational complexity and memory complexity, in this case, are only ((n log n) and ((n).
Sliced Wasserstein distance. Using random projections, the sliced Wasserstein (SW) distance can exploit the closed-form benefit of Wasserstein distance in one dimension. The definition of sliced Wasserstein distance (Bonneel et al. 2015) between two probability measures μ E Pp((R.) and V/ E (R is:
SWP v/) = Es-z(s4 1) p (Wp( 0dv)],
(2)
where (S4-1) is the uniform distribution over the unit-hyper sphere.
Monte Carlo estimation. The expectation in the SW is often intractable, hence, Monte Carlo samples are often used to estimate the expectation:
SW
SW (M, v;
I Σ (1tt).
where projecting directions 01,-.. Or. are drawn i.i.d from (((5-1). When μ and are empirical measures that have at most n supports in d dimension, the time complexity of SW is O(Ln log n Ldn). Here, Ln log for sorting L sets of projected supports and Ldn is for projecting supports to L sets of scalars. Similarly, the space complexity for storing the projecting directions and the projected supports of SW O(Ld + Ln). We refer to Algorithm I in Appendix for the detailed algorithm for the SW.
Variance and Error. By using some simple transformations, we can derive the variance of the Monte (H, Carlo approximation of the SW distance as: Var Sar (H, vr L) + (Ottu, 0tv)]. We also have the error of the Monte Carlo approximation (Nadjahi et al. 2020b is:
SW (SW (U, v; (w,
Var [WF (()), 1/2
(4)
VL
Here, can see that Var Var(WP (W 04v) plays an important role in controlling the approximation error. Therefore, a natural question arises: "Can we construct a function Z(0; μu, v) such that Z (6; μu, v)] = SWP,4, v) while VarlZ (0: n1, < Var Var/Wz (6ttu 0tv)?". If we design such Z(0; μz, v), will have || Et (0 - SW v) < + Varl u. 12 < + VarlWW
VL
which implies that the estimator + Σ 700 (1, is better than SW (1, T.).
3 CONTROL VARIATE SLICED WASSERSTEIN ESTIMATORS
We first adapt notations from the control variates literature to the SW case in Section 3.1 After that, discuss how construct control variates and their computational properties in Section 3.2
3

---

Published as a conference paper at ICLR 2024
3.1 CONTROL VARIATE FOR SLICED WASSERSTEIN DISTANCE
Short Notations. We are approximating the SW which is an expectation with respect to the random variable 0. For convenience, let denote (pt0 05w) as W 80;0; pa, v), we have:
and SW (W L)
Σ w(On),
(5)
L
where 0I....,. Og Lid (0 = 1((
Control Variate. A control variate (Owen 2013 is a random variable C(O) such that its expectation tractable i.e., [[C(()) B. From the control variate, we consider the following variable:
W(0 p. v/) - ((C()) B)
(6)
where y E R. Therefore, it is easy to check that u. ((C( -B) (W v) since [[C B. Therefore, the Monte Carlo estimation of (0: (((()
=
B)]. i... 1. E (W(k:: p, (C(())) B)), is unbiased estimation of (8:
Now, we consider the variance of the variable (0: μ. 1) (C(()) B). VarlW (1, (C(()) B)] (e: - 2.CCov[W(C v), (
The above variance attains its minimum, with respect to y, for 7- (0..)) Var [()
Using the
optimal 7", we the minimum variance of is:
Var Va:[W(O: v) (1- (o: C(0))
VariW 14, Var[C(0)]
Therefore, the variance of (O; μe, v) (C(() - B) with is lower than W W00; (4, if the control variate C(0) has a correlation with W (0;
Definition 1. Given control variate C(0), the corresponding controlled projected one-dimensional Wasserstein distance is:
p(, (0: CovlW Var[C((0]]
(C(#) B).
(8)
In practice, computing 7" might be intractable, hence, we can estimate 7 using Monte Carlo samples.
Control Variate Estimator of the SW distance. Now, we can form the new estimation of SW v). Definition 2. Given a control variate C(0) with [[C(()) = B, the number of projections L > 1, the Control Variate Sliced Wasserstein estimator is:
CV-SW (H, L, SW,,,, + LT. Σ (C(0)) B),
where 0, (0). SW (W,Lm,; Et W(O:u, and the estimated optimal coefficient 7t (::1( with CovlW (0; C),0(;;; (W
[ SW SW,,H, (U,,; ()) B), = (. -)
Remark In Definition 2 the optimal coefficient Y is estimated by reusing Monte Carlo samples 01...., 0g. It is possible to estimate 7 using new samples f'....,6',, to make it independent to SW SW,lu, v; L). Nevertheless, this resampling approach costs doubling computational complexity. Therefore, the estimation in Definition iis preferable in practice (Owen) (2013). It is worth noting that the estimation is only asymptotic unbiased, however, it is negligible since the unbiasedness is always lost after taking the p-rool for computing the value of SW with finite L.
In this section, we have not yet specified C(B). the next session, we will focus on constructing a computationally efficient C()) that is also effective by conditioning it on two probability measures and v. Specifically, we define C(A) == C(A: p.v), v), which involves conditioning on these two probability measures y and
4

---

Published as a conference paper at ICLR 2024
3.2 CONSTRUCTING CONTROL VARIATES
We recall that we are interested in the random variable W (0: (0; W 02v). It is desirable to have a control variate (()) such that C(Ø) R ( p. v) (Owen 2013). Therefore, is natural to also construct C(0)= = C(0; v1, based on two measures 14, V. Moreover, since we know the closed-form solution of Wasserstein distance between two Gaussians, control variates in the form C(O; v) W2 (0; p)), (0:: (m)2(0; o2 v))) could be tractable.
Gaussian Approximation. The question of constructing the control variate i.e., specifying ((m (0: (.), of (8; p)) and (mm2(0:2). (f: v)) now requires finding the Gaussian approximation of two projected measures fig and 0tv. When two measures are discrete, namely, μ = Σi= a,os.
Ei 02 = 1) and v= = >i= B:0g.
2
i== 8 = =1), we perform the following optimization:
(0) (0: = argmax miia?
a; ag
1
exp
1 201
(0T 22 - -m1)²
(10)
m=(0:)), = argmaxma.of
3r Bog
exp
(8T yE - m2)²
(11)
2TO
20
Proposition 1. Let j and p be two discrete probability measures i.e., μ Σi= azoz. El=100 04 1) and v = En Bebyy Ei=1 33 1), we have: m m1(0;1)
Em11: To (0 = EE= az (0 xi - mm (0; m2 = >> 3,0 yo o² =
E 3, (0 yi - v)) are solution of the problems in equation 10 11
It is worth noting that the closed-form in Proposition] I can also be seen as the solution from using moment matching. When and are continuous, we can solve the optimization by using stochastic gradient descent by sampling from μ and v respectively. In addition, we can use corresponding empirical measures of 1 and V i.e., Un = -i=1 ōx, and Vn. + E dy; as proxies (where XI,..., X,, i,dd μ and v). Beyond optimization, we can also utilize the Laplace approximation to obtain
We refer to Appendix C.1 for the detailed proof of Proposition |
Yi...., Y, itd two approximated Gaussians for 0īn and 05v when the push-forward densities are tractable. We refer the reader to Appendix [ for more detail.
Constructing Control Variates. From the closed-form of the Wasserstein-2 distance between two Gaussians in equation
we have: (m of ()). v), a (0:
(m,(0; (0: p) m²(0; v))
20, 20, (0:1002) (000:)))2000000)0)))))))0000)( (0: [1 as the requirement of control vari- ate, we could calculate the expectation of each term i.e., [[(mm((::] - m2 v)) (0]
m0) + o)
To calculate
E/Wa(N (0; p), of v)).
E [o]( and E(o, (6; (0)
Proposition 2. Given two discrete measures μ and v, mı p), m²(0; v). of n)), and of: p) as
in Proposition 7 we obtain:
[m (0: p)) (0:(0
Σ
a
(12)
d
2
E [o7(0:14)
= = Σ 04
-
d aml
1
E [[0(0011) = d cml
m
8
y/s
m
B
(13)
The proof of Proposition 2 given in Appendix
Unfortunately, we cannot compute the expectation for the last term Ela, (00:)))0(:)0)))))))))))))))))) (0: However, we could construct control variates using lower-bound and upper-bound of W2(N (0; 10), Definition 3. Given two Gaussian approximations of 8üp and 0.v i.e. ((m and N (0: v), o v)), we define the following two control variates:
v), o2
Clow (0: pl, v) (m1(0: 14) m²(0:)))
(14)
Cap (0; (m² (0: (4) m2)(0 + (0:0p) v)
(15)
5

---

Published as a conference paper at ICLR 2024
We now discuss some properties of the proposed control variates. Proposition 3. We have the following relationship:
Clow (f: p.v) & (2 μ.) (1(()::))))NNm((..))))).
I (.. (16)
The proof of Proposition 3 follows directly from the construction of the control variates. For completeness, we provide the proof in Appendix
Proposition 4. Let μ = =i== addz, and v > B,dy, using the control variate Crow (0; p1, v) is equivalent to using the following control variate:
(m) oz())))
(17)
el mm) of ((2--ma)
where (f argmax mi.of Ei=1 ar log
(12x)40071)
and (2( argmax ma..a m2.03 E B, log
10 mm) | ((: mz))
(223)11031)
The proof for The Proposition is given in Appendix C.4 The proposition means that using the control variate Clow (0; μu, v) is equivalent to using the Wasserstein-2 distance between two projected location-scale multivariate Gaussians with these location-scale multivariate Gaussians are the approximation of the two original probability measures μ and v on the original space.
Control Variate Estimators of the SW distance. From two defined control variates in Definition 3 we define the corresponding controlled projected one-dimensional Wasserstein distances and control variate sliced Wasserstein estimators.
Definition 4. Given two control variates Crow (0: v),, Cup ing controlled projected mne-dimensional Wasserstein distances are defined as:
(0; v.v)) in Definition 3 the correspond-
Zoow (0; v) [6;; μa. V, Clow (f; p. p)), 2up(0:puv) = Z(0: Cap(O:p.)),
(18)
where 7 70; pu U, C())) is defined in Definition 1
Definition 5. Given Zrow (0, p, v),,, Zup(O, p. v), CV-SW (n, C(O) as defined in Definition
and Definition 4 the lowerbound and upperbound control variate sliced Wasserstein estimators are:
I
LCV-SW,, (1, vs
Σ
Zrow (,,)) = CV-SW (H, L, (1o.
(19)
L
i=1
UCV-SW,
1 Σ
Zup
(20)
We refer to Algorithm 2 for the detailed algorithms for the control variate estimators in Appendix D
Computational Complexities. When dealing with two discrete probability measures and that have at most n supports, projecting the supports of the two measures to sets of projected supports costs ((Ldn) in time complexity. After that, fitting one-dimensional Gaussians also costs (Ldn). From Proposition] 2 and Definition 3 computing the control variate Clow (f:; jr, v) and its expected value Brow (p, v) costs Odd)). Similarly, computing the control variate Cup (6; μu, v) and its expected value Bup also costs (dn). Overall, the time complexity of LCV-SW" p (14, L) and UCV-SW" (u, v; L) is the same the conventional SW, which is O log n Ldn). Similarly, their space complexities are also ( + Ln) as the conventional estimator.
Gaussian Approximation of the sliced Wasserstein distance. recent work Nadjahi al. 2021), the closed-form of Wasserstein between Gaussian distributions is utilized to approximate the value of the SW distance. The key idea based on the concentration random projections in high-dimension, i.e., they are approximately Gaussian. However, the resulting deterministic approximation not very accurate since is only based on the first two moments of two original measures. As result, the proposed approximation can only work well when two measures have weak dimensional dependence. In contrast, we use the closed-fone form Wasserstein between Gaussian distributions design control variates, which can reduce the variance of the Monte Carlo estimator.

---

Published as a conference paper at ICLR 2024
MUST
The empirical errors of the conventional estimator (SW) and the control variate estimators (LCV-SW, UCV-SW) when comparing empirical distributions over MNIST images and point-clouds.
Figure 1:
Table 1: Summary of Wasserstein-2 2 (Flamary et al. 2021 (multiplied by 10"). computational time second
(s) to reach step 8000 of different sliced Wasserstein estimators in gradient flows from three different runs.
Distances
Step 3000 (W,
Step 4000 (Ws
Step 5000 (W: 1)
Step 6000CWs D)
Step 8000 (W,
Time (+.)
SW L-10
305. 5196 4921
157.
0.3611
25,2454 0.3387
0.1264
000222
1.le
26.13 0.04
ICVVSWI-
1001 1787
136.0129
35,0229
0.1450
0.0538
+-0.047
1.5e 0.9c-
27.02 +.2.1
UCV-SW
0.1783
136.0139
0.0021
35.0916
0.1444
0.0535 0.0065
1.cc
SW TW1-0000
300 7127336--- 0.2375
1408 0.3146
33.0253
0.1349
0.0183 + 0.001
14e- 5+ 0.26- 5
220
LCV-SW 1-110
300 3924
0. 0.0068
133-6243
+ 0.0066
33.5100
+.00001
0.0184 = 8.7e
1.4e - 0.3e
221.76 0:35
UCV-SW L=100
300.3927 1: 0.0050
153.62 035.62-12-2 : 0.0065
33 53.50888.. 0.0022
0.0136 1: 0.0000
1.ee 5:2 0.2e-5
233.71 1.006
Our estimators are still stochastic, converge to the true value of the SW when increasing the number of projections, and work well with probability measures that have arbitrary structures and dimensions.
Beyond uniform slicing distribution and related works. In the SW distance, the projecting direction follows the uniform distribution over the unit-hypersphere oo (0) = 2/( In some cases, changing the slicing distribution might benefit downstream applications. For example, the distributional sliced Wasserstein distance (guuyen et al. 2021) can be written as DSWP(u, 2) Ee-r(0) (W7(0pp, (Hip, where a is a distribution over the unit hypersphere. We can directly apply the proposed control variates to the DSW as long as we can calculate Ee~ [00 J. If this is not the case, we can still use the proposed control variates by approximating the DSW via importance sampling. In particular, o(0) then use control variates. We discuss we can rewrite DSWP (u, v) = (e) Wp 0īv) [o("]]
other related works including how we can apply the control variates to other variants of the SW distance in Appendix E
Gradient estimators. In some cases, we might be interested in obtaining the gradient VoSWE va) = V. 0fr,)] e.g., statistical inference. Since the expectation in the SW does not depend on d, we can use the Leibniz rule to exchange differentiation and expecta- tion, namely, V
V.E.WT (O:p,, 0tvs)] E WP (e 0tra)]. After that, we can use Va p. 1%) (VoCrow (0; μt 1&) or Cu (0; (..,.vo)) as the control variate. However, estimating the optimal Cov(V se Wg105m.04us) V., C(Ba.))))))) (for i .,...,d d' with d' is the number of dimensions of Var(V,, s, C(O3(.P6)]
parameters Φ)) is computationally expensive and depends significantly on the specific settings of va- Therefore, we utilize a more computationally efficient estimator of the gradient i.e., E a Z(0; μr, vo)] for (0; μe, v&) is Zlow (0; μr, v&) or zup (0: μt, v&) in Definition! 4 After that, Monte Carlo samples are used to approximate expectations to obtain a stochastic gradient.
4 EXPERIMENTS
Our experiments aim to show that using control variates can benefit applications of the sliced Wasserstein distance. In particular, show that the proposed control variate estimators have lower variance in practice when comparing probability measures over images and point-clouds in Section [.]] After that, we show that using the proposed control variate estimators can help to drive gradient flow to converge faster to a target probability measure from a source probability measure while retaining approximately the same computation Section 4.2 Finally, we show the proposed control variate estimators can also improve training deep generative models on standard benchmark images dataset such as CIFAR10, and CelebA in Section 4.3 Due to the space constraint, additional experiments and detailed experimental settings are deferred to Appendix F
4.1 COMPARING EMPIRICAL PROBABILITY MEASURES OVER IMAGES AND POINT-CLOUDS
Settings. We aim to first test how well the proposed control variates can reduce the variance in practice. To do this we conventional Monte Carlo estimation for the SW3 with very
7

---

Published as a conference paper at ICLR 2024
Figure 2:
Point-cloud gradient flows for L 10 from SW, LCV-SW and UCV-SW respectively,
large L value of 100000 and treat it as the population value. Next, we vary in the set {2, 5, 10, 50, 100, 500, 1000, 5000, 7000, ] and compute the corresponding estimates of the SW with both the conventional estimator and control variate estimators. Finally, we calculate the absolute difference between the estimators and the estimated population as the estimated error. We apply this process to compare the empirical distributions over images of digit 0 with the empirical distribution images of digit in MNIST (LeCun et al., 1998) (784 dimensions, with about 6000 supports). Similarly, we compare two empirical distributions over two randomly chosen point-clouds in the ShapeNet Core-55 dataset (Chang et al. 2015) (3 dimensions, 2048 supports). We also estimate the variance of the estimators via Monte Carlo samples with = 100000 Table S in Appendix F.T
Results. We show the estimated errors and the computational time which are averaged from 5 different runs in Figure From the figure, we observe that control variate estimators reduce considerably the error in both cases. On MNIST, the lower bound control variate estimator (LCV-SW) gives lower error than the upper bound control variate estimator (UCV-SW) while being slightly faster. The computational time of the LCV-SW has nearly identical computational time the conventional estimator (SW). In the lower dimension i.c., the point-cloud case, two control variates give the same quality of reducing error and the same computational time. In this case, the computational time of both two control variates estimators are the same as the conventional estimator. From Table in Appendix [.II] we observe that the variance of the control variate estimators is lower than the conventional estimator considerably, especially the LCV-SW. We refer to Appendix F. experiments on different pairs of digits and point-clouds which show the same phenomenon.
F.T for
4.2 POINT CLOUD GRADIENT FLOWS
Settings. We model distribution j(t) flowing with time t along the sliced Wasserstein gradient flow (t) + SW (u(t), v), which drives it towards a target distribution v (Santambrogio 2015). Here, we set 1/ Σi= dy, as a fixed empirical target distribution and the model distribution (ut) = E.=1 8x,(1). where the time-varying point cloud X(!) = (X:())) (IR&) ". Sttrting at time t= O. we integrate the ordinary differential equation X(1) - x( [SW, (1 E. 8x,0). v)] for each iteration. We choose (0) and are two random point-cloud shapes in ShapeNet Core-55 dataset (Chang et al. 2015) which were used in Section
SWp(u(p
After that, we use different estimators the replacement of the for SW estimate the
of the SW i.e., SW LCV-SW and UCV-SW, gradient including the proposed control variate estimators and the conventional estimator. Finally, we use p and the Euler scheme with 8000 iterations and step size 0.01 solve the flows.
Results. We use the Wasserstein-2 distance as a neutral metric to evaluate how close the model distribution sa(() is the target distribution v. We show the results for the conventional Monte Carlo estimator (denoted as SW), the control variate Clow estimator (denoted as LCV-SW), and the control variate Cap estimator (denoted as UCV-SW), with the number of projections L = 10, 100 in Table [ In the table, we also report the time in seconds that estimators need to reach the last step. In addition, we visually the flows for L = 100 in Figure 2 Moreover, we observe the same phenomenon on different pair of point-clouds in Table 4 and in Figure 6 in Appendix F22
4.3 DEEP GENERATIVE MODELING
Settings, We conduct deep generative modeling on CIFARIO (with image size 32x32) (Krizhevsky et al. 2009), non-croppeed CelebA (with image size 64x64) (Liu al. 2015) with the SNGAN
8

---

Published as a conference paper at ICLR 2024
SW L=10
LCV-SW L=10
UCV-SW L=10
Figure 3: Random generated images of distances on CIFARIO and CclcbA
Table 2: Summary of FID and IS scores from three different runs on CIFARIO (32x32), and CelebA (64x64).
Method
(3233))()2332)
CelebA (61x61)
Method
CIFARIO (3x32))
ColebA (61x64)
FD
FI)
SW1--10 LCV-SW L-10
21.72 0:37
7.50
11:14 1.12
SW
10.05
7.74
0.04
10 0.73
L-1-
0.12
8:20
0.02
9.78
UCV-SW L=10
21.71
0.36
7.57
0.01
11.00 10000:0000
LCV-
13:21 0:49
8.21 0.00
9.51
0.30
architecture (Miyato et al. 2018).
For the framework, we follow the sliced Wasserstein genera-
tor (Deshpande et al.j 2018 Nguyen & Ho 2023) which is described in detail in Appendix FF3
The main evaluation metrics are FID score (Heusel al. 2017) and Inception score (IS) (Salimans et al. (016)) On CelebA, we do not report the IS since it poorly captures the perceptual quality of
face images (Heusel et al. 2017). The detailed settings about architectures, apper-paramttrs.. and evaluation of FID and IS are provided in Appendix F3
Results. We train generators with the conventional estimator (SW), and the proposed control variates estimators (LCV-SW and UCV-SW) with the number of projections L = 10, 1000. We report FID scores and IS scores Table From the table, we observe that the LCV-SW gives the best generative models on both CIFAR 10 and CelebA when L, = 10. In particular, the LCV-SW can reduce about 14% FID score on CIFAR1O, and about 10% FID score on CelebA. Also, the LCV-SW increases the IS score by about 2.6%. For the UCV-SW, it enhance the performance slightly compared with the SW when L = 10. is worth noting that, the computational times of estimators are approximately the same since the main computation in the application is for training neural networks. Therefore, we can see the benefit of reducing the variance by using control variates here. Increasing L to 1000, despite the gap being closer, the control variate estimators still give better scores than the conventional estimator. In greater detail, the LCV-SW can improve about 3.5% FID score on CIFAR 10, about 2% IS score on CIFARIO. and about 2.7% FID score on CelebA compared to the SW. In this setting of L, the UCV-SW gives the best FID scores on both CIFARIO and CelebA. Concretely, it helps to decrease the FID score by about 6.1% on CIFARIO and about 5.4% on CelebA. From the result, the UCV-SW seems to need more projections than the LCV-SW to approximate well the optimal control variate coefficient (+*) since the corresponding control variate of the UCV-SW has two more random terms. For the qualitative result, we show randomly generated images from trained models in Figure 3 for L = 10 and in Figure 7 for L = 1000 in Appendix F3 Overall, we obverse that generated images are visually consistent to the FID scores and IS scores in Table 2
CONCLUSION
We have presented a method for reducing the variance of Monte Carlo estimation of the sliced Wasser- stein (SW) distance using control variates. Moreover, we propose two novel control variates that serve as lower and upper bounds for the Wasserstein distance between two Gaussian approximations of two measures. By using the closed-form solution of the Wasserstein distance between two Gaussians and the closed-form of Gaussian fitting of discrete measures via Kullback Leibler divergence, we demonstrate that the proposed control variates can be computed linear time. Consequently, the control variate estimators have the same computational complexity as the conventional estimator of SW distance. On the experimental side, we demonstrate that the proposed control variate estimators have smaller variances than the conventional estimator when comparing probability measures over images and point clouds. Finally, we show that using the proposed control variate estimators can lead to improved performance of point-cloud SW gradient flows and generative models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to thank Peter Mueller for his insightful discussion during the course of this project. NH acknowledges support from the NSF IFML 2019844 and the NSF AT Institute for Foundations of Machine Learning.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223, 2017.
Clément Bonet, Paul Berg, Nicolas Courty, François Septier, Lucas Drumetz, and Minh Tan Pham. Spherical sliced-wasserstein. In The Eleventh International Conference on Learning Representa- tions, 2023.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Plister. Sliced and Radon Wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 1(51):22-45, 2015.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, al. Shapenet: An information-rich 3d 2015.
model repository. arXiv preprint
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 1959)118331185, 2016.
Bharath Bhushan Damodaran, Benjamin Kellenberger, Rémi Flamary, Devis Tuia, and Nicolas Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 447,463,7 2018.
Ishan Deshpande, Ziyu Zhang, and Alexander Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3483-3491, 2018.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http:
wwto http://apes//v22. 20- 451 html
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626 06636637,44 2017.
Nhat Ho, XuanLong Nguyen, Mikhail Yurochkin, Hung Hai Bui, Viet Huynh, and Dinh Phung. Multilevel clustering via Wasserstein means. In International Conference on Machine Learning, pp. 1501-1509, 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412,694 2014.
Soheil Kolouri, Gustavo K. Rohde, and Heiko Hoffmann. Sliced Wasserstein distance for learning Gaussian mixture models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3427-3436, 2018.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced Wasserstein distances. In Advances in Neural Information Processing Systems, pp. 261-272 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
10

---

Published as a conference paper at ICLR 2024
Thanh Tung Le, Khai Nguyen, shanlin sun, Kun Han, Nhat Ho, and Xiaohui Xie. Diffeomorphic mesh deformation via efficient optimal transport for cortical surface reconstruction. The Tweifth International Conference Learning Representations, 2024a.
Tung Le. Khai Nguyen, Shanlin Sun, Nhat Ho, and Xiaohui Xie. Integrating efficient optimal transport and functional maps for unsupervised shape correspondence learning Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024b.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2) 1998.
Chen-Ya Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht. Sliced Wasserstein discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10285-10295, 2019.
Tao Li, Cheng Meng, Jun Yu, and Hongteng Xu. Hilbert curve projection distance for distribution comparison, arXiv preprint arXiv:2205. 200i::22005/50555 /5059, 2022.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
Kimia Nadjahi, Valentin De Bortoli, Alain Durmus, Roland Badeau, and Umut Şimşekli. Approxi- mate Bayesian computation with the sliced- Sord--aar Wassersten distance. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5470-5474. IEEE, 2020a.
Kimia Nadjahi, Alain Durmus, Lénaic Chizat, Soheil Kolouri, Shahin Shahrampour, and Umut Simsekli. Statistical and topological properties of sliced probability divergences. Advances in Neural Information Processing Systems, 33:20802-208 2020b.
Kimia Nadjahi, Alain Durmus, Pierre E Jacob, Roland Badeau, and Umut Simsekli. Fast approxima- distance using concentration of random projections. Advances in
tion of the Neural Information Processing Systems, 34, 2021.
Khai Nguyen and Nhat Ho. Energy-based sliced Wasserstein distance. Advances in Neural Informa- fion Processing Systems, 2023.
Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional Wrreed-Wssersteiin and applications to generative modeling. In International Conference Learning Representations, 2021.
Khai Nguyen, Dang Nguyen, and Nhat Ho. Self-attention amortized distributional projection opti- mization for sliced wasserstein point-cloud reconstruction. Proceedings of the 40th International Conference Machine Learning, 2023.
Khai Nguyen, Nicola Bariletto, and Nhat Ho. Quasi-monte carlo for 3d sliced Wasserstein. In The Twelfth International Conference on Learning Representations, 2024a.
Khai Nguyen, Shujian Zhang, Tam Le, and Nhat Ho. Sliced Wasserstein with random- path projecting directions. arXiv preprint arXiv:2401. 2024b.
Art Owen. Monte carlo theory, methods and examples. 2013.
Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-6)3555-677, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Chen. Improved techniques for training GANs. Advances in Newral Information Processing Systems, 29, 2016.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94, 2015.
11

---

Published as a conference paper at ICLR 2024
Artem Savkin, Yida Wang, Sebastian Wirkert, Nassir Navab, and Federico Tombari. Lidar upsampling with sliced wasserstein distance. IEEE Robotics and Automation Letters, 8(1):392-399 2022.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto- encoders. In International Conference Learning Representations, 2018.
Cédric Villani. Optimal transport: Old and New. Springer, 2008a.
Cédric Villani. Optimal transport: old and new, volume 338. Springer Science & Business Media, 2008b.
Jie Wang, Rui Gao, and Yao Xie. Two-sample test with kernel projected wasserstein distance. In International Conference on Artificial Intelligence and Statistics, pp. 8022-8055. PMLR, 2022.
Mingxuan Yi and Song Liu. Sliced Wasserstein variational inference. In Fourth Symposium on Advances in Approximate Bayesian Inference, 2021.
12

---

Published as a conference paper at ICLR 2024
Supplement to "Sliced Wasserstein Estimation with Control
Variates"
We first provide the skipped proofs in Appendix C Additionally, present the detailed algorithms for the conventional estimator of the sliced Wasserstein distance and the control variate estimators in Appendix D Sliced Wasserstein variants how to use control variates in those variants in the same appendix. In Appendix F we provide additional experiments and their detailed experimental settings. Finally, report the computational infrastructure in Appendix
discussed in Appendix E and also discuss
A DISCUSSION
Laplace Approximation. We are interested in approximating a continuous distribution μ by an Gaussian A Mm,, o²). We assume that we know the pdf of which is doubly differentiable and referred to as fxx We first find m such that f'(m) = After, we use the second-order Taylor expansion of log (3) around m:
log (xx Re log [(m)
2n (ar m)?
where the first order does not appear since (mmm = 0 and 0²
log f(x)
Sampling-based approximation. We are interested in approximating a continuous distribution f by an Gaussian N (m,, o²). Here, we assume that we do not know the pdf of μ., however, we
sample from X1, X X,, I Em X; and x² = n (X, which is equivalent to doing maximum likelihood estimate, moment matching, and doing optimization in Equation with the proxy empirical measure ph.
iidd
μ.. Therefore,
approximate by N(m, o²) with
1 +E 1= 8x,-
B NON-PARAMETRIC HYPOTHESIS TESTING
We follow the non- non-parametric two-sample test in (Wang et al.) 2022). We use the permutation test for two cases. The first case is with two measures as Gaussians with diagonal covariance matrix and the second case is with two measures as Gaussians with full covariance matrix. We use the same hyper-parameters setting as in Section 5.1 in (Wang et al. 2022) and use SW, LCV-SW, and UCV-SW with L 100 as the testing distance. We obtain the result in Figure
SW
LCV-SW
UCV-SW
Couariance Shifas (tagerst Gauedar
Covadance Stitee diggrrah
Guustan
10
0.8
0.8
68
0.6
06
0.4
04
0.2
.02
0.0
2 2
2
21
0.0
22
2
27
2
2
27
2
Dinambor
Dimandon 0
Figure 4: Testing results on Gaussian distributions across different choices of dimension D. Left: power for Gaussian distributions, where the shifted covariance matrix is still diagonal; Middle: power for Gaussian distributions, where the shifted covariance matrix non-diagonal; Right: Type-I error.
C PROOFS
C.I PROOF OF PROPOSITION 1
We have P and v are two empirical measures i.e., μ Σi= a,d.; (E a, 1) and Dm B:dy, (m Br 1), we have their projected measures 0tyy Ei= 0:00T and Oiv
13

---

Published as a conference paper at ICLR 2024
Ei=1 BeooT y. Now, we have:
(- (~20 (0T 2i - mi )²
argmaxm1.ot mi
12T ai log
exp
209
l=
ai log(of) )
argmax mi
Oi
(0' 34 m1)² )²=
20
2
argmax of (m1, ot).
Taking the derivatives and setting them to O, we have:
d
f(m1,
=
ai
m1
0T x4) (Tx;))=
1
Σ
n
a&
0,
dm
i=1 01
of
i=1
12
do?
[m1
= Σ
2
(0 I; m1)²
20,
20
01
m )2
0g
0.
1=1
Hence, we obtain:
D 1=
aTTT
12 Σ
a;0 a,OJxn Iz
m1
1= a4
E a;(0' T4 m2)²
0 Σラ=1 aj
(,
m1)²
1=1
With similar derivation, we obtain the result for m2, of and complete the proof.
C.2 PROOF OF PROPOSITION 2
We first prove the following lemma.
Lemma 1. Let 0 is uniformely distributed vector on the unit-hypersphere i.e, U(S0-1), we have |[00 1 Ld.
Proof. Since we can obtain a uniformly distributed vector 0 on g by normalizing a unit Gaussian (O. Iz). Therefore, rewrite:
distributed vector i.e., / = = with
||=||a
E
|| 12122
Since the expectation a random matrix is matrix of expectation, we now need to calculate E[0;0,] for all i.j E [[d]].
or
E
Let denote = (f)...., 02) and (=1...., Ed), we have 0' (0I...., - -.....,.a) ,0) (for any 2a) ~
i [[d]] == (1.2...... d) also follow (Sd- since = T with z' = (2), (O La) (linear mapping of Gaussian is Gaussian). Therefore, we have B[0,0,] = 1 which means [00,0,]] 0 for all E [[d]] + i. Now, we calculate the variance E|0 for all E [[d]] have
2
Σ
E
E
E
1.
(=1
Moreover, OI...., Ou are exchangeable with a similar argument with the symmetry and linear mapping property of Gaussian distribution. Therefore, they are indentically distributed i.e., E E[A"1 1 //d for all i,j [[d]] We complete the proof here.
14

---

Published as a conference paper at ICLR 2024
We now go back to the proof of Proposition 2 From Proposition
have:
E (m1 p))
m2(0; E
agt'
2t
3
y1
R
0:24
Σ
172
Beys
|
04
Σ
m
Beyli
d
1
n
Belli
where we use the result of Eg
[00
from Lemma
Next, we have:
27
n
n
E
[[]
E
04
0 F4
a&d Zep
p=11
E [
ai Di
Ogp
ai
1=1
i=
Similarly, we have E [o](0:v)]
E
B 1131 - ==
3ryrr 11.
C.3 PROOF OF PROPOSITION
We recall that:
Clow p12) (0: m2(0;
(op(0; (mi (0:))) +a (0; (0r (0
(0: m2(0v))) (0)(0: o2(0:: v))
Since 0 < p)) m²)0;v))² to the inequalities.
v) we obtain the result by adding (m1 (0: p1)
C.4 PROOF OF PROPOSITION
We have a and v are two empirical measures i.e., 1 = E a,d,, (E a; = 1) and v
Em 3:8gs Em Br = 1). Now, we have:
1 (z, mi || - ((x=)((m)))
mi))
argmax
az log
cxp
(22)
argmax
ai
20
(Iz
m1
m m1) log(at)
argmax o
(m)
Taking the derivatives and setting them 0, we have:
Vm, f(m of) =
Σ
a
xx)
Σ
m, ax3;
= 0,
2d
o7d
5ml
im/l
d
d
d
fm1
0f)
Σ aj
(x) m1 T(x; m1)
do?
202d42
2d+2
20
d
a:||
m1
12
do?
oRd
022
0.
1=1
15

---

Published as a conference paper at ICLR 2024
Hence, we obtain:
2i=1 ii
m2
0.
m1
0
21=1 Oj
of Li=1
20
mi
12
Σ
a || m, 12
Di=1 04
Similarly, we obtain m2 = Ei= 3ith and 03 (Ei= 3:11y1 y - m2 (13) 4
Using the linearity of Gaussian distributions, we have OtN(m1 o;
=N m1. 0f0 10)
N(O of (0 0 1). Similarly, we obtain (HN(mmz I2I)
11mm2.33 σ2). ~N(0mmz
Therefore, we
have:
(O1V 10N00/N000000 ( = (0 m1 00 0Tm²)² + (ar 02)²
Calculating the expectation, we have:
E [W5300NN(m, of (22 [fT) = (mi m2) "E[O m2) + (a) -02)²
m1 mz 12 (o1 - 02)
Let denote C(0) = W2 ofI), 0EN (E(T),,N((m,))) (m2, o I)), we have:
C(0) - [[((()] (0T m; TT m2)²
m+ m21l2
2
1
0
Bith
B:30
d
i=1
i=1
i=1
i=1
1
24
T Ui
Σ
3:84
d
i=1
i=1
Clow (0; (0;H,, - 1E 11)
Similarly, we obtain:
Var [C[0)] = E [[C(0) - ([C(()]) [[Crow (1: v) E[Crow (0; p,))))
Var[Crow (0; p1 p,v)]
Cov [C(0). W (0; E -[CCC)
v) (O:(,;;,
= [Ciow (0; v) E[Cro.n (WW (0; - ([
(0; v)]1
Cov (Clow (0; v), W (0; μ, v],
Therefore, it sufficient to claim that using Clow control variate is equivalent to use W3(0:N(m1,.o I) (m2. the control variate.
D ALGORITHMS
We present the algorithm for computing the conventional Monte Carlo estimator of the sliced Wasserstein distance between two discrete measures in Algorithm [ Similarly, we provide the algorithms for the lower bound control variate estimator and the upper bound control variate estimator in Algorithm 2 and in Algorithm respectively.
E RELATED WORKS
Sliced Wasserstein variants with different projecting functions. The conventional sliced Wasser- stein is based a linear projecting function i.e., the inner product to project measures to one dimension. In some special cases of probability measures, some other projecting functions might be preferred e.g., generalized sliced Wasserstein (Kolouri et al. 2019) distance with circular, polyno- mial projecting function, spherical sliced Wasserstein (Bonet et al. 2023) with geodesic spherical projecting function, and Despite having different projecting functions, all mentioned sliced
16

---

Published as a conference paper at ICLR 2024
Algorithm 1 The conventional estimator of sliced Wasserstein distance.
Input: Probability measures = Ei=1 asd2, and v -i= Bioy P 2 1, and the number of projections L.
Set SW (1, v/; L) = 0
forl=1 1 to L do
Sample 8/ ~ 1($--1)
Compute SW (1, SW, (,, L. + ++ S Foisa()) -Fo (=) (z)
end for Return: SW L;
Algorithm 2 The lower bound control variate estimator of sliced Wasserstein distance.
Input: Probability measures μ = Di=ı ardn; and v Di= 3r0y, p Z 1, and the number of
projections L.
Set SW (1, 1/; L) =
Compute 0 Li=1 a and y = Em= Bi8h
for - L, do
Sample 0₂ ~ (
Compute wi Ju F 0::0u (2)
F (z)l
Compute a = (0 I 4)²
end for
Compute SW p (1, v; L)= = E w1
512 Compute b = -
Compute y /E (wi SWw--SW",,,ee (()
Compute LCV-SW" (M, (H, = SW (wl,,, - - Ei= Ei -b)
Return: LCV-SW,(H,u) LL,
Algorithm 3 The upper bound control variate estimator of sliced Wasserstein distance.
Input: Probability measures = Ei=1 a,dg, and v Ei 3,0y. P 2 1, and the number of
projections L.
Set SW p (1,, vz L) =0
Compute E = -E OgTis and j = Em Biyli
for L do
Sample 02 ~ (1
Compute wi S1 F 0.5u (2) F 0. (z)
Compute = (8 1 - g) + Σi as 8 TT + En B,(0T y: AT 8)
end for
Compute SW (14, (14, 1. = T Σラ Wi
Compute b = 11 012 + E -= 3)1 + 1 Bil((vi 2)
Compute 7 1. El-1(wi (11- (G-
Compute UCV-SW" " (,, SW
E/
Return: UCV-SW (m,
Wasserstein variants utilize random projecting directions that follow the uniform distribution over the unit hypersphere and estimated using the Monte Carlo integration scheme. Therefore, could directly adapt the proposed control variates in these variants.
Applications of the control variate estimators. Since the proposed control variates are used to estimate the SW distance, they can be applied to all applications where sliced Wasserstein exists. We would like to mention some other applications such as domain adaptation (Lee et al. 2019).
17

---

Published as a conference paper at ICLR 2024
Point.Cloud
-8- USY-SW UCV-5W
121
LOW-SW UCV/SW
127
1.7
Ale 2114
Seooet
Spooed
30-1
7W
30
100
Figure 5: The empirical errors of the conventional estimator (SW) and the control variate estimators (LCV-SW.
UCV-SW) when comparing empirical distributions over MNIST images and point-clouds.
Table 3: Estimated variances of conventional estimator's and the control variate estimators.
Estimator
MNIST 0-1
MNIST 1-7
Point-cloud-1
Point-cloud-2
SW
4700.87 59.12
1205.62 + 14.17
12.78 + 0.025
12.79 + 0.026
LCV-SW
0.045 = 0.008
0.0017 + 0.001
0.0025 + 0.0000
0.0021 + 0.0000
UCV-SW
0.061 = 0.016
0.0018 = 0.0001
0.0025 + 0.0000
0.0021 + 0.0000
Table 4: Summary Wasserstein-2 scores (multiplicd by 10") from different runs, computational time in
second (s) to reach step 500 of different sliced Wasserstein variants in gradient flows
Distances
Step 3000 (W,
Step 4000 (W; 1)
Step S0ND (W;
Step 6000W;
Step 8000 (W2 J)
Time (s
L-10
0,4919
0.1054
LCV-SW 1-10
902-9718
0.1788
135.9132 + 0.0922
35.0292
0.1457
0.0452
0.0045
1.76 6+ 0.3e
27.65 0:01
UCV-SW L=10
902 9717 0.1888
135.9132
0.0922
35,0286
0.1458
0.0446 0 0038
2.0e 3 0.4e
29.56 0.01
L-100
0.2375
0.3146
222.06
LCVASW 1-100
300
0.1054
139.52238
0.0065
33.4460
0.0031
D.0I84 5.8e -
1.4e 6++ 0.1e
223. 79 0.82
UCV-SW L=100
900 2631 0.0054
135.5237 0,0065
24.4459-50
0.0030
0.0083 8.5e
285.29 1.65
approximate Bayesian computation NNdjahi al. 2020a). color transfer (Li al. 2022), point-cloud reconstruction (Nguyen et al. 2024a), mesh deformation Le et al. (2024a). diffusion models (Nguyen et al. 2024b), and many other tasks.
F ADDITIONAL EXPERIMENTS
In this section, we provide some additional experiments for applications the main text. In particular, calculate the empirical variances of the conventional estimator and the control variate estimators on different images digits and point-clouds in Appendix We provide the point-cloud gradient flow between two new point-cloud in Appendix F.2 We then provide the detailed training of deep generative models and additional generated images in Appendix F.3
F.1 COMPARING EMPIRICAL PROBABILITY MEASURES OVER IMAGES AND POINT-CILOUDS
Settings. We follow the same settings which are used in the main text. However, for MNIST, we compare probability measures over digit I and digit 7. For point-clouds, we compare two different point-clouds from the main text. We report the estimated variances of ( v), Ziow (0; μ. v), Zap (0: p) defined in Definition in Table 3 We use large number of samples e.g., 100000 Monte Carlo samples. the table, point-cloud-1 denotes the pair in Figure and point-cloud-2 denotes the pair in Figure 6
Results. We show the estimated errors and the corresponding computational time in Figure S From the figure, observe the same phenomenon in the main text. In particular, the control variate estimators reduce considerably the errors in both cases while having approximately the same computational time.
18

---

Published as a conference paper at ICLR 2024
Figure 6: Moit--Cloud gradient flows for 10 from SW, LCV-SW, and UCV-SW.
SW L=1000
LCV-SW L==000
UCV-SW l.=1000
Figure 7: Random generated images of distances CIFARIO and CelebA.
F.2 POINT-CLOUT GRADIENT FLOWS
Settings. We follow the same settings which are used in the main text. However, we use different point-clouds which are also used in Appendix F.T
Results. We show the quantitative results in Table and the corresponding qualitative result in Figure 6 Overall, observe the same phenomenon as in the main text. In particular, the control variate estimators i.c., LCV-SW, UCV-SW help to drive the flows to converge faster to the target point-cloud than the conventional estimator i.e., SW. It is worth noting that the computational time of the control variate estimators is only slightly higher than the conventional estimator for both settings of the number of projections L 10, 100.
F.3 DEEP GENERATIVE MODELING
Setting. We denote a as our data distribution. We then design the model distribution V/ as push forward probability measure that is created by pushing unit multivariate Gaussian (c) through
19

---

Published as a conference paper at ICLR 2024
neural network Go that maps from the realization of the noise to the data space. We use second neural network T3 that maps from data space to a single scalar. We denote Tæ, the sub neural feature space (output of the last Resnet block), and
network of T that maps from the data space Ta that maps from the feature space (image of T3 to a single scalar. More precisely, T3 TBp o7 We the following neural networks for Go and T3:
. CIFAR10:
GGi 2 (~ ((, 4x4x 256 Dense. Linear) - ResBlock up 256 + ResBlock up 256 + ResBlock up 256 + BN, ReLU,
3 conv, 3 Tanh.
-T, x [-1, 1j,11_22522x + ResBlock down 128 + ResBlock down 128
ResBlock down 128 + ResBlock 128 ResBlock 128.
-Tay æ 1(Spectral normalization).
1
ReLU
Global sum pooling(125)
-Ts(x) (Ts,
CelebA:
Got R (~ (~~c e: e: (0, + 2x64xx2(DDDenn Linear) - ResBlock up 256
ResBlock up 256
ResBlock up 256 ResBlock up 256
BN, ReLU,
3 x conv, 3 Tanh
-T,; E [-1, 1],1x3x32x2 + ResBlock down 128 - ResBlock down 128 7
ResBlock down 128 + ResBlock 128 - ResBlock 128.
- -Tay: x E 1288888 + ReLUU + Global sum pooling
(Spectral normalization).
(x) (2, (z)).
We the following bi-optimization problem to train neural networks:
min (Z (min -1 - +T(()) + Ez-e -1 - -1 91.32
min E x~ X~ 2 (8(T FPx, oGo
where the function 7 T3s (z), (T (x)) which the concatenation vector of T3, (x) and Ta, (Ta, (x)). S is estimator of the sliced Wasserstein distance. The number of training iterations set to 100000 on CIFARIO and 50000 in CelebA. We update the generator Go every iterations and we update the feature function T3 every iteration. The mini-batch size m is set 128 in all datasets. We use the Adam (Kingma & Ba 2014) optimizer with parameters (81, (2) = (0. 0.9)
for both G and T3s with the learning rate 0.0002. We use 50000 random samples from estimated generative models G for computing the FID scores and the Inception scores. In evaluating FID scores, we all training samples for computing statistics of datasets.
Results. addition to the result in the main text, we provide generated images from the conventional estimator (SW) and the control variate estimator (LCV-SW and UCV-SW) with the number of projections L = 1000 Figure 7 Overall, we see that by increasing the number of projections, the generated images are visually improved for all estimators. This result is consistent with the FID scores and the IS scores in Table 2
G COMPUTATIONAL INFRASTRUCTURE
For comparing empirical probability measures over images and point-cloud application, and the point-cloud gradient flows application, we use Macbook Pro MI for conducting experiments. For deep generative modeling, experiments are run on single NVIDIA V100 GPU.
20


Reviews:
[{"review_id": "EuQpLnkMSR", "reviewer": "Reviewer_pTc9", "summary": "Developing efficient algorithms to compute slided Wasserstein distance is an important problem in OT. The authors proposed to use control variates, a special variance reduction algorithm, to compute the distance reliably. Compared with standard Monte Carlo sampling, it has a much smaller variance, as observed empirically. The authors carefully designed the control variates, which plays a crucial role in the performance of variance reduction. The computational complexity is also analyzed, and a comprehensive numerical study is performed to show the superior performance of their proposed algorithm.", "questions": "I think the authors also miss an important application on computing sliced Wasserstein (SW) distance. Since the SW distance can be used to quantify the difference between distributions, it can be used for non-parametric two-sample testing, i.e., given samples from two distributions $\\mu$ and $\\nu$, to determine either $H_0: \\mu=\\nu$ or $H_1: \\mu\\ne\\nu$. Related literature [Wang et al. 2021] used the projected Wasserstein distance (seeks the one-dimensional projector that maximizes OT distance, instead of finding the averaged OT distance among all one-dimensional projectors) for two-sample testing, but it can be naturally extended for SW distance. The authors can use their variance reduction technique to finish this task with superior computational efficiency.\n\nRef: Wang J, Gao R, Xie Y. Two-sample test using projected wasserstein distance[C]//2021 IEEE International Symposium on Information Theory (ISIT). IEEE, 2021: 3320-3325.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Although the idea of control Variate is simple and straightforward, the construction of the control variate estimator is highly non-trivial. The authors find a good estimator based on the closed-form expression of the OT distance between two fitted Gaussians. This idea is useful. \n- I agree that it is intractable to evaluate $\\mathbb{E}[W_2^2(\\mathcal{N}(m_1(\\theta;\\mu), \\sigma_1^2(\\theta;\\mu)), \\mathcal{N}(m_2(\\theta;\\mu), \\sigma_2^2(\\theta;\\mu)))]$, and the authors proposed lower and upper bounds on this unknown quantity around Proposition 3, which is valid.\n- The computational complexity for two control variate-based sliced Wasserstein distance is analyzed, which showcases their computational efficiency. \n- It is good to see how to apply the authors' algorithm to other variants of sliced Wasserstien distance, as pointed at the end of Section 3.\n- It is also interesting to see how to apply the control variate to compute the gradient of sliced Wasserstein distance, as pointed at the end of Section 3.\n- Numerical study on three different applications is solid.", "weaknesses": "While I appreciate and understand the authors' main idea, some parts of writing can be improved:\n- It is good that the authors provided strong motivations for using control covariates in Section 3.1, paragraph 2, the wiring in this part is poor. I suggest the authors follow the writing of [A. Shapiro 2021, Section 5.5.2] to re-write this part. \n\nRef: Shapiro A, Dentcheva D, Ruszczynski A. Lectures on stochastic programming: modeling and theory[M]. Society for Industrial and Applied Mathematics, 2021.\n\n- For the paragraph \"Constructing Control Variates\" in Section 3.2, the authors should omit the detailed deviation of the closed-form solution of $W_2^2(\\mathcal{N}(m_1(\\theta;\\mu), \\sigma_1^2(\\theta;\\mu)), \\mathcal{N}(m_2(\\theta;\\mu), \\sigma_2^2(\\theta;\\mu)))$. It is fine to present only the final simplified expression.\n- For the paragraph below Proposition 1, the description for computing the sliced Wasserstein between continuous distributions is brief and confusing. The authors may consider describe the algorithms in detail and present this part in an extra Appendix instead. \n- I feel the theoretical analysis of the variance-reduced estimator is not enough. But when I look at related literature, there is little theoretical guarantees on this part. So I think it is fine regarding the theoretical contribution part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1FflNT9KWx", "reviewer": "Reviewer_dx24", "summary": "The paper proposes novel Monte Carlo estimators for the sliced Wasserstein distance (SW) using the control variates principle. The proposed estimators have reduced variance, and the computation costs remain the same. In addition, the paper provides various experiments and applications, which illustrate that in practice the estimators do outperform naive SW estimator, and can be applied to various tasks including gradient flow and generative modeling.", "questions": "Please see above (section Weaknesses) for details. The reviewer is confident that these can be clarified.\n\nTo make the paper stronger, the reviewer suggest investigating the balance between the computational tractability (e.g. using $m_1,\\sigma_1$) and statistical relevance, as it seems that the easier control variate is to be computed, the less information it provides. Theoretical or empirical evidence would be illuminating.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is overall clear and well presented, and the results are original and novel to the knowledge of the reviewer. The strengths of the paper includes:\n\n1. The statistical aspects of estimator of the SW is mostly studied from the perspective of the marginal populations, and this paper sheds new light on the MC side of estimation of the SW, from a variance reduction point of view. This can be of practical interests.\n2. The construction of the estimators makes extensive use of properties of Wasserstein distance, especially in the 1 dimensional case. The balance of computational tractability and statistical relevance seems an interesting aspect, especially for SW as it utilizes external randomness.\n3. The paper provides extensive experiments, which seem sufficient for justifying the practicality of the proposed estimators.", "weaknesses": "Some weaknesses:\n\n1. A major confusion is proposition 1, in which the paper claims to minimize KL divergence between discrete distribution and a Gaussian distribution. However, to the best of the reviewer's knowledge, there's no way to obtain finite value for KL between discrete distribution and continuous distribution, even under the most generalized setting with Radon–Nikodym derivative. The reviewer acknowledge that this potential flaw does not defeat the purpose of identifying the Gaussian proxy using information of $\\mu,\\nu$ (e.g. as an alternative, through moment matching), but urge the authors to revise this part accordingly.\n2. The usage of 'upper/lower bounds' seems to lack justification, as they are quite different from the Gaussian approach the paper proposes, in terms of how much correlation is changed/lost. In addition, though $\\mathbb{E}[\\sigma_1(\\theta;\\mu)\\sigma_2(\\theta;\\nu)]$ does not have a closed form, it's unclear why it can't be estimated by MC, as is done throughout the paper. (A personal thought: from a statistical perspective, in order to introduce correlation it might be possible to use $\\sigma_1^2(\\theta;\\mu)\\sigma_2^2(\\theta;\\nu)$, if it is at all possible to address.)\n3. The treatment of the resulting estimator from control variates is rough. In particular, the following items need proper discussion: unbiasedness (the biasedness after taking p-root does not justify the biasedness of proposed estimator which is without a root), actual variance of the proposed estimators (which is not the control variate estimator but a further estimator of it), and how good the proposed estimators of coefficients are (e.g. $\\hat{\\gamma^*}$).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VBDzEAlusk", "reviewer": "Reviewer_mJzC", "summary": "Wasserstein (W) distance plays an increasingly preponderant role in many machine learning pipelines, since its ability to capture geometric features of the objects at hand. However, it suffers from a heavy computational cost, $O(n^3\\log(n))$ where $n$ is the number of supports of the probability measures. To overcome this computational bottleneck, sliced Wasserstein distance (SWD) stands as an alternative to Wasserstein distance. SWD is based on slicing the origin measures  $\\mu$ and $\\nu$ by projecting them on a direction $\\theta$ of the unit-hypersphere and then calculating an expectation (average) of the 1-dimensional Wasserstein distances between the projected measures $\\mu_\\theta$ and $\\nu_\\theta$. This expectation is calculated using Monte Carlo (MC) integration to estimate SWD distance. It is known that the error of the MC approximation is $O\\big({L}^{-1/2} \\times \\text{Variance}[W_p^p(\\mu_\\theta, \\nu_\\theta)\\big]\\big)$. \n\nThis paper proposes a computationally efficient control of the term $\\text{Variance}[W_p^p(\\mu_\\theta, \\nu_\\theta)\\big]$ based on the control variates from the literature on variance reduction. The idea behind this is to find left and right control variates estimators, which are Gaussian control approximations of $\\mu_\\theta$ and  \\nu_\\theta$ with a low variance of their Wasserstein distances.  These estimators share the same computational complexity and memory complexity as the conventional estimator of SWD (vanilla MC estimator).", "questions": "### Minor Typos\n- Page 2 (last line): there is an extra $F_\\nu$.\n- Page 4: the term $\\gamma^2 \\text{Var}[C(\\theta)]$ is extra in the derivative $f'(\\gamma)$.\n- Page 4: \"... has a correlation with $W(\\theta)$\" --> \" ... has a correlation with $W(\\theta; \\mu, \\nu)$\"\n- Page 5: in Definition 3 \"... i.e.$\\mathcal{N}$ ...\" missing space\n- Page 12: there is no sign \"-\" on the derivative of $f(m_1, \\sigma^2)$ wrt $m_1$.\n- Page 13:  the notation $[[\\cdot]]$ is not defined.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Proposing a novel control of the variance term in the projection complexity of SWD that leads to a computationally efficient estimation of SWD.\n- The estimators are based on the control variate from the literature of reduction variance, which seems interesting to bridge the OT metrics with reduction variance. \n- Extensive experiments on comparing probability measures over images and point-clouds, point clouds gradient flows, and generative modeling. The left and right estimators have computational and performance gains over the conventional MC approximation of SWD.\n- The presentation of the paper is easy to follow. I checked the proofs and they sound good to me. \n- The code is attached and the experimental protocols are well explained (in the main + supplementary), which guarantees the reproducibility of the experiments.", "weaknesses": "- In Definition 1, the expectation of the controlled projected one-dimensional Wasserstein distance is non-nonegative (unbiased estimator of SWD). But what about the variable itself, $Z(\\theta, \\mu, \\nu)$, is it also non-negative?\n- I'm wondering about the utility of Proposition 4, which states the left control estimator is equivalent to considering a control variate with respect to the 2-Wasserstein distance between a projection of Gaussian approximations of the origin measures. I'm thinking about the following scheme: we first calculate these Gaussian approximations of the origin probability measures then an SWD between these approximations and the controlled SWD is given by: $|\\hat{SWD}(\\text{origin measures}) - \\hat {SWD}(\\text{Gaussian Approximation of the origin measures})|$ without adding the factor $\\gamma$. \nI don't know it may highlight the importance of $\\gamma$ to get more ``tight`` control variate. Any comment(s) will be appreciated for this point.\n- The most related previous work is the Gaussian approximation for the SWD_2 (Nadjahi et al; n NeurIPS'21). Can we expect a fast rate of convergence of the controlled variate estimator, e.g. LCV-SWD to the true Sliced Wasserstein distance? (In Nadjahi et al; n NeurIPS'21, the rate is $O(d^{-1/8})$, see Corollary 1 therein, which is too slow for $d \\gg1$.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WBzAOAsOUu", "reviewer": "Reviewer_yEdx", "summary": "The paper focuses on improving the Monte Carlo estimation scheme used in computing Sliced Wasserstein (SW). While several works in the literature have concentrated on enhancing the sampling scheme, such as max-SW [1], projected W [2], and distributional SW [3], the paper's originality lies in its approach to controlling the variance of the estimation using control variates. The solution can be computed in linear time, similar to SW, and experiments demonstrate that, with a fixed number of lines L, the estimation is improved when compared to SW.\n\n[1] Deshpande, Ishan, et al. \"Max-sliced wasserstein distance and its use for gans.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[2] Rowland, Mark, et al. \"Orthogonal estimation of wasserstein distances.\" The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.\n\n[3] Nguyen, Khai, et al. \"Distributional Sliced-Wasserstein and Applications to Generative Modeling.\" International Conference on Learning Representations. 2020.", "questions": "- how does the method compares with max-sliced SW, distributional SW and projected SW?\n- Is there any results that compare the value provided by CV-SW with SW or Wasserstein ? Do you have any results regarding the sample complexity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses an important problem: how to provide a proxy for the Wasserstein distance using a fast algorithm. It aims to take a step forward in SW estimation (which has a complexity of $O(n \\log(n))$) by carefully selecting the projection lines. For a given number of directions $L$, the algorithms provide improved performance over SW, as evaluated in several experimental setups.\nThe originality of the approach is that it relies on the control variates method, which reduces variance in Monte Carlo methods. The paper is clear, and its claims are supported by empirical and/or theoretical evidence.\n\nMain strengths:\n- Fairly well motivated and original solution\n- Theory and experiments seem sounded", "weaknesses": "The paper lacks positioning with respect to competitors whose goal is also to improve the sampling scheme of SW, or more generally, other SW variants. Evaluating the proposed methods among numerous competitors can be challenging. However, CV-SW is benchmarked only against SW, whereas algorithms such as max-SW (which considers only the 'max' direction), distributional SW (which searches for an 'optimal' direction), or even projected Wasserstein (which uses orthogonal directions) are closely related to the proposed method in spirit. Theoretical and experimental comparisons are missing. \n\n\n\nMinor comments:\n- $W(\\theta)$ (two lines after eq. 7) has not been defined\n- Could you check the value of $f'(\\gamma)$ tow lines after eq. 8 ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)