SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tongzheng Ren
tongzheng@utexas.edu
Sliced Wasserstein (SW) distance suffers from redundant projections due to independent uniform random projecting directions. To partially overcome the issue, max K sliced Wasserstein (Max-K-SW) distance ( $K \geq 1$ ), seeks the best discriminative orthogonal projecting directions. Despite being able to reduce the number of projections, the metricity of the Max-K-SW cannot be guaranteed in practice due to the non-optimality of the optimization. Moreover, the orthogonality constraint is also computationally expensive and might not be effective. To address the problem, we introduce a new family of SW distances, named Markovian sliced Wasserstein (MSW) distance, which imposes a first-order Markov structure on projecting directions. We discuss various members of the MSW by specifying the Markov structure including the prior distribution, the transition distribution, and the burning and thinning technique. Moreover, we investigate the theoretical properties of MSW including topological properties (metricity, weak convergence, and connection to other distances), statistical properties (sample complexity, and Monte Carlo estimation error), and computational properties (computational complexity and memory complexity). Finally, we compare MSW distances with previous SW variants in various applications such as gradient flows, color transfer, and deep generative modeling to demonstrate the favorable performance of the  $\mathrm{MSW}^1$ .
Sliced Wasserstein (SW) [7] distance has been well-known as a great alternative statistical distance for Wasserstein distance [57, 49]. In short, SW takes the average of Wasserstein distances between corresponding pairs of one-dimensional projected measures as the distance between the two original measures. Hence, the SW has a low computational complexity compared to the conventional Wasserstein distance due to the closed-form solution of optimal transport in one dimension. When the probability measures have at most  $n$  supports, the computational complexity of the SW is only  $\mathcal{O}(n\log n)$ . This complexity is much lower than the computational complexity  $\mathcal{O}(n^3\log n)$  of Wasserstein distance and the complexity  $\mathcal{O}(n^2)$  [1, 34, 35, 33] of entropic Wasserstein [11] (Sinkhorn
$^{1}$ Code for this paper is published at https://github.com/UT-Austin-Data-Science-Group/MSW.
divergence). Moreover, the memory complexity of the SW is  $\mathcal{O}(n)$  which is lower than  $\mathcal{O}(n^2)$  of the Wasserstein (Sinkhorn) distance. The reason is that SW does not need to store the cost matrix between supports which cost  $\mathcal{O}(n^2)$ . An additional appealing property of the SW is that it does not suffer from the curse of dimensionality, namely, its sample complexity is  $\mathcal{O}(n^{-1/2})$  [40, 46] compared to  $\mathcal{O}(n^{-1/d})$  [19] of the Wasserstein distance ( $d$  is the number of dimensions).
Due to the scalability, the SW has been applied to almost all applications where the Wasserstein distance is used. For example, we refer to some applications of the SW which are generative modeling [60, 15, 27, 42], domain adaptation [30], clustering [28], approximate Bayesian computation [39], gradient flows [37, 5], and variational inference [61]. Moreover, there are many attempts to improve the SW. The generalized sliced Wasserstein (Gsw) distance that uses non-linear projection is proposed in [26]. Distributional sliced Wasserstein distance is proposed in [44, 45] by replacing the uniform distribution on the projecting directions in SW with an estimated distribution that puts high probabilities for discriminative directions. Spherical sliced Wasserstein which is defined between distributions that have their supports on the hyper-sphere is introduced in [4]. A sliced Wasserstein variant between probability measures over images with convolution is defined in [43].
Despite having a lot of improvements, one common property in previous variants of the SW is that they use independent projecting directions that are sampled from a distribution over a space of projecting direction e.g., the unit-hypersphere. Those projecting directions are further utilized to project two interested measures to corresponding pairs of one-dimensional measures. Due to the independence, practitioners have reported that many projections do not have the power to discriminative between two input probability measures [26, 15]. Moreover, having a lot of projections leads to redundancy and losing computation for uninformative pairs of projected measures. This problem is known as the projection complexity limitation of the SW.
To partially address the issue, the max sliced Wasserstein (Max-SW) distance is introduced in [14]. Max-SW seeks the best projecting direction that can maximize the projected Wasserstein distance. Since the Max-SW contains a constraint optimization problem, the projected subgradient ascent algorithm is performed. Since the algorithm only guarantees to obtain local maximum [46], the performance of empirical estimation Max-SW is not stable in practice [42] since the metricity of Max-SW can be only obtained at the global optimum. Another approach is to force the orthogonality between projecting directions. In particular, K-sliced Wasserstein [50] (K-SW) uses  $K > 1$  orthogonal projecting directions. Moreover, to generalize the Max-SW and the K-SW, max-K sliced Wasserstein (Max-K-SW) distance ( $K > 1$ ) appears in [12] to find the best  $K$  projecting directions that are orthogonal to each other via the projected sub-gradient ascent algorithm. Nevertheless, the orthogonality constraint is computationally expensive and might not be good in terms of reflecting discrepancy between general measures. Moreover, Max-K-SW also suffers from the non-optimality problem which leads to losing the metricity property in practice.
To avoid the independency and to satisfy the requirement of creating informative projecting directions efficiently, we propose to impose a sequential structure on projecting directions. Namely, we choose a new projecting direction based on the previously chosen directions. For having more efficiency in computation, we consider first-order Markovian structure in the paper which means that a projecting direction can be sampled by using only the previous direction. For the first projecting direction, it can follow any types of distributions on the unit-hypersphere that were used in the literature e.g., uniform distribution [7] and von Mises-Fisher distribution [23, 45] to guarantee the metricity. For the transition distribution on the second projecting direction and later, we propose two types of family which are orthogonal-based transition and input-aware transition. For the orthogonal-based transition, we choose the projecting direction uniformly on the unit hypersphere such that it is orthogonal to the previous direction. In contrast to the previous transition which does not use the information from the two input measures, the input-aware transition uses the sub-gradient with respect to the previous projecting direction of the corresponding projected Wasserstein distance between the two measures to design the transition. In particular, the projected sub-gradient update is used to create the new projecting direction. Moreover, we further improve the computational time and computational memory by introducing the burning and thinning technique to reduce the number of random projecting directions.
Contribution. In summary, our contributions are two-fold:
1. We propose a novel family of distances on the space of probability measures, named Markovian sliced Wasserstein (MSW) distances. MSW considers a first-order Markovian
structure on random projecting directions. Moreover, we derive three variants of MSW that use two different types of conditional transition distributions: orthogonal-based and input-aware. We investigate the theoretical properties of MSW including topological properties (metricity, weak convergence, and connection to other distances), statistical properties (sample complexity, and Monte Carlo estimation error), and computational properties (computational complexity and memory complexity). Moreover, we introduce a burning and thinning approach to further reduce computational and memory complexity, and we discuss the properties of the resulting distances.
2. We conduct experiments to compare MSW with SW, Max-SW, K-SW, and Max-K-SW in various applications, namely, gradient flows, color transfer, and deep generative models on standard image datasets: CIFAR10 and CelebA. We show that the input-aware MSW can yield better qualitative and quantitative performance while consuming less computation than previous distances in gradient flows and color transfer, and comparable computation in deep generative modeling. Finally, we investigate the role of hyper-parameters of distances e.g., the number of projections, the number of time-steps, and so on, in applications.
Organization. We first provide background for Wasserstein distance, sliced Wasserstein distance, and max sliced Wasserstein distance in Section 2. In Section 3, we propose Markovian sliced Wasserstein distances and derive their theoretical properties. Section 4 contains the comparison of MSW to previous SW variants in gradient flows, color transfer, and deep generative modeling. We then conclude the paper in Section 5. Finally, we defer the proofs of key results in the paper and supplementary materials to Appendices.
Notation. For  $p \geq 1$ ,  $\mathcal{P}_p(\mathbb{R}^d)$  is the set of all probability measures on  $\mathbb{R}^d$  that have finite  $p$ -moments. For any  $d \geq 2$ , we denote  $\mathcal{U}(\mathbb{S}^{d-1})$  is the uniform measure over the unit hyper-sphere  $\mathbb{S}^{d-1} := \{\theta \in \mathbb{R}^d \mid ||\theta||_2^2 = 1\}$ . For any two sequences  $a_n$  and  $b_n$ , the notation  $a_n = \mathcal{O}(b_n)$  means that  $a_n \leq C b_n$  for all  $n \geq 1$ , where  $C$  is some universal constant. We denote  $\theta \sharp \mu$  is the push-forward measures of  $\mu$  through the function  $f: \mathbb{R}^d \to \mathbb{R}$  that is  $f(x) = \theta^\top x$ .
We start with reviewing the background on Wasserstein distance, sliced Wasserstein distances, their computation techniques, and their limitations.
Wasserstein distance. Given two probability measures  $\mu \in \mathcal{P}_p(\mathbb{R}^d)$  and  $\nu \in \mathcal{P}_p(\mathbb{R}^d)$ , the Wasserstein distance [57, 48] between  $\mu$  and  $\nu$  is:
$$
\mathrm {W} _ {p} ^ {p} (\mu , \nu) = \inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| x - y \| _ {p} ^ {p} d \pi (x, y), \tag {1}
$$
where  $\Pi (\mu ,\nu)$  is set of all couplings that have margins are  $\mu$  and  $\nu$  respectively. The computational complexity and memory complexity of Wasserstein distance are  $\mathcal{O}(n^3\log n)$  and  $\mathcal{O}(n^2)$  in turn when  $\mu$  and  $\nu$  have at most  $n$  supports. When  $d = 1$ , the Wasserstein distance can be computed with a closed form:  $\mathrm{W}_p^p (\mu ,\nu) = \int_0^1 |F_\mu^{-1}(z) - F_\nu^{-1}(z)|^p dz$ , where  $F_{\mu}$  and  $F_{\nu}$  are the cumulative distribution function (CDF) of  $\mu$  and  $\nu$  respectively.
Sliced Wasserstein distance. By randomly projecting two interested high-dimensional measures to corresponding pairs of one-dimensional measures, sliced Wasserstein (SW) distance can exploit the closed-form benefit of Wasserstein distance in one dimension. The definition of sliced Wasserstein distance [7] between two probability measures  $\mu \in \mathcal{P}_p(\mathbb{R}^d)$  and  $\nu \in \mathcal{P}_p(\mathbb{R}^d)$  is:
$$
\mathrm {S W} _ {p} ^ {p} (\mu , \nu) = \mathbb {E} _ {\theta \sim \mathcal {U} \left(\mathbb {S} ^ {d - 1}\right)} \mathrm {W} _ {p} ^ {p} \left(\theta \sharp \mu , \theta \sharp \nu\right). \tag {2}
$$
Monte Carlo samples are often used to approximate the intractable expectation unbiasedly:
$$
\widehat {\mathbf {S W}} _ {p} ^ {p} (\mu , \nu) = \frac {1}{L} \sum_ {l = 1} ^ {L} \mathbf {W} _ {p} ^ {p} \left(\theta_ {l} \sharp \mu , \theta_ {l} \sharp \nu\right), \tag {3}
$$
where  $\theta_{1},\ldots ,\theta_{L}$  are drawn randomly from  $\mathcal{U}(\mathbb{S}^{d - 1})$ . When  $\mu$  and  $\nu$  are discrete measures that have at most  $n$  supports in  $d$  dimension, the computational complexity of SW is  $\mathcal{O}(Ln\log_2n + Ldn)$  and the memory complexity for storing the projecting directions and the projected supports is  $\mathcal{O}(L(d + n))$ .
Here,  $Ln\log_2n$  is for sorting  $L$  sets of projected supports and  $Ld$  is for projecting supports to  $L$  sets of scalars.
Max sliced Wasserstein distance. To select the best discriminative projecting direction, the max sliced Wasserstein (Max-SW) distance [14] between  $\mu \in \mathcal{P}_p(\mathbb{R}^d)$  and  $\nu \in \mathcal{P}_p(\mathbb{R}^d)$  is introduced as follows:
$$
\operatorname {M a x} - \mathrm {S W} _ {p} (\mu , \nu) = \max  _ {\theta \in \mathbb {S} ^ {d - 1}} W _ {p} \left(\theta \sharp \mu , \theta \sharp \nu\right). \tag {4}
$$
Computing Max-SW requires solving the constrained optimization problem. In practice, the projected sub-gradient ascent algorithm with  $T > 1$  iterations is often used to obtain a surrogate projecting direction  $\hat{\theta}_T$  for the global optimum. Hence, the empirical Max-SW distance is  $\widehat{\mathrm{Max - SW}}_p(\mu ,\nu) = W_p(\hat{\theta}_T\sharp \mu ,\hat{\theta}_T\sharp \nu)$ . The detail of the projected sub-gradient ascent algorithm is given in Algorithm 1 in Appendix A.1. The computational complexity of Max-SW is  $\mathcal{O}(Tn\log_2n + Tdn)$  and the memory complexity of Max-SW is  $\mathcal{O}(d + n)$ . It is worth noting that the projected sub-gradient ascent can only yield local maximum [46]. Therefore, the empirical Max-SW might not be distance even when  $T\to \infty$  since the metricity of Max-SW can be only obtained at the global maximum.
K-sliced Wasserstein distance. The authors in [50] propose to estimate the sliced Wasserstein distance based on orthogonal projecting directions. We refer to the distance as K-sliced Wasserstein distance (K-SW). The definition of K-SW between two probability measures  $\mu \in \mathcal{P}_p(\mathbb{R}^d)$  and  $\nu \in \mathcal{P}_p(\mathbb{R}^d)$  is:
$$
\mathrm {K} - \mathrm {S W} _ {p} ^ {p} (\mu , \nu) = \mathbb {E} \left[ \frac {1}{K} \sum_ {i = 1} ^ {K} \mathrm {W} _ {p} ^ {p} \left(\theta_ {i} \sharp \mu , \theta_ {i} \sharp \nu\right) \right], \tag {5}
$$
where the expectation is with respect to  $(\theta_{1},\ldots ,\theta_{K})\sim \mathcal{U}(\mathbb{V}_{k}(\mathbb{R}^{d}))$  with  $\mathbb{V}_K(\mathbb{R}^d) = \left\{(\theta_1,\dots ,\theta_K)\in \mathbb{S}^{d - 1}|\langle \theta_i,\theta_j\rangle = 0\forall i,j\leq K\right\}$  is the Stiefel manifold. The expectation can be approximated with Monte Carlo samples  $(\theta_{1l},\ldots ,\theta_{Kl})_{l = 1}^{L}$  from  $\mathcal{U}(\mathbb{V}_K(\mathbb{R}^d))$ . In the original paper,  $L$  is set to 1. To sample from the uniform distribution over the Stiefel manifold  $\mathcal{U}(\mathbb{V}_k(\mathbb{R}^d))$ , it requires using the Gram-Schmidt orthogonality process which has the computational complexity  $\mathcal{O}(K^2 d)$  (quadratic in  $K$ ). Therefore, the total computational complexity of K-SW is  $\mathcal{O}(LKn\log_2n + LKn + LK^2 d)$  and the memory complexity of K-SW is  $\mathcal{O}(LK(d + n))$ . More detail related to K-SW including Gram-Smith process and sampling uniformly from the Stiefel manifold is given in Appendix A.1.
Max K sliced Wasserstein distance. To generalize both Max-SW and K-SW, Max K sliced Wasserstein is introduced in [12]. Its definition between  $\mu \in \mathcal{P}_p(\mathbb{R}^d)$  and  $\nu \in \mathcal{P}_p(\mathbb{R}^d)$  is:
$$
\operatorname {M a x} - \mathrm {K} - \mathrm {S W} _ {p} ^ {p} (\mu , \nu) = \max  _ {\left(\theta_ {1}, \dots , \theta_ {K}\right) \in \mathbb {V} _ {K} \left(\mathbb {R} ^ {d}\right)} \left[ \frac {1}{K} \sum_ {i = 1} ^ {K} \mathrm {W} _ {p} ^ {p} \left(\theta_ {i} \sharp \mu , \theta_ {i} \sharp \nu\right) \right]. \tag {6}
$$
Similar to Max-SW, a projected sub-gradient ascent algorithm with  $T > 1$  iterations is used to approximate Max-K-SW. We refer the reader to Algorithm 4 in Appendix A.1 for greater detail. Since the projecting operator to the Stiefel manifold is the Gram-Smith process, the computational complexity of Max-K-SW is  $\mathcal{O}(TKn\log_2n + TKdn + TK^2 d)$ . The memory complexity of Max-K-SW is  $\mathcal{O}(K(d + n))$ . Similar to Max-SW, the metricity of Max-K-SW is only obtained at the global optimum, hence, the empirical estimation might not be stable. Moreover, the orthogonality constraint is also computationally expensive i.e., quadratic in terms of the number of orthogonal projections  $K$ .
3 Markovian Sliced Wasserstein distances
We first define Markovian sliced Wasserstein (MSW) distance and discuss its theoretical properties including topological properties, statistical properties, and computational properties in Section 3.1. In Section 3.2, we discuss some choices in designing the Markov chain including the prior distribution and the transition distribution. Finally, we discuss the burning and thinning variant of MSW which can reduce the computational and memory complexity in Section 3.3.
3.1 Definitions, Topological, Statistical, and Computational Properties
We first start with a general definition of Markovian sliced Wasserstein distance in Definition 1.
Definition 1. For any  $p \geq 1$ ,  $T \geq 1$ , and dimension  $d \geq 1$ , the Markovian sliced Wasserstein of order  $p$  between two probability measures  $\mu \in \mathcal{P}_p(\mathbb{R}^d)$  and  $\nu \in \mathcal{P}_p(\mathbb{R}^d)$  is:
$$
M S W _ {p, T} ^ {p} (\mu , \nu) = \mathbb {E} _ {\left(\theta_ {1: T}\right) \sim \sigma \left(\theta_ {1: T}\right)} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} W _ {p} ^ {p} \left(\theta_ {t} \sharp \mu , \theta_ {t} \sharp \nu\right) \right], \tag {7}
$$
where  $T$  is the number of time steps, the expectation is under the projecting distribution  $\theta_{1:T} \sim \sigma(\theta_{1:T})$  with  $\sigma(\theta_{1:T}) = \sigma(\theta_1, \ldots, \theta_T) = \sigma_1(\theta_1) \prod_{l=2}^T \sigma_t(\theta_t | \theta_{t-1})$ , and  $\sigma_1(\theta_1), \sigma_t(\theta_t | \theta_{t-1}) \in \mathcal{P}(\mathbb{S}^{d-1})$  for all  $t = 1, \ldots, T$ .
The first projecting direction  $\theta_{1}$  follows the distribution  $\sigma_{1}(\theta_{1})$  with  $\sigma_{1}(\theta_{1})$  to be any distributions on the unit hyper-sphere, e.g., the uniform distribution, a von Mises-Fisher distribution, and so on. By designing the transition distribution  $\sigma_{l}(\theta_{l}|\theta_{l - 1})$ , we can obtain various variants of MSW. It is worth noting that the MSW can be rewritten as the average of  $T$  expectation of one-dimensional Wasserstein distance,  $\mathrm{MSW}_{p,T}^{p}(\mu ,\nu) = \frac{1}{T}\sum_{t = 1}^{T}\mathbb{E}_{\theta_{t}\sim \sigma_{t}(\theta_{t})}[W_{p}^{p}(\theta_{t}\sharp \mu ,\theta_{t}\sharp \nu)]$ , however,  $\sigma_t(\theta_t) = \int \prod_{i = 1}^t\sigma_1(\theta_1)\prod_{l = 2}^t\sigma_{t'}(\theta_{t'}|\theta_{t' - 1})d\theta_1\dots d\theta_{t - 1}$  are not the same for  $t = 1,\ldots ,T$ . Moreover, sampling directly from  $\sigma_t(\theta_t)$  is intractable, hence, using the definition of the MSW in Definition 1 is more reasonable in terms of approximating the expectation using Monte Carlo samples.
Monte Carlo estimation. Similar to SW, we also need to use Monte Carlo samples to approximate the expectation in Definition 1. We first samples  $\theta_{11},\ldots ,\theta_{L1}\sim \sigma_1(\theta_1)$  for  $L\geq 1$  , then we samples  $\theta_{lt}\sim \sigma_t(\theta_t|\theta_{lt - 1})$  for  $t = 1,\dots ,T$  and  $l = 1,\dots ,L$  . After that, we can form an unbiased empirical estimation of MSW as follows:  $\widehat{\mathbf{MSW}}^{p}_{p,T}(\mu ,\nu) = \frac{1}{LT}\sum_{l = 1}^{L}\sum_{t = 1}^{T}\mathbf{W}_{p}^{p}(\theta_{lt}\sharp \mu ,\theta_{lt}\sharp \nu)$
Before going to the specific design of those distributions, we first discuss the empirical estimation of MSW, and investigate its theoretical properties including topological properties, statistical properties, and computational properties.
Topological Properties. We first state the following assumption: A1. In MSW, the prior distribution  $\sigma_{1}(\theta_{1})$  is supported on all the unit-hypersphere or there exists a transition distribution  $\sigma_{t}(\theta_{t}|\theta_{t - 1})$  being supported on all the unit-hypersphere. The assumption A1 is easy to satisfy and it holds for all later choices of the prior distribution and transition distribution. We now consider the metricity properties of the Markovian sliced Wasserstein distance.
Theorem 1 (Metricity). For any  $p \geq 1$ ,  $T \geq 1$ , and dimension  $d \geq 1$ , if  $A1$  holds, Markovian sliced Wasserstein  $MSW_{p,T}(\cdot ,\cdot)$  is a valid metric on the space of probability measures  $\mathcal{P}_p(\mathbb{R}^d)$ , namely, it satisfies the (i) non-negativity, (ii) symmetry, (iii) triangle inequality, and (iv) identity.
The proof of Theorem 1 is in Appendix B.1. Next, we show that the convergence in MSW implies the weak convergence of probability measures and the reverse also holds.
Theorem 2 (Weak Convergence). For any  $p \geq 1$ ,  $T \geq 1$ , and dimension  $d \geq 1$ , if  $A1$  holds, the convergence of probability measures in  $\mathcal{P}_p(\mathbb{R}^d)$  under the Markovian sliced Wasserstein distance  $MSW_{p,T}(\cdot, \cdot)$  implies weak convergence of probability measures and vice versa.
Theorem 2 means that for any sequence of probability measures  $(\mu_k)_{k \in \mathbb{N}}$  and  $\mu$  in  $\mathcal{P}_p(\mathbb{R}^d)$ ,  $\lim_{k \to +\infty} \mathrm{MSW}_{p,T}(\mu_k, \mu) = 0$  if and only if for any continuous and bounded function  $f: \mathbb{R}^d \to \mathbb{R}$ ,  $\lim_{k \to +\infty} \int f \, \mathrm{d}\mu_k = \int f \, \mathrm{d}\mu$ . The proof of Theorem 2 is in Appendix B.2. Next, we discuss the connection of MSW to previous sliced Wasserstein variants.
Proposition 1. For any  $p \geq 1$  and dimension  $d \geq 1$ ,
(i) For any  $T \geq 1$  and  $\mu, \nu \in \mathcal{P}_p(\mathbb{R}^d)$ ,  $MSW_{p,T}(\mu, \nu) \leq Max-SW_p(\mu, \nu) \leq W_p(\mu, \nu)$ .
(ii) If  $T = 1$  and the prior  $\sigma_{1}(\theta_{1})\coloneqq \mathcal{U}(\mathbb{S}^{d - 1}),MSW_{p,T}(\mu ,\nu) = SW_{p}(\mu ,\nu).$
The proof of Proposition 1 is in Appendix B.3.
Statistical Properties. We investigate the sample complexity (empirical estimation rate) of the MSW.
Proposition 2 (Sample Complexity). Let  $X_{1}, X_{2}, \ldots, X_{n}$  be i.i.d. samples from the probability measure  $\mu$  being supported on compact set of  $\mathbb{R}^d$ . We denote the empirical measure  $\mu_{n} = \frac{1}{n}\sum_{i=1}^{n}\delta_{X_{i}}$ . Then, for any  $p \geq 1$  and  $T \geq 1$ , there exists a universal constant  $C > 0$  such that  $\mathbb{E}[MSW_{p,T}(\mu_n,\mu)] \leq C\sqrt{(d+1)\log n/n}$ , where the outer expectation is taken with respect to the data  $X_{1}, X_{2}, \ldots, X_{n}$ .
The proof of Proposition 2 is in Appendix B.4. The proposition suggests that MSW does not suffer from the curse of dimensionality. Next, we investigate the MSW's Monte Carlo approximation error.
Proposition 3 (Monte Carlo error). For any  $p \geq 1$ ,  $T \geq 1$ , dimension  $d \geq 1$ , and  $\mu, \nu \in \mathcal{P}_p(\mathbb{R}^d)$ , we have:  $\mathbb{E}|\widehat{MSW}_{p,T}^p(\mu, \nu) - MSW_{p,T}^p(\mu, \nu)| \leq \frac{1}{T\sqrt{L}} \text{Var}\left[\sum_{t=1}^{T} W_p^p(\theta_t \sharp \mu, \theta_t \sharp \nu)\right]^{\frac{1}{2}}$ , where the variance is with respect to  $\sigma(\theta_1, \ldots, \theta_T)$ .
The proof of Proposition 3 is in Appendix B.5. From the above proposition, we know that increasing the number of projections  $L$  reduces the approximation error.