SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathcal{F}[\gamma](w) = \int_{\mathbb{R}^{d'}} e^{-i\langle w,x\rangle} d\gamma(x)$  denotes the Fourier transform of  $\gamma \in \mathcal{P}(\mathbb{R}^{d'})$ . By the injectivity of the Fourier transform, we obtain  $\mu = \nu$  which concludes the proof.
(a) We first provide the proof for the inequality  $\mathrm{SW}_p(\mu, \nu) \leq \mathrm{EBSW}_p(\mu, \nu; f)$ . It is equivalent to prove that
$$
\mathbb {E} _ {\theta \sim \mathcal {U} (\mathbb {S} ^ {d - 1})} \left[ \mathrm {W} _ {p} ^ {p} \left(\theta \sharp \mu , \theta \sharp \nu\right) \right] \leq \mathbb {E} _ {\theta \sim \sigma_ {\mu , \nu} (\theta ; f, p)} \left[ \mathrm {W} _ {p} ^ {p} \left(\theta \sharp \mu , \theta \sharp \nu\right) \right].
$$
From the law of large number, it is sufficient to demonstrate that
$$
\frac {1}{L} \sum_ {i = 1} ^ {L} W _ {p} ^ {p} \left(\theta_ {i} \sharp \mu , \theta_ {i} \sharp \nu\right) \leq \sum_ {i = 1} ^ {L} \frac {W _ {p} ^ {p} \left(\theta_ {i} \sharp \mu , \theta_ {i} \sharp \nu\right) f \left(W _ {p} ^ {p} \left(\theta_ {i} \sharp \mu , \theta_ {i} \sharp \nu\right)\right)}{\sum_ {i = 1} ^ {L} f \left(W _ {p} ^ {p} \left(\theta_ {i} \sharp \mu , \theta_ {i} \sharp \nu\right)\right)}, \tag {9}
$$
for any  $L \geq 1$  and  $\theta_1, \ldots, \theta_L \stackrel{\text{i.i.d.}}{\sim} \mathcal{U}(\mathbb{S}^{d-1})$ . To ease the presentation, we denote  $a_i = W_p^p(\theta_i \sharp \mu, \theta_i \sharp \nu)$  and  $b_i = f(W_p^p(\theta_i \sharp \mu, \theta_i \sharp \nu))$  for all  $1 \leq i \leq L$ . The inequality (9) becomes:
$$
\frac {1}{L} \left(\sum_ {i = 1} ^ {L} a _ {i}\right) \left(\sum_ {i = 1} ^ {L} b _ {i}\right) \leq \sum_ {i = 1} ^ {L} a _ {i} b _ {i}. \tag {10}
$$
We prove the inequality (10) via an induction argument. It is clear that this inequality holds when  $L = 1$ . We assume that this inequality holds for any  $L$ . We now verify that the inequality (10) also holds for  $L + 1$ . Without loss of generality, we assume that  $a_1 \leq a_2 \leq \ldots \leq a_L \leq a_{L+1}$ . Since the function  $f$  is an increasing function, it indicates that  $b_1 \leq b_2 \leq \ldots \leq b_L \leq b_{L+1}$ . Applying the induction hypothesis for  $a_1, \ldots, a_L$  and  $b_1, \ldots, b_L$ , we find that
$$
\left(\sum_ {i = 1} ^ {L} a _ {i}\right) \left(\sum_ {i = 1} ^ {L} b _ {i}\right) \leq L \sum_ {i = 1} ^ {L} a _ {i} b _ {i}.
$$
This inequality leads to
$$
(\sum_ {i = 1} ^ {L + 1} a _ {i}) (\sum_ {i = 1} ^ {L + 1} b _ {i}) \leq L \sum_ {i = 1} ^ {L} a _ {i} b _ {i} + (\sum_ {i = 1} ^ {L} a _ {i}) b _ {L + 1} + (\sum_ {i = 1} ^ {L} b _ {i}) a _ {L + 1} + a _ {L + 1} b _ {L + 1}
$$
Therefore, to obtain the conclusion of the hypothesis for  $L + 1$ , it is sufficient to demonstrate that
$$
L \sum_ {i = 1} ^ {L} a _ {i} b _ {i} + (\sum_ {i = 1} ^ {L} a _ {i}) b _ {L + 1} + (\sum_ {i = 1} ^ {L} b _ {i}) a _ {L + 1} + a _ {L + 1} b _ {L + 1} \leq (L + 1) (\sum_ {i = 1} ^ {L + 1} a _ {i} b _ {i}),
$$
which is equivalent to show that
$$
\left(\sum_ {i = 1} ^ {L} a _ {i}\right) b _ {L + 1} + \left(\sum_ {i = 1} ^ {L} b _ {i}\right) a _ {L + 1} \leq \sum_ {i = 1} ^ {L} a _ {i} b _ {i} + L a _ {L + 1} b _ {L + 1}. \tag {11}
$$
Since  $a_{L + 1} \geq a_i$  and  $b_{L + 1} \geq b_i$  for all  $1 \leq i \leq L$ , we have  $(a_{L + 1} - a_i)(b_{L + 1} - b_i) \geq 0$ , which is equivalent to  $a_{L + 1}b_{L + 1} + a_ib_i \geq a_{L + 1}b_i + b_{L + 1}a_i$  for all  $1 \leq i \leq L$ . By taking the sum of these inequalities over  $i$  from 1 to  $L$ , we obtain the conclusion of inequality (11). Therefore, we obtain the conclusion of the induction argument for  $L + 1$ , which indicates that inequality (10) holds for all  $L$ . As a consequence, we obtain the inequality  $\mathrm{SW}_p(\mu, \nu) \leq \mathrm{EBSW}_p(\mu, \nu; f)$ .
(b) We recall the definition of the Max-SW:
$$
\operatorname {M a x} - \operatorname {S W} _ {p} (\mu , \nu) = \max  _ {\theta \in \mathbb {S} ^ {d - 1}} W _ {p} \left(\theta_ {\sharp} ^ {\sharp} \mu , \theta_ {\sharp} ^ {\sharp} \nu\right).
$$
Since  $\mathbb{S}^{d - 1}$  is compact and the function  $\theta \to W_{p}(\theta \sharp \mu ,\theta \sharp \nu)$  is continuous, we have  $\theta^{\star} = \operatorname{argmax}_{\theta \in \mathbb{S}^{d - 1}}W_{p}(\theta \sharp \mu ,\theta \sharp \nu)$ . From Definition 2, for any  $p\geq 1$ , dimension  $d\geq 1$ , energy-function  $f$ , and  $\mu ,\nu \in \mathcal{P}_p(\mathbb{R}^d)$  we have:
$$
\begin{array}{l} \operatorname {E B S W} _ {p} (\mu , \nu) = \left(\mathbb {E} _ {\theta \sim \sigma_ {\mu , \nu} (\theta ; f, p))} \left[ W _ {p} ^ {p} \left(\theta_ {p} ^ {\sharp} \mu , \theta_ {p} ^ {\sharp} \nu\right) \right]\right) ^ {\frac {1}{p}} \\ \leq \left(\mathbb {E} _ {\theta \sim \sigma_ {\mu , \nu} (\theta ; f, p)} \left[ W _ {p} ^ {p} \left(\theta^ {\star} \sharp \mu , \theta^ {\star} \sharp \nu\right) \right]\right) ^ {\frac {1}{p}} = W _ {p} ^ {p} \left(\theta^ {*} \sharp \mu , \theta^ {*} \sharp \nu\right) = \operatorname {M a x - S W} _ {p} (\mu , \nu). \\ \end{array}
$$
Furthermore, by applying the Cauchy-Schwartz inequality, we have:
$$
\begin{array}{l} \operatorname {M a x - S W} _ {p} ^ {p} (\mu , \nu) = \max  _ {\theta \in \mathbb {S} ^ {d - 1}} \left(\inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d}} \left| \theta^ {\top} x - \theta^ {\top} y \right| ^ {p} d \pi (x, y)\right) \\ \leq \max  _ {\theta \in \mathbb {S} ^ {d - 1}} \left(\inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| \theta \| ^ {p} \| x - y \| ^ {p} d \pi (x, y)\right) \\ = \inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| \theta \| ^ {p} \| x - y \| ^ {p} d \pi (x, y) \\ = \inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| x - y \| ^ {p} d \pi (x, y) \\ \leq \inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} | x - y | ^ {p} d \pi (x, y) \\ = W _ {p} ^ {p} (\mu , \nu), \\ \end{array}
$$
after taking the  $p$ -rooth, we completes the proof.
We aim to show that for any sequence of probability measures  $(\mu_k)_{k \in \mathbb{N}}$  and  $\mu$  in  $\mathcal{P}_p(\mathbb{R}^d)$ ,  $\lim_{k \to +\infty} \mathrm{EBSW}_p(\mu_k, \mu; \bar{f}) = 0$  if and only if for any continuous and bounded function  $f: \mathbb{R}^d \to \mathbb{R}$ ,  $\lim_{k \to +\infty} \int f \, \mathrm{d}\mu_k = \int f \, \mathrm{d}\mu$ .
We now show that  $\lim_{k\to \infty}\mathrm{EBSW}_p(\mu_k,\mu ;\bar{f}) = 0$  implies  $(\mu_k)_{k\in \mathbb{N}}$  converges weakly to  $\mu$ . Let  $(\mu_k)_{k\in \mathbb{N}}$  be a sequence such that  $\lim_{k\to \infty}\mathrm{EBSW}_p(\mu_k,\mu ;\bar{f}) = 0$ , we suppose  $(\mu_k)_{k\in \mathbb{N}}$  does not converge weakly to  $\mu$ . So, let  $\mathcal{D}_{\mathcal{P}}$  be the Lévy-Prokhorov metric,  $\lim_{k\to \infty}\mathcal{D}_{\mathcal{P}(\mu_k,\mu)}\neq 0$  that implies
there exists  $\varepsilon > 0$  and a subsequence  $\left(\mu_{\psi(k)}\right)_{k \in \mathbb{N}}$  with an increasing function  $\psi: \mathbb{N} \to \mathbb{N}$  such that for any  $k \in \mathbb{N}$ :  $\mathcal{D}_{\mathcal{P}}(\mu_{\psi(k)}, \mu) \geq \varepsilon$ . Using the Hölder inequality with  $\mu, \nu \in \mathbb{P}_p(\mathbb{R}^d)$ , we have:
$$
\begin{array}{l} \operatorname {E B S W} _ {p} (\mu , \nu ; \bar {f}) = \left(\mathbb {E} _ {\theta \sim \sigma_ {\mu , \nu} (\theta ; \bar {f}, p)} \left[ W _ {p} ^ {p} \left(\theta \sharp \mu , \theta \sharp \nu\right) \right]\right) ^ {\frac {1}{p}} \\ \geq \mathbb {E} _ {\theta \sim \sigma_ {\mu , \nu} (\theta ; \bar {f}, p)} [ W _ {p} (\theta \sharp \mu , \theta \sharp \nu) ] \\ \geq \mathbb {E} _ {\theta \sim \mathcal {U} (\mathbb {S} ^ {d - 1})} \left[ W _ {p} \left(\theta \sharp \mu , \theta \sharp \nu\right) \right] \\ \geq \mathbb {E} _ {\theta \sim \mathcal {U} (\mathbb {S} ^ {d - 1})} [ W _ {1} (\theta \sharp \mu , \theta \sharp \nu) ] \\ = \mathrm {S W} _ {1} (\mu , \nu). \\ \end{array}
$$
Therefore,  $\lim_{k\to \infty}\mathrm{SW}_1(\mu_{\psi (k)},\mu) = 0$  which implies that there exists a subsequence  $(\mu_{\phi (\psi (k))})_{k\in \mathbb{N}}$  with an increasing function  $\phi :\mathbb{N}\rightarrow \mathbb{N}$  such that  $(\mu_{\phi (\psi (k))})_{k\in \mathbb{N}}$  converges weakly to  $\mu$  by Lemma S1 in [27]. Therefore a contradiction appears from the assumption of  $\lim_{k\to \infty}\mathcal{D}_{\mathcal{P}}(\mu_{\phi (\psi (k))},\mu)\neq 0$  Therefore,  $\lim_{k\to \infty}\mathrm{EBSW}_p(\mu_k,\mu ;\bar{f}) = 0$ $(\mu_k)_{k\in \mathbb{N}}$  converges weakly to  $\mu$
When  $(\mu_k)_{k\in \mathbb{N}}$  converges weakly to  $\mu$ , we have  $(\theta \sharp \mu_k)_{k\in \mathbb{N}}$  converges weakly to  $\theta \sharp \mu$  for any  $\theta \in \mathbb{S}^{d - 1}$  by the continuous mapping theorem. From [43], the weak convergence implies the convergence under the Wasserstein distance. So, we have  $\lim_{k\to \infty}W_p(\theta \sharp \mu_k,\mu) = 0$ . Moreover, using the fact that the Wasserstein distance is also bounded, hence, the bounded convergence theorem implies:
$$
\begin{array}{l} \lim  _ {k \rightarrow \infty} \operatorname {E B S W} _ {p} ^ {p} (\mu_ {k}, \mu ; \bar {f}) = \lim  _ {k \rightarrow \infty} \mathbb {E} _ {\theta \sim \sigma_ {\mu , \nu} (\theta ; \bar {f}, p)} \left[ W _ {p} ^ {p} \left(\theta \sharp \mu_ {k}, \theta \sharp \mu\right)\right] \\ = \lim  _ {k \rightarrow \infty} \frac {\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ \mathrm {W} _ {p} ^ {p} \left(\theta \sharp \mu_ {k} , \theta \sharp \mu\right) w _ {\mu_ {k} , \mu , \sigma_ {0} , \bar {f} , p} (\theta) \right]}{\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ w _ {\mu_ {k} , \mu , \sigma_ {0} , \bar {f} , p} (\theta) \right]} \\ = 0. \\ \end{array}
$$
Again, using the continuous mapping theorem with function  $x \to x^{1/p}$ , we have  $\lim_{k \to \infty} \mathrm{EBSW}_p(\mu_k, \mu; \bar{f}) \to 0$ . We conclude the proof.
A.4 Proof of Proposition 2
We first show that the following inequality holds
$$
\mathbb {E} \left[ \operatorname {M a x} - \operatorname {S W} _ {p} \left(\mu_ {n}, \mu\right) \right] \leq C \sqrt {(d + 1) \log n / n}
$$
where  $C > 0$  is some universal constant and the outer expectation is taken with respect to the random variables  $X_{1},\ldots ,X_{n}$ . We now follow the proof technique from in [30]. Let  $F_{n,\theta}^{-1}$  and  $F_{\theta}^{-1}$  be the inverse cumulative distributions of two push-forward measures  $\theta \sharp \mu_{n}$  and  $\theta \sharp \mu$ . Using the closed-form expression of the Wasserstein distance in one dimension, we obtain the following equations and inequalities:
$$
\begin{array}{l} \operatorname {M a x - S W} _ {p} ^ {p} (\mu_ {n}, \mu) = \max  _ {\theta \in \mathbb {S} ^ {d - 1}} \int_ {0} ^ {1} | F _ {n, \theta} ^ {- 1} (u) - F _ {\theta} ^ {- 1} (u) | ^ {p} d u \\ = \max  _ {\theta \in \mathbb {R} ^ {d}: \| \theta \| = 1} \int_ {0} ^ {1} \left| F _ {n, \theta} ^ {- 1} (u) - F _ {\theta} ^ {- 1} (u) \right| ^ {p} d u \\ \leq \operatorname {diam}(\mathcal{X})\max_{\theta \in \mathbb{R}^{d}: \| \theta \| \leq 1}|F_{n,\theta}(x) - F_{\theta}(x)|^{p}. \\ \end{array}
$$
where  $\mathcal{X} \subset \mathbb{R}^d$  is the compact set of the probability measure  $\mu$ . We can check that
$$
\max_{\theta \in \mathbb{R}^{d}: \| \theta \| \leq 1}|F_{n,\theta}(x) - F_{\theta}(x)| = \sup_{A\in \mathcal{H}}|\mu_{n}(A) - \mu (A)|,
$$
where  $\mathcal{H}$  is the set of half-spaces  $\{z\in \mathbb{R}^d:\theta^\top z\leq x\}$  for all  $\theta \in \mathbb{R}^d$  such that  $\| \theta \| \leq 1$ . From VC inequality (Theorem 12.5 in [9]), we have:
$$
\mathbb {P} \left(\sup  _ {A \in \mathcal {H}} | \mu_ {n} (A) - \mu (A) | > t\right) \leq 8 S (\mathcal {H}, n) e ^ {- n t ^ {2} / 3 2}.
$$
with  $S(\mathcal{H}, n)$  is the growth function which is upper bounded by  $(n + 1)^{VC(\mathcal{H})}$  due to the Sauer Lemma (Proposition 4.18 in "High-Dimensional Statistics: A Non-Asymptotic Viewpoint", Wainwright et al). From Example 4.21 in [44], we get  $VC(\mathcal{H}) = d + 1$ .
Let  $8S(\mathcal{H},n)e^{-nt^2 /32}\leq \delta$ , we have  $t^2\geq \frac{32}{n}\log \left(\frac{8S(\mathcal{H},n)}{\delta}\right)$ . Therefore, we obtain
$$
\mathbb {P} \left(\sup  _ {A \in \mathcal {H}} | \mu_ {n} (A) - \mu (A) | \leq \sqrt {\frac {3 2}{n} \log \left(\frac {8 S (\mathcal {H} , n)}{\delta}\right)}\right) \geq 1 - \delta ,
$$
Using the Jensen inequality and the tail sum expectation for non-negative random variable, we have:
$$
\begin{array}{l} \mathbb {E} \left[ \sup  _ {A \in \mathcal {H}} | \mu_ {n} (A) - \mu (A) | \right] \\ \leq \sqrt {\mathbb {E} \left[ \sup  _ {A \in \mathcal {H}} | \mu_ {n} (A) - \mu (A) | \right] ^ {2}} = \sqrt {\int_ {0} ^ {\infty} \mathbb {P} \left(\left(\sup  _ {A \in \mathcal {H}} | \mu_ {n} (A) - \mu (A) |\right) ^ {2} > t\right) d t} \\ = \sqrt {\int_ {0} ^ {u} \mathbb {P} \left(\left(\sup  _ {A \in \mathcal {H}} | \mu_ {n} (A) - \mu (A) |\right) ^ {2} > t\right) d t + \int_ {u} ^ {\infty} \mathbb {P} \left(\left(\sup  _ {A \in \mathcal {H}} | \mu_ {n} (A) - \mu (A) |\right) ^ {2} > t\right) d t} \\ \leq \sqrt {\int_ {0} ^ {u} 1 d t + \int_ {u} ^ {\infty} 8 S (\mathcal {H} , n) e ^ {- n t / 3 2} d t} = \sqrt {u + 2 5 6 S (\mathcal {H} , n) \frac {e ^ {- n u / 3 2}}{n}}. \\ \end{array}
$$
Since the inequality holds for any  $u$ , we optimize for  $u$ . Let  $f(u) = u + 256S(\mathcal{H}, n)\frac{e^{-nu/32}}{n}$ , we have  $f'(u) = 1 + 8S(\mathcal{H}, n)e^{-nu/32}$ . Setting  $f'(u) = 0$ , we obtain the minima  $u^{\star} = \frac{32\log(8S(\mathcal{H}, n))}{n}$ . Therefore, we have:
$$
\mathbb {E} \left[ \sup _ {A \in \mathcal {H}} | \mu_ {n} (A) - \mu (A) | \right] \leq \sqrt {f (u ^ {\star})} \leq \sqrt {\frac {3 2 \log (8 S (\mathcal {H} , n))}{n} + 3 2} \leq C \sqrt {\frac {(d + 1) \log (n + 1)}{n}},
$$
by using Sauer Lemma. As a consequence, we obtain:
$$
\mathbb {E} \left[ \mathrm {E B S W} _ {p} \left(\mu_ {n}, \mu ; f\right) \right] \leq C \sqrt {(d + 1) \log n / n},
$$
B Additional Materials
B.1 Additional Background
Sliced Wasserstein. When two probability measures are empirical probability measures on  $n$  supports:  $\mu = \frac{1}{n}\sum_{i=1}^{n}\delta_{x_i}$  and  $\nu = \frac{1}{n}\sum_{i=1}^{n}\delta_{y_i}$ , the SW distance can be computed by sorting projected supports. In particular, we have  $\theta \sharp \mu = \frac{1}{n}\sum_{i=1}^{n}\delta_{\theta^\top x_i}$ ,  $\theta \sharp \nu = \frac{1}{n}\sum_{i=1}^{n}\delta_{\theta^\top y_i}$ , and  $\mathbf{W}_p^p(\theta \sharp \mu, \theta \sharp \nu) = \frac{1}{n}\sum_{i=1}^{n}(\theta^\top x_{(i)} - \theta^\top y_{(i)})^p$  where  $\theta^\top x_{(i)}$  is the ordered projected supports. We provide the pseudo-code for computing the SW in Algorithm 1.
Max sliced Wasserstein. The Max-SW is often computed by the projected gradient ascent. The sub-gradient is used when the one-dimensional optimal matching is not unique e.g., in discrete cases. We provide the projected (sub)-gradient ascent algorithm for computing the Max-SW in Algorithm 2. Compared to the SW, the Max-SW needs two hyperparameters which are the number of iterations  $T$  and the step size  $\eta$ . Moreover, the empirical estimation of the Max-SW might not converge to the Max-SW when  $T \to \infty$ .
Distributional sliced Wasserstein. To solve the optimization of the DSW, we need to use the stochastic (sub)-gradient ascent algorithm. In particular, we first need to estimate the gradient:
$$
\nabla_ {\psi} \left(\mathbb {E} _ {\theta \sim \sigma_ {\psi} (\theta)} \mathbf {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu)\right) ^ {\frac {1}{p}} = \frac {1}{p} \left(\mathbb {E} _ {\theta \sim \sigma_ {\psi} (\theta)} \mathbf {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu)\right) ^ {\frac {1 - p}{p}} \nabla_ {\psi} \mathbb {E} _ {\theta \sim \sigma_ {\psi} (\theta)} \mathbf {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu).
$$
To estimate the gradient  $\nabla_{\psi}\mathbb{E}_{\theta \sim \sigma_{\psi}(\theta)}\mathrm{W}_p^p (\theta \sharp \mu ,\theta \sharp \nu)$ , we need to use reparameterization trick for  $\sigma_{\psi}(\theta)$  e.g., the vMF distribution. After using the reparameterization trick, we can approximate the gradient  $\nabla_{\psi}\mathbb{E}_{\theta \sim \sigma_{\psi}(\theta)}\mathrm{W}_p^p (\theta \sharp \mu ,\theta \sharp \nu) = \frac{1}{L}\sum_{l = 1}^{l}\nabla_{\psi}\mathrm{W}_p^p (\theta_{l,\psi}\sharp \mu ,\theta_{l,\psi}\sharp \nu)$  where  $\theta_{1,\psi},\ldots ,\theta_{L,\psi}$  are i.i.d reparameterized samples from  $\sigma_{\psi}(\theta)$ . Similarly, we approximate  $\mathbb{E}_{\theta \sim \sigma_{\psi}(\theta)}\mathrm{W}_p^p (\theta \sharp \mu ,\theta \sharp \nu) = \frac{1}{L}\sum_{l = 1}^{L}\mathrm{W}_p^p (\theta_l\sharp \mu ,\theta_l\sharp \nu)$ . We refer to the details in the following papers [6, 32]. We review the algorithm for computing the DSW in Algorithm 3. Compared to the SW, the DSW needs three hyperparameters i.e., the number of projections  $L$ , the number of iterations  $T$ , and the step size  $\eta$ .
Minimum Distance Estimator and Gradient Estimation. In statistical inference, we are given the empirical samples  $X_{1},\ldots ,X_{n}$  from the interested distribution  $\nu$ . Since we do not know the form of  $\nu$ , we might want to find an alternative representation. In particular, we want to find the best member  $\mu_{\phi}$  in a family of distribution parameterized by  $\phi \in \Phi$ . To do that, we want to minimize the
distance between  $\mu_{\phi}$  and the empirical distribution  $\nu_{n} = \frac{1}{n}\sum_{i = 1}^{n}\delta_{X_{i}}$ . This framework is named the minimum distance estimator [45]:
$\min_{\phi \in \Phi}\mathcal{D}(\mu_{\phi},\nu_{n})$  , where  $\mathcal{D}$  is a discrepancy between two distributions. The gradient ascent algorithm is often used to solve the problem. To do so, we need to compute the gradient  $\nabla_{\phi}\mathcal{D}(\mu_{\phi},\nu_{n})$  When using sliced Wasserstein distances, the gradient  $\nabla_{\phi}\mathcal{D}(\mu_{\phi},\nu_{n})$  is often approximated by a stochastic gradient since the SW distances involve an intractable expectation. In previous SW variants, the expectation does not depend on  $\phi$  , hence, we can use directly the Leibniz rule to exchange the gradient and the expectation, then perform the Monte Carlo approximation. In particular, we have  $\nabla_{\phi}\mathbb{E}_{\theta \sim \sigma (\theta)}[\mathrm{W}_p^p (\theta \sharp \mu ,\theta \sharp \nu)] = \mathbb{E}_{\theta \sim \sigma (\theta)}[\nabla_{\phi}\mathrm{W}_p^p (\theta \sharp \mu ,\theta \sharp \nu)]\approx \frac{1}{L}\sum_{l = 1}^{L}\nabla_{\phi}\mathrm{W}_p^p (\theta_l\sharp \mu ,\theta_l\sharp \nu)$  for  $\theta_1,\ldots ,\theta_L\stackrel {i.i.d}{\sim}\sigma (\theta).$
B.2 Importance Sampling
Derivation. We first provide the derivation of the importance sampling estimation of EBSW. From the definition of the EBSW, we have:
$$
\begin{array}{l} \operatorname {E B S W} _ {p} (\mu , \nu ; f) = \left(\mathbb {E} _ {\theta \sim \sigma_ {\mu , \nu} (\theta ; f, p)} \left[ \mathrm {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu) \right]\right) ^ {\frac {1}{p}} \\ = \left(\frac {\int_ {\mathbb {S} ^ {d - 1}} \mathrm {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu) f (\mathrm {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu)) d \theta}{\int_ {\mathbb {S} ^ {d - 1}} f (\mathrm {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu)) d \theta}\right) ^ {\frac {1}{p}} \\ = \left(\frac {\int_ {\mathbb {S} ^ {d - 1}} \mathrm {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu) \frac {f \left(\mathrm {W} _ {p} ^ {p} \left(\theta \sharp \mu , \theta \sharp \nu\right)\right)}{\sigma_ {0} (\theta)} \sigma_ {0} (\theta) d \theta}{\int_ {\mathbb {S} ^ {d - 1}} \frac {f \left(\mathrm {W} _ {p} ^ {p} \left(\theta \sharp \mu , \theta \sharp \nu\right)\right)}{\sigma_ {0} (\theta)} \sigma_ {0} (\theta) d \theta}\right) ^ {\frac {1}{p}} \\ = \left(\frac {\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ \mathrm {W} _ {p} ^ {p} \left(\theta_ {\sharp} ^ {\sharp} \mu , \theta_ {\sharp} ^ {\sharp} \nu\right) \frac {f \left(\mathrm {W} _ {p} ^ {p} \left(\theta_ {\sharp} ^ {\sharp} \mu , \theta_ {\sharp} ^ {\sharp} \nu\right)\right)}{\sigma_ {0} (\theta)} \right]}{\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ \frac {f \left(\mathrm {W} _ {p} ^ {p} \left(\theta_ {\sharp} ^ {\sharp} \mu , \theta_ {\sharp} ^ {\sharp} \nu\right)\right)}{\sigma_ {0} (\theta)} \right]}\right) ^ {\frac {1}{p}} \\ = \left(\frac {\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ \mathbf {W} _ {p} ^ {p} (\theta \sharp \mu , \theta \sharp \nu) w _ {\mu , \nu , \sigma_ {0}} (\theta) \right]}{\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ w _ {\mu , \nu , \sigma_ {0}} (\theta) \right]}\right) ^ {\frac {1}{p}}. \\ \end{array}
$$
Algorithms. We provide the algorithm for the IS estimation of the EBSW in Algorithm 4. Compared to the algorithm of the SW in Algorithm 1, the IS-EBSW can be obtained by only adding one or two lines of code in practice. Therefore, the computation of the IS-EBSW is as fast as the SW while being more meaningful.
Gradient Estimators. Let  $\mu_{\phi}$  be parameterized by  $\phi$ , we derive now the gradient estimator  $\nabla_{\phi} \mathrm{EBSW}_p(\mu, \nu; f)$  through importance sampling. We have:
$$
\begin{array}{l} \nabla_ {\phi} \mathrm {E B S W} _ {p} (\mu_ {\phi}, \nu ; f) = \frac {1}{p} \left(\frac {\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ \mathrm {W} _ {p} ^ {p} (\theta \sharp \mu_ {\phi} , \theta \sharp \nu) w _ {\mu_ {\phi} , \nu , \sigma_ {0} , f} (\theta) \right]}{\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ w _ {\mu_ {\phi} , \nu , \sigma_ {0} , f} (\theta) \right]}\right) ^ {\frac {1 - p}{p}} \\ \nabla_ {\phi} \frac {\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ \mathbf {W} _ {p} ^ {p} (\theta \sharp \mu_ {\phi} , \theta \sharp \nu) w _ {\mu_ {\phi} , \nu , \sigma_ {0} , f} (\theta) \right]}{\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ w _ {\mu_ {\phi} , \nu , \sigma_ {0} , f} (\theta) \right]}. \\ \end{array}
$$
We denote  $A(\phi) = \mathbb{E}_{\theta \sim \sigma_0(\theta)}\left[\mathrm{W}_p^p (\theta \sharp \mu_\phi ,\theta \sharp \nu)w_{\mu_\phi ,\nu ,\sigma_0,f}(\theta)\right], B(\phi) = \mathbb{E}_{\theta \sim \sigma_0(\theta)}\left[w_{\mu_\phi ,\nu ,\sigma_0,f}(\theta)\right]$ , we have
$$
\nabla_ {\phi} \frac {A (\phi)}{B (\phi)} = \frac {B (\phi) \nabla_ {\phi} A (\phi) - A (\phi) \nabla \phi B (\phi)}{B ^ {2} (\phi)}.
$$
Using Monte Carlo samples  $\theta_{1},\ldots ,\theta_{L}\sim \sigma_{0}(\theta)$  after using the Lebnitz rule to exchange the differentiation and the expectation, we obtain:
$$
\left(\frac {\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ \mathbf {W} _ {p} ^ {p} (\theta \sharp \mu_ {\phi} , \theta \sharp \nu) w _ {\mu_ {\phi} , \nu , \sigma_ {0} , f} (\theta) \right]}{\mathbb {E} _ {\theta \sim \sigma_ {0} (\theta)} \left[ w _ {\mu_ {\phi} , \nu , \sigma_ {0} , f} (\theta) \right]}\right) ^ {\frac {1 - p}{p}} \approx \left(\frac {\frac {1}{L} \sum_ {l = 1} ^ {L} \left[ \mathbf {W} _ {p} ^ {p} (\theta_ {l} \sharp \mu_ {\phi} , \theta_ {l} \sharp \nu) w _ {\mu_ {\phi} , \nu , \sigma_ {0} , f} (\theta_ {l}) \right]}{\frac {1}{L} \sum_ {l = 1} ^ {L} \left[ w _ {\mu_ {\phi} , \nu , \sigma_ {0} , f} (\theta_ {l}) \right]}\right) ^ {\frac {1 - p}{p}},
$$
$$
\nabla_ {\phi} A (\phi) \approx \frac {1}{L} \sum_ {l = 1} ^ {L} \nabla_ {\phi} \left(\mathbf {W} _ {p} ^ {p} (\theta_ {l} \sharp \mu_ {\phi}, \theta_ {l} \sharp \nu) w _ {\mu_ {\phi}, \nu , \sigma_ {0}, f} (\theta)\right),
$$
$$
\nabla_ {\phi} B (\phi) \approx \frac {1}{L} \sum_ {l = 1} ^ {L} \nabla_ {\phi} w _ {\mu_ {\phi}, \nu , \sigma_ {0}, f} (\theta),
$$
which yields the gradient estimation. If we construct the slicing distribution by using a copy of  $\mu_{\phi}$  i.e.,  $\mu_{\phi'}$  with  $\phi' = \phi$  in terms of value, the gradient estimator can be derived by: