SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now continue the proof of Theorem 2. We first show that if  $\lim_{k\to \infty}\mathrm{MSW}_{p,T}(\mu_k,\mu) = 0$ ,  $(\mu_k)_{k\in \mathbb{N}}$  converges weakly to  $\mu$ . We consider a sequence  $(\mu_{\phi (k)})_{k\in \mathbb{N}}$  such that  $\lim_{k\to \infty}\mathrm{MSW}_{p,T}(\mu_k,\mu) = 0$  and we suppose  $(\mu_{\phi (k)})_{k\in \mathbb{N}}$  does not converge weakly to  $\mu$ . Therefore, let  $d_{\mathcal{P}}$  be the Lévy-Prokhorov metric,  $\lim_{k\to \infty}d_{\mathcal{P}}(\mu_k,\mu)\neq 0$  that implies there exists  $\varepsilon >0$  and a subsequence  $(\mu_{\psi (k)})_{k\in \mathbb{N}}$  with an increasing function  $\psi :\mathbb{N}\rightarrow \mathbb{N}$  such that for any  $k\in \mathbb{N}$ :  $d_{\mathcal{P}}(\mu_{\psi (k)},\mu)\geq \varepsilon$ . However, we have
$$
\begin{array}{l} \mathrm {M S W} _ {p, T} (\mu , \nu) = \left(\mathbb {E} _ {(\theta_ {1: T}) \sim \sigma (\theta_ {1: T})} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} W _ {p} ^ {p} \left(\theta_ {t} \sharp \mu , \theta_ {t} \sharp \nu\right) \right]\right) ^ {\frac {1}{p}} \\ \geq \mathbb {E} _ {\left(\theta_ {1: T}\right) \sim \sigma \left(\theta_ {1: T}\right)} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} W _ {p} \left(\theta_ {t} \sharp \mu , \theta_ {t} \sharp \nu\right) \right] \\ \geq \mathbb {E} _ {\left(\theta_ {1: T}\right) \sim \sigma \left(\theta_ {1: T}\right)} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} W _ {1} \left(\theta_ {t} \sharp \mu , \theta_ {t} \sharp \nu\right) \right] = \mathrm {M S W} _ {1, T} (\mu , \nu), \\ \end{array}
$$
by the Holder inequality with  $\mu, \nu \in \mathbb{P}_p(\mathbb{R}^d)$ . Therefore,  $\lim_{k\to \infty}\mathrm{MSW}_{1,T}(\mu_{\psi (k)},\mu) = 0$  which implies that there exists a subsequence  $(\mu_{\phi (\psi (k))})_{k\in \mathbb{N}}$  with an increasing function  $\phi :\mathbb{N}\rightarrow \mathbb{N}$  such that  $(\mu_{\phi (\psi (k))})_k\in \mathbb{N}$  converges weakly to  $\mu$  by Lemma 1. Hence,  $\lim_{k\to \infty}d_{\mathcal{P}}(\mu_{\phi (\psi (k))},\mu) = 0$  which contradicts our assumption. We conclude that if  $\lim_{k\to \infty}\mathrm{MSW}_{p,T}(\mu_k,\mu) = 0$ ,  $(\mu_k)_{k\in \mathbb{N}}$  converges weakly to  $\mu$ .
Now, we show that if  $(\mu_k)_{k\in \mathbb{N}}$  converges weakly to  $\mu$ , we have  $\lim_{k\to \infty}\mathrm{MSW}_{p,T}(\mu_k,\mu) = 0$ . By the continuous mapping theorem, we obtain  $(\theta \sharp \mu_k)_{k\in \mathbb{N}}$  converges weakly to  $\theta \sharp \mu$  for any  $\theta \in \mathbb{S}^{d - 1}$ . Since the weak convergence implies the convergence under the Wasserstein distance [58], we obtain  $\lim_{k\to \infty}W_p(\theta \sharp \mu_k,\mu) = 0$ . Moreover, the Wasserstein distance is also bounded, hence the bounded convergence theorem:
$$
\begin{array}{l} \lim  _ {k \rightarrow \infty} \mathbf {M S W} _ {p, T} ^ {p} (\mu_ {k}, \mu) = \mathbb {E} _ {(\theta_ {1: T}) \sim \sigma (\theta_ {1: T})} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} W _ {p} ^ {p} \left(\theta_ {t} \sharp \mu_ {k}, \theta_ {t} \sharp \mu\right)\right] \\ = \mathbb {E} _ {(\theta_ {1: T}) \sim \sigma (\theta_ {1: T})} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} 0 \right] = 0. \\ \end{array}
$$
By the continuous mapping theorem with function  $x \to x^{1/p}$ , we obtain  $\lim_{k \to \infty} \mathrm{MSW}_{p,T}(\mu_k, \mu) \to 0$  which completes the proof.
B.3 Proof of Proposition 1
(i) We recall the definition of Max-SW:
$$
\operatorname {M a x - S W} _ {p} (\mu , \nu) = \max  _ {\theta \in \mathbb {S} ^ {d - 1}} W _ {p} \left(\theta_ {\sharp} ^ {\sharp} \mu , \theta_ {\sharp} ^ {\sharp} \nu\right).
$$
Let  $\theta^{*} = \operatorname{argmax}_{\theta \in \mathbb{S}^{d - 1}} W_{p}(\theta \sharp \mu, \theta \sharp \nu)$ , from Definition 1, for any  $p \geq 1$ ,  $T \geq 1$ , dimension  $d \geq 1$ , and  $\mu, \nu \in \mathcal{P}_p(\mathbb{R}^d)$  we have:
$$
\begin{array}{l} \mathrm {M S W} _ {p, T} (\mu , \nu) = \left(\mathbb {E} _ {\left(\theta_ {1: T}\right) \sim \sigma \left(\theta_ {1: T}\right)} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} W _ {p} ^ {p} \left(\theta_ {t} \sharp \mu , \theta_ {t} \sharp \nu\right) \right]\right) ^ {\frac {1}{p}} \\ \leq \left(\frac {1}{T} \sum_ {t = 1} ^ {T} W _ {p} ^ {p} \left(\theta^ {*} \sharp \mu , \theta^ {*} \sharp \nu\right)\right) ^ {\frac {1}{p}} = W _ {p} \left(\theta^ {*} \sharp \mu , \theta^ {*} \sharp \nu\right) = \operatorname {M a x - S W} _ {p} (\mu , \nu). \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {M a x - S W} _ {p} ^ {p} (\mu , \nu) = \max  _ {\theta \in \mathbb {S} ^ {d - 1}} \left(\inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d}} \left| \theta^ {\top} x - \theta^ {\top} y \right| ^ {p} d \pi (x, y)\right) \\ \leq \max  _ {\theta \in \mathbb {S} ^ {d - 1}} \left(\inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| \theta \| ^ {p} \| x - y \| ^ {p} d \pi (x, y)\right) \\ = \inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| \theta \| ^ {p} \| x - y \| ^ {p} d \pi (x, y) \\ = \inf  _ {\pi \in \Pi (\mu , \nu)} \int_ {\mathbb {R} ^ {d} \times \mathbb {R} ^ {d}} \| x - y \| ^ {p} d \pi (x, y) \\ = W _ {p} ^ {p} (\mu , \nu), \\ \end{array}
$$
(ii) This result can be directly obtained from the definitions of MSW and SW.
B.4 Proof of Proposition 2
In this proof, we denote  $\Theta \subset \mathbb{R}^d$  as the compact set of the probability measure  $\mu$ . From Proposition 1, we find that
$$
\mathbb {E} \left[ \mathrm {M S W} _ {p, T} \left(\mu_ {n}, \mu\right) \right] \leq \mathbb {E} \left[ \operatorname {M a x} - \mathrm {S W} _ {p} \left(\mu_ {n}, \mu\right) \right].
$$
Therefore, the proposition follows as long as we can demonstrate that
$$
\mathbb {E} \left[ \operatorname {M a x} - \operatorname {S W} _ {p} \left(\mu_ {n}, \mu\right) \right] \leq C \sqrt {(d + 1) \log_ {2} n / n}
$$
where  $C > 0$  is some universal constant and the outer expectation is taken with respect to the data. The proof for this result follows from the proof of Proposition 3 in [43]. Here, we provide the proof for the completeness. By defining  $F_{n,\theta}$  and  $F_{\theta}$  as the cumulative distributions of  $\theta \sharp \mu_n$  and  $\theta \sharp \mu$ , the closed-form expression of the Wasserstein distance in one dimension leads to the following equations and inequalities:
$$
\begin{array}{l} \operatorname {M a x - S W} _ {p} ^ {p} (\mu_ {n}, \mu) = \max  _ {\theta \in \mathbb {S} ^ {d - 1}} \int_ {0} ^ {1} | F _ {n, \theta} ^ {- 1} (u) - F _ {\theta} ^ {- 1} (u) | ^ {p} d u \\ = \max  _ {\theta \in \mathbb {R} ^ {d}: \| \theta \| = 1} \int_ {0} ^ {1} | F _ {n, \theta} ^ {- 1} (u) - F _ {\theta} ^ {- 1} (u) | ^ {p} d u \\ \leq \operatorname {diam}(\Theta)\max_{\theta \in \mathbb{R}^{d}: \| \theta \| \leq 1}|F_{n,\theta}(x) - F_{\theta}(x)|^{p}. \\ \end{array}
$$
We can check that
$$
\max  _ {\theta \in \mathbb {R} ^ {d}: \| \theta \| \leq 1} | F _ {n, \theta} (x) - F _ {\theta} (x) | = \sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) |,
$$
where  $\mathcal{B}$  is the set of half-spaces  $\{z\in \mathbb{R}^d:\theta^\top z\leq x\}$  for all  $\theta \in \mathbb{R}^d$  such that  $\| \theta \| \leq 1$ . From VC inequality (Theorem 12.5in [16]), we have
$$
\mathbb {P} \left(\sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) | > t\right) \leq 8 S (\mathcal {B}, n) e ^ {- n t ^ {2} / 3 2}.
$$
with  $S(\mathcal{B}, n)$  is the growth function which is upper bounded by  $(n + 1)^{VC(\mathcal{B})}$  due to the Sauer Lemma (Proposition 4.18 in [59]). From Example 4.21 in [59], we get  $VC(\mathcal{B}) = d + 1$ .
Let  $8S(\mathcal{B},n)e^{-nt^2 /32}\leq \delta$ , we have  $t^2\geq \frac{32}{n}\log \left(\frac{8S(\mathcal{B},n)}{\delta}\right)$ . Therefore, we obtain
$$
\mathbb {P} \left(\sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) | \leq \sqrt {\frac {3 2}{n} \log \left(\frac {8 S (\mathcal {B} , n)}{\delta}\right)}\right) \geq 1 - \delta ,
$$
Now we convert the probability statement to inequality with expectation. Using the Jensen inequality and the tail sum expectation for non-negative random variable, we have:
$$
\begin{array}{l} \mathbb {E} \left[ \sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) | \right] \\ \leq \sqrt {\mathbb {E} \left[ \sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) | \right] ^ {2}} = \sqrt {\int_ {0} ^ {\infty} \mathbb {P} \left(\left(\sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) |\right) ^ {2} > t\right) d t} \\ = \sqrt {\int_ {0} ^ {u} \mathbb {P} \left(\left(\sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) |\right) ^ {2} > t\right) d t + \int_ {u} ^ {\infty} \mathbb {P} \left(\left(\sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) |\right) ^ {2} > t\right) d t} \\ \leq \sqrt {\int_ {0} ^ {u} 1 d t + \int_ {u} ^ {\infty} 8 S (\mathcal {B} , n) e ^ {- n t / 3 2} d t} = \sqrt {u + 2 5 6 S (\mathcal {B} , n) \frac {e ^ {- n u / 3 2}}{n}}. \\ \end{array}
$$
Let  $f(u) = u + 256S(\mathcal{B},n)\frac{e^{-nu / 32}}{n}$ , we have  $f^{\prime}(u) = 1 + 8S(\mathcal{B},n)e^{-nu / 32}$ , hence the minima  $u^{\star} = \frac{32\log(8S(\mathcal{B},n))}{n}$ . Since the inequality holds for any  $u$ , we have:
$$
\mathbb {E} \left[ \sup  _ {B \in \mathcal {B}} | \mu_ {n} (B) - \mu (B) | \right] \leq \sqrt {\frac {3 2 \log (8 S (\mathcal {B} , n))}{n} + 3 2} \leq C \sqrt {\frac {(d + 1) \log (n + 1)}{n}},
$$
by using Sauer Lemma. Putting the above results together leads to
$$
\mathbb {E} \left[ \operatorname {M a x} - \operatorname {S W} _ {p} \left(\mu_ {n}, \mu\right) \right] \leq C \sqrt {(d + 1) \log_ {2} n / n},
$$
where  $C > 0$  is some universal constant. As a consequence, we obtain the conclusion of the proposition.
B.5 Proof of Proposition 3
For any  $p \geq 1$ ,  $T \geq 1$ , dimension  $d \geq 1$ , and  $\mu, \nu \in \mathcal{P}_p(\mathbb{R}^d)$ , using the Holder's inequality, we have:
$$
\begin{array}{l} \mathbb {E} \left| \widehat {\mathbf {M S W}} _ {p, T} ^ {p} (\mu , \nu) - \mathbf {M S W} _ {p, T} ^ {p} (\mu , \nu) \right| \\ \leq \left(\mathbb {E} \widehat {\left| \mathbf {M S W} _ {p , T} ^ {p} (\mu , \nu) - \mathbf {M S W} _ {p , T} ^ {p} (\mu , \nu) \right| ^ {2}}\right) ^ {\frac {1}{2}} \\ = \left(\mathbb {E} \left| \frac {1}{T L} \sum_ {t = 1} ^ {T} \sum_ {l = 1} ^ {L} \mathrm {W} _ {p} ^ {p} \left(\theta_ {t l} \sharp \mu , \theta_ {t l} \sharp \nu\right) - \mathbb {E} _ {\theta_ {1: T} \sim \sigma \left(\theta_ {1: T}\right)} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} W _ {p} ^ {p} \left(\theta_ {t} \sharp \mu , \theta_ {t} \sharp \nu\right) \right] \right| ^ {2}\right) ^ {\frac {1}{2}} \\ = \left(V a r \left[ \frac {1}{T L} \sum_ {t = 1} ^ {T} \sum_ {l = 1} ^ {L} W _ {p} ^ {p} \left(\theta_ {t l} \sharp \mu , \theta_ {t l} \sharp \nu\right) \right]\right) ^ {\frac {1}{2}} \\ = \frac {1}{T \sqrt {L}} \operatorname {V a r} \left[ \sum_ {t = 1} ^ {T} W _ {p} ^ {p} \left(\theta_ {t} \sharp \mu , \theta_ {t} \sharp \nu\right) \right] ^ {\frac {1}{2}}, \\ \end{array}
$$
Algorithm 7 Gradient flow with the Euler scheme
Input. the start distribution  $\mu = \frac{1}{n}\sum_{i=1}^{n}\delta_{X_i}$ , the target distribution  $\nu = \frac{1}{n}\sum_{i=1}^{n}\delta_{Y_i}$ , number of Euler iterations  $T$  (abuse of notation), Euler step size  $\eta$  (abuse of notation), a metric  $\mathcal{D}$ .
$$
X = X - n \cdot \eta \nabla_ {X} \mathcal {D} \left(P _ {X}, P _ {Y}\right)
$$
Output.  $\mu = \frac{1}{n}\sum_{i = 1}^{n}\delta_{X_i}$
Figure 4: The figures show the gradient flows that are from the empirical distribution over the color points to the empirical distribution over S-shape points produced by different distances. The corresponding Wasserstein-2 distance between the empirical distribution at the current step and the S-shape distribution and the computational time (in second) to reach the step is reported at the top of the figure.
In this section, we present the detail of experimental frameworks and additional experiments on gradient flows, color transfer, and deep generative modeling which are not in the main paper.
C.1 Gradient Flows
**Framework.** We have discussed in detail the framework of gradient flow in Section 4.1 in the main paper. Here, we summarize the Euler scheme for solving the gradient flow in Algorithm 7.
Visualization of gradient flows. We show the visualization of gradient flows from all distances (Table 1) in Figure 4. Overall, we observe that the quality of the flows is consistent with the quantitative Wasserstein-2 score which is computed using [18]. From the figures, we see that iMSW and viMSW help the flows converge very fast. Namely, Wasserstein-2 scores at steps 200 of iMSW and viMSW are much lower than other distances. For oMSW, with  $L = 5$ ,  $T = 2$ , it achieves a comparable result to SW, K-SW, and Max-SW while being faster.
Studies on hyper-parameters. We run gradient flows with different values of hyper-parameters and report the Wasserstein-2 scores and computational time in Table 3. From the table and Figure 4, we
Table 3: Summary of Wasserstein-2 scores, computational time in second (s) of different distances in gradient flow application.
Distances	Wasserstein-2 (↓)	Time (↓)	Distances	Wasserstein-2 (↓)	Time (↓)
SW (L=10)	0.0113 × 10-2	0.85	SW (L=100)	0.0096 × 10-2	4.32
Max-SW (T=5)	0.0231 × 10-2	1.02	Max-SW (T=100)	0.0083 × 10-2	10.46
K-SW (L=5,K=2)	0.0104 × 10-2	0.92	K-SW (L=20,K=2)	0.0096 × 10-2	1.97
Max-K-SW (K=2,T=5)	0.0152 × 10-2	1.41	Max-K-SW (K=2,T=100)	0.0083 × 10-2	10.46
iMSW (L=1,T=5)	0.0109 × 10-2	1.07	iMSW (L=5,T=5)	0.0055 × 10-2	2.44
iMSW (L=2,T=10)	0.0052 × 10-2	2.79	iMSW (L=5,T=2)	0.0071 × 10-2	1.14
iMSW (L=2,T=5,M=4)	0.0101 × 10-2	1.2	iMSW (L=2,T=5,M=2)	0.0055 × 10-2	1.25
iMSW (L=2,T=5,M=0,N=2)	0.0066 × 10-2	1.28	iMSW (L=2,T=5,M=2,N=2)	0.0072 × 10-2	1.19
viMSW (L=2,T=5,κ=10)	0.0052 × 10-2	3.12	viMSW (L=2,T=5,κ=100)	0.0053 × 10-2	2.76
Figure 5: The figures show the source image, the target image, and transferred images from different distances. The corresponding Wasserstein-2 distance between the empirical distribution over transferred color palates and the empirical distribution over the target color palette and the computational time (in second) is reported at the top of the figure. The color palates are given below the corresponding images.
see that SW with  $L = 10$  is worse than oMSW, iMSW, and viMSW with  $L = 2$ ,  $T = 5$  (10 total projections). Increasing the number of projections to 100, SW gets better, however, its Wasserstein-2 score is still higher than the scores of iMSW and viMSW while its computational time is bigger. Similarly, Max-(K)-SW with  $T = 100$  is better than Max-(K)-SW with  $T = 5$  and  $T = 10$ , however, it is still worse than iMSW and viMSW in terms of computation and performance. For burning and thinning, we see that the technique can help improve the computation considerably. More importantly, the burning and thinning techniques do not reduce the performance too much. For iMSW, increasing  $L$  and  $T$  leads to a better flow. For the same number of total projections e.g., 10,  $L = 2$ ,  $T = 5$  is better than  $L = 5$ ,  $T = 2$ . For viMSW, it usually performs better than iMSW, however, its computation is worse due to the sampling complexity of the vMF distribution. We vary the concentration parameter  $\kappa \in \{10, 50, 100\}$  and find that  $\kappa = 50$  is the best. Hence, it might suggest that a good balance between heading to the "max" projecting direction and exploring the space of projecting directions is the best strategy.
Framework. In our experiments, we first compress the color palette of the source image and the target image to 3000 colors by using K-Mean clustering. After that, the color transfer application is
Figure 6: The figures show the source image, the target images, and transferred images from different distances. The corresponding Wasserstein-2 distance between the empirical distribution over transferred color palates and the empirical distribution over the target color palette and the computational time (in second) is reported at the top of the figure. The color palates are given below the corresponding images.
Algorithm 8 Color Transfer
Input. source color palette  $X \in \{0, \dots, 255\}^{n \times 3}$ , target color palette  $Y \in \{0, \dots, 255\}^{n \times 3}$ , number of Euler iterations  $T$  (abuse of notation), Euler step size  $\eta$  (abuse of notation), a metric  $\mathcal{D}$ .
$$
X = \operatorname {r o u n d} (X, \{0, \dots , 2 5 5 \})
$$
Output.  $X$
conducted by using Algorithm 8 which is a modified version of the gradient flow algorithm since the color palette contains only positive integer in  $\{0, \dots, 255\}$ . The flow can be seen as an incomplete transportation map that maps from the source color palette to a color palette that is close to the target color palette. This is quite similar to the iterative distribution transfer algorithm [8], however, the construction of the iterative map is different.
Visualization of transferred images. We show the source image, the target image, and the corresponding transferred images from distances in Figure 5 and Figure 6. The color palates are given below the corresponding images. The corresponding Wasserstein-2 distance between the empirical distribution over transferred color palates and the empirical distribution over the target color palette and the computational time (in second) is reported at the top of the figure. First, we observe that
Table 4: Summary of Wasserstein-2 scores, computational time in second (s) of different distances in the color transfer application.
Distances	Wasserstein-2 (↓)	Time (↓)	Distances	Wasserstein-2 (↓)	Time (↓)
SW (L=45)	414.51	41.77	SW (L=15)	421.5	12.96
Max-SW (T=45)	449.42	59.13	Max-SW (T=15)	450.37	19.03
K-SW (L=15,K=3)	411.74	38.86	K-SW (L=5,K=3)	413.16	14.2
Max-K-SW (K=3,T=15)	479.43	53.95	Max-K-SW (K=3,T=5)	510.43	17.46
oMSW (L=3,T=5)	415.06	13.69	oMSW (L=3,T=15)	414.29	38.51
iMSW (L=3,T=5)	16.97	25.91	iMSW (L=3,T=15)	15.23	79.47
iMSW (L=5,T=5)	21.63	39.82	iMSW (L=5,T=3)	24.02	22.27
iMSW (L=3,T=15,M=14)	26.23	48.08	iMSW (L=3,T=15,M=10)	18.67	55.55
iMSW (L=3,T=15,M=0,N=2)	16.6	62.66	iMSW (L=3,T=15,M=10,N=2)	19.2	50.1
viMSW (L=3,T=5,κ=50)	16.48	29.14	viMSW (L=3,T=5,κ=100)	16.49	29.06
the qualitative comparison (transferred images and color palette) is consistent with the Wasserstein scores. We observe that iMSW and viMSW have their transferred images closer to the target image in terms of color than other distances. More importantly, iMSW and viMSW are faster than other distances. Max-SW and Max-K-SW do not perform well in this application, namely, they are slow and give high Wasserstein distances. For oMSW, it is comparable to SW and K-SW while being faster.
Studies on hyper-parameters. In addition to result in Figure 5, we run color transfer with other settings of distances in Table 4. From the table, increasing the number of projections  $L$  lead to a better result for SW and K-SW. However, they are still worse than iMSW and viMSW with a smaller number of projections. Similarly, increasing  $T$  helps Max-SW, Max-K-SW, and iMSW better. As discussed in the main paper, the burning and thinning technique improves the computation and sometimes enhances the performance.
C.3 Deep Generative Models
**Framework.** We follow the generative modeling framework from [20, 42]. Here, we state an adaptive formulation of the framework. We are given a data distribution  $\mu \in \mathcal{P}(\mathcal{X})$  through its random samples (data). Our goal is to estimate a parametric distribution  $\nu_{\phi}$  that belongs to a family of distributions indexed by parameters  $\phi$  in a parameter space  $\Phi$ . Deep generative modeling is interested in constructing  $\nu_{\phi}$  via pushforward measure. In particular,  $\nu_{\phi}$  is implicitly represented by pushing forward a random noise  $\nu_0 \in \mathcal{P}(\mathcal{Z})$  e.g., standard multivariable Gaussian, through a parametric function  $G_{\phi}: \mathcal{Z} \to \mathcal{X}$  (a neural network with weights  $\phi$ ). To estimate  $\phi(\nu_{\phi})$ , the expected distance estimator [54, 41] is used:
$$
\operatorname * {a r g m i n} _ {\phi \in \Phi} \mathbb {E} _ {(X, Z) \sim \mu^ {\otimes m} \otimes \nu_ {0} ^ {\otimes m}} [ \mathcal {D} (P _ {X}, P _ {G _ {\phi} (Z)}) ],
$$
where  $m \geq 1$ ,  $\mathcal{D}$  can be any distance on space of probability measures,  $\mu^{\otimes}$  is the product measures, namely,  $X = (x_{1},\ldots ,x_{m}) \sim \mu^{\otimes}$  is equivalent to  $x_{i} \sim \mu$  for  $i = 1,\dots,m$ , and  $P_{X} = \frac{1}{m}\sum_{i = 1}^{m}\delta_{x_{i}}$ . Similarly,  $Z = (z_{1},\ldots ,z_{m})$  with  $z_{i} \sim \nu_{0}$  for  $i = 1,\dots,m$ , and  $G_{\phi}(Z)$  is the output of the neural work given the input mini-batch  $Z$ .
By using Wasserstein distance, sliced Wasserstein distance, and their variants as the distance  $\mathcal{D}$ , we obtain the corresponding estimators. However, applying directly those estimators to natural image data cannot give perceptually good results [20, 15]. The reason is that Wasserstein distance, sliced Wasserstein distances, and their variants require a ground metric as input e.g.,  $\mathcal{L}_2$ , however, those ground metrics are not meaningful on images. Therefore, previous works propose using a function that maps the original data space  $\mathcal{X}$  to a feature space  $\mathcal{F}$  where the  $\mathcal{L}_2$  norm is meaningful [52]. We denote the feature function  $F_{\gamma}:\mathcal{X}\to \mathcal{F}$ . Now the estimator becomes:
$$
\operatorname * {a r g m i n} _ {\phi \in \Phi} \mathbb {E} _ {(X, Z) \sim \mu^ {\otimes m} \otimes \nu_ {0} ^ {\otimes m}} [ \mathcal {D} (P _ {F _ {\gamma} (X)}, P _ {F _ {\gamma} (G _ {\phi} (Z)))} ].
$$
Max-K-SW
K-SW
oMSW
iMSW
viMSW
Figure 7: Random generated images of distances on CIFAR10.
The above optimization can be solved by stochastic gradient descent algorithm with the following stochastic gradient estimator:
$$
\begin{array}{l} \nabla_ {\phi} \mathbb {E} _ {(X, Z) \sim \mu^ {\otimes m} \otimes \nu_ {0} ^ {\otimes m}} [ \mathcal {D} (P _ {F _ {\gamma} (X)}, P _ {F _ {\gamma} (G _ {\phi} (Z)))} ] = \mathbb {E} _ {(X, Z) \sim \mu^ {\otimes m} \otimes \nu_ {0} ^ {\otimes m}} [ \nabla_ {\phi} \mathcal {D} (P _ {F _ {\gamma} (X)}, P _ {F _ {\gamma} (G _ {\phi} (Z)))} ] \\ \approx \frac {1}{K} \sum_ {k = 1} ^ {K} \nabla_ {\phi} \mathcal {D} \left(P _ {F _ {\gamma} \left(X _ {k}\right)}, P _ {F _ {\gamma} \left(G _ {\phi} \left(Z _ {k}\right)\right)}\right), \\ \end{array}
$$
where  $X_{1},\ldots ,X_{K}$  are drawn i.i.d from  $\mu^{\otimes m}$  and  $Z_{1},\ldots ,Z_{K}$  are drawn i.i.d from  $\nu_0^{\otimes m}$ . There are several ways to estimate the feature function  $F_{\gamma}$  in practice. In our experiments, we use the following objective [15]:
$$
\min  _ {\gamma} \left(\mathbb {E} _ {X \sim \mu^ {\otimes m}} [ \min  (0, - 1 + H (F _ {\gamma} (X))) ] + \mathbb {E} _ {Z \sim \nu_ {0} ^ {\otimes m}} [ \min  (0, - 1 - H (F _ {\gamma} (G _ {\phi} (Z)))) ]\right),
$$
where  $H: \mathcal{F} \to \mathbb{R}$ . The above optimization problem is also solved by the stochastic gradient descent algorithm with the following gradient estimator:
$$
\begin{array}{l} \nabla_ {\gamma} \left(\mathbb {E} _ {X \sim \mu^ {\otimes m}} [ \min  (0, - 1 + H (F _ {\gamma} (X))) ] + \mathbb {E} _ {Z \sim \nu_ {0} ^ {\otimes m}} [ \min  (0, - 1 - H (F _ {\gamma} (G _ {\phi} (Z)))) ]\right) \\ = \mathbb {E} _ {X \sim \mu^ {\otimes m}} \left[ \nabla_ {\gamma} \min  (0, - 1 + H (F _ {\gamma} (X))) \right] + \mathbb {E} _ {Z \sim \nu_ {0} ^ {\otimes m}} \left[ \nabla_ {\gamma} \min  (0, - 1 - H (F _ {\gamma} (G _ {\phi} (Z)))) \right] \\ \approx \frac {1}{K} \sum_ {k = 1} ^ {K} [ \nabla_ {\gamma} \min (0, - 1 + H (F _ {\gamma} (X _ {k}))) ] + \frac {1}{K} \sum_ {k = 1} ^ {K} [ \nabla_ {\gamma} \min (0, - 1 - H (F _ {\gamma} (G _ {\phi} (Z _ {k})))) ], \\ \end{array}
$$
where  $X_{1},\ldots ,X_{K}$  are drawn i.i.d from  $\mu^{\otimes m}$  and  $Z_{1},\ldots ,Z_{K}$  are drawn i.i.d from  $\nu_0^{\otimes m}$ .
Settings. We use the following neural networks for  $G_{\phi}$  and  $F_{\gamma}$ :
CIFAR10: