Distributional Sliced-Wasserstein and Applications to Generative Modeling
Authors: Khai Nguyen
Nhat Ho
Tung Pham
Hung Bui
Venue: ICLR 2021 Spotlight (2021)
Keywords: Deep generative models
Sliced Wasserstein
Optimal Transport

Abstract: Sliced-Wasserstein distance (SW) and its variant, Max Sliced-Wasserstein distance (Max-SW), have been used widely in the recent years due to their fast computation and scalability even when the probability measures lie in a very high dimensional space. However, SW requires many unnecessary projection samples to approximate its value while Max-SW only uses the most important projection, which ignores the information of other useful directions. In order to account for these weaknesses, we propose a novel distance, named Distributional Sliced-Wasserstein distance (DSW), that finds an optimal distribution over projections that can balance between exploring distinctive projecting directions and the informativeness of projections themselves. We show that the DSW is a generalization of Max-SW, and it can be computed efficiently by searching for the optimal push-forward measure over a set of probability measures over the unit sphere satisfying certain regularizing constraints that favor distinct directions. Finally, we conduct extensive experiments with large-scale datasets to demonstrate the favorable performances of the proposed distances over the previous sliced-based distances in generative modeling applications.

Published as a conference paper at ICLR 2021
DISTRIBUTIONAL SLICED-WASSERSTEN WASSERSTEIN AND APPLICATIONS TO GENERATIVE MODELING
Khai Nguyen
Nhat Ho
Tung Pham
VinAI Research, Vietnam Thainbevr iiaiiiii.............m
University of Texas, Austin VinAI Research, Vietnam mi ninhnhat@uteexass
VinAI Research, Vietnam v. Sunnghh@vingg io
Hung Bui
VinAI Research, Vietnam v. hunghhl@vinaa
ABSTRACT
Sliced- Wasserstsnt distance (SW) and its variant, Max -aese--aasnntteiin distance (Max-SW), have been used widely in the recent years due to their fast computation and scalability even when the probability measures lie in very high dimensional space. However, SW requires many unnecessary projection samples to approximate its value while Max-SW only uses the most important projection, which ignores the information of other useful directions. In order account for these weaknesses, we propose novel distance, named Distributional Sliced- Wereeddtsssrrtiinn distance (DSW), that finds an optimal distribution over projections that can balance between exploring distinctive projecting directions and the informativeness of projections themselves. We show that the DSW is a generalization of Max-SW, and it can be computed efficiently by searching for the optimal push-forward measure over a set of probability measures over the unit sphere satisfying certain regularizing constraints that favor distinct directions. Finally, conduct extensive experiments with large-scale datasets to demonstrate the favorable performances of the pro- posed distances over the previous sliced-based distances in generative modeling applications.
INTRODUCTION
Optimal transport (OT) is a classical problem in mathematics and operation research. Due to its appealing theoretical properties and flexibility in practical applications, it has recently become an important tool in the machine learning and statistics community; see for example, (Courty et al., 2017: Arjovsky et al., 2017: Tolstikhin et al., 2018; Guirajani et al., 2017) and references therein. The main usage of OT is to provide distance named Wasserstein distance, measure the discrepancy between two probability distributions. However, that distance suffers from expensive computational complexity, which is the main obstacle using OT in practical applications.
There have been two main approaches to overcome the high computational complexity problem: either approximate the value of OT or apply the OT adaptively to specific situations. The first approach was initiated by (Cuturi, 2013) using an entropic regularizer to speed up the computation of the OT (Sinkhorn, 1967; Knight, 2008). The entropic regularization approach has demonstrated its usefulness in several application domains (Courty et al., 2014; Genevay et al., 2018; Bunne et al., 2019). Along this direction, several works proposed efficient algorithms for solving the entropic OT (Altschuler et al., 2017; Lin et al., 2019b;a) as well as methods to stabilize these algorithms (Chizat et al., 2018; Peyré & Cuturi, 2019; Chizat al., 2018; Schmitzer, 2019). However, these algorithms have complexities of the order (R:2), where l is the number of supports. It is expensive when we need to compute the OT repeatedly, especially in learning the data distribution.
"The work was finished when Nhat Ho worked VinAI Research in the summer of 2020.

---

Published as a conference paper at ICLR 2021
The second approach, known as "slicing", takes rather different perspective. It leverages two key ideas: the OT closed-form expression for two distributions in oue-dimensional space, and the transformation of distribution into set of projected one- me-dimensional distributions by the Radon transform (RT) (Helgason, 2010). The popular proposal along this direction is Sliced- Wasserstein (SW) distance (Bonneel et al., 2015), which samples the projecting directions uniformly over unit sphere in the data ambient space and takes the expectation of the resulting one-dimensional OT distance. The SW distance hence requires significantly lower computation cost than the original Wasserstein distance and is more scalable than the first approach. Due to its solid statistical guarantees and efficient computation, the SW distance has been successfully applied practical tasks (Deshpande et al., 2018: 8: Liutkus et al., 2019; Kolouri et al., 2018; Wu et al., 2019; Deshpande et al., 2019) where has been shown to have comparative performances to other distances and divergences between probability distributions. However, there is an inevitable bottleneck of computing the SW distance. Specifically, the expectation with respect to the uniform distribution of projections in SW is intractable to compute; therefore, the Monte Carlo method is employed to approximate it. Nevertheless, drawing from a uniform distribution of directions in high-dimension can result in overwhelming number of irrelevant directions, especially when the actual data lies in a low-dimensional manifold. Hence, SW typically needs to have a large number of samples to yield an accurate estimation of the discrepancy. Alternatively, in the other extreme, Max (Max-SW) distance (Deshpande et al., 2019) uses only one important direction to distinguish the probability distributions. However, other potentially relevant directions are ignored in Max-SW. Therefore, Max-SW can miss some important differences between the two distributions in high dimension. We note that the linear projections in the Radon transform can be replaced by non-linear projections resulting in the generalized sliced-Waad Wasserstein distance and its variants (Beylkin, 1984; Kolouri et al., 2019).
variety of
in
Apart from these main directions, there are also few proposals that try either to modify them or to combine the advantages of the above-mentioned approaches. In particular, Paty & Cuturi (2019) extended the idea of the max-sliced distance to the max-subspace distance by considering finding an optimal orthogonal subspace. However, this approach is computationally expensive, since could not exploit the closed-form of the one-dimensional Wasserstein distance. Another approach named the Projected Wasserstein distance (PWD), which was proposed in (Rowland et al., 2019), uses sliced decomposition to find multiple one-dimension optimal transport maps. Then, it computes the average cost of those maps equally in the original dimension.
Our contributions. Our paper also follows the slicing approach. However, we address key friction in this general line of work: how to obtain a relatively small number of slices simultaneously to maintain the computational efficiency, but at the same time, cover the major differences between two nigh-dimensional distributions. We take probabilistic view of slicing by using a probability measure on the unit sphere to represent how important each direction From this viewpoint, SW uses the uniform distribution while Max-SW searches for the best -errac distribution over the projections, both can be considered as special cases. In this paper, we propose to search for an optimal distribution of important directions. We regularize this distribution such that it prefers directions that are far away from one another, hence encouraging an efficient exploration of the space directions. In the case of no regularization, our proposed method recovers max- (generalized) SW special case. In summary, our main contributions are two-fold:
1. First, we introduce a novel distance, named Distributional Sliced- Wasserstein distance (DSW), account for the issues of previous sliced distances. Our main idea is to search for not just a single most important projection, but an optimal distribution over projections that could balance between an expansion of the area around important projections and the informativeness of projections themselves, i.c., how well they can distinguish the two target probability measures. We show that DSW is a proper metric in the probability space and possesses appealing statistical and computational properties as the previous sliced distances.
2. Second, we apply the DSW distance to generative modeling tasks based on the generative adversarial framework. The extensive experiments on real and large-scale datasets show that DSW distance significantly outperforms the SW and Max-SW distances under similar computational time on these tasks. Furthermore, the DSW distance helps model distribution converge to the data distribution faster and provides more realistic generated images than the SW and Max-SW distances.
2

---

Published as a conference paper at ICLR 2021
Organization. The remainder of the paper is organized as follows. In Section 2, provide back- grounds for Wasserstein distance and its slice- based versions. In Section 3, we propose distributional (generalized) sliced-Wasserster Wasserstein distance and analyze some of its theoretical properties. Section 4 includes extensive experiment results followed by discussions in Section Finally, we defer the proofs of key results and extra materials in the Appendices.
Notation. For any 8, 0' E R cos(O, 8*) = e T where |I-II is &2 norm. For any d 2 2. Sd denotes the unit sphere in d dimension in l2 norm Furthermore, denotes the Dirac delta function, and (.-)) is the Euclidean inner-prodie For any p 2 1, (RR)) the set of real-valued functions on Rd with finite p-th moment.
2 BACKGROUND
In this section, we provide necessary backgrounds for the (generalized) Radon transform, the Wasserstein, and sliced-Wassenee Wasserstein distances.
2.1 WASSERSTEIN DISTANCE
We start with a formal definition of Wasserstein distance. For any p 2 1, we define (p as the set of Borel probability measures with finite p-th moment defined on a given metric space (IRd, II.I)) For any probability measures μr, defined on X Rd,, denote their corresponding probability density functions as I and Ig. The Wasserstein distance of order p between j and v is given by (Villani, 2008; Peyré & Cuturi, 2019):
( inf J || dT (1),,)))
W Wo(,v)
RETl()))) xxy
where [1(,, v) is all transportation plans т such that the marginal distributions of are p and respectively. In order to simplify the presentation, we abuse the notation by using both (, v) and WF( Iz) interchangeably for the Wasserstein distance between μ and When μ and are one-dimension measures, the Wasserstein distance between y and has a closed-form expression W, v) (.6' (2) F, (=) where F₁. and F, are the cumulative distribution function (CDF) of L, and Lu, respectively.
2.2 (GENERALIZED) RADON TRANSFORMS
Now, we review (generalized) Radon transform maps, which are key to the notion of (generalized)
distance and its variants. The Radon transform (RT) maps function / E IL (R2 to the space of functions defined over space of lines in Rd. In particular, for any t E IR and direction E gd-1, the RT is defined as follows (Helgason, 2010) RI 0) == Sg [(z)5(t (x.0) )x.
The generalized Radon transform (GRT) (Beylkin, 1984) extends the original one from integration over hyperplanes of Rdd integration over hypersurfaces. In particular, is defined as: Gl Jax (tt)((t g(x, 0) )xr, where E IR and A E Ωg. Here, 2g is compact subset of Rd and g: Rd gd-1 R is defining function (cf. Assumptions H1-H4 in (Kolouri et al., 2019) for the definition of defining function) inducing the hypersurfaces. When g(x 8) = (z, 0) and Qg = gd the generalized Radon transform becomes the standard Radon transform.
2.3 (GENERALIZED) SLICED- WASSERSTEIN DISTANCES
The sliced- Wasserstein distance (SW) between two probability measures μ and is defined as (Bon- neel al., 2015): SW,(H, v) = (fga WP (- 8), RI,(-, (L)
Similarly, the gen- eralized diced-Wassersser distance (Kolouri al., 2019) (GSW) is given by GSW,(p, (Ja, (GI (-) (I)) where Qo is the compact set of feasible parameter. How- ever, these integrals are usually intractable. Thus, they are often approximated by using Monte Carlo scheme to draw uniform samples [0,),N, from Sd-1 and Szg In particular, SWP(H, ss
+ > WP (RI,, (-.0,,, RI,(,, and GSWP v) x + EN WP(GI,(- (.,.L,,,))))))) GI). In order obtain good approximation of (generalized) SW distances, N needs to be sufficiently large.
3

---

Published as a conference paper at ICLR 2021
However, important directions not distributed uniformly the sphere. Thus, this approach will draw potentially many unimportant projections that are not only expensive but also greatly reduce the effect of the SW distance.
2.4 MAX (GENERALIZED) SLICED- WASSERSTEIN DISTANCES
An approach to using only informative directions is to simply take the best slice in discriminating two given probability distributions. That distance max pande et al., 2019), which is given by max (H, 2) == maX (R 0), RIv(- (..-)))) 0)). By combining this idea with non-linear projections from generalized Radon transform, we obtain max generalized sliced- Wasserstein distance
distance (Mx-SW) (Desh-
(Kolouri et al., 2019). The formal definition of that distance is: maxGSW, v) == maxeen, Wp(GI,(- 0), (-, 0)). The (generalized) Max- SW distances focus on finding only the most important direction. Meanwhile, other informative directions play no role in the distance. Therefore, (generalized) Max-SW distances can ignore useful information about the structure of high dimensional probability measures.
DISTRIBUTIONAL SLICED- WASSERSTEIN DISTANCE
With the aim of improving the limitations of the previous sliced distances, we propose novel distance, named Distributional Sliced- Wassersiein distance (DSW), that can search for not just a single but distribution of important directions on the unit sphere. We prove that it is a well-defined metric and discuss its connection to the existing sliced-based distances in Section 3.1. Then, we provide procedure approximate DSW based on its dual form in Section 3.2.
3.1 DEFINITION AND METRICITY
We first start with a definition of distributional sliced- Wasserstein distance. We say C' > O admissible if the set Mc of probability measures on S satisfying Er w [0 01] S C' is not empty.
Definition 1. Given two probability measures ja and v on Rdd with finite withh p-th moments where p 2 1 and an admissible regularizing constant C >> The distributional of order between and v is given by:
distance (DSW)
(W = sup oENic
Eaaa
WB [)
where R the Radon transform operator.
The DSW aims to find the optimal probability measure of slices the unit sphere 5d-1, Note that, the Max-SW distance is equivalent to searching for the best Dirac measure on a single point in 1d-1 which puts all weights in only one direction. Meanwhile, the uniform measure in the formulation of SW distance distributes the same weights in all directions. Indeed, the uniform and Dirac measures two special cases, because they view that either all directions are equally important or only one direction is important. That view is too restricted if the data actually lie on low dimensional space. Thus, we aim to find probability measure which concentrates only on areas around important directions. Furthermore, we do not want these directions to lie in only one small area, because under the orthogonal projection of RT, their corresponding one-dimensional distributions will become similar. In order to achieve this, we search for an optimal measure a that satisfies the regularization constraint E, 0 ~a [ 8 < C. By Shuchy-Schwarz inequality, C' is no greater than 1, thus MI, contains all probability measures on the unit sphere. Optimizing over M,, simply returns the best Dirac measure corresponding to the Max-SW distance. When C' is small, the constraint forces the measure a to distribute more weights to directions that are far from each other (in terms of their angles). Thus, a small appropriate value of will help to balance between the distinctiveness and informativeness of these targeted directions. For further discussion about C, see Appendix B.
Next, we show that DSW is well-defined metric on the probability space. Theorem 1. For any p 1 and admissible C > 0. (-:: C') is well-defined metric in the space of Borel probability measures with finite p-th moment. In particular, it is non-negative, symmetric, identity, and satisfies the triangle inequality.
4

---

Published as a conference paper at ICLR 2021
The proof of Theorem is in Appendix A.1. Our next result establishes the topologiaal equivalence between DSW distance and (max)-sliced) Wasserstein and Wasserstein distances. Theorem 2. For any p Z 1 and admissible C O, the following holds
(a) DSW, VI, C) S maxSW,((,,) Wo(,,)).
(b) C 1/d, we have DSW, (p, Vs C) 2 (1) 1/1 maxSW,(1, (2) sW,(u,
As a consequence, when 2 and C 2 1/d, DSW,(:;:: SWp. maxSW,,. and W are topologi- cally equivalent, namely, the convergence of probability measures under (S;,3 implies the convergence of these measures under other metrics and vice versa.
The proof of Theorem is in Appendix A.2. As a consequence Theorem 2, the statistical error of estimating the unknown distribution based on the empirical distribution of n. i.i.d data under DSW distance is C2 n n-12 with high probability where Cj is some universal constant depending on dimension d (see Appendix B.3). Therefore, as other sliced- Wasserstein distances, the DSW distance does not suffer from the curse of dimensionality.
3.2
COMPUTATION OF DSW DISTANCE
Direct computation of DSW distance is challenging. Hence we consider dual form of DSW distance and reparametrization of as follows.
Definition 2. For any p and admissible C O, there exists non-negative constant Ào depending on C' such that the dual form of DSW distance takes the following form
{
(W((L,.:,) -XcEa, [10T0r] } +
DSW (1, us C) = sup EMM
(IE
AdC,
where MI denotes the space of all probability measures on the unit sphere 1d-1
By the Lagrangian duality theory, pS vy; C* 2 DSW vi C*) for any 2 and admissible C 0. In Definition 2, the set Mc disappears and λo plays the tuning role for the regularized term Eq.0'~ [10 8'|]. When Ac is large, Eg,0'~a [ 8') needs to be small, meaning that is small. When Ae is small, the value of Eg,0r~ [ 0'] becomes less important, i.e.. C is large.
For reparametrizing the measure a, we use pushforward of uniform measure on the unit sphere through some measurable function f. In particular, let f be Borel measurable function from Sd-1 1d-1 For any Borel set A C gd-1, we define (A) (dd'(f-((A)) where gd 1 is the uniform probability measure on gd-1, Then for any Borel measurable function gd-1 + R, we have Je (0) Se (g ()) Therefore, we obtain the equivalent dual form of
DSW as follows:
1/P
DSW (H, sup fEF
(Banod-1
(I((
>xB ~or-1 [[(00]
1(0') [1(00]((6]]]]
+ +C00 sup DS(J), fEF
where a class of all Borel measurable functions from dd-1 to gd-11
Finding the optimal f: We parameterize in the dual form (2) by using deep neural network with parameter 6, defined as fa- Then, we estimate the gradient of the objective function DS(fa) in equation (2) with respect to Φ and use stochastic gradient ascent algorithm to update Φ. Since there are expectations over uniform distribution in the gradient ( ((). we use the Monte Carlo method to approximate these expectations. Note that, we can use the fixed point from the stochastic ascent algorithm to approximate the dual value of DSW in equation (2). A detailed argument for this point is in Appendix B.2. Finally, in generative model applications with DSW being the loss function, only need to use the gradient of the function DS(.) to update the parameters interest. Therefore, can treat À regularized parameter and tune it to find suitable value in these applications.
5

---

Published as a conference paper at ICLR 2021
14
-03
4.0
-03
-03
0.5
65
04
3.5 -35
4.
Figure 1: Empirical behavior of optimal measure approximated by 1000 samples, on circle for different values of λc (the constant in the dual form of DSW in Definition 2) when jr and v bivariate Gaussian distributions sharing the same cigenvectors. When Ac == When x increases, C becomes small.
Illustration of the roles of Ao and C: To illustrate the roles of Ao and in finding optimal distribution d, conduct simple experiment two Gaussian distributions with zero means and covariance matrices given by ( 9) and (s 1)- The experiment optimizes the empirical form of Definition 2 with different choices of Ac The results are shown Figure 1 with the reported value of λ and Eg [10] 0' For Àc = O, the obtained distribution concentrates only on one direction. When Àc = 50, optimal a distributes more weights to other directions on the circle. When Ao 1000, optimal is close the discrete distribution concentrated covariance matrices, which are the main directions differentiating the two Gaussian distributions.
2)
eigenvectors of the
Extension of DSW and comparison of DSW to Max-GSW-NN: Similar to SW, we extend DSW to distributional generalized sliced Wasserstein (DGSW) by using the non- non-linear projecting operator via GRT. The definition of the DGSW and its properties are in Appendix C. Finally, in Appendix E.1, we show the distinction of the DSW to Max-GSW-NN (Kolouri et al., 2019) when the neural network defining function MaX-GSW-NN is g(z, 0) (x, f(0)) where f gd-1 Sd-1
4 EXPERIMENTS
In this section, we conduct extensive experiments comparing the performance in both generative quality and computational speed of the proposed DSW distance with other bliced-based distances, namely the SW, Max-SW, Max-GSW-NN (Kolouri al., 2019) and projected robust subspace Wasserstein (PRW) (Paty & Cuturi, 2019 Lin et al., 2020) using the minimum expected distance estimator (MEDE) (Bernton al., 2019) on MNIST (LeCun et al., 1998), CIFARIO (Krizhevsky. 2009), CelebA (Liu et al., 2015) and LSUN (Yu et al., 2015) datasets. The details of the MEDE framework are described different methods may be subject to the differences in the hyperparameter settings and software implementations of different methods. On MNIST dataset, we train generative models with different distances and then evaluate their performances by comparing Wasserstein-2 distances between 10000 random generated images and all images from the MNIST test set. Due to the very large size of other datasets, e.g., 3 million images in LSUN, it is expensive to compute empirical Wasserstein-2 distance as its complexity is of order Okk2 log k) where & is the number of support points. Therefore, after we train generative models, we use FID score (Heusel et al., 2017) evaluate the generative quality of these generators. The FID score is calculated from 10000 random generated images and all training samples using precomputed statistics (Heusel et al., 2017). Finally, for Ao in DSW (see Definition 2), it is chosen in the set {1, 10, 100, 1000) such that its Wasserstein-2 (FID score) (between 1 10000 random generated images and all images from corresponding validation set) the lowest among the four values. Detailed experiment settings are in Appendix G. Finally, we also apply the DSW into color transfer task (Rabin et al., 2010: 2014: Bonneel et al., 2015; Perrot et al., 2016) in Appendix F, where we find that DSW also performs better than SW and Max-SW in this task.
Appendix
We would like to note that the wall-clock timing of
4.1 RESULTS ON MNIST
Generative quality and computational speed: We report the performance of the learned generative models for MNIST in Figure 2(a). To plot this figure, vary the number of projections N

---

Published as a conference paper at ICLR 2021
MAOT
112
PAa
Comsuaaceaal the mibaes
201 2003
1000
1899
(a)
(b)
(c)
(d)
Figure 2: (a) Comparison between DSW. SW, Max-SW, Max- Max--SS-II CSW-NN, PRW and WD based on exccution time and performance. Here, each dot of SW and DSW corresponds to the number of projections chosen in {1, 10, 10 5 x 1 100 in [2, 5, 50); (b) Comparison between SW. DSW, Max-SW and Max-GSW-NN hased Wasserstein-2 distance between distributions of learned model and test over iterations; (c) Computation speed of distances based on the number of minibatch's 's samples (log-log scale); (d) Effect Xe on the mean of absolute values of pairwise cosine similarity between 10 random directions from the found distribution a of DSW.
x 10², 10"). Each dot of PRW corresponds to the dimension of the subspace chosen
{1, 10, 102.5. x 102, 1,,,100,5. x 100,10*) 1003 for the SW, and N E (1, x 10?; 10 5 x 10 } for the DSW. Then measure the computational time per minibatch and the Wasserstein-2 score of the learned generators for each N. We plot the Wasserstein-2 score and computational time of Max-SW and Max-GSW-NN their standard settings (Kolouri al., 2019). Except for the regime with very fast but low-quality learned models, DSW is better than all the existing slice-based baselines in terms of both model quality and computational speed. Moreover, DSW can learn good models with very few projections, e.g., DSW-10 10 achieves better model quality than Max-GSV GSW-NN and Max-SW and is one order-of- of-magnitude faster than these sliced distances. Finally, with a similar computational time, a learned generator by DSW has the Wasserstein-2 score that is roughly 10% lower than the one got from SW. For the qualitative comparison between these distances, we show random generated images from their generative models in Figure 7 in Appendix E. We observe that generated images from DSW are sharper and easier to classify into numbers than those from other baseline distances.
Comparison with projected robust subspace Wasserstein (PRW) and Wasserstein distance: In Figure 2(a), we plot the Wasserstein-2 score and computational time of Wasserstein distance (WD) and PRW, where the subspace dimension of PRW varies in the range {2, 5, 10, 50}. PRW is able to improve upon the model quality of slice-based methods including DSW, however at the cost of being an order of magnitude slower than DSW with 10 projections (DSW-10). We observe that DSW-10 obtains a better Wasserstein-2 score than PRW with 5-dimensional subspace, while its corresponding computational time is 30 times faster than that of PRW-5. Using 50 dimension, PRW's Wasserstein-2 score improves about 29% to that of DSW-10 10 but the computational cost is also around 40 times slower. The model trained by WD gives good Wasserstein-2 score: however, expensive (about 40 times slower than DSW-10). The main computational advantage of DSW comes from the exact calculation Wasserstein distance one-dimension. The visual comparison between PRW, WD and DSW based on their generated images is in Figure 12 in Appendix E.2.
computational
Convergence behavior: Figure 2(b) shows that DSW learns better models at a faster speed of convergence than other baseline distances with very small number of projections, e.g., DSW-10 is the second lowest curve compared to curves from other based distances.
Scalability over sample size of minibatch: Results in Figure 2(c) show that DSW has computa- tional complexity the order (A log k), which similar to those of other sliced-based distances, where k is the number of samples per batch.
Effect of the regularization parameter Ac: For each value of A E {1,10, 100, 1000]. we find the optimal distribution a of DSW with N = 10 projections, and then calculate A = N ΣN 10_ 0 an approximation of the regularized term 1Eg,8~~ [10 0']] in the dual form of DSW in equation (2), where (0.) ~ a. The results are shown Figure 2(d). We observe that when λo increases, AN goes down. When A = 0, i.e., no regularization, Ax gets close to 1, meaning that all projected directions collapse to one direction. When Ào = 1000, Ay is close to 0.1, suggesting that all projected directions are nearly orthogonal.
1
7

---

Published as a conference paper at ICLR 2021
Celeba
C3F4310
116
Nax- SW
Sii
132
Max- Sa
517
Nac-WW
Sii
Max- CDV hi Dow
Man-CSW-WN
Nax CSW 10V
120
DGW
++
DSW
30
XD
XD
85
70-
83
08-
37 Compurational tine mirtbakch 1200 150
150 1.0
60
1.00
280
10 20
40
40
Compurxtions//iee time per witatrh (NCO- 163
Compuntions mwm.mrtttr 1630 (1)
Figure 3: Comparison between DSW, SW, Max-SW and Max-GSW-ND in terms of execution time and performance. Here, each dot of SW and DSW corresponds the number of projections chosen in {10?, x 10', 10 5 x 10', 10"). We set the minibatch size 512 on CelebA and CIFAR, and be 4096 on LSUN.
Additional experiments: We also investigate how the number of gradient-steps used for updating distribution of directions and how the size of minibatches affects the quality of DSW (see Appendix E. The results show that an increasing number of gradient steps to update a leads to better performance of DSW but also slows down the computation speed. Furthermore, we carry out experiments with DGSW, an extension of DSW to non-linear projections, and test the new proposed distances in training encoder-generator models on MNIST using joint contrastive inference (JCI) in Appendices E.1 and E.3. The description of these models is in Appendix D.
4.2 RESULTS ON LARGE-SCALE DATASETS
Next, we conduct lcrge-scale experiments on a range of more realistic image datasets. We train generative models using CIFARIO, CelebA, and LSUN datasets (all these datasets rescaled to 64x64 resolution). When working with high dimensional distributions, Deshpande et al. (2018) proposed a trick improve the quality of the generator by learning feature function which maps data to a new feature space that is more manageable in size. When the feature function is fixed, the generator is trained to match the distribution of features. When the generator is fixed, the feature function tries to tease apart the data empirical features from the generated feature distribution. For the experiments in this section, we use the same technique with DSW and all other baseline distances.
We compare DSW with SW, Max-SW, and Max-GSW-NN in both generative quality (FID score) and computational time in Figure We could not compare DSW with PRW on the large-scale datasets since PRW is computationally expensive to train to obtain good generated images. On CelebA and CIFARIO 10, we let N. the number of projections of both DSW and SW, vary the set (10",5 5 x 102 10 5 103, 104). For LSUN, since it takes considerably longer time to train each model, we only vary N in the set [10?, 10 On all these large datasets, DSW outperforms all the other baselines in both FID score of the learned model and computational efficiency. The gap of FID scores between DSW and other methods is especially large on CIFAR 10 and LSUN. For example, on CIFAR1O, with the same computational time, FID scores of DSW always lower than those of SW about 20 units. On LSUN, with 100 projections, DSW can achieve an FID score of 46 while SW with 10000 projections still has worse FID score of over 60. It is interesting note that on these high-dimensional datasets, Max-SW performs rather poorly: it obtains the highest FID scores among all distances while requires heavy computation. Max Max-GSW-NN has better FID scores than (Max)-SW; however, it is still worse than DSW and while being slower. This is consistent with the intuition that as the number of dimension of the data grows, use of a single important slice in Max-SW becomes less efficient approximation. DSW, on the other hand, is able to make use of more important slices, and at the same time avoids SW's inefficiency of uniform slice-sampling.
Generated images from CelebA, CIFARIO and LSUN are deferred to Appendix E. Comparing to other sliced-based Wasserstein distances, generated samples obtained from the DSW's generative model are also more visually realistic. Further experiments compare DGSW with GSW, Max-GSW, and Max-GG are also given in the Appendix E. Based on these experiments, we can conclude that the distributional approach also improves the generative quality of non-linear slicing distances.
8

---

Published as a conference paper at ICLR 2021
5 CONCLUSION
In this paper, we have presented the novel distributional sliced- Wasserstein (DSW) distances between two probability measures. Our main idea is to search for the best distribution of important directions while regularizing towards orthogonal directions. We prove that they are well-defined metrics and provide their theoretical and computational properties. We compare our proposed distances to other sliced-based distances in variety of generative modeling tasks, including estimating generative models and jointly estimating both generators and inference models. Extensive experiments demonstrate that our new distances yield significantly better models and convergence behaviors during training than the previous sliedd-based distances. One important future direction is to investigate theoretically the optimal choice of the regularization parameter Ae such that the DSW distance can capture all the important directions that can distinguish two target probability measures well.

---

Published as a conference paper at ICLR 2021
REFERENCES
Jason Altschuler, Jonathan Niles-Weed, and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration. In Advances neural information processing systems, pp. 1964-1974, 2017.
Luca Ambrogioni, Umut Güşliü, Yagmur Güçlütürk, Max Hinne, Marcel AJ van Gerven, and Eric Maris. Wasserstein variational inference. In Advances in Neural Information Processing Systems, pp.
2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223, 2017.
Erhan Bayraktar and Gaoyue Guo. Strong equivalence between metrics of Wasserstein type. arXiv preprint arXiv:1912.08244 2019.
Espen Bernton, Pierre E Jacob, Mathieu Gerber, and Christian P Robert. On parameter estimation with the Wasserstein distance. Information and Inference: Journal of the IMA, 8(4):657-676. 2019.
Gregory Beylkin. The inversion problem and applications of the generalized Radon transform. Communications on pure and applied mathematics, 37(5):579-599, 1984.
Nicolas Bonneel and David Coeurjolly. Spot: sliced partial optimal transport. ACM Transactions on Graphics (TOG), 38(4):1-133 2019.
Nicolas Bonneel, Julien Rabin, Gabriel Peyré, and Hanspeter Plister. Sliced and Radon Wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 1(51):22 45, 2015.
Nicolas Bonnotte. Unidimensional and evolution methods for optimal transportation. PhD thesis, Paris 11, 2013.
Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie Jegelka. Learning generative models across incomparable spaces. In International Conference on Machine Learning. 2019.
Xiongjie Chen, Yongxin Yang, and Yunpeng Li. Augmented sliced Wasserstein distances. arXiv preprint arXiv:2006.0880 2020.
Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavier Vialard. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Computation, 87(314):2563-2609 2018.
Nicolas Courty, Rémi Flamary, and Devis Tuia. Domain adaptation with regularized optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 274-289. Springer, 2014.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamoniy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, pp. 3730-3739, 2017.
Marco Cuturi. Sinkhorn distances:
Lightspeed computation of optimal transport. In Advances in neural information processing systems, pp. 2292-2300, 2013.
Ishan Deshpande, Ziyu Zhang, and Alexander Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3483-3491, 2018
Ishan Deshpande, Yuan- Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander Schwing- Max-sliced Wasserstein distance and its use for GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10648-10656, 2019.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv: 1606 00704, 2016.
10

---

Published as a conference paper at ICLR 2021
R'emi Flamary and Nicolas Courty. Pot python optimal transport library, 2017. URL https: //pythonot. wigghiihhhb.... io/.
Aude Genevay, Gabriel Peyre, and Marco Cuturi. Learning generative models with Sinkhorn divergences. In International Conference Artificial Intelligence and Statistics, pp. 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein Gans. In Advances in neural information processing systems, pp. 2767-5777,2011
Sigurdur Helgason. Integral geometry and Radon transforms. Springer Science & Business Media, 2010.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Berhhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp.
2017.
Diederik Kingma and Max Welling.
Auto-encoding variational Bayes.
arXiv preprint
arXiv: 14, 2013.
Philip A Knight. The Baachorn-Knopp algorithm: convergence and applications. SIAM Journal on Matrix Analysis and Applications, 30(1):261-275, 2008.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch,
2020.
Soheil Kolouri, Phillip E Pope. Charles E Martin, and Gustavo K Rohde. Sliced Wasserstein
auto-encoders. In International Conference on Learning Representations, 2018.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. Generalized sliced Wasserstein distances. In Advances in Neural Information Processing Systems, pp. 261-272, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-2333 1998.
Bing Li, Shaoli Wang, et al. On directional regression for dimension reduction. Journal of the American Statistical Association, 102:997-1008, 2007.
Ker-Chau Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical
Association, 86(414):316-3) 1991.
Tianyi Lin, Nhat Ho, and Michael Jordan. On efficient optimal transport: An analysis of greedy and accelcrated mirror descent algorithms. In International Conference on Machine Learning, pp. 3982-3991, 2019a.
Tianyi Lin, Nhat Ho, and Michael I Jordan. On the acceleration of the Sinkhorn and Greenkhorn algorithms for optimal transport. arXiv preprint arXiv: (et.:/9966..777 01437, 2019b.
Tianyi Lin, Chenyou Fan, Nhat Ho, Marco Cuturi, and Michael I. Jordan. Projection robust Wasser- stein distance and Riemannian optimization. arXiv preprint arXiv:2006.07455 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Antoine Liutkus, Umut Simsekli, Szymon Majewski, Alain Durmus, and Fabian-Robae Stöter. Wasserstin flows: Nonparametric generative modeling via optimal transport and diffusions. In International Conference on Machine Learning, pp. 4104-4113, 2019.
Cheng Meng, Yuan Ke, Jingyi Zhang, Mengrui Zhang, Wenxuan Zhong, and Ping Ma. Large-scale optimal transport map estimation using projection pursuit. Advances Neural Information Processing Systems, pp. 8118 18-8129, 2019.
11

---

Published as a conference paper at ICLR 2021
Boris Muzellec and Marco Cuturi. Subspace detours: Building transport plans that optimal on subspace projections. In Advances in Neural Information Processing Systems, pp. 1928,, 2019.
Kimia Nadjahi, Alain Durmus, Umut Simsekli, and Roland Badeau. Asymptotic guarantees for learning generative models with the sliced- Wasserstein distance. In Advances in Neural Information Processing Systems, pp. 250-260 260-260,200 2019.
François-Pierre Paty and Marco Cuturi. Subspace robust Wasserstein distances. In International Conference Machine Learning, pp. 5072-5081, 2019.
Michaël Perrot, Nicolas Courty, Rémi Flamary, and Amaury Habrard. Mapping estimation for discrete optimal transport. Advances in Neural Information Processing Systems, 29:4197-4205, 2016.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends@ in Machine Learning, 1 1(5-6))355-607, 2019.
Julien Rabin, Julie Delon, and Yann Gousseau. Regularization of transportation maps for color and contrast transfer. In 2010 IEEE International Conference on Image Processing, pp. 1933-1936.
IEEE, 2010.
Julien Rabin, Sira Ferradans, and Nicolas Papadakis. Adaptive color transfer with relaxed optimal transport. In 2014 IEEE International Conference on Image Processing (ICIP), pp. 4852-4856.
IEEE, 2014.
Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. Color transfer between images. IEEE Computer graphics and applications, 21(5):34-41, 2001.
Mark Rowland, Jiri Hron, Yunhao Tang, Krzysztof Choromanski, Tamas Sarlos, and Adrian Weller. Orthogonal estimation of Wasserstein distances. The 22nd International Conference on Artificial Intelligence and Statistics, pp.
2019.
Berharrd Schmitzer. Stabilized sparse scaling algorithms for entropy regularized transport problems.
SIAM Journal on Scientific Computing, 413341114431118, 2019.
Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathematical Monthly, 74(4):402-44) 1967.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.
Cédric Villani. Optimal transport: Old and New. Springer, 2008.
Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint. Cambridge University Press, 2019.
Jiging Wu, Zhiwu Huang, Dinesh Acharya, Wen Li, Janine Thoma, Danda Pani Paudel, and Lue Van Gool. Sliced Wasserstein generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3713-3722, 2019.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large- scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:/506.082 03366 2015.
12


Reviews:
[{"review_id": "yVW9nMnkzGU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a well written paper with some interesting results.  This paper is to propose a distributional sliced-Wasserstein distance \nto address the limitations of standard SW and Max-SW. The proposed method finds an optimal distribution over projections that can balance between exploring distinctive projecting directions and the informativeness of projections. Some theoretical results are presented in both the main paper and its supplementary document. This reviewer personally enjoys reading this paper. \nHere are a few additional comments. \n\n1. How to select $\\lambda_C$ in practice? The authors need to discuss it in details. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WRoyQAhcLRb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper describes a family of sliced Wasserstein divergences that maximize the distribution over slices subject to constraints on the concentration of slices. Extremes of the family are the sliced Wasserstein and max-sliced Wasserstein distance. In between these the divergence is sensitive to informative discrepancies in multiple subspaces, while still leveraging the relatively fast computation the Wasserstein distance in one dimension for empirical distributions. A dual formulation provides a variational approximation using a (possibly deep) neural network to instantiate the slicing distribution through a pushforward approach. Basic theory prove the divergence is a distance metric between measures. Extensive experiments show improvement over sliced and max-sliced Wasserstein distances and related projection based approaches. \n\nStrengths:\nOverall the paper is a clear and original contribution to the field of sliced Wasserstein distances. The paper's appendix shows multiple applications of the methodology where it outperforms existing divergences. \n\nWeaknesses:\nThe specific parameterization of the mapping $f$ is not clear (it is not given in the main body and it is poorly described in Appendix F as a \"single multi layer perceptron (MLP) layer with normalized output\"). In the main body it is called a deep neural network. \n\nAlso I am a bit wary of interpreting the wall-clock timing differences due to differences in software implementation and hyper-parameter choices for optimization and number of gradient steps (Figure 2(a,c) and Figure 3). Learning curves are more fair, but even these use the best value of  $\\lambda_C$ versus default settings for other methods. As the paper recognizes the max sliced generalized Wasserstein with a deep neural network is the closest competitor and also requires solving a max-min saddle point optimization. It is shown in the appendix as Figure 6(a–e) (green trace). Even in this case the differences in learning curve and timing could possibly be due to software implementation and hyper-parameter selection in the optimization.  \n\nOverall this is a clear accept as it a meaningful improvement to sliced Wasserstein approaches. There a few points that need to be clarified, and I would argue for less emphasis on the wall-clock timing of methods. \n\nHow many parameters are used in the neural network defining $f_\\phi$? Would an empirical distribution optimized over a corresponding number of fixed slices (originally randomly drawn) perform as well? \n\nMinor comments (main body):\nP.1 use Author (year) instead of Author (Author year). \n\nP.1  The case of empirical sample has not been described so what is meant by the term \"atoms\" to describe $k$ is not apparent to the reader.  Then on page 6 \"$k$ is the number of support points\". The terminology could be more consistent. \n\nP.2  Double parentheses \"))\"  after Deshpande et al., 2019. \n\nReferences. Capitalization of conference names and book titles is not consistent. Sinkhorn is a proper noun and should be capitalized. \n\nMinor comments (appendix):\n\nThe appendix doesn't appear to be carefully proofread. \n\nSometimes the notation uses $.$ instead of $\\cdot$ for the arguments of multivariate functions, this is harder to read as in  $\\mathrm{DSW}_p(.,.;C)$ versus $\\mathrm{DSW}_p(\\cdot,\\cdot;C)$ \n\nThere are a number of missing definite articles for example \"that length of side of right triangle\" -> \"that the length of the side of the right triangle\".\n\nThroughout, \"close-form\" -> \"closed-form\".\n\nP.18 \"subspcaae\" and full stop \".\" beginning new line. \n\nP.19 \"as the al time increases considerably\" \n\nP.19 \" DSW and DGSW are slower than Max-SW, Max-GSW (50 gradient updates to find the max direction), and Max-GSW-NN (50 update times for the defining neural net function).\"  Is there any meaning to the subtle variation between \"gradient updates\" and \"update times\".\n\nP.23 \"single multi layer perceptron (MLP) layer\" ? This is not clear. \n\nP.23  \"as the f function in the dual empirical forms of DSW and DGSW for the dual empirical forms of these distances)\" Redundant wording and extra parentheses. \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8CJ-35vgjW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper presents a novel variant of the Sliced Wasserstein (SW) distance. Wasserstein distances have been used recently in lot of machine learning problems. One of the major problem is that, in its primal form, it is computationally expensive. In order to alleviate this problem, a class of methods, called sliced, leverage on the fact that Wasserstein has a closed form expression in 1D (which amount to sort the samples). It replaces the original Wasserstein distance by an expectation of 1D sub-problems over directions drawn uniformly on the unit hypersphere (akin to a Radon transform). Observing that not all the directions are meaningful, the authors propose a variant of SW where the expectation over all the directions is replaced by an expectation over a distribution of directions. The ‘extent’ of this distribution is controlled by an extra parameter. Interestingly, the authors show that this formulation is computationally tractable if one parametrizes this distribution by a measurable function, expressed as a neural network. This result is obtained by deriving the Lagrangian dual of the original problem. Comparisons with previous works are then given in two GAN scenarii: one on MNIST to explore the importance of the different parameters, and another on larger and more complicated datasets, where the FID score is exposed. \n\nThe paper is well written, very clear and easy to follow. Related works are correctly cited. There is one missing work that should be cited though: \n\nMeng, Cheng, et al. \"Large-scale optimal transport map estimation using projection pursuit.\" Advances in Neural Information Processing Systems. 2019.\n\nWhere authors choose the most relevant directions in a way similar to projection pursuit. I really believe this paper could also lead to interesting comparisons. \n\nAll in all, the presented method is a variant of SW that builds on several previous works exploring a similar idea (how to better sample the directions), such as max-SW or subspace robust Wasserstein distances. As such, it can be considered relatively incremental, but this should not totally prevent publications if the computational benefits/performances are very good. Regarding this point, I have some questions about the experimental section:\n - it seems that Figures 2.a (W_2 score) and 3.a (FID) are obtained at convergence. From appendix F, it seems that the number of training epochs is fixed for all methods. I wonder if this setting is fair for comparing max-SW, as far as only one direction is contained (the max direction), the gradient might gather less information. I suspect that more iterations might be needed, why not going until the full convergence  of the model ?\n - when comparing SW and DSW, are the directions drawn randomly for every batches ? (I guess this the common practice when using SW). As such, I do not understand what is the meaning of fixing the number of directions in advance.\n - finally it seems that DSW is computed on mini-batches of samples. While I acknowledge there is a common practice to do so, I think computing a 1D Wasserstein on a mini batch is not the same as computing the 1D W on the full dataset. As such, this mini batch version of SW is not the same as computing the true SW. In the end, i) the size of the mini-batch might have an impact on the estimation quality, that should be discussed ii) if computing 1D W on mini batches, why not computing and comparing with the mini batch version of the original version of W ? This has been done in several papers and has shown to give good results (See the recent study on this theme\nFatras, Kilian, et al. \"Learning with minibatch Wasserstein: asymptotic and gradient properties.\" the 23nd International Conference on Artificial Intelligence and Statistics. Vol. 108. 2020.\n) , plus it does not have the limitation/artefacts of the sliced Wasserstein.\n\nFinally, I would have liked to see other types of experiments than a GAN to really assess the power of this new methodology (SW has been used for classification, domain adaptation, computing gradient flows, etc.). Why limiting the applications to generative modeling ? If generative modeling is the target, then I would expect to see comparisons with other types of metrics than SW (wether it be dual version of W_1 as in WGAN, MMD, or any other divergence from the model zoo).\n\nFor all these reasons my recommendation will only be ‘above acceptance threshold’.\n\n## after author response\nI thank the authors for providing sensible answers to my questions, and analysis of their method compared to the minibatch version. I am still not sure about the fact that the rate of convergence of  SW is indeed indepedent of the dimension, as the dimension is indeed hiddent in the scalar product. Nevertheless, I believe this question is out of the scope of the paper, and I changed my final rating accordingly to the new version of the manuscript.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a new max-sliced Wasserstein distance. Specifically, the proposed method is a multiple sliced variants of the existing max-sliced Wasserstein distance. Compared to the subspace Robust Wasserstein distance, the proposed method can be efficiently computed.

Overall, the proposed method is a good extension of the max-sliced Wasserstein and can be used in various applications. All authors agree to accept the paper, so, I also vote for acceptance.
Decision: Accept (Spotlight)