Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This is a well written paper with some interesting results.  This paper is to propose a distributional sliced-Wasserstein distance 
to address the limitations of standard SW and Max-SW. The proposed method finds an optimal distribution over projections that can balance between exploring distinctive projecting directions and the informativeness of projections. Some theoretical results are presented in both the main paper and its supplementary document. This reviewer personally enjoys reading this paper. 
Here are a few additional comments. 

1. How to select $\lambda_C$ in practice? The authors need to discuss it in details. 


review_title: Distributional Sliced-Wasserstein and Applications to Generative Modeling

final_score: rating: 7: Good paper, accept
review_content: Summary:
The paper describes a family of sliced Wasserstein divergences that maximize the distribution over slices subject to constraints on the concentration of slices. Extremes of the family are the sliced Wasserstein and max-sliced Wasserstein distance. In between these the divergence is sensitive to informative discrepancies in multiple subspaces, while still leveraging the relatively fast computation the Wasserstein distance in one dimension for empirical distributions. A dual formulation provides a variational approximation using a (possibly deep) neural network to instantiate the slicing distribution through a pushforward approach. Basic theory prove the divergence is a distance metric between measures. Extensive experiments show improvement over sliced and max-sliced Wasserstein distances and related projection based approaches. 

Strengths:
Overall the paper is a clear and original contribution to the field of sliced Wasserstein distances. The paper's appendix shows multiple applications of the methodology where it outperforms existing divergences. 

Weaknesses:
The specific parameterization of the mapping $f$ is not clear (it is not given in the main body and it is poorly described in Appendix F as a "single multi layer perceptron (MLP) layer with normalized output"). In the main body it is called a deep neural network. 

Also I am a bit wary of interpreting the wall-clock timing differences due to differences in software implementation and hyper-parameter choices for optimization and number of gradient steps (Figure 2(a,c) and Figure 3). Learning curves are more fair, but even these use the best value of  $\lambda_C$ versus default settings for other methods. As the paper recognizes the max sliced generalized Wasserstein with a deep neural network is the closest competitor and also requires solving a max-min saddle point optimization. It is shown in the appendix as Figure 6(a–e) (green trace). Even in this case the differences in learning curve and timing could possibly be due to software implementation and hyper-parameter selection in the optimization.  

Overall this is a clear accept as it a meaningful improvement to sliced Wasserstein approaches. There a few points that need to be clarified, and I would argue for less emphasis on the wall-clock timing of methods. 

How many parameters are used in the neural network defining $f_\phi$? Would an empirical distribution optimized over a corresponding number of fixed slices (originally randomly drawn) perform as well? 

Minor comments (main body):
P.1 use Author (year) instead of Author (Author year). 

P.1  The case of empirical sample has not been described so what is meant by the term "atoms" to describe $k$ is not apparent to the reader.  Then on page 6 "$k$ is the number of support points". The terminology could be more consistent. 

P.2  Double parentheses "))"  after Deshpande et al., 2019. 

References. Capitalization of conference names and book titles is not consistent. Sinkhorn is a proper noun and should be capitalized. 

Minor comments (appendix):

The appendix doesn't appear to be carefully proofread. 

Sometimes the notation uses $.$ instead of $\cdot$ for the arguments of multivariate functions, this is harder to read as in  $\mathrm{DSW}_p(.,.;C)$ versus $\mathrm{DSW}_p(\cdot,\cdot;C)$ 

There are a number of missing definite articles for example "that length of side of right triangle" -> "that the length of the side of the right triangle".

Throughout, "close-form" -> "closed-form".

P.18 "subspcaae" and full stop "." beginning new line. 

P.19 "as the al time increases considerably" 

P.19 " DSW and DGSW are slower than Max-SW, Max-GSW (50 gradient updates to find the max direction), and Max-GSW-NN (50 update times for the defining neural net function)."  Is there any meaning to the subtle variation between "gradient updates" and "update times".

P.23 "single multi layer perceptron (MLP) layer" ? This is not clear. 

P.23  "as the f function in the dual empirical forms of DSW and DGSW for the dual empirical forms of these distances)" Redundant wording and extra parentheses. 




review_title: Slicing from a tuned distribution of slices is better than one best slice or uniform slicing

final_score: rating: 7: Good paper, accept
review_content: 
The paper presents a novel variant of the Sliced Wasserstein (SW) distance. Wasserstein distances have been used recently in lot of machine learning problems. One of the major problem is that, in its primal form, it is computationally expensive. In order to alleviate this problem, a class of methods, called sliced, leverage on the fact that Wasserstein has a closed form expression in 1D (which amount to sort the samples). It replaces the original Wasserstein distance by an expectation of 1D sub-problems over directions drawn uniformly on the unit hypersphere (akin to a Radon transform). Observing that not all the directions are meaningful, the authors propose a variant of SW where the expectation over all the directions is replaced by an expectation over a distribution of directions. The ‘extent’ of this distribution is controlled by an extra parameter. Interestingly, the authors show that this formulation is computationally tractable if one parametrizes this distribution by a measurable function, expressed as a neural network. This result is obtained by deriving the Lagrangian dual of the original problem. Comparisons with previous works are then given in two GAN scenarii: one on MNIST to explore the importance of the different parameters, and another on larger and more complicated datasets, where the FID score is exposed. 

The paper is well written, very clear and easy to follow. Related works are correctly cited. There is one missing work that should be cited though: 

Meng, Cheng, et al. "Large-scale optimal transport map estimation using projection pursuit." Advances in Neural Information Processing Systems. 2019.

Where authors choose the most relevant directions in a way similar to projection pursuit. I really believe this paper could also lead to interesting comparisons. 

All in all, the presented method is a variant of SW that builds on several previous works exploring a similar idea (how to better sample the directions), such as max-SW or subspace robust Wasserstein distances. As such, it can be considered relatively incremental, but this should not totally prevent publications if the computational benefits/performances are very good. Regarding this point, I have some questions about the experimental section:
 - it seems that Figures 2.a (W_2 score) and 3.a (FID) are obtained at convergence. From appendix F, it seems that the number of training epochs is fixed for all methods. I wonder if this setting is fair for comparing max-SW, as far as only one direction is contained (the max direction), the gradient might gather less information. I suspect that more iterations might be needed, why not going until the full convergence  of the model ?
 - when comparing SW and DSW, are the directions drawn randomly for every batches ? (I guess this the common practice when using SW). As such, I do not understand what is the meaning of fixing the number of directions in advance.
 - finally it seems that DSW is computed on mini-batches of samples. While I acknowledge there is a common practice to do so, I think computing a 1D Wasserstein on a mini batch is not the same as computing the 1D W on the full dataset. As such, this mini batch version of SW is not the same as computing the true SW. In the end, i) the size of the mini-batch might have an impact on the estimation quality, that should be discussed ii) if computing 1D W on mini batches, why not computing and comparing with the mini batch version of the original version of W ? This has been done in several papers and has shown to give good results (See the recent study on this theme
Fatras, Kilian, et al. "Learning with minibatch Wasserstein: asymptotic and gradient properties." the 23nd International Conference on Artificial Intelligence and Statistics. Vol. 108. 2020.
) , plus it does not have the limitation/artefacts of the sliced Wasserstein.

Finally, I would have liked to see other types of experiments than a GAN to really assess the power of this new methodology (SW has been used for classification, domain adaptation, computing gradient flows, etc.). Why limiting the applications to generative modeling ? If generative modeling is the target, then I would expect to see comparisons with other types of metrics than SW (wether it be dual version of W_1 as in WGAN, MMD, or any other divergence from the model zoo).

For all these reasons my recommendation will only be ‘above acceptance threshold’.

## after author response
I thank the authors for providing sensible answers to my questions, and analysis of their method compared to the minibatch version. I am still not sure about the fact that the rate of convergence of  SW is indeed indepedent of the dimension, as the dimension is indeed hiddent in the scalar product. Nevertheless, I believe this question is out of the scope of the paper, and I changed my final rating accordingly to the new version of the manuscript.

review_title: Interesting novel variant of Sliced Wasserstein, but somehow incremental

Meta-review: In this paper, the authors propose a new max-sliced Wasserstein distance. Specifically, the proposed method is a multiple sliced variants of the existing max-sliced Wasserstein distance. Compared to the subspace Robust Wasserstein distance, the proposed method can be efficiently computed.

Overall, the proposed method is a good extension of the max-sliced Wasserstein and can be used in various applications. All authors agree to accept the paper, so, I also vote for acceptance.
Decision: Accept (Spotlight)