Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This manuscript introduces the concept of augmented sliced Wasserstein distances. The main idea is to extend the sliced Wasserstein distance based on mapping samples to higher -dimensional hypersurfaces. The proposed distance is shown to be a metric. Moreover, given that the optimal choice of the nonlinear maps is rather computationally intensive to obtain, an approximation based on neural networks is proposed. Several experiments are shown where a better performance is obtained with respect to existing methods.
main_review: Strengths
- The paper is well written and results are easy to follow.
- The new metric is well justified with respect to existing literature, and many of its theoretical properties are analyzed.
- The amount of numerical experiments and the variety of problems is very impressive!

Weaknesses
- Many of the points where mathematical formality is required is unfortunately omitted. For example at the beginning of page 4, I'm still wondering what are those "certain conditions". How large or small, or scalable is the approximation in (10) or (11)? What are the "certain non-trivial requirements" in Section 3.1. What are the "certain conditions" in Remark 1? What are the "certain constraints " in Section 3.2 Optimization objective?
summary_of_the_review: Correctness: 
- I believe the statements are correct, but I did not checked ever single line of all the proofs. 

Novelty:
- Many of the concepts are natural extensions of the sliced Wasserstein distances. However, there is sufficient novelty.

Empirical novelty:
- A very complete numerical examples set is presented.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the augmented sliced Wasserstein distance (ASWD), a new variety of sliced Wasserstein distance (SWD), that allows comparing two probability distributions by combining a nonlinear embedding of the sample data points to a higher-dimensional space with a slicing scheme to calculate 1D Wasserstein distances between uniformly projection directions. The authors introduce the spatial Radon transform, which includes the standard Radon transform and the special case of polynomial generalized Random transform (introduced in Kolouri et al. 2019). They further prove that ASWD is a valid metric if and only if the mapping is injective. Several experiments are conducted on generative modelling (CIFAR10, CelebA,  MNIST, color transferring).
main_review: #### Strength: 
The paper is well-written and the approach is mostly well presented. The authors introduce the ASWD and derive a set of theoretical properties and propose a numerical algorithm to approximate ASWD. They finally conduct several experiments to illustrate the advantages of ASWD over the state-of-the-art of sliced-based Wasserstein distance like SWD, GSWD, and DSWD.
The code is given, results seem reproducible.

#### Weakness: 

- The authors argue that projecting the original data samples of the source and target distributions into higher-dimensional space leads to maximally discriminating projected samples, by constraining the $k$-th moments of the projected samples. In my opinion, the latter constraint could be well motivated and clarified (or at least well explained or discussed). 

- Since the projection is made by the same mapping $g$, It might be not guaranteed maximal discrimination over the projected samples. Intuitively, I think the mapping $g$ should be data-dependent and/or making a trade-off with different regularization parameters, i.e. the functional $L(\mu, \nu, \lambda; g)$ becomes $L(\mu, \nu, \lambda_1, \lambda_2; g)$.

- As was shown in Remark 3 that if the tuning parameter $\lambda>1$ then ASWD is a proper metric, whereas the authors conducted their experiments with values $\lambda<1$, because it leads to outperforming the other sliced-based Wasserstein metrics. In my opinion, this restricts the applications of ASWD, since one has to investigate a panel of tuning parameters $\lambda<1$ to first check if ASWD is still a valid distance then if it gives good performance compared to the other SWDs. 

- The non-linearity of the injective mapping $g(\cdot) = [Id, \phi_\omega(\cdot)]$ seems a simple injective NN, allowing to gain efficient time complexity. Does it possible to construct $g$ by other injective NNs? It is probably that the fact for a small $lambda <1$ ASWD outperforms the sliced-based Wassrstein distance is linked to the choice of this special parameterization of the injective NN.

- How is the dimensionality of the latent space $d_\theta$ decided and is its impact? The paper does not seem to have included discussions on this seemingly important hyperparameter.

#### Typo:
- page 4: (Definition 1) "Given an measurable" -> "Given a measurable"
summary_of_the_review: The authors introduce the ASWD a new method of SWDs and derive a set of theoretical properties and propose a numerical algorithm to approximate ASWD. The ASWD is a proper metric if and only if the original data samples embedding in a higher-dimensional space through a nonlinear injective mapping. In my opinion, the optimizing ASWD with respect to the injective mapping is strongly depending on the choice of the tuning parameter which guarantees that ASWD is a valid metric if it is greater than $1$. However, in practice, it is better to conduct experiments with values less than $1$. This creates some "broken pieces" between theoretical fundings and experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new slice-based approach to efficiently compute the Wasserstein distance between two distributions $\nu$ and $\mu$. The method termed ASWD (augmented sliced Wasserstein Distance) first  projects the samples from $\nu$ and $\mu$ onto a higher dimensional space using a non linear injective mapping function and then uses the classical random linear projections onto 1D to compute the sliced Wasserstein Distance. Overall, the procedure amounts to applied a spatial Radon Transform to perform the slicing. Theoretical results establish conditions  under which ASWD is a metric. A numerical algorithm is given along with the design of the injective mapping using NN. Empirical evaluations on simulation datasets and on generative modeling highlight the potential of the proposed method over existing approaches.
main_review: - Overall, the paper is well written. It thoroughly  reviews the background and main works on SWD.  The rationale behind the proposed method is justified and aims to design an appropriate non-linear projection mapping that renders the induced sliced-Wasserstein distance (SWD) a valid metric. Theoretical analyses of conditions under which the metric property is preserved are provided.  In this regard the contribution is of interest and significative. Intensive empirical evaluations support these developments and show how the approach contributes efficient SWD computation.

- Specifically,  instead of seeking a generalized Radon Transform (defined by non-linear defining functions that have to follow stringent constraints) or finding the distribution of (non-linear) random projections that maximize the expected 1D Wasserstein distances, the main idea is rather to learn a non-linear mapping function $g$ onto a high-dimension hyper-surfaces. The function $g$ acts as a measurable mapping that generates push-forward measures of the distributions to be compared. Hence, linear random projections onto 1D are performed  to obtain the related augmented SWD (ASWD) as expected 1D Wasserstein distances. The procedure results in defining a spatial Radon transform based on $g$ to compute the SWD which is clearly a clever idea. Therefore, establishing that ASWD is a metric simply derives from the injective property $g$. This facilitates the theoretical study of ASWD. 

- As an another interesting contribution, learning $g$ from data is achieved by maximizing a regularized version of ASWD. The range of the regularization parameter $\lambda$ ensuring that the obtained $g$ leads to a valid metric  is stated. To guarantee the injection property, a neural network (NN) is considered and $g$ is get as the concatenation of the input and the output of the NN. The idea inspired from DenseNet is tricky and presents a clear advantage of avoiding learning complex NN. 

- Intensive empirical evaluations are conducted. They show impressive results compared to existing methods as SWD, distributional SWD (DSWD), generalized SWD (GSWD). In a synthetic problem of evolving source distribution to target one, ASWD achieves the smallest distance. The ASWD is also used to train a generative model for image generation. The reported performances (FID score) highlight the effectiveness of ASWD.

- ASWD is shown to be a valid metric. What are the topological properties of ASWD compared to plain Wasserstein distance, SWD? Beyond empirical evaluations, can one theoretically relate ASWD to WD and SWD?

- The main feature of ASWD is to project the samples onto a $d_\theta$-high-dimensional hyper-surface using $g$.  Due to the design of $g$ in Eq. 19, $d_\theta > d$. Does the choice of $d_\theta$ impact the quality of the obtained ASWD?

- How evolve the computation time according to the batch size? In Nguyen et al., 2021 the computation complexity tends to scale in $O(n \lg n)$.

- Unless mistaken, it seems that the FID scores in Table 1 for CIFAR 10 and CelebA are not competitive with the ones reported on Figure 3 in Nguyen et al, 2021. Indeed Nguyen et al. 2021 show smaller FID of order 60 for CIFAR 10 and 70 for CelebA. Can the authors explain the shift in the reported performances in Table 1? 


Minor comments
- In remark 4, the expression 'rank of the augmented' is not clear and is undefined.

*After rebuttal*

I read authors rebuttal. They address most  raised points. Indeed they study the influence of $d_\theta$ on  the computed ASWD as long as the computational complexity wich is similar to the one  of distributional SWD up to a constant. Also the mismatch in the reported FID scores is clarified and  fixed. One left out concern is  the topological property of ASWD.
summary_of_the_review: The paper is well written and proposes a new sliced Wasserstein distance using the spatial Radon defined Transform defined as line integrals of a function along all hyperplances over a hyper-surface induced by a non-linear injective mapping. The new distance is supported by theoretical analysis and empirical experiments showing its effectiveness. Topological properties of the distance and some details on the empirical  evaluations remain to be clarified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a variant of sliced Wasserstein distance, named augmented sliced Wasserstein distance.
ASWD maps input data points to hypersurfaces using neural networks, then calculates SWD on the hypersurfaces.
ASWD alleviates the low efficiency problem of SWD for high-dimensional data.
Various tasks including flow, generative modeling, and barycenters show the advantage of AWSD against some existing methods.
main_review: 
This paper is overall well written, and ASWD is well motivated.
The proposed method seems to have some originality, though the technique of using non-linear functions to improve the efficiency of slicing has been proposed in GSWD-NN.
The author theoretically show that their proposed method is a valid distance metric even if they use neural networks for the non-linear functions.
I did not check the theoretical results carefully, but they are correct as far as I can tell.
The significance of ASWD has been shown through various experiments such as flow, generalize model, barycenter, and color transfer.

My concern is about the difference between ASWD and GSWD-NN.
To my understanding, the main differences between them are
(1) GSWD-NN is not a valid metric because they uses non-injective function, and (2) ASWD calculates SWD after the non-linear mappings, while max-GSWD-NN directly calculates 1-dimensional Wasserstein distance after the mapping.

Questions:
- (a) As the authors claim, GSWD-NN is not a valid metric. However, if they use an injective function, would GSWD-NN be a valid metric? In other words, is the claim against the inappropriate choice of $g(x)$? Or is GSWD inherently difficult to combine with NNs?
- (b) The second question is about the cause of the difference in performance. The experiment in Sec G.2 indicates that there is not much difference between ASWD and ASWD-non-injective in performances when optimizing NN. So is the difference in performance between ASWD and GSWD-NN due to the random projection (or some other reasons, e.g., regularization term $L_\lambda$)?

summary_of_the_review: This paper is well written and well motivated. The proposed method seems to have some originality, though the technique of using non-linear functions has been proposed. Various experiments show its significance over existing methods.
review_title: null

Meta-review: The paper presents a variant of sliced wasserstein distance , where the slicing operation is performed with a neural network. The resulting distance is studied and experiments on synthetic data and as cost in generative modeling are performed.

While the idea of the paper is not that novel, the work is overall well executed. Reviewers agreed that the paper is borderline weak accept. Accept as a poster.
Decision: Accept (Poster)