Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes to use amortized optimization to predict the optimal projection direction which can be used for learning Wasserstein generative models. The authors derived a family of amortized models which can be used for the framework. Empirical results show that the proposed approach is able to achieve good performance on benchmark datasets.
strengths_and_weaknesses: **Strength**
1. The paper is relatively well-written.
2. The experiments are detailed.
3. The proposed approach could be interesting for the community. However, learning the projection direction also seems straightforward. 

**Weakness**
1. The technical novelty could be limited. The proposed approach seems like a direct application of amortized inference to sliced Wasserstein generative models. There are also no extra theoretical insights/analyses provided.
2. The amortized model considered in the paper could be restrictive. It would be good to perform theoretical analyses on how powerful the model family is.
3. Empirical results are good but not impressive. The improvements in Table 1 are marginal. In Table 2, the memory cost for the proposed approach is also larger than the baselines. 

questions: 1. Proposition 1 mentions that $A-SW(\mu, \nu)=0$ does not imply $\mu=\nu$. Would the proposed method still guarantee convergence to the right distribution?
2. As mentioned in Line 233, the optimization forms a min-max problem. Does it cause training instabilities?
3. Does the batch size affect the performance? In other words, does the proposed approach require significantly larger batch sizes to achieve good performance?
 
limitations: The authors have not adequately addressed the limitations and potential negative societal impact of their work. It would be good to add more discussions.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The presented draft proposes a method for estimating the Max-SW by learning the projection leading to the largest possible SW metric.
strengths_and_weaknesses: ### Strengths 
Generative modeling is an important subject in machine learning.

The paper is exceptionally well written and simple to understand.
The proposed method is simple yet very powerful as it simplifies computation for Wasserstein generative models while improving model performance.

The background is very well explained. The method section is easy to understand.

The empirical evaluation is extensive, covering four data sets of different scales, and the results are impressive.

I greatly appreciate the submission of the code, which is well structured.


### Weaknesses

#### Miscellaneous Issues
* The presented paper uses a different font / font size or line distance. As the font / line distance is increased, I will not consider it for the review score, but it has be fixed for the final version.

#### Typos
* 35: there have been remained certain problems
* 85: we make some conclusion. Suggestions: we provide a conclusion.
* 86: preposition: defer … in -> defer … to 
* 174: measure -> measures
* 223: leads to three corresponding amortized sliced Wasserstein [a word seems to be missing here], …

questions: Could the method be extended to “top-k SW” instead of max-SW? 
Orthogonality could be enforced by a orthogonality loss via a Lagrangian.

limitations: The method requires additional memory and model evaluations but this is extensively discussed in the presented draft.

In practice with large models, i.e., in the demonstrated experiments, the computational cost of the Wasserstein loss is often small, which means that the utility of the method is limited in some applications. However, due to the favorable asymptotical complexity of the method, it allows learning in new settings which were previously impossible or hardly feasible. Further, even in the demonstrated cases the method achieve superior performance.

What I would have liked to see would be using SW and Max-SW with even larger L / T. It would be interesting to see at which point the methods break even, even if the baselines take a very long time. 
It would strengthen the paper even further, and with such an evaluation, I would consider raising my score.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In the paper, the authors investigate the application of amortized optimization procedures to find the most distinguishing direction that can be used to calculate the sliced Wasserstein distance between two empirical latent distributions. This leads to the construction of three variants of the amortized sliced Wasserstein model: linear amortized ($\mathcal{LA}$-SW), generalized linear amortized ($\mathcal{GA}$-SW), and non-linear amortized ($\mathcal{NA}$-SW). The authors conduct experiments on CIFAR10, CelebA(-HQ), and  STL10 datasets, which prove the performance of their methods compared to SNGAN and (max-)SW models. They also analyze the computational complexity of the proposed solutions.
strengths_and_weaknesses: The proposed method seems to be novel, well-motivated, and with sufficient theoretical background. Also, the experimental results show significant improvement over the SOTA techniques. Hence, I think this paper may be interesting for the ML community. However, I found some presentation issues (see below for the details).
questions: My final opinion may depend on the response to issues (1) and (2).

(1)  I think the presentation (including the English of the paper) needs some improvements. For example, the authors quite often use simplified phrases such as (amortized, sliced, max) Wasserstein, to name the models, distances, or losses (without adding these words, what I find inappropriate), e.g.:

l. 43: Sliced Wasserstein is defined $\rightarrow$ Sliced Wasserstein distance is defined,

l. 69: named amortized sliced Wasserstein $\rightarrow$ named amortized sliced Wasserstein losses,

l. 141: mini-batch max-sliced Wasserstein $\rightarrow$ mini-batch max-sliced Wasserstein loss,

l. 222-223: leads to three corresponding amortized sliced Wasserstein $\rightarrow$ leads to three corresponding amortized sliced Wasserstein models.

(2) Do the sets of vectors listed in l. 192, 202, and 217 indeed form bases of linear subspaces in $\mathbb{R}^d$, or do only span these spaces (without independence), and why does $w_0$ appear only one time? Moreover, what do you exactly mean by writing "one-one "reshape" mapping"?

(3) Minor issues:

l. 27: $\phi$ is a parameter of NN (not NN),

l. 23 and 30: $\mathcal{D}(\cdot,\cdot)$ denotes an arbitrary discrepancy or an OT distance?,

l. 38: what is $d$?

l. 38: needs precising: I hope $m$ means the size of the sum of supports of two mini-batch measures,

l. 91-92:  a language issue in this sentence ("as the product measure which has the support is $m$ random variables follows $\mu$"?),

l. 127: huge/large,e.g., $\rightarrow$ huge/large, e.g., (missing gap),

l. 133: samples $\rightarrow$ sample,

l. 172 and 184: equation (5)/5 $\rightarrow$ Equation (5),

l. 174: measure $\rightarrow$ measures,

l. 213: the one-one mapping $\rightarrow$ the one-one "reshape" mapping,

l. 235: $\ldots$ $\rightarrow$ $, \ldots,$.
limitations: I would recommend including a separate paragraph to describe limitations (as it was done for potential impact, see Appendix F).
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Modern generative models aim at fitting the probability distribution of the observed data. To this end, a common strategy consists in minimizing a chosen divergence between the observed empirical distribution and a parametric distribution, over the set of parameters. In recent years, models based on the minimization of optimal transport (OT) metrics have attracted significant interest. However, the standard OT metric, namely the Wasserstein distance, suffers from an expensive computational cost and sample complexity in large-scale settings (i.e. when the compared distributions are high-dimensional or supported on a large number of samples). For that reason, using practical alternatives to the traditional Wasserstein distance is more convenient in generative modeling. 

Due to its lower computational complexity and favorable theoretical properties, the sliced Wasserstein distance has been successfully integrated within the generative modeling framework. Nevertheless, the sliced Wasserstein distance (SW) is defined as an expectation which is intractable in general. In practice, this distance is thus traditionally approximated with a simple Monte Carlo algorithm: the expectation is replaced by a finite sum over L projection directions. Computing the sliced Wasserstein distance then scales linearly with L, meaning that improving the accuracy of the Monte Carlo estimator (by increasing L) leads to higher computational requirements. This is a limitation of SW-based generative algorithms, since they need to compute SW at each training iteration. One solution is to pick "more informative" samples in the Monte Carlo strategy, which motivated the formulation of maximum sliced Wasserstein (max-SW): the expectation is replaced by a maximum operator.

In this paper, the authors argue that the mini-batch formulation of max-SW (mini-batch max-SW, equation (5)) can be computationally expensive when used in generative modeling, and they develop alternative pseudo-metrics inspired by amortized optimization to overcome this problem. Their main contributions are summarized below.

1) This paper introduces a new family of pseudo-metrics, coined "amortized sliced Wasserstein" (A-SW), by reformulating the mini-batch max-SW using amortized optimization (Definition 2). In other words, the optimal projection directions found in mini-batch max-SW are approximated with a family of parametric functions (the amortized model).

2) The authors prove that amortized SW are positive, symmetric (Proposition 1) and lower than mini-batch max-SW (Proposition 1).

3) Three instances of amortized SW are proposed by specifying the amortized model (Definitions 3, 4 and 5) and an analysis of their computational complexity as compared to mini-batch-max-SW is provided. 

4) Finally, GANs based on A-SW are proposed and compared against SNGAN and (max-)SW-based generators in image generation in terms of training speed, memory and quality of results, on 4 standard datasets (Section 5). According to the empirical results, A-SW is able to produce images of higher quality, at the price of a higher execution time and memory due to the optimization of the amortized model (depending on its parameterization).
strengths_and_weaknesses: Strengths:
- The problem addressed in this paper is adequate for NeurIPS: generative models based on optimal transport metrics, in particular variants of sliced Wasserstein distance, have gained popularity in the machine learning community, and it is important to develop strategies which further reduce their computational requirements.
- The main idea behind the paper is quite original since, to the best of my knowledge, amortized optimization (which has been previously deployed in other areas such as variational inference and auto-encoders, for example) had not been applied in optimal transport. 
- Overall, I found the paper clearly written and easy to follow. Some typos are reported in "Questions".

Weaknesses:

Unclear conclusion: In the end, the message of this paper is confusing to me: in the introduction, the authors motivate the development of A-SW as a strategy to reduce the computational complexity of mini-batch max-SW, but this aspect is not supported by the computational complexity analysis (l.237-247) nor the experimental section (l.324-326: A-SW is "comparable to Max-SW ... about the computational memory and the computational time", or even slower and more memory-intensive when the amortized model is not linear (Table 2)). According to Table 1, the main advantage in using A-SW instead of mini-batch-max-SW in generative models is not the speeding-up of the training, but the improved quality of the results (Table 1). However, it is not clear why this happens, which leads me to the next point ("Lack of theoretical justification").

Lack of theoretical justification: the proposed methodology is not theoretically-grounded in my opinion and the empirical analysis is not comprehensive enough to compensate this lack of insights, which makes me question the performance of the proposed generative algorithm. More specifically,
1) Proof of Proposition 2 (Appendix A.2): The conditions guaranteeing the existence of $\phi^*$ should be specified.
2) The parameterization of the amortized models is not clearly motivated. Are the proposed parameterizations standard in amortized optimization (if so, please add some references)? In particular, the amortization gap seems to be an important aspect in amortized optimization and is usually taken into account in the final methodology [1,2], but the authors chose not to study it (l.162-163). Are there any guarantees on whether the optimal projection directions lie in the space parameterized by Definitions 3, 4 or 5? 
3) A-SW outperforms mini-batch max-SW in terms of FID/IS scores in the image generation problem, and the authors conjecture that mini-batch max-SW performs poorly because it is stuck in a local optimum. What prevents A-SW from suffering from the same issue? This aspect could have also been explored empirically, which relates to my last point ("No uncertainty measure in the experiments").

No uncertainty measure in the experiments: In Section 5, if I am not mistaken, generative models have been trained only once on each dataset. In the checklist, the authors specified that they reported error bars, but I cannot find these anywhere. Having multiple runs is essential and can support the conjecture that A-SW-based generative models do not get stuck in a local optimum or at least less often than mini-batch max-SW generators (I am not fully convinced by this aspect for now).

[1] "Amortized Inference Regularization", Shu et al. (NeurIPS 2018)

[2] "Iterative Amortized Policy Optimization", Marino et al. (NeurIPS 2021)
questions: To address my concerns, I encourage the authors to:
- Clarify the motivation and take-home message of the paper: A-SW does not seem to provide computational advantages over mini-batch max-SW but instead, provides more accurate results ; this might be due to the design of the amortized models, which constraints the shape of the optimal projections, thus regularizes the underlying optimization problem.
- Add a more extensive discussion on the amortization gap to better motivate the proposed amortized models and the training procedure.
- Further illustrate the empirical superior performance of A-SW by: 1) running the experiments multiple times and report the associated error bars (Table 1, Figure 1); 2) finding an experiment where the training of mini-batch-max-SW-generator does not get stuck in a bad local optimum, and compare its performance against A-SW. It would also be interesting to see how mini-batch max-SW performs when the number of mini batches k is equal to 1. 

Typos:
Main document: l.92, l.263 (unclear sentences); l.102 (definition of $W_p$: the norm should be Euclidean so indexed by 2); equation (2) ($\nu_\phi$ instead of $\nu$); l.131 ($Y_{\phi,i}$ instead of $Y_{\phi_i}$); l.174 ("two probability measureS")

Appendix: derivations below l.523 (parenthesis issue, equation repeated twice); l.553; legend of Table 7.
limitations: The limitations of A-SW are discussed in Section 5 and seem to be the computational complexity (which, again, contradicts the original motivation). The potential negative societal impact is discussed briefly in Appendix F.
review_title: null

Meta-review: metareview: During the author-reviewer discussions, the authors have addressed most of the concerns raised by the reviewers, leading to original scores being raised. During the reviewer discussions, the disagreement among reviewers about the demonstration of computational benefits was discussed. At this point, the merits of the paper, including the originality of its contribution and the sufficient experimental validation, outweigh the doubts remaining with one of the reviewers. Therefore, the recommendation is to accept this submission.

I would like to thank the authors and reviewers for engaging in discussions.

confidence: Less certain
Decision: Accept