Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed symmetric Wasserstein autoencoders (SWAE), which is a new type of generative autoencoders within the framework of optimal transport (OT). This work is based on Wasserstein autoencoders (WAE), but leverage a symmetric distance between the encoding distribution and the decoding distribution to better preserve the local structure of the data in the latent space. A reconstruction loss based regularization is suggested for better performance. The paper is well written and easy to follow. My only concern is that the improvement over WAE seems to be marginal. Below are some minor issues that need to be addressed. 

1. Please provide the appendix.

2. The data sets used in the experiment are kind of too simple. Can the author provide results for full CIFAR10? 

3. The FID of WAE-GAN and WAE-MMD in Table 2 does not look right, can the author provide implementation info? Did you use your own code?


review_title: A nice generative autoencoder using optimal transport

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to treat the encoding and the decoding pairs symmetrically as a solution to OT problems. SWAE minimizes $p(x_d, z_d)$ and $p(x_e, z_e)$ in a jointly manner and shows better latent representation learning and generation. Moreover, the symmetric treatment for encoding and decoding shows an advantage in data denoising. 


----------

It is interesting to minimize the distance between $p(x_d, z_d)$ and $p(x_e, z_e)$ with the OT formulation.  The usage of the deterministic encoder and decoder could solve the problem of $W(p_{x_e}, p_{e_d})$ minimization, while it is difficult for the latent code since the latent codes from the prior and the posterior are not paired. Here it is good to see the authors apply aggregated posterior methods to solve this. The usage of the closest pseudo-inputs in constructing the pair of $z_e$ and $z_d$ could be a good approximation for $W(p_{z_e}, p_{z_d})$.

My questions mainly lie in the experiment part:

The authors report SWAE with three configurations, where $\beta=1$, $\beta=0.5$ and $\beta=0$ respectively. When $\beta \rightarrow 1$, SWAE is only minimizing the $d( p(x_d, z_d), p(x_e, z_e) ) $; when $\beta \rightarrow 0$, SWAE behaves like an autoencoder with regularization in latent space.  The classification results and the reconstruction results of the case $\beta = 0$  are missing. I am curious how these two cases perform, are they perform like VAE and Vampprior VAE? 

The authors visualize the latent space with t-sne where the dimensionality of $z$ is 80 before the dimension reduction. As far as I know, when the dimension of $z$ is large enough, the t-sne results of VAEs and WAEs are similar to figure 3a, 3b and 3g. The results shown in c,d,e and f seem degenerated.  Especially 3c, it does not look like what we have seen in previous papers. For example, figure 2 in Makhzani et al, 2015 (https://arxiv.org/pdf/1511.05644.pdf) shows that the latent representations produced by VAE have several modes.  I doubt this could be the problem of dimension reduction. It will be more convincing if the author could show the visualization of these models where $dim(z) = 2$ and without any dimension reduction. 

For the reconstruction results (not the denoising reconstruction) shown in Figure 9 and 10, the images seem to be dynamically binarized. However, in the description of the experiment details, this part is not mentioned. Considering SWAE is not applying a Bernoulli decoder, the dynamic binarization is not necessary. Moreover, the reconstruction from SWAE ($\beta=0.5$) and SWAE ($\beta=0$) seem to be binarized, which is not reasonable.

When $\beta=1$, a good baseline for SWAE will be ALI (Dumoulin et al., 2017), since ALI is minimizing $JS(p(x_d, z_d)|| p(x_e, z_e))$. I am curious how SWAE ($\beta=1$) outperforms/underperforms compared to ALI and some corresponding analysis from the authors will be appreciated.

In the random generation results, only SWAE ($\beta^\star$) is shown. The generation results by SWAE ($\beta=1$) and SWAE ($\beta=0$) are missing while the quantitative results are shown in table 2.


---------------

In general, the idea of symmetrically treating encoding and decoding to solve with the Wasserstein distance is interesting and is worthy of study. However, some details in the experiments remain unclear; some experiments results and the corresponding analysis are missing. The authors might need to dig out more to support their method. 

review_title: Interesting method, but the experiments need to be refined

final_score: rating: 5: Marginally below acceptance threshold
review_content: In the paper, the authors propose a new family of generative auto-encoders, named Symmetric Wasserstein Autoencoders (SWAEs), based on replacing the KL divergence between the encoding and decoding distributions on the traditional VAE framework into the Wasserstein metric between these distributions. Then, they also carried out experiments to demonstrate the favorable performance of their auto-encoder over previous base-line autoencoders.

I think the SWAEs is quite interesting but lacks novelty. Here are my comments with the paper:

(1) The result of Theorem 1 is under the assumption that both the encoder and decoder are deterministic, which is quite restrictive. Can the authors provide some understandings when either one of them is random? The result of Theorem 1 is also quite similar to the main result in the Wasserstein autoencoder work. I think the authors should at least cite this point properly.

(2) Simply replacing KL divergence between the encoding and decoding distributions by Wasserstein metric to have symmetric properties sounds not novel. The KL divergence has a good interpretation in terms of ELBO. What can we interpret the Wasserstein metric between the encoding and decoding distributions? Theorem 1 does not seem convincing to me.

(3) In the experiments, the choice of $\beta = 1/2$ seems to yield best results, i.e., we should balance the reconstruction loss and the discrepancy in the data space in the objective function (4). Can the authors provide some intuition behind that?

(4) The related work with generative modeling based on optimal transport lacks several relevant recent works; see for examples [1], which achieves SOTA among sliced-based Wasserstein distances in deep generative models, or [2], which proposes max-sliced Wasserstein distance for deep generative models.

References:

[1] K. Nguyen, N. Ho, T. Pham, H. Bui. Distributional sliced-Wasserstein and applications to deep generative modeling. Arxiv preprint Arxiv: 2002.07367, 2020.

[2] I. Deshpande, Y. Hu, R. Sun, A. Pyrros, N. Siddiqui, S. Koyejo, Z. Zhao, D. Forsyth, A. Schwing. Max-sliced Wasserstein distance and its use for GANs. CVPR, 2019.
review_title: The proposal of Symmetric Wasserstein Autoencoders is quite interesting but lacks novelty.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:
 
This works proposes an new auto-encoder variant based on an Optimal Transport (OT) penalty.  While there are many such previous works of OT and auto-encoders, this work proposes a joint OT penalty on data and latent space. As the scalability of computing OT penalties in high dimensions is a concern, the authors address this by restricting to deterministic encoders and decoders in Theorem 1, an extension to joint distributions of Theorem 1 of Tolstikhin 2018. The resulting algorithm amounts to a loss involving L2 penalties for (1) the reconstruction loss (2) decoded latents (conditional on "pseudo-inputs") and real samples (3) encoded samples and the conditional latents. Next experimental results are shown on small-scale datasets (MNIST, Fashion-MNIST, Coil20, subest of CIFAR-10) and compared against the VAE, WAE-{GAN,MMD}, VampPrior, and MIM.

##########################################################################

Reasons for score: 

Overall I vote for rejection. Given the numerous prior works (properly cited in the paper), the novelty of the proposed loss appears limited to me. Theorem 1 also appears to be limited novelty over the Theorem 1 of Tolstikhin 2018. Additionally, the empirical evaluation of the method is only limited to small-scale datasets. 
 

##########################################################################

Pros: 
* Easy to read
* Reasonable selection of prior models to compare against

##########################################################################

Cons: 
* Only small-scale datasets are evaluated. Thus the empirical advantage is unclear
* Theorem 1 appears to be of limited novelty over Theorem 1 in Tolstikin 2018


##########################################################################

Questions during rebuttal period: 
 
Why is your theorem novel over Theorem 1 in Tolstikin 2018?

What is the performance of your method on more complex datasets such as CelebA and LSUN Bedroom?

#########################################################################

POST-REBUTTAL RESPONSE:

Thanks for clarification on Theorem 1 of this paper, i.e. that the novelty is in interpretation. I agree that the "denoising" between observed and generated data is an interesting idea.

I read the author's additional experiments on CelebA. In Figure 10, VampPrior is still qualitatively superior to the author's best result $SWAE(\beta^*=0.5)$. I have some skepticism over the reported results for WAE-{GAN/MMD}, which are much worse than the results in the original paper (Tolstikhin 2018). The authors appear to have used different encoder/decoder architectures, which complicates the comparison. Is WAE's decreased performance due to choice of architecture or algorithm? FID scores on CelebA would be also helpful.

All told, I raise my score, but still harbor some doubts over the empirical advantage of this work.


review_title: Unclear novelty of Theorem 1 over Tolstikhin 2018 / Empirical evaluations limited to small-scale datasets

Meta-review: In this paper, the authors proposed a new variant of the Wasserstein autoencoder (WAE), which matches the joint distribution of data and the latent codes induced by the encoder and the joint distribution induced by the decoder in the framework of optimal transport. Because of matching the distributions that are not considered by existing autoencoders like WAEs or VAEs, I agree with the authors that the proposed method is novel to some degrees. 

However, the experimental part does not support the superiority of the method well. For example, some reviewers (including me) think the results of the baselines shown in Figures 8-10 are underestimated. According to my personal experience, the WAE should perform much better on CelebA than that shown in Figure 10. The experiments in Figures 11-17 provide more reasonable results, but the advantage of the proposed method is not convincing.

Here is my suggestion:
1) Because the proposed method can achieve flexible prior, besides randomly generating data, the authors can consider adding some experiments on conditional generation, i.e., generating data from a single modality of the learned prior. I believe the proposed method will be more convincing if it can show some advantages in the conditional generation task.
2) The runtime comparison for the method and the baselines in the training phase should be discussed. 
3) The short name "SWAE" is in conflict with an earlier work "Sliced Wasserstein Autoencoder", which is also called "SWAE".
Decision: Reject