Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: # Paper Summary

The paper extends the variational autoencoder framework with a richer prior distribution to model more complex correlations in the latent variable distribution. They start with a Gaussian mixture distribution as the prior for the latent variables, and add an encoder network to allow richer correlation structure in the latent variables.  Training the prior distribution requires an optimization between the prior distribution and the latent encoded distribution of the training data set. The paper starts with an existing method of optimizing the prior by computing an approximation of the Wasserstein distance between prior and encoded training distribution that uses an average over slices through the prior and encoded training distribution. The paper replaces linear projections used in prior work with a non-linear projection. The paper also employs a structural consistency term which has been used in prior work, however, the paper employs this term differently than prior work by applying it between encoder features and latent variables rather than inputs and latent variables. Since the latent variable space is now a complex and possibly a nonconvex submanifold, points in the latent space R^D  lying between points corresponding to training data points may not actually fall in the training distribution. The paper therefore proposes a method of interpolating between points in the manifold by constructing a graph between points sampled from the manifold and then choosing points lying along lines in the graph. The paper tests the method on three datasets, a synthetic 40 dimensional spiral dataset, the venerable MNIST dataset and a scaled down CELEB A dataset. Plots of the latent space trained on the spiral dataset shows that the latent space can in fact have complex internal structure. 

# Pros and Cons

Improving the ability of generative models to capture high-dimensional empirical distributions accurately is a key problem in machine learning and central to the representation learning theme of the ICLR community. 

The paper clearly states contributions up front, namely: using an encoder to generate richer priors for the latent variable distribution, non-linear projections for sliced Wasserstein approximation, and a graph based interpolation method.  They also alter the structural consistency term so that it applies to more abstract features instead of inputs. 

The paper does a thorough and clear job of covering prior work and technical background such as the Wasserstein metric, perhaps even excessively so.  

The particular choice of a sinusoidal non-linear projection, a key contribution according to the paper, is not motivated in the text. On first glance, the sinusoidal term seems like an odd choice for a non-linearity. After looking at the results which include a test on spirals, it is clearer why this might have be chosen, but is a sinusoidal term likely to be helpful for non periodic data? It might be possible to shed some light on this by investigating whether the coefficients in the non-linear term, zeta and gamma, are significantly different from zero after training on MNIST or the CELEB A data set used in the paper.  

Figure 3 comparing EPSWAE and SWAE doesn’t clearly illustrate the benefits of the EP component. In the print version, both grids of images seem blurry and prone to oddities such as overly large and dark eyes. Even when one blows up the image to 3X size using the digital version the advantage does not jump out. While I recognize that evaluating generative models is hard, the observations do not clearly support the author's hypothesis that the EP component provides an advantage. Maybe they could evaluate Freschet Inception Distance over the whole data set? Use blind human reviewers to choose between EPSWAE and SWAE based on realism and report preference scores? Interestingly, there is one duplicate image in the EPSWAE grid: row 1, col 5 and row 5, col 8 look identical. Seems odd to get identical images: is this because of sampling from a discrete graph structure? The highly similar but not identical images row 1, col 1 and row 1, col 8 are more what I would expect. Maybe the advantage could be made clearer by helping us focus on relevant features. For instance, it might be that EPSWAE is a little bit less likely to generate large black eyes? I can't tell from this small sample, but a grid that focuses on this might make the point. Hair versus background also seems to be a challenge. SWAE image row 3, col 4 seems to have two hair regions for the same face, but, EPSWAE images such as row 4, col 7 also look like they have two distinct hair regions. There are a couple of SWAE images that seem particularly ill formed: row 5, col 5  and maybe row 5, col 8, and row 7, col 4. It might be worth focusing on a few of the worst examples from both EPSWAE and SWAE to show differences in tails rather than the mean? I wonder if you could do a leave-one-out kind of analysis where you check the probability of held out training data points under the prior for EPSWAE vs. SWAE to assess if the prior is capturing the empirical distribution better? You would have to invert the prior network with gradient descent to do this ... but it might work. 

It might make sense to compress some of the tutorial material up front to make room for more results demonstrating the efficacy of the model. The MNIST results, fig 7 in appendix D, shows some advantage for EPSWAE: For instance, figure 7b and 7c both have more bloated numbers … especially 8, 3 and 0.  Also figure 7b has one degenerate number in position 1,2 … possibly a 2?  

The paper argues that the structured latent space improves generative power. Is there any evidence of structure in the latent space after training on CELEB A? If we plot 2 or 3D projections, or plot projections of structure preservering factorizations such as PCA, do we see structure in the encoding training data points or are they distributed independently? One might also try independence tests between variables in the encoded latent space to see if pairs of variables are being encoded with correlations. This could be compared easily between EPSWAE and SWAE. 

Figure 4 in section 5.4 on interpolation shows smooth interpolation between two points A and B in latent space presumably drawn from training data. The interpolations are pretty smooth. Nice! However, the authors claim is that the graph embedding gives better interpolations than linear interpolation between points in the latent space. To make this point, we would also need to see interpolations between points using linear interpolations. The plots on spiral might make this point, but it is not clear.  

The ability of VAE’s to disentangle the dimensions of an empirical distribution into indpendent latent variables is sometimes seen as a feature, not a bug. For instance, if the training data truly lie along a spiral, isn’t this really a 1D latent space and not a 3D space? While I can see the appeal of improving generated distribution realism, some discussion by the paper on the merits of improving encoder and decoder vs. complexifying the latent space would help to motivate this approach. 

It isn't clear to me that the structural consistency term is a good idea in general. Ideally we want the latent space to capture something fundamental about the underlying structure of the data and not features of the input. Moving the structural consistency from input to more abstract features addresses this concern somewhat, but aren't the latent values themselves the ultimate goal? 

# Recommendation

 I recommend a rejection of the paper. The hypotheses (that richer priors and geodesic interpolation generate better images on realistic images) are not clearly supported by the experimental results provided.

# Questions

 For the spiral training, what was the dimensionality of the latent space? Was it 3D?   

Figure 4 caption contains statement "through an intermediate sample corresponding to the midpoint in latent space". Are you actually literally using the midpoint? I thought graph embedding was being used to avoid using midpoints? Maybe the sentence is just ambiguous? 

How many samples are used in the Wasserstein approximation? How were the coefficients in the multi-term loss function defined (alpha, beta and kappa)? Oh - I see these are in the appendix... Seems like appendices are becoming pretty integral to papers these days ... Probably worth including these for the main results in section 5 for the two results presented. 

Step 3 in the graph embedding didn't make sense to me after reading it a couple of times. It wasn't quite clear how this sample specific weighting works. It would probably be worth expanding this a bit at the expense of background material, as it is one of the contributions of the paper. 

# Other Feedback

Page 2 "Adversarial methods are harder to train" ... also adversarial methods are implicit distributions -- you can sample from them but you cannot easily calculate the likelihood of an image under the adversarial model (although you can use gradient descent to try to find the latent parameters). This makes things like outlier detection difficult. 

Page 7, Section 5.2, last sentence refers to Fig 6, but I think this should be Appendix D, Fig 7? 

If you flipped the name around from EPSWAE to SWEP-AE, you would have a much more memorable acronym for people to take away from your paper/talk/poster although I recognize this doesn’t have the same “build” on previous work dynamic.  
review_title: Richer Priors and Geodesic Interpolation Might Improve Image Generation in VAEs  

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces an additional prior-encoder network to autoencoders to learn an unconstrained prior. The autoencoder and prior-encoder networks are iteratively trained with the sliced Wasserstein distance (SWD). To strengthen SWD, this paper further applies nonlinear transformations with a structural consistency term for better match between two distributions. For better interpolation on the latent space, it also introduces a graph-based algorithm. 

While the paper cites some works that also aims at addressing the drawback of SWD, it still misses some important related works like [a, b, c]. The paper is highly expected to make more discussion and suitable empirical comparison with them.

[a] Deshpande et al., Generative modeling using the sliced wasserstein distance, CVPR 2018.

[b] Wu et al., Sliced wasserstein generative models, CVPR 2019.

[c] Liutkus et al., Sliced-Wasserstein Flows: Nonparametric Generative Modeling via Optimal Transport and Diffusions, ICML 2019.

The motivation of using nonlinear transformation sounds not convincing. It is indeed known that traditional SW approximation generally requires a large amount of linear transformations. The bottleneck has been overcome by some existing works (Chen et al., 2020b; Kolouri et al., 2019; Nguyen et al., 2020; Deshpande et al., 2019, 2018). Why can the suggested nonlinear transformation avoid suffering from this issue? It is also not clear why to choose Eq.(6) for the nonlinear transformation. Are there any more excellent properties of the suggested nonlinear transformation compared to the existing methods? It is also necessary to make more discussions on the application of other nonlinear transformations.

The motivation of using the additional prior-encoder is not clear to me? The introduction states that it learns an unconstrained prior distribution that matches any data manifold topology. Unfortunately, I cannot find any clear explanation about this in the proposed method part. 

The evaluation is highly insufficient. The paper merely compares  the proposed method with SWAE which was published in 2018. More recent methods like [Deshpande et al. 2019, Wu et al. 2019, Liutkus et al. 2019] should be compared for a more complete study. Moreover, new generative modeling methods should be evaluated quantitatively using popular metrics like inception score, FID etc. Unfortunately, this paper does study this at all. In addition, the visual results of the proposed method seems not comparable with the state of the art on the CelebA dataset.

review_title: missing some related works, unclear motivation, and insufficient evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses the issues of representation learning with VAEs and propose EPSWAE as a solution. EPSWAE applies a prior encoder to construct an implicit prior, which is more flexible. Moreover, the authors apply the sliced Wasserstein distance for the matching between the posterior and the prior, enhance the conventional SWD with non-linear transformations and make the latent space similar to the feature space with a structural consistency loss. This paper also proposes a graph-based algorithm for minimizing the pathwise energy to achieve the manifold walking to improve the interpolation in the latent space.

-------------------

The paper is well written. The pipeline is clear and easy to understand. The representation learning with VAEs is a widely studied topic. Using a flexible implicit prior will boost the learning of the latent codes. The usage of the encoder (or generator in the adversarial cases) in the latent space is widely discussed in previous works, such as vampprior, semi-implicit VI, doubly semi-implicit VI, etc. Thus the contribution in this part is limited.  

The sliced Wasserstein distance is an efficient approximation of the Wasserstein distance for the distribution matching. To avoid the projections that contain useless information, as cited in this paper, a lot of papers generalize the typical random linear transform to non-linear transformation (Chen et al., 2020b; Kolouri et al., 2019; Nguyen et al., 2020; Deshpande et al., 2019). Here the authors propose the non-linear sliced Wasserstein (NSW) distance in Equation 5 with a transformation shown in Equation 6, which appears to be a special case that satisfies the four conditions discussed in (Kolouri et al., 2019). Is there any difference between NSW distance and the generalized SWD in Kolouri et al., 2019? 

The graph-based method is an interesting way for the manifold walking and is much better than conventional ways such as linear interpolation. I think this part should be discussed more in the paper. 

-------------------

Some detailed questions about the technique

* The authors claim the usage of FSC encourages pairwise distances of the latent code to be similar to the pairwise distances of the data features. I am curious if FSC is necessary for manifold learning. In figure 5, it does not show much difference with/without the FSC loss. 

* As claimed in the paper, the adversarial methods in latent space are expensive, while in the experiment part, there is no computation (such as time per update step) comparison with the adversarial methods. The usage of FSC also needs to compute the pairwise distance. Is that expensive too?

* In the abstract and the pseudocode shown in the appendix, the prior encoder is trained with sliced Wasserstein loss. Why we choose SWD for the training instead of NSW? 

* It is also confusing that in the method part, the prior encoder is trained with NSW loss, which is not consistent as claimed in the other parts of the paper.

* In the experiments, the authors claim the generations are better, while the quantitative results, such as fid, are not provided. 

* For the spiral toy dataset: views in Fig. 5 seem not consistent and are hard to compare. How about we compare in 2d cases?

In conclusion, the paper has the merits, and these investigations may be helpful for this problem, but it is not enough and need to dig out more to be an ICLR publication.  

---

Update after the discussions

I appreciate the efforts that the authors make in their responses, some of which address my concerns and improve the quality of the paper.

I have raised my rating. However, taking into account all information during the discussion phase, I stick to my original review that this paper still needs to explore more to be a mature publication. For example, if the main contribution comes from the prior encoder, as I said the contribution is limited since the usage of the encoder (or generator in the adversarial cases) in the latent space is widely discussed in previous works, such as vampprior, semi-implicit VI, doubly semi-implicit VI, etc. This also seems to make the contribution of the sliced Wasserstein part incremental. Plus, as this paper has several components, their relations need to be discussed in a more clear way. Thus, more ablation studies are needed to help this paper to present its insight in a much more clear way.

Thanks again for the efforts that the authors make and I hope my reviews could help them to polish this paper to be a nice publication. 

review_title: This paper addresses the representation learning of VAEs and is reasonable, but need to dig more.

Meta-review: All three referees have provided detailed comments, both before and after the author response period. While the authors have carefully revised the paper and provided detailed responses, leading to clearly improved clarity and quality, there remain clear concerns on novelty (at least not sufficiently supported with ablation study) and experiments (neither strong enough nor sufficient to support the main hypotheses). The authors are encouraged to further improve their paper for a future submission.
Decision: Reject