Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
- I think that VAEs are rather forced to be interpreted from an information theoretic point of view for the sake of it, rather than for the sake of a clear and unequivocal contribution from the perspective of VAEs and latent-variable models themselves. How is that useful for a VAE? 

- "The left vertical line corresponds to the zero rate setting. ...": All these limits are again from an information theoretic point of view and no formulation nor demonstration is provided on how this can actually be as useful. As mentioned earlier in the paper, there are well-known problems with taking this information theory perspective, e.g. difficulties in estimating MI values, etc.

- Breaking (some of) the long sentences and paragraphs in page 3 with an unequivocal mathematical formulation would smooth the flow a bit.

- "(2) an upper bound that measures the rate, or how costly it is to transmit information about the latent variable.": I am not entirely sure about this one and why it is massively important to be compromised against the obviously big first term.

- Toy Model experiment: I do not see any indication of how this is not just a lucky catch and that VAEs consistently suffer from a problem leading to such effect.

- Section 5: "can shed light on many different models and objectives that have been proposed in the literature... ": Again the contribution aspect is not so clear through the word "shed light".


Minor
- Although apparently VAEs represent the main and most influential latent-variable model example, I think switching too much between citing them as VAEs and then as latent-variable models in general was a bit confusing. I propose mentioning in the beginning (as happened) that VAEs are the seminal example of latent-variable models and then going on from this point onwards with VAEs without too much alternation between latent-variable models and VAEs.

- page 8: "as show"
review_title: Good analysis. Contributions are not so clear though.

final_score: rating: 7: Good paper, accept
review_content: EDIT:  I have reviewed the authors revisions and still recommend acceptance. 


Summary

This paper proposes assessing VAEs via two quantities: rate R (E[ KLD[q(z|x) || p(z)] ]) and distortion D (E[ log p(x|z) ]), which can be used to bound the mutual information (MI) I(x,z) from above and below respectively (i.e. H[x] - D <= I(x,z) <= R).  This fact then implies the inequality H[x] <= R + D, where H[x] is the entropy of the true data distribution, and allows for the construction of a phase diagram (Figure 1) with R and D on the x and y axis respectively.  Models can be plotted on the diagram to show the degree to which they favor reconstruction (D) or sampling diversity (R).  The paper then reports several experiments, the first being a simulation to show that a VAE trained with vanilla ELBO cannot recover the true rate in even a 1D example.  For the second experiment, 12 models are trained by varying the encoder/decoder strength (CNN vs autoregressive) and prior (fact. Gauss vs autoregressive vs VampPrior).  Plots of the D vs R and ELBO vs R are shown for the models, revealing that the same ELBO value can be decomposed into drastically different R and D values.  The point is further made through qualitative results in Figure 4.   


Evaluation

Pros:  While no one facet of the paper is particularly novel (as similar observations and discussion has been made by [1-4]), the paper, as far as I’m aware, is the first to formally decompose the ELBO into the R vs D tradeoff, which is natural.  As someone who works with VAEs, I didn’t find the conclusions surprising, but I imagine the paper would be valuable to someone learning about VAEs.  Moreover, it’s nice to have a clear reference for the unutilized-latent-space-behavior mentioned in various other VAE papers.  The most impressive aspect of the paper is the number of models trained for the empirical investigation.  Placing such varied models (CNN vs autoregressive vs VampPrior etc) onto the same plot from comparison (Figure 3) is a valuable contribution.       

Cons:  As mentioned above, I didn’t find the paper conceptually novel, but this isn’t a significant detraction as its value (at least for VAE researchers) is primarily in the experiments (Figure 3).  I do think the paper---as the ‘Discussion and Further Work’ section is only two sentences long---could be improved by providing a better summary of the findings and recommendations moving forward.  Should generative modeling papers be reporting final R and D values in addition to marginal likelihood?  How should an author demonstrate that their method isn’t doing auto-decoding?  The conclusion claims that “[The rate-distortion tradeoff] provides new methods for training VAE-type models which can hopefully advance the state of the art in unsupervised representation learning.”  Is this referring to the constrained optimization problem given in Equation #4?  It seems to me that the optimal R-vs-D tradeoff is application dependant; is this not always true?      

Miscellaneous / minor comments:  Figure 3 would be easier to read if the dots better reflected their corresponding tuple (although I realize representing all tuple combinations in terms of color, shape, etc is hard).  I had to keep referring to the legend, losing my place in the scatter plot.  I found sections 1 and 2 rather verbose; I think some text could be cut to make room for more final discussion / recommendations.  For example, I think the first two whole paragraphs could be cut or at least condensed and moved to the related works section, as they are just summarizing research history/trends.  The paper’s purpose clearly starts at the 3rd paragraph (“We are interested in understanding…”).  The references need cleaned-up.  There are several conference publications that are cited via ArXiv instead of the conference (IWAE should be ICLR, Bowman et al. should be CoNLL, Lossy VAE should be ICLR, Stick-Breaking VAE should be ICLR, ADAM should be ICLR, Inv Autoregressive flow should be NIPS, Normalizing Flows should be ICML, etc.), and two different versions of the VAE paper are cited (ArXiv and ICLR).  


Conclusions

I found this paper to present valuable analysis of the ELBO objective and how it relates to representation learning in VAEs.  I recommend the paper be accepted, although it could be substantially improved by including more discussion at the end.  



1.  S. Zhao, J. Song, and S. Ermon.  “InfoVAE: Information Maximizing Variational Autoencoders.”  ArXiv 2017.

2.  X. Chen, D. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Shulman, I. Sutskever, and P. Abbeel.  “Variational Lossy Autoencoder.”  ICLR 2017.

3.  I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. “Beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.”  ICLR 2017

4.  S. Bowman, L. Vilnis, O. Vinyas,  A. Dai, R. Jozefowicz, and S. Bengio.  “Generating Sentences from a Continuous Space.”  CoNLL 2016.
review_title: Review: Valuable formulation of VAE training tradeoffs

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper optimizes the beta-VAE objective and analyzes the resulting models in terms of the two components of the VAE loss: the reconstruction error (which the authors refer to as distortion, “D”) and the KL divergence term (which the authors refer to as rate, “R”). Various VAEs using either PixelCNN++ or a simpler model for the encoder, decoder, or marginal distribution of a VAE are trained on MNIST (with some additional results on OMNIGLOT) and analyzed in terms of samples, reconstructions, and their rate-distortion trade-off.

Review:

I find it difficult to point my finger to novel conceptual or theoretical insights in this paper. The idea of maximizing information for unsupervised learning of representations has of course been explored a lot (e.g., Bell & Sejnowski, 1995). Deeper connections between variational inference and rate-distortion have been made before (e.g., Balle et al., 2017; Theis et al., 2017), while this paper merely seems to rename the reconstruction and KL terms of the ELBO. Variational lower and upper bounds on mutual information have been used before as well (e.g., Barber & Agakov, 2003; Alemi et al., 2017), although they are introduced like new results in this paper. The derived “sandwich equation” only seems to be used to show that H - D - R <= 0, which also follows directly from Gibbs’ inequality (since the left-hand side is a negative KL divergence). The main contributions therefore seem to be the proposed analysis of models in the R-D plane, and the empirical contribution of analyzing beta-VAEs.

Based on the R-D plots, the authors identify a potential problem of generative models, namely that none of the trained models appear to get close to the “auto-encoding limit” where the distortion is close zero. Wouldn’t this gap easily be closed by a model with identity encoder, identity decoder, and PixelCNN++ for the marginal distribution? Given that autoregressive models generally perform better than VAEs in terms of log-likelihood, the model’s performance would probably be closer to the true entropy than the ELBO plotted in Figure 3a). What about increasing the capacity of the used in this paper? This makes me wonder what exactly the R-D plot can teach us about building better generative models.

The toy example in Figure 2 is interesting. What does it tell us about how to build our generative models? Should we be using powerful decoders but a lower beta?

The authors write: “we are able to learn many models that can achieve similar generative performance but make vastly different trade-offs in terms of the usage of the latent variable”. Yet in Figure 3b) it appears that changing the rate of a model can influence the generative performance (ELBO) quite a bit?
review_title: Potentially interesting but insights not clear

Meta-review: This paper gives a coding theory interpretation of VAEs and uses it to motivate an additional knob for tuning and evaluating VAEs: namely, the tradeoff between the rate and the distortion. This is a useful set of dimensions to investigate, and past work on variational models has often found it advantageous to penalize the latent variable and observation coding terms differently, for broadly similar motivations. This paper includes some careful experiments analyzing this tradeoff for various VAE formulations, and provides some interesting visualizations. However, as the reviewers point out, it's difficult to point to a single clear contribution here, as the coding theory view of variational inference is well established, and the VAE case has been discussed in various other works. Therefore, I recommend rejection.

Decision: Reject