Venue: ICLR 2018 Workshop

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors measure the Fisher information in generative models (VAEs) and show that this quantity decreases for each layer in the neural networks used to parameterize the approximate and generative distributions, which is explained by vanishing gradients in deep neural networks. The authors then suggest that we can mitigate this effect by including skip connections in the models.

A series of experiments is performed using VAEs with deep encoders or decoders, both with or without skip connections. The results are compared using the ELBO and a score of how accurate an SVM can classify the MNIST digit using the latent representation as input.

I liked the experiments comparing deep encoders and decoders and how it affects the latent representation. My main concern with the paper is that there seems to be few new contributions e.g. skip connections are well known to improve deep neural networks (e.g. RESnets or Maaloee et. al 2016 [deep auxiliary generative models]) and secondly that the fisher informaton have been explored in e.g. Disjardins et. al mentioned in the paper. Lastly the paper could use some more proof reading (e.g. fishier -> fisher and other)

Overall i think the paper in its present state have to few new contributions to support a publication at the ICLR-workshop



review_title: Review

final_score: rating: 3: Clear rejection
review_content: 
Summary

This paper proposes measuring information loss within the deterministic encoder/decoder layers via the Fisher Information at that layer.  Adding skip connections to the VAE is then recommended to mitigate this information loss.  

Evaluation

Pros:  The experiments---albeit on only MNIST---do show improvements when skip-connections are added to the VAE.

Cons:  

Firstly, regarding the proposal to use Fisher Info.: Proposition #1 is concerned with the parameters of the encoder /  decoder but not the latent variable itself.  This makes the treatment not specific to VAEs and a general statement about NNs.  Moreover, I’m not sure en/decoder Fisher Information is as revealing a quantity for VAEs as is the mutual information between x (data) and z (latent code), as studied in [1], although I’m sure the two can be related.

Secondly, the paper’s proposal to use skip-connections is not a novel contribution.  Skip connections have been used within VAEs dating back to 2016 [2, 3].  The paper makes no mention of this, even though [2] is cited.   


1.  S. Zhao, J. Song, and S. Ermon.  “InfoVAE: Information Maximizing Variational Autoencoders.”  ArXiv 2017.

2.  D. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling.  “Improving Variational Inference with Inverse Autoregressive Flow.”  NIPS 2016.

3.  L. Maaloe, C. Sonderby, S. Sonderby, and O. Winther. “Auxiliary Deep Generative Models.”  ICML 2016.

review_title: Review: Low novelty, omission of related work

final_score: rating: 3: Clear rejection
review_content: Intuitively, the basic hypothesis in this paper makes sense: It might be challenging for deeper en- and decoders in VAEs to maintain a high mutual information between the input and the latents or the input and the reconstruction respectively. But I’m sceptical whether fisher information is the right metric to analyze this: The FI measures the amount of information a hidden representation carries about the encoders *parameters*. It seems easy to conceive cases where an encoder maintains high mutual information between its input and output variables, but a low FI estimates for its parameters and vice versa.

It should also be noted out that ResNet-style en- and decoder architectures have been used for a long time in VAEs for image modeling -- and ResNet blocks contain skip-connections to ease training.

In the experimental section the authors should clarify how they computed the NLLs: Does the decoder use a binary / Bernoulli likelihood? How were the inputs binarized? (e.g. dynamically sampled proportional to grey-level during training?). I’m mentioning this because the baseline result of 87.9 nats (testset) for 1-layer fully-connected VAE seems implausible good for the std. binarization. Archiving ~80.2 nats with any kind of fully connected architecture would be spectacular.

review_title: Skip connections in VAE en- and decoders

Meta-review: Based on the reviews, this paper has not been accepted for presentation at the ICLR workshop. However, the conversation and updates can continue to appear here on OpenReview.
Decision: Reject