Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper provides an analysis of different VAE architectures with both Gaussian and categorical latent variables from the perspective of unsupervised clustering.
The authors propose a simple modification of the GMVAE model which removes the need of fiddling with the ELBO in order to make the model use the discrete latent variables.
The authors demonstrate empirically (on MNIST) that small differences in the model's architecture can have a substantial impact on how the discrete latent variables are used as measured by classification accuracy and conditional entropy.

Novelty

The contribution of this paper is in noticing that a relatively small change to the architecture of the GMVAE model allows it to more efficiently use the discrete latent variables while retaining a principled loss function.

Clarity

The different architectures explored, the experiment performed on MNIST and the experimental results are clearly explained. 
The analysis/discussion of model properties lacks some clarity. In particular, the analysis in section 3 offers little insight into the problem.

Significance

While the authors expose an intriguing aspect of the behavior of (GM)VAEs with discrete latent variables, the effect of these properties in semi-supervised learning is only conjectural at this point as no experiments were performed to analyse that.
More experiments are needed to assess the relevance of the observations made in this work.
Also note that, while having an interesting effect on how the model interacts with the discrete latent variables, the resulting ELBO of the proposed GMVAE is worse than that of M2.

Quality

The paper is ok overall. The text could be improved by focusing on the experimental results and on less speculative analysis of the model properties.
The strength of this paper would substantially improve if the authors had performed semi-supervised classification experiments.
review_title: The authors expose an intriguing aspect of the behavior of (GM)VAEs

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors seek to analyze (deep) discrete latent variable models
with variational inference---with specific focus on "unsupervised
clustering". However, the work lacks attribution to related work in
this problem, offer a nonstandard baseline, and do not discuss
scalability concerns of their proposal.

There is no discussion of any relevant work about (deep) discrete
latent variable models for clustering. This makes it difficult to
understand what insights to glean from the paper. For example,
Ranganath et al.  (2015) have applied deep exponential families with
variational inference for learning mixed memberships in text
corpora. Johnson et al. (2016) have studied Gaussian mixture models
with neural networks. There are other works in the area of topic
models and deep learning (e.g., Chien and Lee, 2017)---which get at
even more complicated (mixed membership) structures than the single
assignment in Gaussian mixtures. It would be useful to mention, if not
compare to, at least one model/inference that has been applied to this
domain.

Following the above, the authors analyze a flaw in a model no one has
actually used for unsupervised clustering leveraging neural nets.
Comparing to, e.g., Ranganath et al. (2015), would be more sensible if
the aim is to better understand the differences in models. Comparing
to, e.g., Johnson et al. (2016), would be more sensible if the aim is
to better understand the differences in inference.

Regarding the algorithm itself, the authors don't mention the scalability
issues present in the original GMVAE paper—as also described by the
reviewers (https://openreview.net/forum?id=SJx7Jrtgl&noteId=SJx7Jrtgl).

Minor remarks

+ I would recommend against the use of "variational models" in the
  title. This work analyzes specific extensions of the variational
  auto-encoder. It is not an interchangeable term with variational
  models, which are about rich posterior approximations (e.g., Tran et
  al., 2016).

References

Chien, Jen-Tzung, and Chao-Hsi Lee. "Deep Unfolding for Topic Models." IEEE Transactions on Pattern Analysis and Machine Intelligence (2017).

Johnson, M. J., Duvenaud, D., Wiltschko, A. B., Datta, S. R., & Adams, R. P. (2016). Composing graphical models with neural networks for structured representations and fast inference. In Neural Information Processing Systems.

Ranganath, R., Tang, L., Charlin, L., & Blei, D. M. (2015). Deep Exponential Families. In Artificial Intelligence and Statistics.

Tran, D., Ranganath, R., & Blei, D. M. (2016). The Variational Gaussian Process. In International Conference on Learning Representations.
review_title: Review: A Note on Deep Variational Models for Unsupervised Clustering

Decision: Reject