Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Instead of adding the ad-hoc \eta, wouldn't it be more principled to get a similar effect by putting the assumptions/constraints in the prior? For example, a hierarchical prior on z. I understand that that could introduce additional difficulties, but I was curious to get a comment from the authors.


review_title: The ad-hoc parameter

final_score: rating: null
review_content: I fail to understand what is the advantage of using a learned approximate posterior q_{\phi_z}(z|y) when in equation (6), you derive the *exact* form of p(z|x,w). Why don't you use the approximate posterior q_{\phi_w}(w|y)q_{\phi_x}(x|y)p(z|x,w) instead of q_{\phi_w}(w|y)q_{\phi_x}(x|y)q_{\phi_z}(z|y) ? Both are valid approximate posteriors, you have less parameters to train and your approximation would be less biased. I don't seem to see that experiment in the paper and I don't think it corresponds to any value of (\eta, \alpha).
The only justification I could see is that the integrals are not closed form anymore, but that have hardly been a major issue (if at all) when training variational auto-encoders with full differentiability. Is there something that I am not understanding ?
review_title: Advantage of q_{\phi_z}(z|y) ?

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This submission proposes an approach to adapting the variational auto-encoder framework (VAE) to the clustering scenario. First the model has to be adapted (with a Gaussian mixture as a prior) and then the inference has to become consistent (by introducing a regularization term). 

A general positive point about this paper is that the model construction is kept simple. Indeed, the assumption about the mixture prior is simple but reasonable, and the inference follows the VAE framework where appropriate with only changing parts that do not conform with the clustering task. These changes are also motivated by some analysis. The presentation is also kept simple: the linking to VAE and related methods is made in an clear and honest way, so that it's easy to follow the paper and understand how everything fits together. 

Also, the regularization term is a well motivated and reasonable addition. Given the VAE context in this paper, I'd be interested in seeing a discussion on the variance of the samples in (6).

A negative issue of this paper is that all crucial regularizations rely upon ad-hoc parameters that control their strength, namely \eta (eq. (3)) and \alpha (eq. 7). According to the authors adjusting these parameters is crucial, and there seems to be no principled way of adjusting them. It also seems that these two parameters interact, since they both regularize z in different ways. This makes the search space over them grow multiplicatively, since the tuning problem now becomes combinatorial. The authors mention that they tune the trade-off between these two regularizers, but I'd be interested in a comment concerning how this is done (what's the space of parameters to search on). In practical clustering applications, high sensitivity to tuned parameters is undesirable, since one also needs to cross-validate values of K at the same time.

I really liked the experiments section. It is not very exhaustive in terms of comparison, but it is very exploratory in terms of demonstrating the model components, strengths and weaknesses. This is much more useful than reporting unintuitive percentage improvements relative to arbitrarily selected baselines and datasets.


Overall, I am a little concerned about the practicality of this approach, given the tuning it requires. However, I am in favor of accepting this paper because it makes its strong and weak points very clear through good explanation and demonstration. Therefore, I expect further research to be built on top of this paper, so that the aforementioned issues will hopefully be alleviated in the future. Finally, the theoretical intuitions given in the paper (and author comments) improve its usefulness as a scientific manuscript.

review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors proposes to a variant of Variational Auto-Encoders using a mixture distribution to enable unsupervised clustering that they combine with an information-theoretical regularization. To demonstrate the merit of such approach, they perform experiments on a synthetic dataset, MNIST and SVHN.
The use of a mixture of VAE is an incremental idea if novel.
I would like to see the comparison with the more straightforward use of a mixture of gaussians prior. This model is more complex and I would like to see a justification of this additional complexity.
The results in Table 1 are questionable. First of all, GMVAE+ seems to outperform other methods with M=1, there should be a run of GMVAE+ with M=10 for proper comparison. But what I find more disturbing in this table is the variance of the results, especially since you are taking the "Best Run". Was the best run maximizing the validation performance or the test performance ? Moreover, the average performance (higher) and standard deviation (lower) of Adversarial Autoencoder makes me question the claim that "we have advanced the state of the art in deep unsupervised clustering both in theory and practice".
The consistency violation regularization might be interesting. But the cluster degeneracy problem is, as far as I know, also a problem in plain mixture of gaussians models. So making an experiment on this simpler model on a synthetic dataset should also be done.
In general, I would recommend running more experiments as to solidify your claims.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors posit a mixture of Gaussian prior for variational
auto-encoders. They also consider a regularization term motivated
from information theory.

The modeling extension is simple and the inference follows
mechanically from what's already standard in the literature. Instead
of using discrete latent variable samples they collapse the expected
KL; this works for few mixture components and has been considered
before in more general contexts, e.g., Titsias and Lazaro-Gredilla
(2015). It will not scale to many mixture components.

I find the discussion in Section 3.2.2 difficult to parse and, if I
understood it correctly, not necessarily correct. Many arguments are
introduced and few fleshed out. First, there is a claim that a
multinomial prior with equal class probabilities assigns the same
number of data points to each class on average; this is true a priori
but certainly not true given data. Second, they claim the KL
regularizer forces the approximate posterior to be close to this
uniform; this is only true for small data, certainly the energy term
in the ELBO (expected log-likelihood) will overpower the regularizer;
is this not the case in a mean-field approximation to a mixture of
Gaussians model? Third, there is a claim that "under the mean-field
approximation, this constraint is enforced on each sample"; how does
the mean-field approximation enforce a constraint on the effect of
Monte Carlo sampling? Fourth, they argue Johnson et al. (2016) can
overcome this issue partly due to SVI; how does data subsampling
affect this behavior? Fifth, they derive the exact posterior in
Equation 6; so to what extent are these arguments relevant?

The experiments are limited on toy data and only a few mixture
components are considered (not enough where the collapsed approach
will not scale).

+ Titsias, M. K., & Lázaro-Gredilla, M. (2015). Local Expectation Gradients for Black Box Variational Inference. In Neural Information Processing Systems.

review_title: Review: Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

Meta-review: The reviewers have looked through both the responses, updates, and had much discussion. We agree that the paper is well executed and exposes ideas that are of value and interest. At the same same, the extent to which the methods can be applied in practice and scaled to different types of problems remains unclear, especially in high-dimensional settings. For this reason, we feel that the paper is not yet ready for acceptance in this years conference.
Decision: Reject