Title: Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
Authors: Nat Dilokthanakul
Pedro A. M. Mediano
Marta Garnelo
Matthew C.H. Lee
Hugh Salimbeni
Kai Arulkumaran
Murray Shanahan
Venue: ICLR 2017
Keywords: Unsupervised Learning
Deep learning

Abstract: We study a variant of the variational autoencoder model (VAE) with a Gaussian mixture as a prior distribution, with the goal of performing unsupervised clustering through deep generative models. We observe that the known problem of over-regularisation that has been shown to arise in regular VAEs also manifests itself in our model and leads to cluster degeneracy. We show that a heuristic called minimum information constraint that has been shown to mitigate this effect in VAEs can also be applied to improve unsupervised clustering performance with our model. Furthermore we analyse the effect of this heuristic and provide an intuition of the various processes with the help of visualizations. Finally, we demonstrate the performance of our model on synthetic data, MNIST and SVHN, showing that the obtained clusters are distinct, interpretable and result in achieving competitive performance on unsupervised clustering to the state-of-the-art results.

TL;DR: We study a variant of the variational autoencoder model with a Gaussian mixture as prior distribution and discuss its optimization difficulties and capabilities for unsupervised clustering.

Review discussion:
I fail to understand what is the advantage of using a learned approximate posterior q_{\phi_z}(z|y) when in equation (6), you derive the *exact* form of p(z|x,w). Why don't you use the approximate posterior q_{\phi_w}(w|y)q_{\phi_x}(x|y)p(z|x,w) instead of q_{\phi_w}(w|y)q_{\phi_x}(x|y)q_{\phi_z}(z|y) ? Both are valid approximate posteriors, you have less parameters to train and your approximation would be less biased. I don't seem to see that experiment in the paper and I don't think it corresponds to any value of (\eta, \alpha).
The only justification I could see is that the integrals are not closed form anymore, but that have hardly been a major issue (if at all) when training variational auto-encoders with full differentiability. Is there something that I am not understanding ?
Hi,
Thank you for your insightful suggestion. We absolutely agree that using p(z|x,w) instead of q(z|y) is going to give us a better approximate posterior.
In fact, we just finished coded this up and ran some experiments. One benefit of using q(z|y) would be, q(z|y) would give us a convenient/cheap way to infer the classification labels.
But, not using an approximation would surely out weight this.
For the synthetic dataset, the modified posterior doesn't change our main results where adjusting eta and alpha is crucial for successful clustering. For MNIST with 1 mc-sample, it improves the best classification result for K = 16 by 4 per cent. No conclusive improvement for the case of K = 10 without extensive experiment. More experiments will be run and we will discuss this in the next version of the paper.
Instead of adding the ad-hoc \eta, wouldn't it be more principled to get a similar effect by putting the assumptions/constraints in the prior?
For example, a hierarchical prior on z. I understand that that could introduce additional difficulties, but I was curious to get a comment from the authors.
Thank you for your comment. We agree that our method could be extended with a hierarchical prior on z;
however, as you mentioned, this does introduce more difficulties. In our work we use eta to illustrate the effect of the z-prior term on clustering – as shown in Figure 2, turning off the effect of z-prior term improves the model’s ability to form separate clusters. With CV regularization, the z-prior term no longer needs to be turned off, thereby highlighting the benefit of information-theoretic regularization.
Although we are considering hierarchical priors or alternative priors for future work, we believe that we have shown that CV regularization is beneficial, and empirically it removes the need for the ad-hoc eta.
As a side note, CV regularisation could be interpreted as imposing an assumption that the Gaussians are not supposed to be overlapping. It would be interesting to make more connection to a sparse prior assumption on z.
The authors proposes to a variant of Variational Auto-Encoders using a mixture distribution to enable unsupervised clustering that they combine with an information-theoretical regularization. To demonstrate the merit of such approach, they perform experiments on a synthetic dataset, MNIST and SVHN.
The use of a mixture of VAE is an incremental idea if novel.
I would like to see the comparison with the more straightforward use of a mixture of gaussians prior. This model is more complex and I would like to see a justification of this additional complexity.
The results in Table 1 are questionable.
First of all, GMVAE+ seems to outperform other methods with M=1, there should be a run of GMVAE+ with M=10 for proper comparison.
But what I find more disturbing in this table is the variance of the results, especially since you are taking the "Best Run". Was the best run maximizing the validation performance or the test performance ?
Moreover, the average performance (higher) and standard deviation (lower) of Adversarial Autoencoder makes me question the claim that "we have advanced the state of the art in deep unsupervised clustering both in theory and practice".
The consistency violation regularization might be interesting.
But the cluster degeneracy problem is, as far as I know, also a problem in plain mixture of gaussians models.
So making an experiment on this simpler model on a synthetic dataset should also be done.
In general, I would recommend running more experiments as to solidify your claims.
We would like to thank you for your review and the very useful suggestion with regards to the inference model. We addressed a justification of the additional complexity by comparing with GMM in section 4.1. We do find that our model performs worse than adversarial autoencoders and we have included our hypothesis as to why this is the case in section 4.2. With regards to the concern about the results, we confirm that the results we report are testing results. Please also refer to our summary of the paper revision for a more comprehensive overview of the changes we have made in response to all of the reviewers.
The authors posit a mixture of Gaussian prior for variational
auto-encoders. They also consider a regularization term motivated
from information theory.
The modeling extension is simple and the inference follows
mechanically from what's already standard in the literature.
Instead
of using discrete latent variable samples they collapse the expected
KL; this works for few mixture components and has been considered
before in more general contexts, e.g., Titsias and Lazaro-Gredilla
(2015). It will not scale to many mixture components.
I find the discussion in Section 3.2.2 difficult to parse and, if I
understood it correctly, not necessarily correct. Many arguments are
introduced and few fleshed out.
First, there is a claim that a
multinomial prior with equal class probabilities assigns the same
number of data points to each class on average; this is true a priori
but certainly not true given data.
Second, they claim the KL
regularizer forces the approximate posterior to be close to this
uniform; this is only true for small data, certainly the energy term
in the ELBO (expected log-likelihood) will overpower the regularizer;
is this not the case in a mean-field approximation to a mixture of
Gaussians model?
Third, there is a claim that "under the mean-field
approximation, this constraint is enforced on each sample"; how does
the mean-field approximation enforce a constraint on the effect of
Monte Carlo sampling?
Fourth, they argue Johnson et al. (2016) can
overcome this issue partly due to SVI; how does data subsampling
affect this behavior?
Fifth, they derive the exact posterior in
Equation 6;
so to what extent are these arguments relevant?
The experiments are limited on toy data and only a few mixture
components are considered (not enough where the collapsed approach
will not scale).
+ Titsias, M. K., & Lázaro-Gredilla, M. (2015). Local Expectation Gradients for Black Box Variational Inference. In Neural Information Processing Systems.
Thank you for suggesting an interesting related paper. We have included it in section 3, where we acknowledge previous work on backpropagation through stochastic variables. We emphasise that our choice was made to make the inference model as simple as possible.
While we agree that scaling up sideways might not be the best idea as the model scales up linearly with the number of clusters, we believe that stacking GMVAEs on top of each other could in theory increases the effective number of clusters much better.
We do agree that our old argument about mean-field is not very well explained and not necessarily correct as you have mentioned.
As a result, we have approached this problem from a new angle and explained our rationale in section 3.3, 3.4 and in the experiments on the synthetic dataset. Please also refer to our summary of the paper revision for a more comprehensive overview of the changes we have made in response to all of the reviewers.
This submission proposes an approach to adapting the variational auto-encoder framework (VAE) to the clustering scenario.
First the model has to be adapted (with a Gaussian mixture as a prior) and then the inference has to become consistent (by introducing a regularization term).
A general positive point about this paper is that the model construction is kept simple.
Indeed, the assumption about the mixture prior is simple but reasonable, and the inference follows the VAE framework where appropriate with only changing parts that do not conform with the clustering task. These changes are also motivated by some analysis. The presentation is also kept simple: the linking to VAE and related methods is made in an clear and honest way, so that it's easy to follow the paper and understand how everything fits together.
Also, the regularization term is a well motivated and reasonable addition. Given the VAE context in this paper, I'd be interested in seeing a discussion on the variance of the samples in (6).
A negative issue of this paper is that all crucial regularizations rely upon ad-hoc parameters that control their strength, namely \eta (eq. (3)) and \alpha (eq. 7). According to the authors adjusting these parameters is crucial, and there seems to be no principled way of adjusting them. It also seems that these two parameters interact, since they both regularize z in different ways. This makes the search space over them grow multiplicatively, since the tuning problem now becomes combinatorial. The authors mention that they tune the trade-off between these two regularizers, but I'd be interested in a comment concerning how this is done (what's the space of parameters to search on). In practical clustering applications, high sensitivity to tuned parameters is undesirable, since one also needs to cross-validate values of K at the same time.
I really liked the experiments section. It is not very exhaustive in terms of comparison, but it is very exploratory in terms of demonstrating the model components, strengths and weaknesses. This is much more useful than reporting unintuitive percentage improvements relative to arbitrarily selected baselines and datasets.
Overall, I am a little concerned about the practicality of this approach, given the tuning it requires.
However, I am in favor of accepting this paper because it makes its strong and weak points very clear through good explanation and demonstration.
Therefore, I expect further research to be built on top of this paper, so that the aforementioned issues will hopefully be alleviated in the future.
Finally, the theoretical intuitions given in the paper (and author comments) improve its usefulness as a scientific manuscript.
We would like to thank you for the constructive and encouraging review. We hope that our new model without ad-hoc parameters is more attractive from a practical point of view. Please also refer to our summary of the paper revision for a more comprehensive overview of the changes we have made in response to all of the reviewers.
We would like to thank the reviewers for the constructive reviews which have, indeed, been very useful. We apologize for the late reply as we had to redo the majority of the experiments and rewrite a big part of the paper accordingly. The following is an outline of the major modifications to the paper:
1.) We have changed our inference model and use the approximate posterior suggested by AnonReviewer2. This means the model we now call GMVAE corresponds to what we called GMVAE+ in the old version of the paper. (See section 3.2).
2.) The entire analysis is now specific to this model. We have removed the argument about the mean-field problem which, as argued by AnonReviewer3, might not be strictly correct.
Instead, we approach the problem from a different angle and address the problem of over-regularisation (over-pruning) in VAEs. We have high confidence that this argument is both better suited and more grounded. (See section 3.3 and section 3.4).
3.) Following the analysis in section 3.4, we solve the problem with a heuristic called "minimum information constraint (free-bits)" that was introduced by Kingma et al. (2016).
As a result we do not resort to ad-hoc modifications like the eta-term in the previous version, but introduce a different tuning parameter, lambda, instead. We study this parameter, lambda, on the synthetic dataset and visualize its behaviour. We found that the GMVAE has a high tolerance region for the possible values of lambda and moreover does not require this heuristic for the MNIST experiments at all.
Therefore, from a practical point of view we have improved our model and alleviated the concerns raised by AnonReviewer1. (see section 4.1 and 4.2)
4.) The MNIST and SVHN experiments work successfully without any regularisation using the new inference model suggested by AnonReviewer2.
As a result, we decided to take out the section on consistency violation that introduced an extra tuning parameter with unintuitive behaviour.
5.) Crucially, we study the problem of over-regularisation in GMVAEs which manifests itself differently from regular VAEs. We explicitly show the behaviour of the "minimum information constraint" heuristic that we use to address the problem.
In conclusion, we have redone all experiments using the new inference model which led to a considerable increase in performance and allowed us to remove all ad-hoc tuning parameters. We show that, although we can train GMVAE using the normal ELBO on the MNIST dataset, the over-regularisation problem persists when trained on our synthetic dataset.
Finally, we show that a heuristic introduced to solve the over-regularisation problem in standard VAEs can be used to improve our model as well.
We hope the reviewers see the intuitions gained from the analysis and the experiments as useful contributions.
Although simple, we believe that GMVAEs or deep GMVAEs (stacked GMVAEs) are a very interesting class of model which has the potential to become very powerful as we gain more understanding about the underlying inference process.

Meta-review:
role: PC
data: comment: The reviewers have looked through both the responses, updates, and had much discussion. We agree that the paper is well executed and exposes ideas that are of value and interest. At the same same, the extent to which the methods can be applied in practice and scaled to different types of problems remains unclear, especially in high-dimensional settings. For this reason, we feel that the paper is not yet ready for acceptance in this years conference.

Decision: Reject