Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding
Authors: Yigit Ugur
George Arvanitakis
Abdellatif Zaidi
Venue:  (2020)
Keywords: clustering
Variational Information Bottleneck
Gaussian Mixture Model

Abstract: In this paper, we develop an unsupervised generative clustering framework that combines variational information bottleneck and the Gaussian Mixture Model. Specifically, in our approach we use the variational information bottleneck method and model the latent space as a mixture of Gaussians. We derive a bound on the cost function of our model that generalizes the evidence lower bound (ELBO); and provide a variational inference type algorithm that allows to compute it. In the algorithm, the coders’ mappings are parametrized using neural networks and the bound is approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on real datasets are provided to support the efficiency of our method.

Under review as a conference paper at ICLR 2020
VARIATIONAL INFORMATION BOTTLENECK FOR UN-
SUPERVISED CLUSTERING: DEEP GAUSSIAN MIX- TURE EMBEDDING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, develop unsupervised generative clustering framework that combines variational information bottleneck and the Gaussian Mixture Model. Specifically, our approach we use the variational information bottleneck method and model the latent space as a mixture of Gaussians. We derive a bound on the cost function of our model that generalizes the evidence lower bound (ELBO); and provide a variational inference type algorithm that allows to compute it. In the algorithm, the coders' mappings are parametrized using neural networks and the bound is approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on real datasets are provided to support the efficiency of our method.
INTRODUCTION
Clustering consists in partitioning given data set into various groups (clusters) based some similarity metric, such as Euclidean distance, Lı norm, L2 norm, Los norm, the popular logarithmic loss measure or others. The principle that each cluster should contain elements of the data that closer to each other than to any other element outside that cluster, in the sense of the defined similarity measure. If the joint distribution of the clusters and data is not known, one should operate blindly in doing so, i.e., using only the data elements at hand; and the approach is called unsupervised clustering. Unsupervised clustering is perhaps one of the most important tasks of unsupervised machine learning algorithms nowadays, due to a variety of application needs and connections with other problems.
Examples of unsupervised clustering algorithms include the so- co-popular K-means (Hartigan Wong 1979) and expectation maximization (EM) (Dempster al. 1977). The K-means algorithm partitions the data in manner that Euclidean distance among the members of each cluster is minimized. With the EM algorithm, the underlying assumption is that the data comprises mixture of Gaussian samples, namely a Gaussian Mixture Model (GMM); and one estimates the parameters of each com- ponent of the GMM while simultaneously associating each data sample Although they offer some advantages in the context of clustering, these algorithms suffer from some strong limitations. For example, order of the data and scaling; and the obtained accuracy depends strongly on the initial seeds (in addition that it does not predict the number of clusters K --value). The EM algorithm suffers mainly from low convergence, especially for high dimensional data.
of those components.
well known that the K K-means is highly sensitive both the
Recently, new approach has emerged that seeks to perform inference on a transformed domain (generally referred to as latent space), not the data itself. The rationale is that because the latent space often has fewer dimensions on it rather than on the high dimensional data directly. A key aspect then is how to design latent space that is amenable to accurate low-complex unsupervised clustering, i.e., one that preserves only those features of the observed high dimensional data that are useful for clustering while removing out all redundant or non-relevant information. Along this line of work, we can mention (Ding & He 2004) which utilizes Principal Component Analysis (PCA) (Wold et al. 1987) for dimensionality reduction followed by K -eaass for clustering the obtained reduced dimension data; or (Roweis) 1997) which uses combination of PCA and the EM algorithm. Other works that alternatives
more convenient computationally to perform inference (clustering)

---

Under review as a conference paper at ICLR 2020
for the linear PCA include Kernel PCA (Hofmann et al. 2008), which employs PCA in non-linear fashion to maximize variance in the data.
The usage of deep neural networks (DNN) for unsupervised clustering of high dimensional data on a lower dimensional latent space has attracted considerable attention, especially with the advent of autoencoder (AE) learning and the development of powerful tools to train them using standard backpropagation techniques (Kingma & Welling, 2014 Rezende et al.] (014).. Advanced forms include Variational autoencoders (VAE) (Kingma & Welling, 2014 Rezende al., 2014) which are generative variants of AE that regularize the structure of the latent space and the more general Variational Information Bottleneck (VIB) of (Alemi et al. (2017) which is technique that is based on the Information Bottleneck method (Tishby et al. 1999) and seeks a better trade-off between accuracy and regularization than VAE via the introduction of Lagrange-typp parameter & which controls that trade-off and whose optimization is similar to deterministic annealing (Slonim) 2002) or stochastic relaxation.
In this paper, we develop an unsupervised generative clustering framework that combines VIB and the Gaussian Mixture Model. Specifically, in our approach we use the variational information bottleneck method and model the latent space mixture of Gaussians. The encoder and decoder of the model are parametrized using neural networks (NN). The cost-function is calculated approximatively by Markov sampling and optimized with stochastic gradient descent. Furthermore, the application of our algorithm to the unsupervised clustering of various datasets, including the MNIST (Lecun et al., 1998), REUTERS (Lewis et al., 2004) and STL-10 (Coates et al., 2011). 1 allows better clustering accuracy than previous state of the art algorithms. For instance, we show that our algorithm performs better than the variational deep embedding (VaDE) algorithm of (Jiang et al. 2017) which is based on VAE and performs clustering by maximizes the ELBO and can be seen as a specific case of our algorithm (Section 3.1). Our algorithm also generalizes the VIB of (Alemi et al.] (017) which models the latent space as an isotropic Gaussian which is generally not expressive enough for the purpose of unsupervised clustering. Other related works, but which are of lesser relevance to the contribution of this paper, are the deep embedded clustering (DEC) of (Xie et al., 2016). the improved deep embedded clustering (IDEC) of (Guo et al., 2017) and (Dilokthanakul et al. 2017). For a detailed survey of clustering with deep learning, the readers may refer to (Min et al. 2018).
To the best of our knowledge, our algorithm performs the best in terms of clustering accuracy by using deep neural networks without any prior knowledge regarding the labels (except the usual assumption regarding the number of the classes) compared to the state-of-thh--ut algorithms of this category. In order to achieve the aforementioned accuracy, i) derive cost-function that contains the IB hyper parameter s that controls the trade-off between over-fit and generalization of the model and we used an approximation of KI. divergence that avoid assumptions which do not hold in the beginning of the learning process and lead to convergence issues; ii) evaluate the arper-parameter by following an annealing approach that improves both the convergence and the accuracy of the proposed algorithm.
Encoder
U~ E. N(u Ee)
Decoder
x
X
fe
IeD
Figure 1: Variational Information Bottleneck with Gaussian Mixtures.
2 PROBLEM DEFINITION AND MODEL.
Consider dataset that is composed of N samples {x, J.N which we wish to partition into |C] > 1 clusters. Let = {1...., |C]] be the set of all possible clusters; and C' designate a categorical random variable that lies in C and stands for the index of the actual cluster. If X is random variable that models elements of the dataset, given X = x; induces a probability distribution on C which the learner should learn. Thus, mathematically the problem is that of estimating the values of the
2

---

Under review as a conference paper at ICLR 2020
unknown conditional probability (c)x(-lx,) for all elements x, of the dataset. The estimates sometimes referred to as the assignment probabilities.
As mentioned previously, resulting model is depicted in Figure where the parameters Te. Mex Σ., for all values of с E C, to be optimized jointly with those of the employed NNs instantiation of the coders. Also, the assignment probabilities are estimated based on the values of latent space vector instead of the observation themselves, i.e., Peiu Qciu. In the rest of this section, we elaborate the inference and generative network models for our method, which are illustrated below.
the VIB framework and model the latent space GMM. The
P
Pux
Que
Qx
x
X
Figure 2: Inference Network
Figure 3: Generative Network
2.1 INFERENCE NETWORK MODEL
We assume that observed data is generated from GMM with |C| components. Then, the latent representation u is inferred according the following procedure:
One of the components of the GMM is chosen according to a categorical variable C.
2. The data x is generated from the c-th competent of the GMM, i.e., Pxic ~ NX(x jec, Ee). latent representation according to Pu x N(uo Eg).
3. Encoder maps
3.1. The encoder is modeled with DNN fo which maps to the parameters of Gaussian distribution, i.e., [14,, Σn] = [x).
3.2. The representation u is sampled from ((p. Eg).
For the inference network, shown in Figure 2 the following Markov chain holds
C
(1)
2.2 GENERATIVE NETWORK MODEL
Since encoder extracts useful representations of the dataset and we assume that the dataset is generated from a GMM, we model our latent space also with mixture of Gaussians. To do so, the categorical variable C' is embedded with the latent variable U. The reconstruction of the dataset is generated according to the following procedure:
1. One of the components of the GMM is chosen according categorical variable C, with prior distribution Qc.
2. The representation u is generated from the e-th component, i.e., Quic ~ W(u: Me. E.).
3. The decoder maps the latent representation u to & which is the reconstruction of the source
x by using the mapping Qxiu-
3.1. The decoder modeled with a DNN go. that maps u to the estimate x, i..., [ = (u).
For the generative network, shown in Figure the following Markov chain holds
C+UeX.
PROPOSED METHOD
In this section we present our clustering method. First, we provide general cost function for the problem of the unsupervised clustering that study here based the variational IB framework; and we show that it generalizes the ELBO bound developed in (Jiang et al. 2017). We then parametrize model using NNs whose parameters optimized jointly with those of the GMM. Furthermore, we discuss the influence of the syper-parameter that controls optimal trade-offs between accuracy and regularization.
3

---

Under review as a conference paper at ICLR 2020
3.1 BRIEF REVIEW OF VARIATIONAL INFORMATION BOTTLENECK FOR UNSUPERVISED LEARNING
As described in Chapter the stochastic encoder Puix maps the observed data a representation u. Similarly, the stochastic decoder Qx|u assigns an estimate & of x based on the vector u. As per the IB method (Tishby et al., 1999) suitable representation U should strike the right balance between capturing all information about the categorical variable C' that is contained in the observation X and using the most concise representation for This leads to maximizing the following Lagrange problem
C.(P) I(C; U) s/ (X: U).
where 0 designates the Lagrange multiplier and for convenience P denotes the conditional distribution Puix.
Instead of equation which is not always computable in unsupervised clustering setting, use a modified version (so-called unsupervised IB objective (Alemi et al. 2017) given by
(P) H(X - s (H(U H()
(4)
Fx rux [log Px s log Pu s log Puxx]]
(5)
For variational distribution Qu on Ui (instead of the unknown Pt) and a variational stochastic decoder Qx (instead of the unknown optimal decoder Px(U), let Q == (Qxlu. Qu). Also, let
P. Q) == Exx [EPoix [log Qx $DKL ( [u]]]
(6)
Lemma 1. For given P, we have
CVB (P for all Q
In addition, there exists a unique Q that achieves the maximum maxq CVB (P, Q) = LP)(), and is given by
Øx = -x Pu.
Using Lemma [ maximization of equation 4 can be written term of the variational IB cost as follows
max C' max C CP
Q
Remark 1. As we already mentioned in the beginning of this chapter, the related work (Jang et al. 2017; performs unsupervised clustering by combining VAE with GMM. Specifically, it maximizes the following ELBO bound
VaDE Epx [Erujx [log Qxiu - DKr. (Prxxxla - -E EPrex xIIQu
(8)
Let, for arbitrary non-negative parameter s, L LVaDE be generalization of the ELBO bound in equation [ of (iann et al.) (207)) given by
CY = Epx [EPujx [log Qxiu! DDI.
PP (9)
Investigating the RHS of equution!
we get
VV CBB Q) = VVDE + SEY EErox [KL (Qc]u)]]
(
(10)
Thus, by the non-negativity of relative entropy it is clear that VaDE is a lower bound on C (P Q) Also, if variational distribution Q is such that the conditional marginal Qciu is equal Pcix the bound is tight since the relative entropy term is zero in this case.
3.2 PROPOSED ALGORITHM; VIB-GMM
In order to compute equation we parametrize the distributions Puix and QxIu using DNNs. For instance, let the stochastic encoder Pujx be DNN fo and the stochastic decoder Qxju be a DNN
Go. That
Po(u)x) -N(u; sso En) (x(x (u) [X]
where [mo En] fo(x).
(1I)
4

---

Under review as a conference paper at ICLR 2020
where 9 and are the weight and bias parameters of the DNNs. Furthermore, the latent space is modeled as GMM with |C] components with parameters &b (Tes Me: Eejen.
Q+(u) Σ Te (u: Ee)
(12)
Using the parametrizations above, the optimization of equation 7 can be rewritten as
max
C. (0, Φ, 1b)
(13)
,,0,0
where the cost function LNN((P, (0, Φ, 16) given by
C. (0, 0. = Epx (UX)
- -DK.]]()ss((
(14)
Then, for given observations of N samples, i.., {x)) equation 13 can be approximated in terms of an empirical cost as follows
Cam (0,
(15)
6.0.0
where C (0, o, v)) is the empirical cost for the i-th observation x,, and given by
C (0, Φ ) = Ep,(U./x,) Q (X, U.)) 8 DKL (PU
(16)
Furthermore, the first term the RHS of equation 16 can be computed using Monte Carlo sampling and the pa-aammtrrization trick (Kingma & Welling 2014). In particular, Po (u(x) can be sampled
by first sampling a random variable Z with distribution Pz, i.c. Pz = (N I), then transforming the samples using some function fo X Z U, i.e., u = fo(x, 2). Thus,
M
EP,(U,Ix,) (X, =
Σ M ===1
log ((x: ui,), U1,m = M4.i Σ D0 Em, 6m ~(O,
where M the number of samples for the Monte Carlo sampling step.
The second term of the RHS of equation 16 is the KL divergence between a single component multivariate Gaussian and a Gaussian Mixture Model with |C] components. An exact closed-form solution for the calculation of this term does not exist. However, a variational lower bound approxi- mation (Hershey & Olsen 2007) of it can be obtained as
C
Dki. DPU.(P(UU/XX))IQQ)UUU
log Te exp (-DKI Ea,i) E.))
(17)
In particular, in the specific case in which the covariance matrices are diagonal, i.e., Σg, = diag nh 1 and Σ. == diag(fož 1)m)), with n. denoting the latent space dimension, equa- tion 17 can be computed follows
DKI.(Po(U, (x
[C]
Bu (pa.s Mo,g)² + log
02 1
log exp
(18)
Tb...
where μo.t.j and ob are the mean and variance of the i-th representation in the j-th dimension of the latent space. Furthermore, Me.j and o² represent the mean and variance of the e-th component of the GMM in the j-th dimension of the latent space.
Finally, train NNs to maximize the cost function equation 14 over the parameters 0 ó. well as those & of the GMM. For the training step, we use the ADAM optimization tool (Kingma & Ba 2015). The training procedure is detailed in Algorithm
5

---

Under review as a conference paper at ICLR 2020
Once model is trained, assign the given dataset into the clusters. As mentioned in Section 2 we do the assignment from the latent representations, i.c. QcIu = Paix- Hence, the probability that the observed data x, belongs to the e-th cluster is computed follows
p(c|x;)) (() u:) Tip (c) (uile)
". (u;; 14 2n)
(19)
qv- (u.)
E. T Nu, Σ
where indicates optimal values of the parameters as found at the end of the training phase. Finally, the right cluster picked based the largest assignment probability value.
Algorithm VIB-GMM algorithm for unsupervised learning
1: input: Dataset D == {x, parameter 2 0.
2: output: Optimal DNN weights H", o* and GMM parameters 3** { Ma Ex
3: initialization Initialize 8, Φ, t/.
4: repeat
5:
Randomly select b mini-batch samples (x:}²- from D.
6:
Draw random i.dd samples (z, { from Pz.
7: Compute m samples u.j fo(x, 2j)
8: For the selected mini-batch, compute gradients of the empirical cost equation 15
9: Update 0, é, s using the estimated gradient (e.g. with SGD ADAM).
until convergence of 0, Φ. th.
Remark 2. It is worth mention that with the use of the KL approximation in equation our algorithm does not use the assumption Pciu Qucu (not as in Jiang et al.) (2017)). which does not hold in the beginning of the training phase and leads to convergence issues. This assumption is only used in the final assignment after the training phase is over.
3.3 EFFECT OF THE HYPER-PARAMETER
As we already mentioned, the hyper-parameter controls the trade-off between the relevance of the representation U and its complexity, As be seen from equation 14 for small val- ues of 8, it is the cross-entropy term that dom- inates, i.e., the algorithm trains the parameters so as to reproduce X as accurate as possible. For large values of 8, however, portant for the NN to produce an encoded ver- sion of whose distribution matches the prior distribution of the latent space, i.e., the term [I (U)) is nearly zero.
Algorithm 2 Annealing Algorithm Pseudo-Codi
input: Dataset D := := }
hyper-parameter interval [8min Smaxx
output: Optimal DNN weights 0r, 6², GMM
parameters &i* f M Egler
assignment probability Paix.
most im-
initialization Initialize 0, o, repeat
Apply VIB-GMM algorithm.
Update v. 0, Φ.
Update 8, e.g., (1 ex)
until does not exceed Smax-
In the beginning of the training process, the GMM components are randomly selected; and starting with a large value of the hyper- parameter is likely steer the solution towards an irrelevant prior. Hence, for the tunning of the hyper-parameter in practice it is more efficient to start with small value of and gradually increase it with the number of epochs. This has the advantage avoid possible local minimas, an aspect that is reminiscent of deterministic annealing (Slonim, 22002), where s plays the role of the temperature parameter. The experiments that will be reported the next section show that proceeding in the above described manner for the selection of the parameter helps getting better accuracy results and better robustness to the initialization (i.e., no need for strong pretraining). A pseudo-code for annealing is given in Algorithm 2 We note that tuning is very critical, such that the step size e, update of s should be chosen carefully, otherwise phase transitions might be skipped that would cause bad ACC score.
4 EXPERIMENTS
4.1 DESCRIPTION OF USED DATASETS
In our empirical experiments, apply our algorithm the clustering of the following datasets.

---

Under review as a conference paper at ICLR 2020
MNIST: A dataset of gray-scale images of 70000 handwritten digits of dimensions 28 28 pixel.
STL-10: A dataset of color images collected from 10 categories. Each category consists 1300 images of size of 96 96 (pixels) x3 (rgb code). Hence, the original input dimension n. is 27648. For this dataset, we use a pretrained convolutional NN model, i.c., ResNet-50 (He et al. 2016) to reduce the dimensionality of the input. This preprocessing reduces the input dimension to 2048. Then, our algorithm and other baselines are used for clustering.
REUTERSIOK: A dataset that is composed of 810000 English stories labeled with category tree. As in (Xie et al. 2016). 4 root categories (corporate/industria) government/oocial, markets, economics) selected labels and all documents with multiple labels discarded. Then, tf-idf features are computed on the 2000 most frequently occurring words. Finally, 10000 samples are taken randomly, which are referred as REUTERSIOK datasct.
4.2 NETWORK SETTINGS AND OTHER PARAMETERS
We the following network architecture: the encoder modeled with NNs with 3 hidden layers with dimensions f 500 - 500 2000 - J, where is the input dimension and n., is the dimension of the latent space. The decoder consists of NNs with dimensions n., 2000 500 500 22g All layers are fully connected. For comparison purposes, we chose the architecture of the hidden layers as well as the dimension of the latent space n. of (Xie et al. (016)) and the VaDE algorithm of (Jiang et al., 2017). All except the last layers of the encoder and decoder are activated with ReLU function. For the last (i.e., latent) layer the encoder we use linear activation; and for the last (i.e., output) layer of the decoder we use sigmoid function MNIST and linear activation for the remaining datasets. The batch size 100 and the variational bound equation 15 is maximized by the Adam optimizer of (Kingma & Ba 2015). The learning rate initialized with 0.002 and decreased gradually every 20 epochs with a decay rate of 0.9 until reaches small value (0.0005 is experiments). The reconstruction loss is calculated by using the cross-entropy criterion for MNIST and mean squared error function for the other datasets.
to coincide with those made for the DEC algorithm
MNIST
STL-10
REUTERS TOK
GMM
50.4
77.1
53.74
DEC
84.3
80.6'
72.17*
VaDE
94.51
84.3
79.87
VIB-GMM 96.2
91.6
80.4
values are taken from VaDE (Jiang et al.
(017)
values are taken from DEC (Xie et al.) 2016)
Table 1: Comparison of clustering accuracy of various algorithms.
650
spach
0.9
000
Sport
0.8
650
Chastesag Jarunay
Ampren
0.7
500
0.6
450
VIBI CSEM
0.5
VaDEE
400
GMMM
50
100
150
200
250
300
350
400
400
500
350
10
Kpociss
EXorggenon
F2X
Figure 4: Accuracy number of epochs for the STL-10 10 dataset.
Figure 5: Information plane for the STL-10 dataset.
7

---

Under review as a conference paper at ICLR 2020
90
0
-30
-40
-30
(a) Initial accuracy = %10
(b) 1-st epoch, accuracy %41
80
80
40
-0
-10
-00
30
20
20
80
-40
20
41
(c) 5-th epoch, accuracy %66
(d) Final, accuracy %91.6
Figure 6: Visualization of the latent space before training; and after 1, 5 and 500 epochs.
4.3 CLUSTERING ACCURACY
We evaluate the performance of our algorithm in terms of the so-called unsupervised clustering accuracy (ACC), which is widely used metric in the context of unsupervised learning (Min et al. 2018). For comparison purposes, we also present those of algorithms from previous art.
For each of the aforementioned datasets, we run our VIB-GMM algorithm for various values of hyper-parameter inside interval [Smin: $max], starting from the smaller valuer $1 and gradually increasing the value of s every Nepach epochs. For the MNIST dataset, we set (Smin Smax Nepoch) = (1, 5, 500); and for the STL-10 dataset and the REUTERS10K datset choose these parameters to be (1, (1,20, 500) and (1, 5, 100), respectively. The obtained ACC accuracy results are reported in the Table from which can be seen that our algorithm outperforms significantly the DEC algorithm of (Xie et al. 2016) as well as the VaDE algorithm of (Jiang et al., 2017) and GMM on the same datsets. Important to note, for the MNIST dataset the reported ACC accuracy of 96.2% using our VIB- GMM algorithm is obtained as the best case run out of ten times run all with random initializations. For instance, do not use any pretrained values for the initialization of our algorithm in sharp contrast with the VaDE of (Tiang et al. 2017) and the DEC of (Xie et al. 2016). For the STL-10 dataset, none of the compared algorithms use a pretrained network except intimal ResNet-50 for dimensionality reduction. For REUTERS 1OK. used the same pretrain parameters as DEC and VaDE. Figure depicts the evolution the ACC accuracy with iterations (number of epochs) for the four compared algorithms.
Figure shows the evolution of the reconstruction loss of our VIB-GMM algorithm for the STL-10 dataset, function of simultaneously varying values of the papemmeareetter and the number of epochs (recall that, as per-the described methodology, we start with s = s and we increase its value gradually every Nepaahh 500 epochs). As it can be seen from the figure, the few first epochs are spent
8

---

Under review as a conference paper at ICLR 2020
almost entirely reducing the reconstruction loss (i.e., fitting phase) and most of the remaining epochs are spent in making the found representation more concise (i.e., smaller KL-divergence), This reminiscent of the two-phase (fitting compression) that was observed for supervised learning using VIB in (Schwartz-2iv & TTssbyy 2017).
4.4 VISUALIZAATION ON THE LATENT SPACE
In this section, we investigate the evolution of the unsupervised clustering of the 1TL-10 dataset on the latent space using our VIB-GMM algorithm. For this purpose, we find it convenient to visualize the latent space through application of the t-SNE algorithm of (van der Maaten & Hinton 2008 in order to generate meaningful representations in two-dimensional space. Figure 6 shows 4000 randomly chosen latent representations before the start of the training process and respectively after 1, 5 and 500 epochs. The shown points (with a marker in the figure) represent latent representations of data samples whose labels identical. Colors used to distinguish between clusters. Crosses (with an x marker in the figure) correspond to the centroids of the clusters. More specifically, Figure (a) shows the initial latent space before the training process. If the clustering is performed on the initial representations it allows ACC accuracy of as small as 10%. i.c., as bad as a random assignment. Figure (b) shows the latent space after one epoch, from which a partition of some of the points starts to be already visible. With five epochs, that partitioning is significantly sharper and the associated clusters can be recognized easily. Observe, however, that the cluster centers seem still not to have converged. With 500 epochs, the ACC accuracy of our algorithm reaches %91.6 6 and the clusters and their centroids are neater as visible from Figure (d).
REFERENCES
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In Proceedings of the 5th International Conference on Learning Representations, 2017.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 215 - 223, 2011.
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39:1 -38, 1977.
Nat Dilokthanakul, Pedro A. M. Mediano, Marta Gamelo, Matthew C.H. Lec. Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahani. Deep unsupervised clustering with Gaussian mixture variational autoencoders. arXiv: /61 1. 1.02648, 2017.
Chris Ding and Xiaofeng He. K-means clustering via principal component analysis. In Proceedings of the 21 " International Conference on Machine Learning, 2004.
Xifeng Guo, Long Gao, Xinwang Liu, and Jianping Yin. Improved deep embedded clustering with local structure preservation. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 1753- 1759, 2017.
J. A. Hartigan and M. A. Wong. Algorithm AS 136: A k-means clustering algorithm. Journal of the Royal Statistical Society, 28:100- 108, 1979.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 770 - 778, 2016.
John R. Hershey and Peder A. Olsen. Approximating the Kullback Leibler divergence between Gaussian mixture models. In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 317-3177300
- 320, April 2007.
Thomas Hofmann, Bernhard Schölkopf. and Alexander Smola. Kernel methods machine learning. The Annaly of Statistics, 36:1 171 1220, June 2008

---

Under review as a conference paper at ICLR 2020
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, and Hanning Zhou. Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pp. 1965 1972, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2 International Conference on Learning Representations, 2014.
Yann Lecun, Leon Bottou, Yoshua Bengio, and Patrick Haffnnr. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pp. 2278 2324, 1998.
David D. Lewis, Yiming Yang, Tony G. Rose. and Fan Li. A new benchmark collection for text categorization research. The Journal of Machine Learning Research, 5:361 397, 2004.
Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang, Jianjing Cui, and Jun Long. A survey of clustering with deep learning: From the perspective of network architecture. IEEE Access, 6:39501 39514, 2018.
Danilo J. Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approxi- mate inference in deep generative models. In Proceedings of the 31 International Conference Machine Learning, pp. 1278 1286, 2014.
Sam Roweis. EM algorithms for PCA and SPCA. Advances Neural Information Processing Systems 10, pp. 626 632, 1997.
Ravid Schwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv: 1703.00810, 2017.
Noam Slonim. The information bottleneck: Theory and applications. PhD dissertation, Hebrew University, 2002.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing, pp. 368 377, 1999.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research 9, pp. 2579 2605, November 2008.
Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and Intelligent Laboratory Systems, 2:37 - 52, August 1987.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In Proceedings of the 33d International Conference on Machine Learning, pp. 478 487, 2016.
A THE PROOF OF LEMMA
First, we expand E, (P) as follows
(P) H(X[U) (T(U))ss((X;U)) sI(;UU)
H(X(U)
= -/ p(u, log p(x/u) du dx
+ / p(u) log log du - // p(u, x) logp(u(x)) du dx.
Then, (P Q) is defined as follows
(P, Q) = /f ux
p(u, x) log (x((u) du
+ / p(u) log (u) du //
plu, x) log (ogp(())) du dx.
(20)
10

---

Under review as a conference paper at ICLR 2020
Hence, we have the following relation
(P)
L3 P. Q) - Exx DKI. (Pquul(x)l)XI (Du.Pl 20
where equality holds under equalities QxJU
Px and Qu = Pu. We note that 2 0.
B THE PROOF OF ALTERNATIVE EXPRESSION LVADE
Here we show how we obtained equation 10
To do so,
C CYaDE Epx EU [log Qxju] DDRI(PU slEpvix [Pka(Pxxll(Rcu)) (
-Epx EPux flo Qxjull -8 p(x) J. p(ulx) log q(u)
p(u[x) du dx
p(x) / p(ulx) Σ p(((x) log p(cx) dudx ((clu)
*
Exx BRuix [log Qx|u]] - /// p(x)p(ulx) p(u]x) q())
du dx
p(elx) du s II ux Σ xx)))
q(clu)
-Bxx EPax Dlg Qxjv|l -s -s/ Σ p(u, c, x) p(ujx)p(c(x)) du dx
q(u)q(clu)
=Epx [EPuix loo Qx - //1 ux Σ c(
p(c|x) p(ulx) dudx
g(c) s(lcc)
=Epx Dog Qxju]] / Ep(c,)) log p(c]x)
dx
g(c)
[I Σ p(xxp((cxx)puulxxx))lo log
p(ux)
du dx
(u(c)
E rx
Eroxx [log QxIu] sDKL (
SErO (Rux(
B P. Q)
sEPx Eroi DKL [Pppx(]
where (a) and (b) follow due to the Markov chain C' + X - U: (c) follows from the definition of LV (P, Q) in equation
C KL DIVERGENCE BETWEEN MULTIVARIATE GAUSSIAN DISTRIBUTIONS
The KL divergence between two multivariate Gaussian distributions Pa N(p1 En) and P2 ~ Nu22, E2) in Rd is
1 {((m -22)² " (22) + log Σ2 log Σ ur( ur(Eg +r((++1E2))) -1)) (21) 2
DEL (P.
For the case in which Σ, and Σn covariance matrices are diagonal, i.e., E, = diag(toi,))) and
En = (o, equation 21 boils down to the following
(KL(PP (11.) 42.5)2 + log
2
+
(22)
02,3
o
2

---

Under review as a conference paper at ICLR 2020
D KL DIVERGENCE BETWEEN GAUSSIAN MIXTURE MODELS
An exact close form for the calculation of the KI. divergence between two Gaussian mixture models does not exist. In this paper, we use variational lower bound approximation for calculations of KL between two Gaussian mixture models. Let f and be GMMs and the marginal densities of x under and g are
M Σ
M
f(x)
win (a;
Σ
win (
C
g(x) Σ
2
(n)e
C=1
The KL divergence between two Gaussian mixtures an g can be approximated as follows
M
EmrEMim Wmr exp (- DRI (m.l(((fmllmm)))
DeKt. Dukl) =
Wm log
(23)
Eem1 Toe exp (-DKL ml1)))
In this paper, we are interested, in particular, M Hence, equation 23 simplifies to
Dvkl(fllg) = - loog
To exp (-DkL Σ
(24)
c=l
where (-I-)) is the KL divergence between single component multivariate Gaussian distribution, defined as equation 21
12


Reviews:
[{"review_id": "rkgw2B4Jqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper purposes to cluster data in an unsupervised manner that estimates the distribution with GMM in latent space instead of original data space. Also, to better describe the distribution of latent code, the authors involve VIB to constraint the latent code.\n\nThe whole pipeline is clear and makes sense. And the experiment proves it's effective. \n\nHowever, in my eyes, it's almost like an extension of VaDE which combines VIB and GMM too. The main difference is the authors use some hyperparameters to control the optimization of the whole model and make some more proper hypothesis. All of those make sense but may not contribute much to the research field.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HJl3w-spFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the autoencoder model combining the usual information bottleneck and the Gaussian mixture model (GMM). Using an approximation to deal with GMMs, the authors derive a bound on the cost function generalizing the ELBO. The performance of the proposed method is tested on three benchmark datasets and compared with existing methods combining VAE with GMM.\n\nWhile the framework and the performance of the proposed method are interesting and promising, some of its main parts are unclearly explained.\n\n- Although Remark 1 well explains the difference between VaDE and the proposed objective functions, little is discussed how this difference affects the learnt model.\n- I wonder why Q_\\phi(x|u) in Eq. (11) doesn’t have to be a distribution. What does the expression [\\hat{x}] mean?\n- Eq. (17) is not explained clearly. Since the equality symbol is used, it is unclear where is approximation. Although this approximation is one of the main parts of the proposed method, little is discussed on the influence of this approximation.\n- Are the information plane and latent representations in Figs 5 and 6 also available for DEC and VaDE and not limited to the proposed method?\n\nMinor comments:\np.5, the math expression between Eqs. (16) and (17): The distribution q(x_i|u_i,m) is undefined.\np.6, l.4 from the bottom: ACC is defined later.\np.7, l.14: Should J be n_u?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byx_WBDRtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The author(s) posit a Mixture of Gaussian's prior for a compressed latent space representation of high-dimensional data (e.g. images and documents). They propose fitting this model using the Variational Information Bottleneck paradigm and explicate its derivation and tie it to the variational objective used by similar models. They empirically showcase their model and optimization methodology on the MNIST, STL-10, and Reuters10k benchmarks.\n\nThe idea of using a latent mixture of Gaussian's to variationally encode high-dimensional data is not new. The author(s) appropriately cite VaDE (Jiang, 2017) and DEC (Xie, 2016), \"Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders\" (Dilokthanakul, 2017). However, the author(s) unfortunately did not mention \"Approximate Inference for Deep Latent Gaussian Mixtures\" (Nalisnick, 2016). Nalisnick et al. consider the exact generative process as the proposed by the author(s), but with the addition of a Dirichlet prior on the Categorical distribution. Nalisnick et al. fit their model with a VAE and circumvent Dirichlet intractability by positing a Kumarswamy stick-breaking variational posterior (the resulting distribution is on the simplex). They achieve 91.58% accuracy, but do so after fitting a KNN to the latent space. New techniques such as \"Implicit Reparameterization Gradients\" (Figurnov, 2018) and \"Pathwise Derivatives Beyond the Reparameterization Trick\" (Jankowiak, 2018) allow direct use of a Dirichlet posterior in VAEs. These methods are respectively implemented in the TensorFlow and PyTorch APIs. My point here is that there are many ways to fit this pre-existing model. From my review of these other works we have, for \"best run\" on MNIST, that author(s) > VaDE > Nalisnick > Dilokthanakul > DEC. Thus, the author(s) are SoTA for this particular generative process for their best run. It would be nice to see their standard deviation to gauge how statistically significant their results are. However, I am dubious of the SoTA claim as I detail later.\n\nThe author(s) derivation was sound, but a bit confusing for me. In particular, I found keeping track of P's and Q's very burdensome after reading sections 2.1 and 2.2. In my experience, Q is typically reserved for a variational distribution that approximates some intractable P distribution, while P is used to describe the generative process (i.e. likelihood) and other exact distributions. Furthermore, one typically introduces the generative process first using P distributions. Once, I got past this confusion everything else made sense. I might suggest introducing the generative process first and with P distributions instead of Q's. The variational model can follow with the Q distributions as the author(s) have it. Equations 4, 5, and 6 all exactly match the unsupervised information bottleneck objective (Alemi, 2017)--see appendix B. I am therefore confident in those equations. I carefully checked their derivation of the VADe comparison (equation 10 and appendix B). Their derivation is straightforward. The principle trick is using the MC assumption C->X->U for P distributions to claim p(u|x,c) = p(u|x). Equations 12-16 all follow naturally from equation 11. If equation 17 is indeed an approximation or bound approximation, I would suggest not using the equals sign. Instead, consider another appropriate operator or rename Dkl to indicate it is the approximated version (just as in equation 24).\n\nIf the author(s) like my suggestion regarding generative vs variational nomenclature, I would also change the second sentence of page 6 to something like \"We use our variational Qc|u distribution to compute assignments.\" Thereafter, I would drop the star indicator for optimal parameters. These parameters are not necessarily optimal given the non-convexity of the DNN. Replace with, \"after convergence, we ...\" Similarly, drop optimal from line 2 of algorithm 1.\n\nCircling back to the experiments, the author(s) use reported values from DEC and VaDE. Those works compute cluster assignments using a KNN classifier on the latent space. This paper however uses the arg max of equation 19. I much prefer this article's method, but for comparison purposes, the author(s) should similarly use a KNN classifier on their latent space to compute accuracy in the same manner. The use of KNN in these other works allows them to consider a number of latent clusters larger than the number of true classes (e.g. 20 clusters for MNIST). I like that the author(s) stick to 10 clusters for MNIST, but for comparison I would have liked to see a KNN generated accuracy alongside their equation 19 based accuracy. It looks like the author(s) implemented VADe for STL-10 based on table 1. If so, it seems they could easily implement equation 19 for their STL-10 value for VADe. If not, please correct this. Not to belabor further, but I would really like to see a table 2 that reports both equation 19 and KNN accuracies when available. Namely, the author(s) should report both values for their model and can leave equation 19 accuracies blank for reported values.\n\nLastly, I always raise an eyebrow when I see tSNE latent space representations. STL-10 was used to generate figure 6, where one needs dim(u) >> 2. However, MNIST can be well reconstructed using just 2 latent dimensions. In this case, tSNE is unnecessary. The author(s) state \"Colors are used to distinguish between clusters.\" This statement is unclear as to whether the author(s) are using the class label or learned latent cluster. If it is the former, figure 6 makes sense in that it shows misclassifications (i.e. red dots in the green cluster). However, if it is the latter then I am concerned tSNE is doing something weird.\n\nTo summarize, I enjoyed the paper. My only concern is novelty. Being the first to pair an existing model with an existing method, in my eyes, does not necessarily meet the ICLR bar. The author(s) seemingly achieve SoTA, but without KNN-based accuracies for their model it is hard to compare to cited works. Having these KNN results and error bars would strengthen their case substantially to: achieving SoTA for an existing model by being the first to pair it with an existing method.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use a mixture of Gaussians to variationally encode high-dimensional data through a latent space. The latent codes are constrained using the variational information bottleneck machinery. 

While the paper is well-motivated and relatively well-written, it contains minimal novel ideas. The consensus in reviews and lack of rebuttal make it clear that this paper should be significantly augmented with novel material before being published to ICLR.

Decision: Reject