Variational Information Bottleneck for Unsupervised Clustering: Deep Gaussian Mixture Embedding

In this paper , we develop an unsupervised generative clustering framework that combines variational information bottleneck and the Gaussian Mixture Model . Specifically , in our approach we use the variational information bottleneck method and model the latent space as a mixture of Gaussians . We derive a bound on the cost function of our model that generalizes the evidence lower bound ( ELBO ) ; and provide a variational inference type algorithm that allows to compute it . In the algorithm , the coders ’ mappings are parametrized using neural networks and the bound is approximated by Markov sampling and optimized with stochastic gradient descent . Numerical results on real datasets are provided to support the efficiency of our method . 1 INTRODUCTION . Clustering consists in partitioning a given data set into various groups ( clusters ) based on some similarity metric , such as Euclidean distance , L1 norm , L2 norm , L∞ norm , the popular logarithmic loss measure or others . The principle is that each cluster should contain elements of the data that are closer to each other than to any other element outside that cluster , in the sense of the defined similarity measure . If the joint distribution of the clusters and data is not known , one should operate blindly in doing so , i.e. , using only the data elements at hand ; and the approach is called unsupervised clustering . Unsupervised clustering is perhaps one of the most important tasks of unsupervised machine learning algorithms nowadays , due to a variety of application needs and connections with other problems . Examples of unsupervised clustering algorithms include the so-popular K-means ( Hartigan & Wong , 1979 ) and expectation maximization ( EM ) ( Dempster et al. , 1977 ) . TheK-means algorithm partitions the data in a manner that the Euclidean distance among the members of each cluster is minimized . With the EM algorithm , the underlying assumption is that the data comprises a mixture of Gaussian samples , namely a Gaussian Mixture Model ( GMM ) ; and one estimates the parameters of each component of the GMM while simultaneously associating each data sample to one of those components . Although they offer some advantages in the context of clustering , these algorithms suffer from some strong limitations . For example , it is well known that the K-means is highly sensitive to both the order of the data and scaling ; and the obtained accuracy depends strongly on the initial seeds ( in addition to that it does not predict the number of clusters or K-value ) . The EM algorithm suffers mainly from low convergence , especially for high dimensional data . Recently , a new approach has emerged that seeks to perform inference on a transformed domain ( generally referred to as latent space ) , not the data itself . The rationale is that because the latent space often has fewer dimensions it is more convenient computationally to perform inference ( clustering ) on it rather than on the high dimensional data directly . A key aspect then is how to design a latent space that is amenable to accurate low-complex unsupervised clustering , i.e. , one that preserves only those features of the observed high dimensional data that are useful for clustering while removing out all redundant or non-relevant information . Along this line of work , we can mention ( Ding & He , 2004 ) which utilizes Principal Component Analysis ( PCA ) ( Wold et al. , 1987 ) for dimensionality reduction followed by K-means for clustering the obtained reduced dimension data ; or ( Roweis , 1997 ) which uses a combination of PCA and the EM algorithm . Other works that use alternatives for the linear PCA include Kernel PCA ( Hofmann et al. , 2008 ) , which employs PCA in a non-linear fashion to maximize variance in the data . The usage of deep neural networks ( DNN ) for unsupervised clustering of high dimensional data on a lower dimensional latent space has attracted considerable attention , especially with the advent of autoencoder ( AE ) learning and the development of powerful tools to train them using standard backpropagation techniques ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) . Advanced forms include Variational autoencoders ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) which are generative variants of AE that regularize the structure of the latent space and the more general Variational Information Bottleneck ( VIB ) of ( Alemi et al. , 2017 ) which is a technique that is based on the Information Bottleneck method ( Tishby et al. , 1999 ) and seeks a better trade-off between accuracy and regularization than VAE via the introduction of a Lagrange-type parameter s which controls that trade-off and whose optimization is similar to deterministic annealing ( Slonim , 2002 ) or stochastic relaxation . In this paper , we develop an unsupervised generative clustering framework that combines VIB and the Gaussian Mixture Model . Specifically , in our approach we use the variational information bottleneck method and model the latent space as a mixture of Gaussians . The encoder and decoder of the model are parametrized using neural networks ( NN ) . The cost-function is calculated approximatively by Markov sampling and optimized with stochastic gradient descent . Furthermore , the application of our algorithm to the unsupervised clustering of various datasets , including the MNIST ( Lecun et al. , 1998 ) , REUTERS ( Lewis et al. , 2004 ) and STL-10 ( Coates et al. , 2011 ) , allows a better clustering accuracy than previous state of the art algorithms . For instance , we show that our algorithm performs better than the variational deep embedding ( VaDE ) algorithm of ( Jiang et al. , 2017 ) which is based on VAE and performs clustering by maximizes the ELBO and can be seen as a specific case of our algorithm ( Section 3.1 ) . Our algorithm also generalizes the VIB of ( Alemi et al. , 2017 ) which models the latent space as an isotropic Gaussian which is generally not expressive enough for the purpose of unsupervised clustering . Other related works , but which are of lesser relevance to the contribution of this paper , are the deep embedded clustering ( DEC ) of ( Xie et al. , 2016 ) , the improved deep embedded clustering ( IDEC ) of ( Guo et al. , 2017 ) and ( Dilokthanakul et al. , 2017 ) . For a detailed survey of clustering with deep learning , the readers may refer to ( Min et al. , 2018 ) . To the best of our knowledge , our algorithm performs the best in terms of clustering accuracy by using deep neural networks without any prior knowledge regarding the labels ( except the usual assumption regarding the number of the classes ) compared to the state-of-the-art algorithms of this category . In order to achieve the aforementioned accuracy , i ) we derive a cost-function that contains the IB hyper parameter s that controls the trade-off between over-fit and generalization of the model and we used an approximation of KL divergence that avoid assumptions which do not hold in the beginning of the learning process and lead to convergence issues ; ii ) evaluate the hyper-parameter s by following an annealing approach that improves both the convergence and the accuracy of the proposed algorithm . 2 PROBLEM DEFINITION AND MODEL . Consider a dataset that is composed of N samples { xi } Ni=1 which we wish to partition into |C| ≥ 1 clusters . Let C = { 1 , . . . , |C| } be the set of all possible clusters ; and C designate a categorical random variable that lies in C and stands for the index of the actual cluster . If X is a random variable that models elements of the dataset , given X = xi induces a probability distribution on C which the learner should learn . Thus , mathematically the problem is that of estimating the values of the unknown conditional probability PC|X ( ·|xi ) for all elements xi of the dataset . The estimates are sometimes referred to as the assignment probabilities . As mentioned previously , we use the VIB framework and model the latent space as a GMM . The resulting model is depicted in Figure 1 , where the parameters πc , µc , Σc , for all values of c ∈ C , are to be optimized jointly with those of the employed NNs as instantiation of the coders . Also , the assignment probabilities are estimated based on the values of latent space vector instead of the observation themselves , i.e. , PC|U = QC|U . In the rest of this section , we elaborate on the inference and generative network models for our method , which are illustrated below . C X U PX|C PU|X Figure 2 : Inference Network C U X QU|C QX|U Figure 3 : Generative Network 2.1 INFERENCE NETWORK MODEL . We assume that an observed data x is generated from a GMM with |C| components . Then , the latent representation u is inferred according the following procedure : 1 . One of the components of the GMM is chosen according to a categorical variable C. 2 . The data x is generated from the c-th competent of the GMM , i.e. , PX|C ∼ N ( x ; µ̃c , Σ̃c ) . 3 . Encoder maps x to a latent representation u according to PU|X ∼ N ( µθ , Σθ ) . 3.1 . The encoder is modeled with a DNN fθ which maps x to the parameters of a Gaussian distribution , i.e. , [ µθ , Σθ ] = fθ ( x ) . 3.2 . The representation u is sampled from N ( µθ , Σθ ) . For the inference network , shown in Figure 2 , the following Markov chain holds C − −X− −U . ( 1 ) 2.2 GENERATIVE NETWORK MODEL . Since encoder extracts useful representations of the dataset and we assume that the dataset is generated from a GMM , we model our latent space also with a mixture of Gaussians . To do so , the categorical variable C is embedded with the latent variable U . The reconstruction of the dataset is generated according to the following procedure : 1 . One of the components of the GMM is chosen according to a categorical variable C , with a prior distribution QC . 2 . The representation u is generated from the c-th component , i.e. , QU|C ∼ N ( u ; µc , Σc ) . 3 . The decoder maps the latent representation u to x̂ which is the reconstruction of the source x by using the mapping QX|U . 3.1 . The decoder is modeled with a DNN gφ , that maps u to the estimate x̂ , i.e. , [ x̂ ] = gφ ( u ) . For the generative network , shown in Figure 3 , the following Markov chain holds C − −U− −X . ( 2 ) 3 PROPOSED METHOD . In this section we present our clustering method . First , we provide a general cost function for the problem of the unsupervised clustering that we study here based on the variational IB framework ; and we show that it generalizes the ELBO bound developed in ( Jiang et al. , 2017 ) . We then parametrize our model using NNs whose parameters are optimized jointly with those of the GMM . Furthermore , we discuss the influence of the hyper-parameter s that controls optimal trade-offs between accuracy and regularization .

Summary: This paper considers the autoencoder model combining the usual information bottleneck and the Gaussian mixture model (GMM). Using an approximation to deal with GMMs, the authors derive a bound on the cost function generalizing the ELBO. The performance of the proposed method is tested on three benchmark datasets and compared with existing methods combining VAE with GMM.