Learning Latent Superstructures in Variational Autoencoders for Deep Multidimensional Clustering

We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features . In general , our superstructure is a tree structure of multiple super latent variables and it is automatically learned from data . When there is only one latent variable in the superstructure , our model reduces to one that assumes the latent features to be generated from a Gaussian mixture model . We call our model the latent tree variational autoencoder ( LTVAE ) . Whereas previous deep learning methods for clustering produce only one partition of data , LTVAE produces multiple partitions of data , each being given by one super latent variable . This is desirable because high dimensional data usually have many different natural facets and can be meaningfully partitioned in multiple ways . 1 INTRODUCTION . Clustering is a fundamental task in unsupervised machine learning , and it is central to many datadriven application domains . Cluster analysis partitions all the data into disjoint groups , and one can understand the structure of the data by examining examples in each group . Many clustering methods have been proposed in the literature ( Aggarwal & Reddy , 2013 ) , such as k-means ( MacQueen et al. , 1967 ) , Gaussian mixture models ( Christopher , 2016 ) and spectral clustering ( Von Luxburg , 2007 ) . Conventional clustering methods are generally applied directly on the original data space . However , it is challenging to perform cluster analysis on high dimensional and unstructured data ( Steinbach et al. , 2004 ) , such as images . It is not only because the dimensionality is high , but also because the original data space is too complex to interpret , e.g . there are semantic gaps between pixel values and objects in images . Recently , deep learning based clustering methods have been proposed that simultanously learn nonlinear embeddings through deep neural networks and perform cluster analysis on the embedding space . The representation learning process learns effective high-level representations from high dimensional data and helps the cluster analysis . This is typically achieved by unsupervised deep learning methods , such as restricted Boltzmann machine ( RBM ) ( Hinton et al. , 2006 ; Hinton & Salakhutdinov , 2006 ) , autoencoders ( AE ) ( Vincent et al. , 2008 ; 2010 ) , variational autoencoders ( VAE ) ( Kingma & Welling , 2014 ) , etc . Previous deep learning based clustering methods ( Xie et al. , 2016 ; Guo et al. , 2017 ; Jiang et al. , 2017 ; Yang et al. , 2017 ) assume one single partition over the data and that all attributes define that partition . In real-world applications , however , the assumptions are usually not true . High-dimensional data are often multifaceted and can be meaningfully partitioned in multiple ways based on subsets of attributes ( Chen et al. , 2012 ) . For example , a student population can be clustered in one way based on course grades and in another way based on extracurricular activities . Movie reviews can be clustered based on both sentiment ( positive or negative ) and genre ( comedy , action , war , etc. ) . It is challenging to discover the multi-facet structures of data , especially for high-dimensional data . To resolve the above issues , we propose an unsupervised learning method , latent tree variational autoencoder ( LTVAE ) to learn latent superstructures in variational autoencoders , and simultaneously perform representation learning and structure learning . LTVAE is a generative model , where the data is assumed to be generated from latent features through neural networks , while the latent features themselves are generated from tree-structured Bayesian networks with another level of latent variables as shown in Fig . 1 . Each of those latent variables defines a facet of clustering . The proposed method automatically selects subsets of latent features for each facet , and learns the dependency structure among different facets . This is achieved through systematic structure learning . Consequently , LTVAE is able to discover complex structures of data rather than one partition . We also propose efficient learning algorithms for LTVAE with gradient descent and Stepwise EM through message passing . The rest of the paper is organized as follows . The related works are reviewed in Section 2 . We introduce the proposed method and learning algorithms in Section 3 . In Section 4 , we present the empirical results . The conclusion is given in Section 5 . 2 RELATED WORKS . Clustering has been extensively studied in the literature in many aspects ( Aggarwal & Reddy , 2013 ) . More complex clustering methods related to structure learning using Bayesian nonparametrics have been proposed , like Dirichlet Process ( Blei et al. , 2006 ) , Hierarchical Dirichlet Process ( HDP ) ( Teh et al. , 2006 ) . However , those are with conventional clustering methods that apply on raw data . Recently , deep learning based clustering methods have drawn more and more attention . A simple two-stage approach is to first learn low-dimensional embeddings using unsupervised feature learning methods , and then perform cluster analysis on the embeddings . However , without any supervision , the representation learning do not necessarily reveal the true cluster structure of the data . DEC ( Xie et al. , 2016 ) is a method that simultaneously learns feature representations and cluster assignments through deep autoencoders . It gradually improves the clustering by driving the deep network to learn a better mapping . Improved Deep Embedded Clustering ( Guo et al. , 2017 ) improves DEC by keeping the decoder network and adding reconstruction loss to the original clustering loss in DEC. Variational deep embedding ( Jiang et al. , 2017 ) is a generative method that models the data generative process using a Gaussian mixture model combined with a VAE , and also performs joint learning of representations and clustering . Similarly , GMVAE ( Dilokthanakul et al. , 2016 ) performs joint learning of a GMM and a VAE , but instead generates the mixture components through neural networks . Deep clustering network ( DCN ) ( Yang et al. , 2017 ) is another one that jointly learns an autoencoder and performs k-means clustering . These joint learning methods consistently achieve better clustering results than conventional ones . The method proposed in ( Yang et al. , 2016 ) uses convolutional neural networks and jointly learns the representations and clustering in a recurrent framework . All these methods assume flat partitions over the data , and do not attempt the structure learning issue . An exception is hierarchical nonparametric variational autoencoders proposed in ( Goyal et al. , 2017 ) . It uses nCRP as the prior for VAE to allow infinitely deep and branching tree hierarchy structure and focuses on learning hierarchy of concepts . However , it is still one partition over the data , only that the partitions in upper levels are more general partitions , while those in lower levels more fine-grained . Different from it , our work focuses on multifacets of clustering , for example , the model could make one partition based on identity of subjects , while another partition based on pose . 3 THE PROPOSED METHOD . In this section , we present the proposed latent tree variational autoencoder and the learning algorithms for joint representation learning and structure learning for multidimensional clustering . 3.1 LATENT TREE VARIATIONAL AUTOENCODER . Deep generative models assume that data x is generated from latent continuous variable z through some random process . The process consists of two steps : ( 1 ) a value z is generated from some prior distribution p ( z ) ; ( 2 ) the observation x is generated from the conditional distribution pθ ( x|z ) , which is parameterized through deep neural networks . Thus , it defines the joint distribution between 𝑍 '' 𝑌 '' 𝑌 $ 𝑍 $ 𝑍 '' , 𝑌 '' 𝑍 & , 𝑌 '' 𝑌 '' , 𝑌 $ 𝑍 ' , 𝑌 $ 𝑃 ( 𝑍 '' |𝑌 '' ) 𝑃 ( 𝑍 $ |𝑌 '' ) 𝑃 ( 𝑍 & |𝑌 $ ) 𝑃 ( 𝑌 $ |𝑌 '' ) 𝑃 ( 𝑌 '' ) 𝑃 ( 𝑍 '' = 𝑧 '' |𝑌 '' ) 𝑃 ( 𝑍 & = 𝑧 & |𝑌 '' ) 𝑃 ( 𝑍 ' = 𝑧'|𝑌 $ ) 𝑃 ( 𝑌 $ |𝑌 '' ) 𝑃 ( 𝑌 '' ) Clique Tree 𝑍 & 𝑍' 𝑃 ( 𝑍'|𝑌 $ ) 𝑍 $ , 𝑌 '' 𝑃 ( 𝑍 '' = 𝑧 '' |𝑌 '' ) Figure 2 : Inference and gradient through message passing . Solid-arrows denote collecting message , and dashed-arrows denote distributing message . observation x and latent variable z : p ( x , z ) = p ( z ) pθ ( x|z ) ( 1 ) This process is hidden from our view , and we learn this process by maximizing the marginal loglikelihood p ( x ) over the parameters θ and latent variable z from data . After the learning , the latent variable z can be regarded as the deep representations of x since it captures the most relevant information of x . Thus , the learning process is also called representation learning . In order to learn the latent structure of z , for example multidimensional cluster structure , we introduce a set of latent variables Y1 , ... , Yl on top of z . A single z or multiple z ’ s form a node zb . Suppose variables in z form B nodes of z1 , · · · , zB . Each latent variable Y may be only connected to a subset of nodes , and the dependency of each zb and its parent Y is characterized by a conditional Gaussian distribution . Furthermore , the latent variables Y1 , ... , Yl are connected to each other , and the dependency of a latent variable Y on its parent Y ′ is characterized by a conditional distribution P ( Y |Y ′ ) . This essentially forms a Bayesian network . And if we restrict the network to be treestructured , the z and Y together form a latent tree model ( Zhang , 2004 ; Poon et al. , 2010 ; 2013 ; Mourad et al. , 2013 ; Pearl , 2014 ; Zhang & Poon , 2017 ) with z being the observed variables and Y being the latent variables . For multidimensional clustering , each latent variable Y is taken to be a discrete variable , where each discrete state y of Y defines a cluster . Each latent variable Y thus defines a facet partition over the data based on subset of attributes and multiple Y ’ s define multiple facets . Given a value y of Y , zb follows a conditional Gaussian distribution P ( zb|y ) = N ( µy , Σy ) with mean vector µy and covariance matrix Σy . Thus , each zb and its parent constitute a Gaussian mixture model ( GMM ) . Suppose the parent of a node is denoted as π ( · ) , the maginal distribution of z is defined as follows p ( z ) = ∑ Y l∏ j=1 p ( yj |π ( Yj ) ) B∏ b=1 N ( zb|µπ ( zb ) , Σπ ( zb ) ) , ( 2 ) which sums over all possible combinations of Y states . As a matter of fact , a GMM is a Gaussian LTM that has only one latent variable connecting to all observed variables . Let the latent structure of Y be S , defining the number of latent variables in Y , the number of discrete states in each variable Y and the connectivity structure among all variables in z and Y . And let the parameters for all conditional probabilities in the latent structure be Θ . Both the latent structure S and the latent parameters Θ are unknown . We aim to jointly learn data representations and the latent structure . The proposed LTVAE model is shown in Fig . 1 . The latent structure S are automatically learned from data and will be discussed in a later section . Due to the existence of the generation network , the inference of the model is intratable . Instead , we do amortized variational inference for the latent variable z by introducing an inference network ( Kingma & Welling , 2014 ) and define an approximate posterior qφ ( z|x ) . The evidence lower bound ( ELBO ) LELBO of the marginal loglikelihood of the data given ( S , Θ ) is : LELBO ( x ) = Eqφ ( z|x ) [ log pθ ( x|z ) ] + Eqφ ( z|x ) [ log ∑ y pS ( z , y ; Θ ) ] + H [ qφ ( z|x ) ] , ( 3 ) where log ∑ y pS ( z , y ; Θ ) is the marginal loglikelihood of the latent variable z under the latent tree model , and H [ · ] is the entropy . The conditional generative distribution pθ ( x|z ) could be a Gaussian distribution if the input data is real-valued , or a Bernoulli distribution if binary , parameterized by the generation network . Using Monte Carlo sampling , the ELBO can be asymptotically estimated by LELBO ( x ) ' 1 M M∑ i=1 log pθ ( x|z ( i ) ) + log ∑ y pS ( z ( i ) , y ; Θ ) + H [ qφ ( z|x ) ] , ( 4 ) where z ( i ) ∼ qφ ( z|x ) . The term H [ qφ ( z|x ) ] can be computed analytically if we choose the form of qφ ( z|x ) to be a Gaussian distributionN ( z ; µx , σx ) : H [ qφ ( z|x ) ] = J2 log ( 2π ) + 1 2 ∑J j=1 ( 1+log σ 2 j ) , where J is the dimensionality of z . Furthermore , the marginal loglikelihood log ∑ y pS ( z ( i ) , y ; Θ ) can be computed efficiently through message passing . Message passing is an efficient algorithm for inference in Bayesian networks ( Koller & Friedman , 2009 ; Poon et al. , 2013 ) . In message passing , we first build a clique tree using the factors in the defined probability density . Because of the tree structure , each zb along with its parent form a clique with the potential ψ ( zb , y ) being the corresponding conditional distribution . This is illustrated in Fig . 2 . With the sampled z ( i ) , we can compute the message ψ′ ( y ) by absorbing the evidence from z . During collecting message phase , the message ψ′ ( y ) are sent towards the pivot . After receiving all messages , the pivot distributes back messages towards all z . Both the posterior of Y and the marginal loglikelihood of z ( i ) thus can be computed in the final normalization step .

Summary: The authors propose to augment the Variational AutoEncoder [1] with a latent prior modeled by a Gaussian Latent Tree Model [2], allowing to introduce a hierarchical structure of clusters in the learned representation. The LT-VAE not only learns the location of each cluster to best represent the data, but also their number and the hierarchical structure of the underlying tree. This is achieved by a three-step learning algorithm. Step 1 is a traditional training of the encoder and decoder neural networks to improve their fitting of the data. Step 2 is an EM-like optimization to better fit the parameters of latent prior to the learned posterior. And step 3 adapts the structure of the latent prior to improve its BIC score [3], which balances a good fit of the latent posterior with the number of parameter (and thus complexity) of the latent prior.