Learning Latent Superstructures in Variational Autoencoders for Deep Multidimensional Clustering

We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features . In general , our superstructure is a tree structure of multiple super latent variables and it is automatically learned from data . When there is only one latent variable in the superstructure , our model reduces to one that assumes the latent features to be generated from a Gaussian mixture model . We call our model the latent tree variational autoencoder ( LTVAE ) . Whereas previous deep learning methods for clustering produce only one partition of data , LTVAE produces multiple partitions of data , each being given by one super latent variable . This is desirable because high dimensional data usually have many different natural facets and can be meaningfully partitioned in multiple ways . 1 INTRODUCTION . Clustering is a fundamental task in unsupervised machine learning , and it is central to many datadriven application domains . Cluster analysis partitions all the data into disjoint groups , and one can understand the structure of the data by examining examples in each group . Many clustering methods have been proposed in the literature ( Aggarwal & Reddy , 2013 ) , such as k-means ( MacQueen et al. , 1967 ) , Gaussian mixture models ( Christopher , 2016 ) and spectral clustering ( Von Luxburg , 2007 ) . Conventional clustering methods are generally applied directly on the original data space . However , it is challenging to perform cluster analysis on high dimensional and unstructured data ( Steinbach et al. , 2004 ) , such as images . It is not only because the dimensionality is high , but also because the original data space is too complex to interpret , e.g . there are semantic gaps between pixel values and objects in images . Recently , deep learning based clustering methods have been proposed that simultanously learn nonlinear embeddings through deep neural networks and perform cluster analysis on the embedding space . The representation learning process learns effective high-level representations from high dimensional data and helps the cluster analysis . This is typically achieved by unsupervised deep learning methods , such as restricted Boltzmann machine ( RBM ) ( Hinton et al. , 2006 ; Hinton & Salakhutdinov , 2006 ) , autoencoders ( AE ) ( Vincent et al. , 2008 ; 2010 ) , variational autoencoders ( VAE ) ( Kingma & Welling , 2014 ) , etc . Previous deep learning based clustering methods ( Xie et al. , 2016 ; Guo et al. , 2017 ; Jiang et al. , 2017 ; Yang et al. , 2017 ) assume one single partition over the data and that all attributes define that partition . In real-world applications , however , the assumptions are usually not true . High-dimensional data are often multifaceted and can be meaningfully partitioned in multiple ways based on subsets of attributes ( Chen et al. , 2012 ) . For example , a student population can be clustered in one way based on course grades and in another way based on extracurricular activities . Movie reviews can be clustered based on both sentiment ( positive or negative ) and genre ( comedy , action , war , etc. ) . It is challenging to discover the multi-facet structures of data , especially for high-dimensional data . To resolve the above issues , we propose an unsupervised learning method , latent tree variational autoencoder ( LTVAE ) to learn latent superstructures in variational autoencoders , and simultaneously perform representation learning and structure learning . LTVAE is a generative model , where the data is assumed to be generated from latent features through neural networks , while the latent features themselves are generated from tree-structured Bayesian networks with another level of latent variables as shown in Fig . 1 . Each of those latent variables defines a facet of clustering . The proposed method automatically selects subsets of latent features for each facet , and learns the dependency structure among different facets . This is achieved through systematic structure learning . Consequently , LTVAE is able to discover complex structures of data rather than one partition . We also propose efficient learning algorithms for LTVAE with gradient descent and Stepwise EM through message passing . The rest of the paper is organized as follows . The related works are reviewed in Section 2 . We introduce the proposed method and learning algorithms in Section 3 . In Section 4 , we present the empirical results . The conclusion is given in Section 5 . 2 RELATED WORKS . Clustering has been extensively studied in the literature in many aspects ( Aggarwal & Reddy , 2013 ) . More complex clustering methods related to structure learning using Bayesian nonparametrics have been proposed , like Dirichlet Process ( Blei et al. , 2006 ) , Hierarchical Dirichlet Process ( HDP ) ( Teh et al. , 2006 ) . However , those are with conventional clustering methods that apply on raw data . Recently , deep learning based clustering methods have drawn more and more attention . A simple two-stage approach is to first learn low-dimensional embeddings using unsupervised feature learning methods , and then perform cluster analysis on the embeddings . However , without any supervision , the representation learning do not necessarily reveal the true cluster structure of the data . DEC ( Xie et al. , 2016 ) is a method that simultaneously learns feature representations and cluster assignments through deep autoencoders . It gradually improves the clustering by driving the deep network to learn a better mapping . Improved Deep Embedded Clustering ( Guo et al. , 2017 ) improves DEC by keeping the decoder network and adding reconstruction loss to the original clustering loss in DEC. Variational deep embedding ( Jiang et al. , 2017 ) is a generative method that models the data generative process using a Gaussian mixture model combined with a VAE , and also performs joint learning of representations and clustering . Similarly , GMVAE ( Dilokthanakul et al. , 2016 ) performs joint learning of a GMM and a VAE , but instead generates the mixture components through neural networks . Deep clustering network ( DCN ) ( Yang et al. , 2017 ) is another one that jointly learns an autoencoder and performs k-means clustering . These joint learning methods consistently achieve better clustering results than conventional ones . The method proposed in ( Yang et al. , 2016 ) uses convolutional neural networks and jointly learns the representations and clustering in a recurrent framework . All these methods assume flat partitions over the data , and do not attempt the structure learning issue . An exception is hierarchical nonparametric variational autoencoders proposed in ( Goyal et al. , 2017 ) . It uses nCRP as the prior for VAE to allow infinitely deep and branching tree hierarchy structure and focuses on learning hierarchy of concepts . However , it is still one partition over the data , only that the partitions in upper levels are more general partitions , while those in lower levels more fine-grained . Different from it , our work focuses on multifacets of clustering , for example , the model could make one partition based on identity of subjects , while another partition based on pose . 3 THE PROPOSED METHOD . In this section , we present the proposed latent tree variational autoencoder and the learning algorithms for joint representation learning and structure learning for multidimensional clustering . 3.1 LATENT TREE VARIATIONAL AUTOENCODER . Deep generative models assume that data x is generated from latent continuous variable z through some random process . The process consists of two steps : ( 1 ) a value z is generated from some prior distribution p ( z ) ; ( 2 ) the observation x is generated from the conditional distribution pθ ( x|z ) , which is parameterized through deep neural networks . Thus , it defines the joint distribution between 𝑍 '' 𝑌 '' 𝑌 $ 𝑍 $ 𝑍 '' , 𝑌 '' 𝑍 & , 𝑌 '' 𝑌 '' , 𝑌 $ 𝑍 ' , 𝑌 $ 𝑃 ( 𝑍 '' |𝑌 '' ) 𝑃 ( 𝑍 $ |𝑌 '' ) 𝑃 ( 𝑍 & |𝑌 $ ) 𝑃 ( 𝑌 $ |𝑌 '' ) 𝑃 ( 𝑌 '' ) 𝑃 ( 𝑍 '' = 𝑧 '' |𝑌 '' ) 𝑃 ( 𝑍 & = 𝑧 & |𝑌 '' ) 𝑃 ( 𝑍 ' = 𝑧'|𝑌 $ ) 𝑃 ( 𝑌 $ |𝑌 '' ) 𝑃 ( 𝑌 '' ) Clique Tree 𝑍 & 𝑍' 𝑃 ( 𝑍'|𝑌 $ ) 𝑍 $ , 𝑌 '' 𝑃 ( 𝑍 '' = 𝑧 '' |𝑌 '' ) Figure 2 : Inference and gradient through message passing . Solid-arrows denote collecting message , and dashed-arrows denote distributing message . observation x and latent variable z : p ( x , z ) = p ( z ) pθ ( x|z ) ( 1 ) This process is hidden from our view , and we learn this process by maximizing the marginal loglikelihood p ( x ) over the parameters θ and latent variable z from data . After the learning , the latent variable z can be regarded as the deep representations of x since it captures the most relevant information of x . Thus , the learning process is also called representation learning . In order to learn the latent structure of z , for example multidimensional cluster structure , we introduce a set of latent variables Y1 , ... , Yl on top of z . A single z or multiple z ’ s form a node zb . Suppose variables in z form B nodes of z1 , · · · , zB . Each latent variable Y may be only connected to a subset of nodes , and the dependency of each zb and its parent Y is characterized by a conditional Gaussian distribution . Furthermore , the latent variables Y1 , ... , Yl are connected to each other , and the dependency of a latent variable Y on its parent Y ′ is characterized by a conditional distribution P ( Y |Y ′ ) . This essentially forms a Bayesian network . And if we restrict the network to be treestructured , the z and Y together form a latent tree model ( Zhang , 2004 ; Poon et al. , 2010 ; 2013 ; Mourad et al. , 2013 ; Pearl , 2014 ; Zhang & Poon , 2017 ) with z being the observed variables and Y being the latent variables . For multidimensional clustering , each latent variable Y is taken to be a discrete variable , where each discrete state y of Y defines a cluster . Each latent variable Y thus defines a facet partition over the data based on subset of attributes and multiple Y ’ s define multiple facets . Given a value y of Y , zb follows a conditional Gaussian distribution P ( zb|y ) = N ( µy , Σy ) with mean vector µy and covariance matrix Σy . Thus , each zb and its parent constitute a Gaussian mixture model ( GMM ) . Suppose the parent of a node is denoted as π ( · ) , the maginal distribution of z is defined as follows p ( z ) = ∑ Y l∏ j=1 p ( yj |π ( Yj ) ) B∏ b=1 N ( zb|µπ ( zb ) , Σπ ( zb ) ) , ( 2 ) which sums over all possible combinations of Y states . As a matter of fact , a GMM is a Gaussian LTM that has only one latent variable connecting to all observed variables . Let the latent structure of Y be S , defining the number of latent variables in Y , the number of discrete states in each variable Y and the connectivity structure among all variables in z and Y . And let the parameters for all conditional probabilities in the latent structure be Θ . Both the latent structure S and the latent parameters Θ are unknown . We aim to jointly learn data representations and the latent structure . The proposed LTVAE model is shown in Fig . 1 . The latent structure S are automatically learned from data and will be discussed in a later section . Due to the existence of the generation network , the inference of the model is intratable . Instead , we do amortized variational inference for the latent variable z by introducing an inference network ( Kingma & Welling , 2014 ) and define an approximate posterior qφ ( z|x ) . The evidence lower bound ( ELBO ) LELBO of the marginal loglikelihood of the data given ( S , Θ ) is : LELBO ( x ) = Eqφ ( z|x ) [ log pθ ( x|z ) ] + Eqφ ( z|x ) [ log ∑ y pS ( z , y ; Θ ) ] + H [ qφ ( z|x ) ] , ( 3 ) where log ∑ y pS ( z , y ; Θ ) is the marginal loglikelihood of the latent variable z under the latent tree model , and H [ · ] is the entropy . The conditional generative distribution pθ ( x|z ) could be a Gaussian distribution if the input data is real-valued , or a Bernoulli distribution if binary , parameterized by the generation network . Using Monte Carlo sampling , the ELBO can be asymptotically estimated by LELBO ( x ) ' 1 M M∑ i=1 log pθ ( x|z ( i ) ) + log ∑ y pS ( z ( i ) , y ; Θ ) + H [ qφ ( z|x ) ] , ( 4 ) where z ( i ) ∼ qφ ( z|x ) . The term H [ qφ ( z|x ) ] can be computed analytically if we choose the form of qφ ( z|x ) to be a Gaussian distributionN ( z ; µx , σx ) : H [ qφ ( z|x ) ] = J2 log ( 2π ) + 1 2 ∑J j=1 ( 1+log σ 2 j ) , where J is the dimensionality of z . Furthermore , the marginal loglikelihood log ∑ y pS ( z ( i ) , y ; Θ ) can be computed efficiently through message passing . Message passing is an efficient algorithm for inference in Bayesian networks ( Koller & Friedman , 2009 ; Poon et al. , 2013 ) . In message passing , we first build a clique tree using the factors in the defined probability density . Because of the tree structure , each zb along with its parent form a clique with the potential ψ ( zb , y ) being the corresponding conditional distribution . This is illustrated in Fig . 2 . With the sampled z ( i ) , we can compute the message ψ′ ( y ) by absorbing the evidence from z . During collecting message phase , the message ψ′ ( y ) are sent towards the pivot . After receiving all messages , the pivot distributes back messages towards all z . Both the posterior of Y and the marginal loglikelihood of z ( i ) thus can be computed in the final normalization step .

Summary: This paper introduces a new VAE model, the latent tree VAE (LTVAE), which aims to learn models with multifaceted clustering, that is separate clusterings are enforced on different subsets of the latent features.  This is achieved using a tree-structured prior on a set of discrete "super latent variables" (Y_1,...,Y_L) that identify which cluster the datapoint falls into for each separate facet (i.e. there is a separate clustering associated with each Y_n).  The subset of the standard latent variables z then form a Gaussian mixture model (GMM) for each Y_n.   Both the structure of this setup (i.e. the associated graphical model) and the parameters (i.e. means and variances of the clusters) are learned during training.  This introduces a number of computational challenges not usually seen in for VAE training, for which, seemingly well thought through, novel schemes are introduced, most notably a message passing scheme for calculating gradients of the log marginal p(z).