Deep Goal-Oriented Clustering

1 INTRODUCTION . Much of the advances in supervised learning in the past decade are due to the development of deep neural networks ( DNN ) , a class of hierarchical function approximators that are capable of learning complex input-output relationships . Prime examples of such advances include image recognition ( Krizhevsky et al. , 2012 ) , speech recognition ( Nassif et al. , 2019 ) , and neural translation ( Bahdanau et al. , 2015 ) . However , with the explosion of the size of modern datasets , it becomes increasingly unrealistic to manually annotate all available data for training . Hence , understanding inherent data structure through unsupervised clustering is of increasing importance . Several approaches to apply DNNs to unsupervised clustering have been proposed in the past few years ( Caron et al. , 2018 ; Law et al. , 2017 ; Xie et al. , 2016 ; Shaham et al. , 2018 ) , centering around the concept that the input space in which traditional clustering algorithms operate is of importance . Hence , learning this space from data is desirable , in particular , for complex data . Despite the improvements these approaches have made on benchmark clustering datasets , the ill-defined , ambiguous nature of clustering still remains a challenge . Such ambiguity is particularly problematic in scientific discovery , sometimes requiring researchers to choose from different , but potentially equally meaningful clustering results when little information is available a priori ( Ronan et al. , 2016 ) . When facing such ambiguity , using direct side-information to reduce clustering ambivalence proves to be a fruitful direction ( Xing et al. , 2002 ; Khashabi et al. , 2015 ; Jin et al. , 2013 ) . Direct sideinformation is usually available in terms of constraints , such as the must-link and the can not -link constraints ( Wang & Davidson , 2010 ; Wagstaff & Cardie , 2000 ) , or via a pre-conceived notion of similarity ( Xing et al. , 2002 ) . However , defining such direct side-information requires human expertise , which could be labor intensive and potentially vulnerable to labeling errors . On the contrary , indirect , but informative , side-information might exist in abundance , and may not require human expertise to obtain . Being able to learn from such indirect information to form a congruous clustering strategy is thus immensely valuable . Main Contributions We propose Deep Goal-Oriented Clustering ( DGC ) , a probabilistic model that is capable of using indirect , but informative , side-information to form a pertinent clustering strategy . Specifically : 1 ) We combine supervision via side-information and unsupervised data structure modeling in a probabilistic manner ; 2 ) We make minimal assumptions on what form the supervised side-information might take , and assume no explicit correspondence between the sideinformation and the clusters ; 3 ) We train DGC end-to-end so that the model simultaneously learns from the available side-information while forming a desired clustering strategy . 2 RELATED WORK . Most related work in the literature can be classified into two categories : 1 ) Methods that utilize extra side-information to form better , less ambiguous clusters ; however , such side-information needs to be provided beforehand and can not be learned ; 2 ) Methods that can learn from the provided labels to lessen the ambiguity in the formed clusters , but these methods rely on the cluster assumption ( detailed below ) , and usually assume that the provided labels are discrete and the ground truth labels . This excludes the possibility of learning from indirectly related , but informative side-information . We propose a unified framework that allows using informative side-information directly or indirectly to arrive at better formed clusters . Latent space sharing among different tasks has been studied in a VAE setting ( Le et al. , 2018 ; Xie & Ma , 2019 ) . In this work we utilize this latent space sharing framework , but instead focus on clustering with the aid of general , indirect side-information . Side-information as constraints Using side-information to form better clusters is well-studied . Wagstaff & Cardie ( 2000 ) consider both must-link and can not -link constraints in the context of K-means clustering . Motivated by image segmentation , Orbanz & Buhmann ( 2007 ) proposed a probabilistic model that can incorporate must-link constraints . Khashabi et al . ( 2015 ) proposed a nonparametric Bayesian hierarchical model to incorporate noisy side-information as soft-constraints . Vu et al . ( 2019 ) utilize constraints and cluster labels as side information . Mazumdar & Saha ( 2017 ) give complexity bounds when provided with an oracle that can be queried for side information . Wasid & Ali ( 2019 ) incorporate side information through the use of fuzzy sets . In supervised clustering , the side-information is the a priori known complete clustering for the training set , which is being used as a constraint to learn a mapping between the data and the given clustering ( Finley & Joachims , 2005 ) . In contrast , we do not assume that the constraints are given a priori . Instead , we let the side-information guide the clustering procedure during the training process . Semi-supervised methods & the cluster assumption Semi-supervised clustering approaches generally assume that they only have access to a fraction of the true cluster labels . Via constraints as the ones discussed , the available labels are propagated to unlabeled data , which can help mitigate the ambiguity in choosing among different clustering strategies ( Bair , 2013 ) . The generative approach to semi-supervised learning introduced in Kingma et al . ( 2014 ) is based on a hierarchical generative model with two variational layers . Although it was originally meant for semi-supervised classification tasks , it can also be used for clustering . However , if used for clustering , it has to strictly rely on the cluster assumption , which states that there exists a direct correspondence between labels/classes and clusters ( Färber et al. , 2010 ; Chapelle et al. , 2006 ) . We show that this approach is a special case of our framework without the probabilistic ensemble component ( see Sec . 4.2 ) and when certain distributional assumptions are made . Sansone et al . ( 2016 ) proposed a method for joint classification and clustering to address the stringent cluster assumption most approaches make by modeling the cluster indices and the class labels separately , underscoring the possibility that each cluster may consist of multiple class labels . Deploying a mixture of factor analysers as the underlying probabilistic framework , they also used a variational approximation to maximize the joint log-likelihood . In this work , we generalize the notion of learning from discrete , ground truth labels to learning from indirect , but informative side-information . We make virtually no assumptions on the form of y nor its relations to the clusters . This makes our approach more applicable to general settings . 3 BACKGROUND & PROBLEM SETUP . 3.1 BACKGROUND—VARIATIONAL DEEP EMBEDDING . The starting point for DGC is the variational auto-encoder ( VAE ) ( Kingma & Welling , 2014 ) with the prior distribution of the latent code chosen as a Gaussian mixture distribution . This is introduced in Jiang et al . ( 2017 ) as VaDE . We briefly review the generative VaDE approach here to provide the background for DGC . We adopt the notation that lower case letters denote samples from their corresponding distributions ; bold , lower case letters denote random variables/vectors ; and bold upper case letters denote random matrices . Assume the prior distribution of the latent code , z , belongs to the family of Gaussian mixture distributions , i.e . p ( z ) = ∑ c p ( z|c ) p ( c ) = ∑ c πcN ( µc , σ2c I ) where c is a random variable , with prior probability πc , indexing the normal component with mean µc and variance σ2c . VaDE allows for the clustering of the input data in the latent space , with each component of the Gaussian mixture prior representing an underlying cluster . A VAE-based model can be efficiently described in terms of its generative process and inference procedure . Given an input x ∈ Rd , the following decomposition of the joint probability p ( x , z , c ) details VaDE ’ s generative process : p ( x , z , c ) = p ( x|z ) p ( z|c ) p ( c ) . In words , we first sample the component index c from a prior categorical distribution p ( c ) , then sample the latent code z from the component p ( z|c ) , and lastly reconstruct the input x through the reconstruction network p ( x|z ) . To perform inference and learn from the data , VaDE is constructed to maximize the log-likelihood of the input data x by maximizing its evidence lower bound ( ELBO ) : log p ( x ) ≥ Eq ( z|x ) log p ( x|z ) − Eq ( c|x ) log q ( c|x ) p ( c ) − Eq ( z , c|x ) log q ( z|x ) p ( z|c ) ( 1 ) where , given the input x , q ( z , c|x ) denotes the variational posterior distribution over the latent variables , and Ed denotes the expectation wrt . distribution d. With proper assumptions on the prior and variational posterior distributions , the ELBO in Eq . 1 admits a closed-form expression in terms of the parameters of those distributions . We refer readers to Jiang et al . ( 2017 ) for additional details . 3.2 PROBLEM SETUP Unlike the unsupervised settings , we do assume we have a response variable y , and our goal is to leverage y to inform a better clustering strategy . Abstractly , given the input-output random variable pair ( x , y ) , we seek to divide the probability space of x into non-overlapping subspaces that are meaningful in explaining the output y . In other words , we want to use the prediction task of mapping data points , x , sampled from the probability space of x to their corresponding sampled outcomes y as a teaching agent , to guide the process of dividing the probability space of x into subspaces that optimally explain y . Since our goal is to discover the subspace-structure without knowing a priori whether such a structure indeed exists , a probabilistic framework is more appropriate due to its ability to incorporate and reason with uncertainty . To this end , we use and extend the VaDE framework , with the following assumption imposed on the latent code that specifically caters to our setting . Assume the input x carries predictive information with respect to the output y . Since the latent code z should inherit sufficient information from which the input x can be reconstructed , it is reasonable to assume that z also inherits that predictive information . This assumption implies that x and y are conditionally independent given z , i.e . p ( x , y|z ) = p ( x|z ) p ( y|z ) . 4 DEEP GOAL-ORIENTED CLUSTERING . 4.1 GENERATIVE PROCESS . In order to incorporate y into a probabilistic model , recall from our previous discussion that y might manifest with respect to the input differently across different subspaces of the input space . Viewing p ( y|z ) as a conditional probability distribution over y resulting from a functional transformation from z to the space of probability distributions over y , we can assume that the ground truth transformation function , gc , is different for each subspace indexed by c. If z ∼ p ( z|c ) for some index c , we assume p ( y|z , c ) ∝ gc ( z ) for some subspace-specific gc . As a result , we learn a different mapping function for each subspace . The overall generative process of our model is as follows : 1 . Generate c ∼ Cat ( π ) ; 2 . Generate z ∼ p ( z|c ) ; 3 . Generate x ∼ p ( x|z ) ; 4 . Generate y ∼ p ( y|z , c ) . The Bayesian network that underlies DGC is shown in Fig . 1 , and the joint distribution of x , y , z , and c can be decomposed as : p ( x , y , z , c ) = p ( y|z , c ) p ( x|z ) p ( z|c ) p ( c ) .

Summary: This paper introduces Deep Goal Oriented Clustering, an approach for joint clustering and classification. The approach shares a latent embedding for the data between the two tasks. The latent embedding is parameterized by a mixture of Gaussians. The approach gives a probabilistic, VAE-based formulation and derives the variational lower bound for the model. The authors run experiments investigating the effectiveness of their approach and the impact of the clustering-component of their approach.