A Probabilistic Approach to Constrained Deep Clustering

1 INTRODUCTION . The ever-growing amount of data and the time cost associated with its labeling has made clustering a relevant task in the field of machine learning . Yet , in many cases , a fully unsupervised clustering algorithm might naturally find a solution which is not consistent with the domain knowledge ( Basu et al. , 2008 ) . In medicine , for example , clustering could be driven by unwanted bias , such as the type of machine used to record the data , rather than more informative features . Moreover , practitioners often have access to prior information about the types of clusters that are sought , and a principled method to guide the algorithm towards a desirable configuration is then needed . Constrained clustering , therefore has a long history in machine learning as it enforces desirable clustering properties by incorporating domain knowledge , in the form of constraints , into the clustering objective . Following recent advances in deep clustering , constrained clustering algorithms have been recently used in combination with deep neural networks ( DNN ) to favor a better representation of highdimensional data sets . The methods proposed so far mainly extend some of the most widely used deep clustering algorithms , such as DEC ( Xie et al. , 2016 ) , to include a variety of loss functions that force the clustering process to be consistent with the given constraints ( Ren et al. , 2019 ; Shukla et al. , 2018 ; Zhang et al. , 2019b ) . Although they perform well , none of the above methods model the data generative process . As a result , they can neither uncover the underlying structure of the data , nor control the strength of the clustering preferences , nor generate new samples ( Min et al. , 2018 ) . To address the above issues , we propose a novel probabilistic approach to constrained clustering , the Constrained Variational Deep Embedding ( CVaDE ) , that uncovers the underlying data distribution conditioned on domain knowledge , expressed in the form of pairwise constraints . Our method extends previous work in unsupervised variational deep clustering ( Jiang et al. , 2017 ; Dilokthanakul et al. , 2016 ) to incorporate clustering preferences as Bayesian prior probabilities with varying degrees of uncertainty . This allows systematical reasoning about parameter uncertainty ( Zhang et al. , 2019a ) , thereby enabling the ability to perform Bayesian model validation , outlier detection and data generation . By integrating prior information in the generative process of the data , our model can guide the clustering process towards the configuration sought by the practitioners . Our main contributions are as follows : ( i ) We propose a constrained clustering method ( CVaDE ) to incorporate given clustering preferences , with varying degrees of certainty , within the Variational Auto-Encoder ( VAE ) framework . ( ii ) We provide a thorough empirical assessment of our model . In particular , we show that ( a ) a small fraction of prior information remarkably increases the performance of CVaDE compared to unsupervised variational clustering methods , ( b ) our model shows superior clustering performance compared to state-of-the-art deep constrained clustering models on a wide range of data sets and , ( c ) our model proves to be robust against noise as it can easily incorporate the uncertainty of the given constraints . ( iii ) We show that our model can drive the clustering performance towards different desirable configurations , depending on the constraints used , and it successfully generates new samples on challenging real-world image data . 2 THEORETICAL BACKGROUND & RELATED WORK . Constrained Clustering . A constrained clustering problem differs from the classical clustering scenario as the user has access to some pre-existing knowledge about the desired partition of the data . The constraints are usually expressed as pairwise constraints ( Wagstaff & Cardie , 2000 ) , consisting of must-links and can not -links , which indicate whether two samples are believed to belong to the same cluster or to different clusters . Such pairwise relations contain less information than the labels used in classification tasks but are usually easier to obtain . Traditional clustering methods have been then extended to enforce pairwise constraints ( Lange et al. , 2005 ) . COP-KMEANS ( Wagstaff et al. , 2001 ) and MPCK-mean ( Bilenko et al. , 2004 ) adapted the well-known K-means algorithm , while several methods proposed a constrained version of the Gaussian Mixture Models ( Shental et al. , 2003 ; Law et al. , 2004 ; 2005 ) . Among them , penalized probabilistic clustering ( PPC , Lu & Leen ( 2004 ) ) is most related to our work as it expresses the pairwise constraints as Bayesian priors over the assignment of data points to clusters . However , PPC , as well as the previous models , shows poor performance and high computational complexity on high-dimensional and large-scale data sets . Deep Constrained Clustering . To overcome the limitations of the above models , constrained clustering algorithms have been lately used in combination with DNNs . Hsu & Kira ( 2015 ) train a DNN to minimize the Kullback-Leibler ( KL ) divergence between similar pairs of samples , while Chen ( 2015 ) performs semi-supervised maximum margin clustering of the learned features on a DNN . More recently , many extensions of the widely used DEC model ( Xie et al. , 2016 ) have been proposed to include a variety of loss functions to enforce pairwise constraints . Among them , SDEC ( Ren et al. , 2019 ) includes a distance loss function that forces the data points with a must-link to be close in the latent space and vice-versa . C-IDEC ( Zhang et al. , 2019b ) , uses , instead , a KL divergence loss , extending the work of Shukla et al . ( 2018 ) . Other works have focused on discriminative clustering methods by self-generating pairwise constraints from either Siamese networks or KNN graphs ( Smieja et al. , 2020 ) ( Fogel et al. , 2019 ) . As none of the approaches proposed so far is based on generative models , the above methods fail to uncover the underlying data distribution . Additionally , DEC-based architectures rely on heavy pretraining of the autoencoder , resulting in no theoretical guarantee that the learned latent space is indeed suitable for clustering ( Min et al. , 2018 ) . VAE-based deep clustering . Many models have been proposed in the literature to perform unsupervised clustering through deep generative models ( Li et al. , 2019 ; Yang et al. , 2019 ; Manduchi et al. , 2019 ; Kopf et al. , 2019 ) . Among them , the Variational Deep Embedding ( VaDE , Jiang et al . ( 2017 ) ) and the Gaussian Mixture Variational Autoencoder ( GMM-VAE , Dilokthanakul et al . ( 2016 ) ) propose a variant of the VAE ( Kingma & Welling ( 2014 ) ; Rezende et al . ( 2014 ) ) in which the prior is a Gaussian Mixture distribution . With this assumption , they construct an inference model that can be directly optimised in the framework of stochastic gradient variational Bayes . However , variational deep clustering methods , such as the VaDE , can not incorporate domain knowledge and clustering preferences . Even though a semi-supervised version on the VAE has been proposed by Kingma et al . ( 2014 ) , the latter can not be naturally applied to clustering . For this reason , we aim at extending the above methods to incorporate clustering preferences in the form of constraints , modeled as Bayesian priors , to guide the clustering process towards a desirable configuration . 3 CONSTRAINED VARIATIONAL DEEP EMBEDDING . In the following section , we propose a novel constrained clustering model ( CVaDE ) to incorporate clustering preferences , with varying degree of certainty , in a VAE-based deep clustering setting . In particular , we use the VaDE ( Jiang et al. , 2017 ) generative assumptions of the data , conditioned on the domain knowledge . We then illustrate how our model can be trained efficiently in the framework of stochastic gradient variational Bayes by optimizing the Conditional Variational Lower Bound . Additionally , we define concrete prior formulations to incorporate our preferences , with a focus on pairwise constraints . 3.1 THE GENERATIVE ASSUMPTIONS . Let us consider a data set X = { xi } Ni=1 consisting of N samples with xi ∈ RM that we wish to cluster into K groups according to some prior information encoded as G. For example , we may know a priori that certain samples should be clustered together with different degree of certainty . HenceG encodes both our prior knowledge on the data set and the degree of confidence . We assume the data is generated from a random process consisting of three steps . First , the cluster assignments c = { ci } Ni=1 , with ci ∈ { 1 , . . . , K } , are sampled from a distribution conditioned on the prior information , c ∼ p ( c|G ) . Next , for each cluster assignment ci , a continuous latent embedding , zi ∈ RD , is sampled from a Gaussian distribution , whose mean and variance depend on the selected cluster ci . Finally , the sample xi is generated from a distribution conditioned on zi . Given ci , the generative process can be summarized as : zi ∼ p ( zi|ci ) = N ( zi|µci , σ2ciI ) ( 1 ) xi ∼ pθ ( xi|zi ) = { N ( xi|µxi , σ2xiI ) with [ µxi , σ 2 xi ] = f ( zi ; θ ) if xi is real-valued Ber ( µxi ) with µxi = f ( zi ; θ ) if xi is binary ( 2 ) where µci and σ 2 ci are mean and variance of the Gaussian distribution corresponding to cluster ci in the latent space and the function f ( z ; θ ) is a neural network , called decoder , parametrized by θ . Without prior information , that is when p ( c|G ) = p ( c ) = ∏ i p ( ci ) = ∏ i Cat ( ci|π ) , the cluster assignments are independent and identical distributed as they follow a categorical distribution with mixing parameters π . In that case , the generative assumptions described above are equal to those of Jiang et al . ( 2017 ) and the parameters of the model can be learned using the unsupervised VaDE method ( see Appendix C ) . In the following , we explore the case when p ( c|G ) 6= p ( c ) . 3.2 CONDITIONAL VARIATIONAL LOWER BOUND . Given the data generative assumptions illustrated in Sec . 3.1 , the objective is to infer the parameters π , µc , σ2c and θ which better explain the data X given prior information on the cluster assignments G. We achieve this by maximizing the marginal log-likelihood conditioned onG , that is : log p ( X|G ) = log ∫ Z ∑ c p ( X , Z , c|G ) , ( 3 ) where Z = { zi } Ni=1 is the collection of the latent embeddings corresponding to the data setX . The conditional joint probability is derived from Eq 1/2 and can be factorized as : p ( X , Z , c|G ) = pθ ( X|Z ) p ( Z|c ) p ( c|G ) = p ( c|G ) N∏ i=1 pθ ( xi|zi ) p ( zi|ci ) . ( 4 ) Since the conditional log-likelihood is intractable , we derive a lower bound of the log marginal conditional probability of the data , which we call Conditional ELBO ( C-ELBO , LC ) : LC ( X|G ) = Eqφ ( Z , c|X ) log p ( X , Z , c|G ) qφ ( Z , c|X ) ( 5 ) Similarly to Jiang et al . ( 2017 ) and Dilokthanakul et al . ( 2016 ) , we employ the following amortized mean-field variational distribution : qφ ( Z , c|X ) = qφ ( Z|X ) p ( c|Z ) = N∏ i=1 qφ ( zi|xi ) p ( ci|zi ) with p ( ci|zi ) = p ( zi|ci ) p ( ci ) ∑ k p ( zi|k ) p ( k ) , ( 6 ) where qφ ( zi|xi ) is a Gaussian distribution with mean µ ( xi ) and variance σ2 ( xi ) I which are the outputs of a neural network , called encoder , parametrized by φ and p ( ci = k ) is denoted as p ( k ) for simplicity . It is important to note that , in this formulation , the variational distribution does not depend on G. This approximation is used to retain a mean-field variational distribution if the cluster assignments , conditioned on the prior information , are not independent ( Sec 3.4.1 ) , that is when p ( c|G ) 6= ∏ i p ( ci|G ) .

Summary: This work proposes CVaDE which is an extension of variational based deep clustering model (VaDE) with additional incorporation of prior clustering preferences as supervision. These priors guide the underlying clustering process towards a user-desirable partitioning of input data. The priors are provided in the form of pairwise constraints indicating which pair of samples belongs to same or different class. Clustering process is modelled using variational Bayes in which the clustering constraints are incorporated into prior probabilities with varying degree of uncertainty. The empirical results shows that in comparison to unconstrained clustering the small amount of pairwise constraints significantly improves clustering performance. Further, it demonstrates CVaDE's robustness to noise, generation capability as well as successful incorporation of different desirable preferences to drive clustering performance towards completely different partitioning.