A Probabilistic Approach to Constrained Deep Clustering

1 INTRODUCTION . The ever-growing amount of data and the time cost associated with its labeling has made clustering a relevant task in the field of machine learning . Yet , in many cases , a fully unsupervised clustering algorithm might naturally find a solution which is not consistent with the domain knowledge ( Basu et al. , 2008 ) . In medicine , for example , clustering could be driven by unwanted bias , such as the type of machine used to record the data , rather than more informative features . Moreover , practitioners often have access to prior information about the types of clusters that are sought , and a principled method to guide the algorithm towards a desirable configuration is then needed . Constrained clustering , therefore has a long history in machine learning as it enforces desirable clustering properties by incorporating domain knowledge , in the form of constraints , into the clustering objective . Following recent advances in deep clustering , constrained clustering algorithms have been recently used in combination with deep neural networks ( DNN ) to favor a better representation of highdimensional data sets . The methods proposed so far mainly extend some of the most widely used deep clustering algorithms , such as DEC ( Xie et al. , 2016 ) , to include a variety of loss functions that force the clustering process to be consistent with the given constraints ( Ren et al. , 2019 ; Shukla et al. , 2018 ; Zhang et al. , 2019b ) . Although they perform well , none of the above methods model the data generative process . As a result , they can neither uncover the underlying structure of the data , nor control the strength of the clustering preferences , nor generate new samples ( Min et al. , 2018 ) . To address the above issues , we propose a novel probabilistic approach to constrained clustering , the Constrained Variational Deep Embedding ( CVaDE ) , that uncovers the underlying data distribution conditioned on domain knowledge , expressed in the form of pairwise constraints . Our method extends previous work in unsupervised variational deep clustering ( Jiang et al. , 2017 ; Dilokthanakul et al. , 2016 ) to incorporate clustering preferences as Bayesian prior probabilities with varying degrees of uncertainty . This allows systematical reasoning about parameter uncertainty ( Zhang et al. , 2019a ) , thereby enabling the ability to perform Bayesian model validation , outlier detection and data generation . By integrating prior information in the generative process of the data , our model can guide the clustering process towards the configuration sought by the practitioners . Our main contributions are as follows : ( i ) We propose a constrained clustering method ( CVaDE ) to incorporate given clustering preferences , with varying degrees of certainty , within the Variational Auto-Encoder ( VAE ) framework . ( ii ) We provide a thorough empirical assessment of our model . In particular , we show that ( a ) a small fraction of prior information remarkably increases the performance of CVaDE compared to unsupervised variational clustering methods , ( b ) our model shows superior clustering performance compared to state-of-the-art deep constrained clustering models on a wide range of data sets and , ( c ) our model proves to be robust against noise as it can easily incorporate the uncertainty of the given constraints . ( iii ) We show that our model can drive the clustering performance towards different desirable configurations , depending on the constraints used , and it successfully generates new samples on challenging real-world image data . 2 THEORETICAL BACKGROUND & RELATED WORK . Constrained Clustering . A constrained clustering problem differs from the classical clustering scenario as the user has access to some pre-existing knowledge about the desired partition of the data . The constraints are usually expressed as pairwise constraints ( Wagstaff & Cardie , 2000 ) , consisting of must-links and can not -links , which indicate whether two samples are believed to belong to the same cluster or to different clusters . Such pairwise relations contain less information than the labels used in classification tasks but are usually easier to obtain . Traditional clustering methods have been then extended to enforce pairwise constraints ( Lange et al. , 2005 ) . COP-KMEANS ( Wagstaff et al. , 2001 ) and MPCK-mean ( Bilenko et al. , 2004 ) adapted the well-known K-means algorithm , while several methods proposed a constrained version of the Gaussian Mixture Models ( Shental et al. , 2003 ; Law et al. , 2004 ; 2005 ) . Among them , penalized probabilistic clustering ( PPC , Lu & Leen ( 2004 ) ) is most related to our work as it expresses the pairwise constraints as Bayesian priors over the assignment of data points to clusters . However , PPC , as well as the previous models , shows poor performance and high computational complexity on high-dimensional and large-scale data sets . Deep Constrained Clustering . To overcome the limitations of the above models , constrained clustering algorithms have been lately used in combination with DNNs . Hsu & Kira ( 2015 ) train a DNN to minimize the Kullback-Leibler ( KL ) divergence between similar pairs of samples , while Chen ( 2015 ) performs semi-supervised maximum margin clustering of the learned features on a DNN . More recently , many extensions of the widely used DEC model ( Xie et al. , 2016 ) have been proposed to include a variety of loss functions to enforce pairwise constraints . Among them , SDEC ( Ren et al. , 2019 ) includes a distance loss function that forces the data points with a must-link to be close in the latent space and vice-versa . C-IDEC ( Zhang et al. , 2019b ) , uses , instead , a KL divergence loss , extending the work of Shukla et al . ( 2018 ) . Other works have focused on discriminative clustering methods by self-generating pairwise constraints from either Siamese networks or KNN graphs ( Smieja et al. , 2020 ) ( Fogel et al. , 2019 ) . As none of the approaches proposed so far is based on generative models , the above methods fail to uncover the underlying data distribution . Additionally , DEC-based architectures rely on heavy pretraining of the autoencoder , resulting in no theoretical guarantee that the learned latent space is indeed suitable for clustering ( Min et al. , 2018 ) . VAE-based deep clustering . Many models have been proposed in the literature to perform unsupervised clustering through deep generative models ( Li et al. , 2019 ; Yang et al. , 2019 ; Manduchi et al. , 2019 ; Kopf et al. , 2019 ) . Among them , the Variational Deep Embedding ( VaDE , Jiang et al . ( 2017 ) ) and the Gaussian Mixture Variational Autoencoder ( GMM-VAE , Dilokthanakul et al . ( 2016 ) ) propose a variant of the VAE ( Kingma & Welling ( 2014 ) ; Rezende et al . ( 2014 ) ) in which the prior is a Gaussian Mixture distribution . With this assumption , they construct an inference model that can be directly optimised in the framework of stochastic gradient variational Bayes . However , variational deep clustering methods , such as the VaDE , can not incorporate domain knowledge and clustering preferences . Even though a semi-supervised version on the VAE has been proposed by Kingma et al . ( 2014 ) , the latter can not be naturally applied to clustering . For this reason , we aim at extending the above methods to incorporate clustering preferences in the form of constraints , modeled as Bayesian priors , to guide the clustering process towards a desirable configuration . 3 CONSTRAINED VARIATIONAL DEEP EMBEDDING . In the following section , we propose a novel constrained clustering model ( CVaDE ) to incorporate clustering preferences , with varying degree of certainty , in a VAE-based deep clustering setting . In particular , we use the VaDE ( Jiang et al. , 2017 ) generative assumptions of the data , conditioned on the domain knowledge . We then illustrate how our model can be trained efficiently in the framework of stochastic gradient variational Bayes by optimizing the Conditional Variational Lower Bound . Additionally , we define concrete prior formulations to incorporate our preferences , with a focus on pairwise constraints . 3.1 THE GENERATIVE ASSUMPTIONS . Let us consider a data set X = { xi } Ni=1 consisting of N samples with xi ∈ RM that we wish to cluster into K groups according to some prior information encoded as G. For example , we may know a priori that certain samples should be clustered together with different degree of certainty . HenceG encodes both our prior knowledge on the data set and the degree of confidence . We assume the data is generated from a random process consisting of three steps . First , the cluster assignments c = { ci } Ni=1 , with ci ∈ { 1 , . . . , K } , are sampled from a distribution conditioned on the prior information , c ∼ p ( c|G ) . Next , for each cluster assignment ci , a continuous latent embedding , zi ∈ RD , is sampled from a Gaussian distribution , whose mean and variance depend on the selected cluster ci . Finally , the sample xi is generated from a distribution conditioned on zi . Given ci , the generative process can be summarized as : zi ∼ p ( zi|ci ) = N ( zi|µci , σ2ciI ) ( 1 ) xi ∼ pθ ( xi|zi ) = { N ( xi|µxi , σ2xiI ) with [ µxi , σ 2 xi ] = f ( zi ; θ ) if xi is real-valued Ber ( µxi ) with µxi = f ( zi ; θ ) if xi is binary ( 2 ) where µci and σ 2 ci are mean and variance of the Gaussian distribution corresponding to cluster ci in the latent space and the function f ( z ; θ ) is a neural network , called decoder , parametrized by θ . Without prior information , that is when p ( c|G ) = p ( c ) = ∏ i p ( ci ) = ∏ i Cat ( ci|π ) , the cluster assignments are independent and identical distributed as they follow a categorical distribution with mixing parameters π . In that case , the generative assumptions described above are equal to those of Jiang et al . ( 2017 ) and the parameters of the model can be learned using the unsupervised VaDE method ( see Appendix C ) . In the following , we explore the case when p ( c|G ) 6= p ( c ) . 3.2 CONDITIONAL VARIATIONAL LOWER BOUND . Given the data generative assumptions illustrated in Sec . 3.1 , the objective is to infer the parameters π , µc , σ2c and θ which better explain the data X given prior information on the cluster assignments G. We achieve this by maximizing the marginal log-likelihood conditioned onG , that is : log p ( X|G ) = log ∫ Z ∑ c p ( X , Z , c|G ) , ( 3 ) where Z = { zi } Ni=1 is the collection of the latent embeddings corresponding to the data setX . The conditional joint probability is derived from Eq 1/2 and can be factorized as : p ( X , Z , c|G ) = pθ ( X|Z ) p ( Z|c ) p ( c|G ) = p ( c|G ) N∏ i=1 pθ ( xi|zi ) p ( zi|ci ) . ( 4 ) Since the conditional log-likelihood is intractable , we derive a lower bound of the log marginal conditional probability of the data , which we call Conditional ELBO ( C-ELBO , LC ) : LC ( X|G ) = Eqφ ( Z , c|X ) log p ( X , Z , c|G ) qφ ( Z , c|X ) ( 5 ) Similarly to Jiang et al . ( 2017 ) and Dilokthanakul et al . ( 2016 ) , we employ the following amortized mean-field variational distribution : qφ ( Z , c|X ) = qφ ( Z|X ) p ( c|Z ) = N∏ i=1 qφ ( zi|xi ) p ( ci|zi ) with p ( ci|zi ) = p ( zi|ci ) p ( ci ) ∑ k p ( zi|k ) p ( k ) , ( 6 ) where qφ ( zi|xi ) is a Gaussian distribution with mean µ ( xi ) and variance σ2 ( xi ) I which are the outputs of a neural network , called encoder , parametrized by φ and p ( ci = k ) is denoted as p ( k ) for simplicity . It is important to note that , in this formulation , the variational distribution does not depend on G. This approximation is used to retain a mean-field variational distribution if the cluster assignments , conditioned on the prior information , are not independent ( Sec 3.4.1 ) , that is when p ( c|G ) 6= ∏ i p ( ci|G ) .

Summary: This paper extends the variational deep embedding VaDE model (a VAE-based clustering method) to integrate pairwise constraints between objects, i.e., must-link and cannot-link. The constraints are integrated a priori as a condition. That is, the prior over the cluster labels is conditioned on the constraints. The whole model, referred to as Constrained VaDE (CVaDE), takes the form of a conditional VAE tailored for constrained clustering. Experiments are curried out on various real-world datasets, and the proposed method is compared to VaDE as well as to recent and classical constrained clustering methods.