Deep Conditional Gaussian Mixture Model for Constrained Clustering
Authors: Laura Manduchi
Kieran Chin-Cheong
Holger Michel
Sven Wellmann
Julia E Vogt
Venue: NeurIPS 2021 Poster (2021)
Keywords: constrained clustering
deep generative models
representation learning

Abstract: Constrained clustering has gained significant attention in the field of machine learning as it can leverage prior information on a growing amount of only partially labeled data. Following recent advances in deep generative models, we propose a novel framework for constrained clustering that is intuitive, interpretable, and can be trained efficiently in the framework of stochastic gradient variational inference. By explicitly integrating domain knowledge in the form of probabilistic relations, our proposed model (DC-GMM) uncovers the underlying distribution of data conditioned on prior clustering preferences, expressed as \textit{pairwise constraints}. These constraints guide the clustering process towards a desirable partition of the data by indicating which samples should or should not belong to the same cluster. We provide extensive experiments to demonstrate that DC-GMM shows superior clustering performances and robustness compared to state-of-the-art deep constrained clustering methods on a wide range of data sets. We further demonstrate the usefulness of our approach on two challenging real-world applications.

Deep Conditional Gaussian Mixture Model for Constrained Clustering
Laura Manduchi
Kieran Chin-Cheong
Department of Computer Science
Department of Computer Science
ETH Zürich muura.manduch@@m ethz ch
ETH Zürich
kieran. chincheongeinf ethz. ch
Holger Michel
Department of Neonatology
University Children's Hospital Regensburg (KUNO)
University of Regensburg, Germany
holger mi
de
Sven Wellmann
Julia E. Vogt
Department of Neonatology
Department of Computer Science
University Children's Hospital Regensburg (KUNO)
ETH Zürich
University of Regensburg, Germany
julia. vogt@inf wtw.e ch
sven. wee..welllmmn@k uni regensburg..ee
Abstract
Constrained clustering has gained significant attention in the field of machine learning it can leverage prior information growing amount of only partially labeled data. Following recent advances in deep generative models, we propose a novel framework for constrained clustering that is intuitive, interpretable, and be trained efficiently in the framework of stochastic gradient variational inference. By explicitly integrating domain knowledge in the form of probabilistic relations, our proposed model (DC-GMM) uncovers the underlying distribution of data conditioned on prior clustering preferences, expressed as pairwise constraints. These constraints guide the clustering process towards a desirable partition of the data by indicating which samples should or should not belong the same cluster. We provide extensive experiments to demonstrate that DC-GMM shows superior clustering performances and robustness compared to state-of-the-ar deep constrained clustering methods on a wide range of data sets. We further demonstrate the usefulness our approach on two challenging real-world applications.
1 Introduction
The ever-growing amount of data and the time cost associated with its labeling has made clustering relevant task in the field of machine learning. Yet, in many cases, a fully unsupervised clustering algorithm might naturally find solution that not consistent with the domain knowledge (Basu et al. 2008). In medicine, for example, clustering could be driven by unwanted bias, such as the type of machine used record the data, rather than more informative features. Moreover, practitioners often have access to prior information about the types of clusters that are sought, and principled method guide the algorithm towards desirable configuration then needed. Therefore. constrained clustering has long history in machine learning as it enforces desirable clustering properties by
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

incorporating domain knowledge, in the form of instance-level constraints Wagstaft & Cardie 2000).
into the clustering objective.
variety of methods have been proposed to extend deterministic deep clustering algorithms, such as DEC (Xie et al., 2016), to force the clustering process to be consistent with given constraints (Ren et al., 2019; Zhang et al. 2019). This results in wide range of empirically motivated loss functions that are rather obscure in their underlying assumptions. Further, they are unable uncover the distribution of the data, preventing them from being extended to other tasks beyond clustering, such as Bayesian model validation, outlier detection, and data generation (Min et al., 2018). Thus, we restrict our search for constrained clustering approach to the class of deep generative models. Although these models have been successfully used in the unsupervised setting (Jiang al.) 2017 Dilokthanakul et al. 2016). their application to constrained clustering has been under-explored.
In this work we propose a novel probabilistic approach constrained clustering, the Deep Conditional Gaussian Mixture Model (DC-GMM), that employs a deep generative model to uncover the underlying data distribution conditioned on domain knowledge, expressed in the form of pairwise constraints. Our model assumes a Conditional Eexture-of-Gausiial prior on the latent representation of the data. That is, a Gaussian Mixture Model conditioned on the user's clustering preferences, based e.g. on domain knowledge. These preferences are expressed as Bayesian prior probabilities with varying degrees of uncertainty. By integrating prior information the generative process of the data, our model can guide the clustering process towards the configuration sought by the practitioners. Following recent advances variational inference (Kingma & Welling 2014 Rezende et al. 2014). we derive scalable and efficient training scheme using amortized inference.
Our main contributions are as follows: (i) We propose new paradigm for constrained clustering (DC-GMM) to incorporate instance-level clustering preferences, with varying degrees of certainty, within the Variational Auto-Encoder (VAE) framework. (ii) We provide a thorough empirical assessment of our model. In particular, we show that (a) a small fraction of prior information remarkably increases the performance of DC-GMM compared to unsupervised variational clustering methods, (b) our model shows superior clustering performance compared to state-of titte--fftheeeee the-hhhtttt art deep constrained clustering models on wide range of data sets and, (c) our model proves to be robust against noise as it can casily incorporate the uncertainty of the given constraints. (iii) Additionally, we demonstrate on two challenging real-world applications that our model can drive the clustering performance towards different desirable configurations, depending on the constraints used.
2 Deep Conditional Gaussian Mixture Model
In the following section, we propose a probabilistic approach to constrained clustering (DC-GMM) that incorporates clustering preferences, with varying degrees of certainty, in VAE-based setting. In particular, we first describe the generative assumptions of the data conditioned on the domain knowledge, for which VaDE (Jiang et al. 2017) and GMM-VVE (Dilokthanakul et al., (016) are special cases. We then define concrete prior formulation to incorporate pairwise constraints and derive new objective, the Conditional ELBO, to train the model in the framework of stochastic gradient variational Bayes. Finally, we discuss the optimization procedure and the computational complexity of the proposed algorithm.
2.1 The Generative Assumptions
W
Let us consider a data set X [rr) consist- ing of N samples with x, E RM that we wish cluster into K groups according to instance-level prior information encoded as W E RNNNN For example, we may know priori that certain sam- ples should (or should not) be clustered together. However, the prior information often comes from different sources with different noise levels. As an example, the instance-level annotations could be obtained from both very experienced domain ex- perts and less experienced users. Hence, W should
Z
μ,
N
Figure 1: The proposed graphical model.

---

encode both our prior knowledge of the data set, expressed in the form of constraints, and its degree of confidence.
We assume the data is generated from a random process consisting of three steps, as depicted in Fig. I
First, the cluster assignments {c; with e, {I,..., K), are sampled from distribution conditioned on the prior information:
c c~ p(c[W;;]]
The prior distribution of the cluster assignments without domain knowledge W, i.c. p(c). follows a categorical distribution with mixing parameters я. Second, for each cluster assignment C; continuous latent embedding, Z; IR is sampled from Gaussian distribution, whose mean and variance depend on the selected cluster Finally, the sample is generated from distribution conditioned on Z,. Given C₁ the generative process can be summarized as:
2, p(z))c)) NZ; [4cc T.=T
x, ~ px( = [N U2,, of 1) real-valued (Ber(aa;)) if binary
(3)
where Me; and o are mean and variance of the Gaussian distribution corresponding to cluster e; the latent space, In the case where real-valued then [Mx,, - fzz; 0). x is binary then Mz. f(;;; 0). The function (=; 0) denotes neural network, called decoder, parametrized by O.
is worth noting that, given W, the cluster assignments are not necessarily independent, i.e. there might be certain j E {l,.... ... N} for which (c, JKC cj (:). This important detail prevents the use of standard optimization procedure
will be explored in the following Sections.
contrary, if there is no prior information, that is ()) = p(c) [I. p(c,) II. (al(c. rT), the cluster assignments are independent and identical distributed. In that particular case, the generative assumptions described above are equal those of Jiang et al. (2017); Dilokthanakul et al. (2016) and the parameters of the model can learned using the unsupervised VaDE method. As a result, VaDE (or RMM-VAED can be seen as special case of our framework.
2.2 Conditional Prior Probability with Pairwise Constraints
We incorporate the clustering preference through the conditional probability p(c W). We focus on pairwise constrains consisting of must-links, if two samples are believed to belong to the same cluster, and cammot- links, otherwise. However, different types of constraints can be included, for example, triple-constraints (see Appendix C
Definition 1. Given a dataset X = ( the pairwise prior information W E NxN is defined as a symmetric matrix containing the pairwise preferences and confidence. In particular
> 0 if there is must-link constraint between x; and xz
= 0 if there is no prior information on samples &; and Wi
O
if there is cannot-link constraint between æ; and Dj
where the value Weal E [0, oo) reflects the degree of certainty in the constraint.
Definition 2. Given the pairwise prior information W, the conditional prior probability p(c W; T) defined as:
he W) P[c TT) EIII, ne (c W) (((T) II ac, W),
1
(4)
where π = [ are the weights associated to each cluster, e; is the cluster assignment of sample x,, ((T)) is the normalization factor and hy W) is a weighting function of the form
h, W)
II exp (Wi
(5)
follows that (e, W) assumes large values if c; agrees with our belief with respect to and low values otherwise. If W., -oo then e; and Φy must be assigned to different clusters otherwise
3

---

P[c[W) + 0 (hard constraint). On the other hand, smaller values indicate soft preference as they admit some degree of freedom in the model. An heuristic select [W, presented in See
The conditional prior probability with pairwise constraints has been successfully used by traditional clustering methods in the past (Lu & Leen 2004), but the best of our knowledge, it has never been applied in the context of deep generative models. It can also be seen as the posterior of the superparamagnetic clustering method (Blatt al. 1996), with loss function given by fully connected Potts model Wu 1982).
2.3 Conditional Evidence Lower Bound
Given the data generative assumptions illustrated in Sec. 2.1 the objective is to infer the parameters of the model, 0, π, and V/ {Mt, o k c= given both the observed data X and the pairwise prior information the cluster assignments W. This could be achieved by maximizing the marginal log-likelihood conditioned on W, that is:
logp X W) = log
Σ 2
p(X,
(6)
where Z = { is the collection of the latent embeddings corresponding to the data set X. The conditional joint probability derived from Eq. and Eq. and can factorized as:
p(X, Z, po (
W π I (2(
Since the conditional log-like ligllikeli intractable, we derive an alternative tractable objective. Definition 3. Lc, the Conditional ELBO (C-ELBO), defined as Co Φ, v, n W := (Z. [log pe (X) -DI Dxc.l///Z.cC/)llzzzcc.....
(8)
with 9o (Z, c(X) being the following amortized mean-field variational distribution:
9 (, c/X) 9o((4/))pclZzz..) II 9olls//.))pcc//..
(9)
The first term of the Conditional ELBO is known as the reconstruction term, similarly to the VAE. The second term, on the other hand, is the Kullback-Leiler (KL) divergence between the variational posterior and the Conditional Gaussian Mixture prior. By maximizing the C-ELBO, the variational posterior mimics the true conditional probability of the latent embeddings and the cluster assignments. This results in enforcing the latent embeddings to follow Gaussian mixture that agrees on clustering preferences.
From Definition 3 and Equations and we can directly derive Lemma Lemma 1. It holds that
1. The C-ELBO Le a lower bound of the marginal log- likelihood conditioned W, that log pl X W) 2 Le Φb X(W).
2. log logp(X(W) Lc(0, Φ0 n, X if and only if gob Z - c,
For the proof we refer the Appendix B does not depend on W. This approximation is used to retain a mean-field variational distribution when the cluster assignments, conditioned on the prior information, are independent (Sec2.1)
is worth noting that in Eq. g the variational distribution
that is when p(c) (c * P(c; W). Additionally, the probability p(c (2;) can be easily computed using the Bayes Theorem, yieldingg
(2, Mage o )Te
p(c. (c.ZZ, Σ. ((z pkx m
(10)
while we define the variational distribution qs (x,) be Gaussian distribution with mean ps and variance (zx)) parametrized by neural network, also known as encoder.

---

2.4 Optimisation & Computational Complexity
The parameters of the generative model and the parameters of the variational distribution are optimised by maximising the C-ELBO. From Definitions 2 and Bl we derive Lemma
Lemma 2. The Conditional ELBO Le factorizes as follow:
(o U, = log ((T) +
Bo (n, x..) logpo(x;|zz)) - log 90 (2/1x)))
1=1
(z)(1//x) (x.)
pkklzz)
log logp(z,1k) + log Te log logl(k))))) (11)
k=
N
Σ
Eng
Σ
pkk p(( W.y
ifj=1
where p(k|z,) pp(c k/2;; M, T) and p p(z. (k) -p(zicc k;
For the complete proof we refer to the Appendix Maximizing Eq. w.r.t π poses computational problems due the normalization factor 52 1.) Crude approximations are investigated in (Basu et al. 2008), however we choose to fix the parameter T 1/ K to make z uniformly distributed in the latent space, as in previous works (Dilokthanakul al. 2016). Hence the normalization factor be treated as a constant. The Conditional ELBO can then be approximated using the SGVB estimator and the reparameterization trick (Kingma & Welling 2014] to be trained efficiently using stochastic gradient descent. We refer to the Appendix [ for the full derivation. We observe that the pairwise prior information only affects the last term, which scans through the dataset twice. To allow for fast iteration we simplify it by allowing the search of pairwise constraints to be performed only inside the considered batch, yielding
B ΣΣΣE'E
Σ
Σ +zj=1k=1
plc: - k/2 >(c; = kl2, Wass
(12)
T
where L denotes the number of Monte Carlo samples and B the batch size. By doing so, the overhead in computational complexity of single joint update of the parameters (LLBKKC)) where Cp the cost of evaluating p(c; = kllz;) with z; E R... The latter is O(KD).
Related Work
Constrained Clustering. A constrained clustering problem differs from the classical clustering scenario as the user has access to some pre-existing knowledge about the desired partition of the data expressed as instance-level constraints (Lange et al. (005). Traditional clustering methods, such as the well-knows K-means algorithm, have been extended to enforce pairwise constraints (Wagstaff)
al. (001). Bilenko et al. (2004)) Several methods also proposed constrained version of the
Gaussian Mixture Models (Shental et al. 2003 Law et al., 2004] (005). Among them, penalized probabilistic clustering (PPC, Lu & Leen 2004 is the most related our work expresses the
pairwise constraints as Bayesian priors over the assignment of data points to clusters, similarly to our model. However, all previous mentioned models shows poor performance and high computational complexity on high-dimensional and large-scale data sets.
Constrained Deep Clustering. To overcome the limitations of the above models, constrained clustering algorithms have lately been used in combination with deep neural networks (DNNs). Hsu Kira (2015) train DNN to minimize the Dullback-Leible (KL) divergence between similar pairs of samples, while Chen 2015) performs semi- -upervised maximum margin clustering on the learned features of DNN. More recently, many extensions of the widely used DEC model (Xie al.) 2016)
have been proposed to include a variety of loss functions to enforce pairwise constraints. Among them, SDEC, (Ren et al. 2019) includes a distance loss function that forces the data points with must-link to be close in the latent space and vice-versa. Constrained IDEC ZZhang et al. 2019). uses a KI. divergence loss instead, extending the work of Shuuka et al. (2018). [Smieja et al. (2020) focuses on discriminative clustering methods by self-generating pairwise constraints from Siamese networks.
5

---

As none of these approaches are based on generative models, the above methods fail to uncover the underlying data distribution.
Deep Generative Models. Although a wide variety of generative models have been proposed in the
literature to perform unsupervised clustering (Li al. 2019 Yang et al. 2019 Manduchi et al. 202T
Jiang ct al. 2017). not much effort has been directed towards extending them to incorporate domain knowledge and clustering preferences. Nevertheless, the inclusion of prior information on growing amount of unlabelled data is of profound practical importance in a wide range of applications (Kingma et al. 2014). The only exception the work of Luo et al. (2018), where the authors proposed the SDCD algorithm, which has remarkably lower clustering performance compared to state-of-the-an constrained clustering models, as we will show in the experiments. Different from our approach, SDCD models the joint distribution of data and pairwise constraints. The authors adopts the two-coin Dawid-Skene model from Raykar et al. (2010) to model p( ([(c), resulting in a different graphical model (see Fig. T. Instead, we consider simpler and more intuitive scenario, where we assume the cluster assignments are conditioned on the prior information, p(c|W;: πT) (see Definition 2 In other words, we assume that different clustering structures might be present within a data set and the domain knowledge should indicate which one is preferred over the other. It is also worth mentioning the work of Tang et al. (2019), where the authors propose to augment the prior distribution of the VAE to take into account the correlations between samples. Contrarily to the DC-GMM, their proposed approach requires additional lower bounds the ELBO cannot be directly applied. Additionally, it also requires post-hoc clustering of the learnt latent space to compute the cluster assignments.
4 Experiments
In the following, we provide a thorough empirical assessment of our proposed method (DC-GMM) with pairwise constraints using a wide range of data sets, First, we evaluate our model's 's performance compared to both the unsupervised variational deep elustering method and state-of-the-art constrained clustering methods. As a next step, we present extensive evidence of the ability of our model to handle noisy constraint information. Additionally, we perform experiments on a challenging real medical data set consisting of pediatric heart ultrasound videos, as well as face image data set to demonstrate that our model can reach different desirable partitions of the data, depending on the constraints used, with real-world, noisy data.
Baselines & Implementation Details. As baselines, we include the traditional pairwise constrained K-means (PCKmeans, Basu et al. (2004)) and two recent deterministic deep constrained clustering methods based on DEC (SDEC, Ren et al. (019), and Constrained IDEC, Zhang et al. (2019) as they achieve state-of-the-art performance in constrained clustering. For simplicity, we will refer to the latter as C-IDEC, We also compare our method to generative models, the semi-supervised SCDC (Lo et al. 2018) and the unsupervised VaDE (Jiang et al.) 2017). To implement were careful in maintaining a fair comparison with the baselines. In particular, we adopted the same encoder and decoder feed-forward architecture used by the basclines: four layers of 500, 500, 2000, D units respectively, where D- = 10 unless stated otherwise. The VAE is pretrained for 10 epochs while the DEC-based baselines need more complex layer-wise pretraining of the autoencoder which involves 50 epochs of pretraining for each layer and 100 epochs of pretraining as finetuning. Each data set is divided into training and test sets, and all the reported results are computed on the latter. We employed the same hyper-parameters for all data sets, see Appendix F.T for details. The pairwise constraints are chosen randomly within the training set by sampling two data points and assigning a must-link if they have the same label and a cannot-link otherwise. Unless stated otherwise, the values of W are set to 10 for all data sets, and 6000 pairwise constraints are used for both our model and the constrained clustering baselines. Note that the total amount of pairwise annotations in a data set of length N is O(V2). Given that N is typically larger than 10000, the number of pairwise constraints used in the experiments represents a small fraction of the total information.
model,
Constrained clustering. We first compare the clustering performance of our model with the baselines on four different standard data sets: MNIST (LeCun et al., 2010), Fashion MNIST (Xiao) jet al., 2017), Reuters (Xie et al., 2016) and STL-10 (Coates et al. (011)) (see Appendix A). More complex data sets will be explored in the following paragraphs. Note that we pre-processed the Reuters data by computing the tf-idf features on the 2000 most frequent words on a random subset of 10 000 documents and by selecting 4 root categories (Xie et al. 2016). Additionally, we extracted

---

Table 1: Clustering performances (%) of our proposed method DC-GMM compared with baselines. All methods use 6000 pairwise constraints except the unsupervised VaDE and the SCDC. Means and standard deviations are computed across 10 runs with different random model initialization. *VaDE results are different from (Jiang et al. 2017) as they only report their best performance. **Results taken from (uo al. 2018).
Dataset
Metric
VaDE*
PCKmeans
SDEC
C-IDEC
SCDD**
DC-GMM (ours)
MNIST
Acc
89.0 #5.0
56.4 12.0
86.2 20.1
96.3 =0.2
84.2
96.6 10.1
NMI
82.8 #3.0
50.5
#1.3
84.2 #0.1
91.8 #1.0
81.2
91.5 #0:2
ARI
80.9 #5.0
38.8 #1.9
80.1 20.1
92.1 =0.4
92.7 #0.3
FASHION Acc
55.1
#2.
53.9 2299
54.0 2.02
68.1 #3.0
80.0 41.0
NMI
57.9 #2.7
50.8 +1.3
57.3 #0.1
66.7 #2.0
71.8 #0.5
ARI
41.6
36.1 #1.7
40.2
#0.1
52.3 #3.0
65.8
REUTERS Acc
76.0 10.7
71.5 12.4
82.1 #0.1
94.7 20.6
95.4 1:022
NMI
50.1 #1.3
48.2 +3.8
62.3 #0.1
81.4
#0.7
82.7 #0.7
ARI
58.0 #1.4
46.5 44.22
66.7 #0.1
87.7 #0.9
89.0 #0.8
STL-10
Acc
77.3 20.5
70.3 442
79.2 20.1
81.6 23.8
89.5 10.5
NMI
70.6 20.4
71.6 #1.3
78.6 20.1
77.3
#1.7
80.2 10.7
ARI
62.7 #0.4
58.4 42.1
71.0 #0.1
71.8 #3.4
78.4 4:0.9
features from the STL-10 image data set using a ResNet-50 (He et al. 2016 as in previous works (Jiang al. 2017). Accuracy, Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI) are used as evaluation metrics. In Table | we report the mean and standard deviation of the clustering performance across 10 runs of both method and the baselines. The only exception the SDCD, for which we only report their original results (Luo et al, 2018) computed with a higher number of constraints. The provided code with 6000 constraints produced highly unstable and sub-optimal results (see Appendix D).
We observe that our model reaches state-of-the-arr clustering performance in almost all metrics and data sets. As C-IDEC turns out to be the strongest baseline, we performed additional comparison to investigate the difference in performance under different settings. In Figure 2 we plot the clustering performance in terms of ARI using a varying number of constraints, Ne. For additional metrics, we refer the Appendix E.T We observe that our method outperforms the strongest baseline C-IDEC by a large margin on all four data sets when fewer constraints are used. When N₂ gets close to N, the two methods tend to saturate on the Reuters and STL data sets.
0.35
0.255
3.65
0340
0.55
0.35
GMM
035
DC-
0.70
DC
DC
0.23
CIDEC
350
CIIDEC
62
da 14 n.
0. ss 10
8. 64 as
(a) MNIST
(b) Fashion MNIST
(c) Reuters
(d) STL10
Figure 2: Comparison of clustering performance of our proposed DC-GMM model and the strongest baseline C-IDEC on four different data sets. The number of constraints vary between 0.1 x N and N, where N the length of the data set. ART is used as evaluation metric.
Constrained clustering with noisy labels. In real-world applications it is often the case that the additional information comes from different sources with different confidence levels. Hence, the ability to integrate constraints with different degrees of certainty into the clustering algorithm is of significant practical importance. this experiment, we consider the case in which the given pairwise constraints have three different noise levels, q {0.1, 0.2, 0.3). where q determines the fraction
7

---

of pairwise constraints with flipped signs (that is, when must-link is turned into cannot-ling link and vice-versa). In Fig. B we show the ARI clustering performance of our model compared the strongest baseline derived from the previous section, C-IDEC. For all data sets, we decrease the value
of the pairwise confidence of our method using the heuristic [Wayl
log ( with a = 1000.
Also, we use grid search to choose the hyper-paramettrs of C-IDEC for the different noise levels (in particular we set the penalty weight of their loss function to 0.01, 0.005, and 0.001 respectively). Additionally, report Accuracy and NMI in Appendix B. DC-GMM clearly achieves better performance on all three noise levels for all data sets. In particular, the higher the noise level, the greater the difference in performance. We conclude that our model is more robust than its main competitor on noisy labels and it can easily include different sources of information with different degrees of uncertainty.
1.40
0.55
1.0
0.50
DC GMM
DC-GMM
DC. DC-GHM
DC-GMM
3.00
C-IDEC
C-IDEC
0.5
C-DEC
0.85
C-IDEC
3.50
0.65
0.7
9.05
0.65
0.55
0.6
0.85
0:50
0.00
0.5
Noise level
Noise level
Noise level
Noise level
(a) MNIST
(b) Fashion MNIST
(c) Reuters
(d) STL10
Figure 3: Comparison of clustering performance of our proposed DC-GMM model and the strongest
baseline C-IDEC on four different data sets with noisy labels. ARI is used as evaluation metric.
Heart Echo. We evaluate the capability of our model in real-world application by using data set consisting of 305 infant echo cardiogram videos obtained from the Hospital Barmherzige Brüder Regensburg. The videos taken from five different angles (called views), denoted by [LA, KAKL, KAPAP, KAAP, 4CV]. Preprocessing of the data includes cropping the videos, resizing them to 64 x 64 pixels and splitting them into total of 20000 individual frames. We investigated two different constrained clustering settings. First, we cluster the echo video frames by view (Zhang et al. 2018). Then, cluster the echo video frames by infant maturity birth, following the WHO definition of premature birth categories ("Preterm") "). We believe that these two clustering tasks demonstrate that our model admits degree of control in choosing the underlying structure of the learned clusters. For both experiments, we compare the performance of our method with the unsupervised VaDE and with C-IDEC. Additionally, we include a variant of both our method and VaDE in which we use VGG-like convolutional neural network (Simonyan & Zisserman 2015) for details on the implementation we refer the Appendix F2 The results are shown in Table The DC-GMM outperforms both baselines by significant margin in accuracy, NMI, and ARI, as well as in both clustering experiments. We also observe that C-IDEC performs poorly on real-world noisy data. We believe this is due to the heavy pretraining of the autoencoder, required by DEC-based methods, it
Table 2: Clustering performance (%) using the heart echo cardiogram data with fully connected layers on the left and convolutional layers (CNN-) on the right. All methods use 6000 pairwise constraints except the VaDE. Mcans and standard deviations computed across 10 runs with different random model initialization.
Clustering
Metric
VaDE
C-IDEC
DC-GMM
CNN-VaDE
CNN-DC-GMM
View
Acc
33.4 13.3
55.1 116.0
83.2 114
41.7 15.2
92.5 11.4
NMI
8.9 12.7
33.3 115.9
64.9 12.3
19.7 47.4
82.6 43.0
ARI
6.5 #2.5
31.2 416.1
63.7 42.7
13.7 46.3
83.1 43.1
Preterm
Acc
41.4 45.4
69.6 #1.1
72.3 11.5
36.7 42.5
73.3 11.1
NMI
6.4 21.
8.3 211.8
25.1 23.4
6.5 24.0
32.0 42.4
ARI
2.3 42.6
13.7 4:1.3
45.1 44.1
3.9 43.8
48.1 42.8
8

---

does not always lead to a learned latent space that is suitable for the clustering task. Additionally, we illustrate a PCA decomposition of the embedded space learned by both the DC-GMM and the unsupervised VaDE baseline for both tasks in Figure 4 Our method is clearly able to learn an embedded space that clusters both the different views and the different preterms more effectively than the unsupervised VaDE. This observation, together with the quantitative results, demonstrates that adding domain knowledge is particularly effective for medical purposes.
Looa
(a) DC-GMM: Acc 0.925
(b) VaDE: Aec 0.417
(c) DC-GMM: Ace 0.733
(d) VaDE: Acc 0.367
Figure 4: PCA decomposition of heart ultrasound imaging test set examples in the embedded space and generative samples using the unsupervised VaDE and our model DC-GMM with 6000 constraints for (a)-(b) View, (c)-(d) Preterm.
Face Images. We further evaluate the performance of our model using the UTKFace data set Zhang al., 2017). This data set contains over 20000 images of male and female faces, aged from I to 118 years old, with multiple ethnicities represented. We use VGG nets (Simonyan & Zisserman 2015) for the VAE (the implementation details are described in the Appendix F.3 As in the Heart
Echo experiment, we chose two different clustering tasks. First we cluster the data using the gender prior information, then we select a sub- uub-sample of individuals between 18 and 50 years of age (approx. 11000 samples) and cluster by ethnicity (White, Black, Indian, Asian) In Fig. 5 we illustrate the PCA decomposition of the embedded space learned by the VaDE and our model. For both tasks we use 2N pairwise constraints where N is the length of the data set, which requires labels for 1.5% of the entire data set. Specifically, on the gender task and the ethnicity task our model achieves an accuracy of 0.89 and 0.85, which outperforms VaDE with relative increase ratio of 74.5% and 49.1%. In terms of NMI, the unsupervised VaDE performance is close to in both tasks, while our model performance is 0.52 and 0.54 respectively. Visually, we observe neat division of the selected clusters in the embedding space with the inclusion of domain knowledge. The unsupervised approach is not able to distinguish any feature of interest. We conclude that it is indeed possible guide the clustering process towards preferred configuration, depending on what the practitioners are seeking in the data, by providing different pairwise constraints. Finally, we tested the generative capabilities of our model by sampling from the learnt generative process of Sec For a visualization of the generated sample refer the Appendix
F
Label
Label
Female
Block
White
Asian
Indian
(a) DC-GMM: Acc 0.89
(b) VaDE: Acc 0.51
(c) DC-GMM: Acc 0.85
(d) VaDE: Acc 0.57
Figure 5: PCA decomposition of test set examples in the embedded space and generative samples using VaDE and DC-GMM for (a)-() Gender, (c)-(d) Ethnicity. In this configuration, DC-GMM obtains a NMI of 0.52 (gender) and 0.58 (ethnicity) while VaDE obtains a NMI close to 0 for both tasks.

---

Conclusion
In this work, we present a novel constrained deep clustering method called DC-GMM, that incor- porates clustering preferences in the form of pairwise constraints, with varying degrees of certainty. In contrast to existing deep clustering approaches, DC-GMM uncovers the underlying distribution of the data conditioned prior clustering preferences. With the integration of domain knowledge, we show that our model can drive the clustering algorithm towards the partitions of the data sought by the practitioners, achieving sfate-of-the-ar constrained clustering performance in real-world and complex data sets, Additionally, our model proves to be robust to noisy constraints as it can efficiently include uncertainty into the clustering preferences. As result, the proposed model can be applied to a variety of applications where the difficulty of obtaining labeled data prevents the use of fully supervised algorithms.
Limitations & Future Work The proposed algorithm requires that the mixing parameters of the clusters are chosen a priori. This limitation is mitigated by the prior information W. which permits more flexible prior distribution if enough information is available (see Definition 2 The analysis of different approaches to learn the weights и represents a potential direction for future work. Additionally, the proposed framework could also be used in a self-supervised manner, by learning W from the data using, e.g. contrastive learning approaches (Chen et al. 2020
Wu et al. 2018
6 Code and Data Availability
The code is available in GitHub repository: datasets are publicly available except the Heart Echo data. The latter is not available due to medical confidentiality.
All
Acknowledgments and Disclosure of Funding
We would like to thank Luca Corinzia (ETH Zurich) for the fruitful discussions that contributed shape this work. We would also like thank Vincent Fortuin (ETH Zurich), Thomas M. Sutter (ETH Zurich), Imant Daunhawer (ETH Zurich), and Ričards Marcinkevičs (ETH Zurich) for their helpful comments and suggestions. This project has received funding from the PHRT SHFN grant # #1-000018-0577 SWISSHEART.
References
Basu, S., Banerjee, A., and Mooney, R. Active semi-supervision for pairwise constrained clustering. In SDM, 2004.
Basu, S., Davidson, I., and Wagstaff, K. Constrained clustering: Advances in algorithms, theory, and applications. 2008
Bilenko, M., Basu, S., and Mooney, R. Integrating constraints and metric learning in sumi-supervised clustering. In /CML '04, 2004.
Blatt, Wiseman, and Domany. Superparamagnetic clustering of data. Physical review letters, 76 18: 3251-3254, 1996.
Chen, G. Deep transductive sumi-super maximum margin clustering. ArXiv, 0ass/1501.062377 2015.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, simple framework for contrastive learning of visual representations. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 Proceedings of Machine Learning Research, pp- 1597-1607. PMLR, 13-18 Jul 2020.
Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
10

---

Dilokthanakul, N., Mediano, P. A. M.. Garnelo, M., Lee, M. C. H., Salimbeni, H., Arulkumaran, K., and Shanahan, M. Deep unsupervised clustering with gaussian mixture variational autoencoders. CoRR, abs/16/ 12//61115 1.02448, 2016.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Hsu, Y.-. and Kira, Z. Neural network-based clustering using pairwise constraints. CoRR, abs/1511.063211 2015.
Jiang, Z., Zheng, Y., Tan, H.., Tang, B., and Zhou, H. Variational deep embedding: An unsupervised and generative approach to clustering. In ICCAI, 2017.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In Bengio, Y. and LeCun, Y. (eds.), 2nd International Conference on Learning Representations, ICLR, 2014.
Kingma, D. P.. Mohamed. S., Rezende. D. J., and Welling, M. Semi-supervised learning with deep generative models. In NIPS, 2014.
Lange, T., Law, M. H. C., Jain, A. K., and Buhmann, J. Learning with constrained and unlabelled data. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), 1:731-738 vol. 1, 2005.
Law, M. H. C., Topchy, A., and Jain, A. K. Clustering with soft and group constraints. In SSPR/SPR, 2004.
Law, M. H. C., Topchy, A., and Jain, A. K. Model-based clustering with probabilistic constraints. In SDM, 2005.
LeCun, Y., Cortes, C., and Burges, C. Mnist handwritten digit database. ATT Labs Availal: 2, 2010.
Available:
Lewis, D. D., Yang. Y., Rose, T. G., and Li, Revl: A new benchmark collection for text categorization research. J. Mach. Learn. Res., 5:361-397, December 2004. ISSN 1532-4435.
Li, X., Chen, Z., Poon, L. K. M., and Zhang, N. L. Learning latent superstructures in variational autoencoders for deep multidimensional clustering. In ICLR, 2019.
Lu, Z. and Leen, T. Semi-supervised learning with penalized probabilistic clustering, In NIPS, 2004.
Luo, Y., TIAN, T., Shi, J.. Zhu, J., and Zhang, B. Semi-crowdsourced clustering with deep generative models. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 3212-3222 Curran Associates,
Inc., 2018
Manduchi, L., Hüser, M., Faltys, M., Vogt, J., Rätsch, G., and Fortuin, V. T-dpsom: An interpretable clustering method for unsupervised learning of patient health states. In Proceedings of the Conference on Health, Inference, and Learning. CHIL '21, pp. 236-245, New York, NY, USA, 2021. Association for Computing Machinery.
Min, E., Guo, X., Liu, Q., Zhang, G., Cui, J., and Long, J. A survey of clustering with deep learning: From the perspective of network architecture. IEEE Access, 6:39501-39514 2018.
Raykar, V. C., Yu, S., Zhao, H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L. Learning from crowds. Journal of Machine Learning Research, 1 1443): 2010.
Ren, Y., Hu, K., Dai, X., Pan, L., Hoi, S. C. H., and Xu, Z. Semi-supervised deep embedded clustering. Neurocomputing, 325:121-130, 2019.
Rezende, D. Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. ICML, 2014.
Shental, N., Hir-Hillel, A., Hertz, T., and Weinshall, D. Computing gaussian mixture models with using equivalence constraints. In NIPS, 2003.

---

Shukla, A., Cheema, G. S., and Anand, S. Semi-supervised clustering with neural networks. arXiv: Learning, 2018.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409 1556, 2015.
Smieja, M., Struski, L.., and Figueiredo, M. A. T. A classification-basel approach to semi-supervised clustering with pairwise constraints. Neural networks the official journal of the International Neural Network Society, 127:193-203. 2020.
Tang, D., Liang, D., Jebara, T., and Ruozzi, N. Correlated variational auto-encoders, ICML, 2019.
Wagstaff, K. and Cardie, Clustering with instance-level constraints. In AAAI/IAAI, 2000.
Wagstaff, K., Cardie, C., Rogers, S., and Schrödl, Constrained k-means clustering with background knowledge. In ICML, 2001.
Wu, F. Y. The potts model. Rev. Mod. Phys., 54:235-268, Jan 1982. doi: 10.1 10.110//rvMo// 235. https::::://inn aps org/ 11t..0///1/1//////...eee.e. 54. 235
Wu, Z., Xiong, Y. Yu, S., and Lin, D. Unsupervised feature learning via non-parametric instance discrimination. 2018 IEFE/CVE Conference Computer Vision and Pattern Recognition, pp- 3733-3742, 2018.
Xiao, H., Rasul, K., and Vollgraf, R. fushion-mniste novel image dataset for benchmarking machine learning algorithms, 2017.
Xie, J., Girshick, R., and Farhadi, A. Unsupervised deep embedding for clustering analysis. volume 48 of Proceedings of Machine Learning Research, pp. 478-487, New York, New York, USA, 20-22 Jun 2016. PMLR.
Yang, L., Cheung, N., Li, and Fang. J. Deep clustering by gaussian mixture variational autoencoders with graph embedding. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 6439-6448, 2019.
Zhang, H., Basu, S., and Davidson, I. A framework for deep constrained clustering algorithms and advances. In ECML/PKDD, 2019.
Zhang, J., Gajjala, S., Agrawal, P., Tison, G., Hallock, I.., Beussink, L.,, Lassen, M., Fan, E.,, Aras, M., Jordan, C., Fleischmann, K., Melisko, M., Qasim, A., Shah, S., Bajesy, R., and Deo, R. Fully automated echocardiogram interpretation in clinical practice: Feasibility and diagnostic accuracy. Circulation, 138: 138:1623-166 10 2018. doi: 10. COL.IG//CRCCLLATTONAAAAA.....
Zhang, Z., Song. Y.. and Qi, H. Age progressionnregressinn by conditional adversarial autoencoder. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2017.
12


Reviews:
[{"review_id": "JXhn-nnHj0j", "reviewer": "Reviewer_yJfq", "summary": "This paper provides a new probabilistic constrained clustering algorithm, Deep Conditional Gaussian Mixture Model (DC-GMM).\nDC-GMM is based on VaDE or GMM-VAE, but DC-GMM adopts a new prior to incorporate the prior information, constrained clustering. They provide the prior formulation in definition 2, and its corresponding evidence lower bound term in Eq. (11).\nDC-GMM shows better performance on MNIST, Fashion, Reuters, STL-10 datasets. Furthermore, they compare their methods and baselines on realistic settings, constrained clustering with noisy labels.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "< strength >\n1. Constrained clustering is an important but not well-explored research area. This paper contributes an important research area, constrained clustering.\n2. The proposed methods are so simple, and they can be adapted to many models.\n3. They provide realistic experiments, realistic settings (noisy labels), and a realistic dataset (Heart Echo).\n\n< weakness >\n1. Table1 shows that the DC-GMM performs better than other methods. However, the intuitive description or reason is not well explained.\n2. SCDC is one of the most important baselines, but there is no reported value for Fashion, Reuters, and STL-10 in Table 1\n3. This paper focuses on probabilistic clustering to utilized the learned representation to the other tasks. However, this paper only performs clustering tasks.\n\n< question >\n1. There can be many candidate ways to construct prior p(c|W;\\pi).\nWhat are the intuition and the advantage of prior modeling in Eq. (4) instead of other candidate ways? \n2. The value of SCDC in Table1 is taken from the original paper. However, the experiment settings of this paper and the SCDC paper are something different. Is it a fair evaluation? \n3. There are other directions to incorporate relation information into the model [1,2,3,4]. I suggest that the authors include the discussion and experimental comparison. \n4. There are recent works for constrained clustering (even though the methods do not focus on deep learning).\nIt will be interesting that DC-GMM performs better than the combination of recent advanced VAE + recent advanced constrained clustering algorithm\n \n\n< reference >\n1. Xu, Hongteng, et al. \"Learning autoencoders with relational regularization.\" International Conference on Machine Learning. PMLR, 2020.\n2. Mylonas, Charilaos, Imad Abdallah, and Eleni Chatzi. \"Relational VAE: A Continuous Latent Variable Model for Graph Structured Data.\" arXiv preprint arXiv:2106.16049 (2021).\n3. Bai, Liang, JiYe Liang, and Fuyuan Cao. \"Semi-supervised clustering with constraints of different types from multiple information sources.\" IEEE transactions on pattern analysis and machine intelligence (2020).\n4. Tang, Da, et al. \"Correlated variational auto-encoders.\" International Conference on Machine Learning. PMLR, 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations of the paper.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "5NTwcDXPhuk", "reviewer": "Reviewer_2qnh", "summary": "This paper developed a new generative clustering method. Compared with existing methods, this paper incorporates the pairwise prior knowledge, and then derived ELBO as the objective function to learn model parameters. The authors then conduct experiments to verify the performance of the proposed method. ", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper developed a new generative clustering method. Compared with existing methods, this paper incorporates the pairwise prior knowledge, and then derived ELBO as the objective function to learn model parameters. The authors then conduct experiments to verify the performance of the proposed method. \n\nPros:\n1. This paper is well written and easy to follow. \n\n2. The authors conduct extensive experiments to verify the performance of the proposed paper. \n\n3. The problem that incorporates the prior knowledge is important.\n\nCons:\n1. The novelty is limited. In fact, incorporating the pairwise prior knowledge is trivial here and it is easy to derive the lower bound by following existing works.  From Eq.(11-12), it can be seen that the difference between this method and existing methods is the weight W. If W=I, then it is exactly VAE.\n\nThe authors should clearly state the challenge when incorporating the pairwise prior knowledge. \n\n2. The prior knowledge in this paper is fixed. Hence, the contribution is incremental. A more practical choice is to assume that W is a random variable. But this paper didn't study this general scenario and therefore the contribution is limited. \n\n3. This paper missed an important related work [1], which also studied the pairwise prior knowledge.\n\n[1]. Correlated Variational Auto-Encoders\n\nDa Tang, Dawen Liang, Tony Jebara, Nicholas Ruozzi; Proceedings of the 36th International Conference on Machine Learning, PMLR 97:6135-6144 \n\n2. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4qBADYKeG6_", "reviewer": "Reviewer_5L8k", "summary": "The paper introduces a new method for constrained clustering with varying degree of certainty. The framework is based on the variational auto-encoder. The numerical experiments are extensive. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths. The paper is well and clearly written. The numerical experiments are well conducted. The technical details are provided. \n\nWeaknesses. Although I think that the contribution is interesting and important, my major concern is the setting of the considered problem. The matrix W that reflects the prior knowledge of the connections between data points and also degree of confidence, is in some sense, the direct indicator on how the points should be clustered. It seems that W turns, in some sense, the problem of clustering into a classification problem, since the information provided by W is very strong and rich. Probably this explains such a good empirical performance? \n\nMy next question is very related to my previous concern. If a human expert who provides W, indicated which points are closely related and which are not, what is the principled difference if the expert provides directly class labels? I would appreciate some examples of real applications showing that the matrix W provides significantly different information from supervised learning.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "3KaOsZrV6sf", "reviewer": "Reviewer_2A4z", "summary": "The paper proposes a method for *constrained clustering* (a semi-supervised setting in which pairs of samples are labeled as belonging to the same or to different clusters). The method is based on learning a GMM prior for the latent space of a VAE where the cluster assignments depend on the pairwise constrains. The authors develop a conditional version of the ELBO for the constrained case and show how it can be optimized using SGVB and reparameterization, like in regular VAE. The proposed method (DC-GMM) is shown to achieve best clustering results for six different datasets (e.g. MNIST where 6000 random training pairs where used as pairwise constraints) compared to recent *constrained clustering* methods and to be more robust to label noise and to small amount of supervision. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** As the authors indicate, the idea of learning a GMM as the prior distribution of the VAE latent space is not by itself new (an additional example of deep clustering with a GMM prior is [1] below). The setting of a GMM latent prior conditional on the pairwise constraints matrix and the proposed optimization method seems to be novel and interesting.  \n\n**Clarity:** The paper is very well written - the motivation is well explained, the setting is clearly described as a graphical model and the method is clearly and accurately explained. \n\n**Quality:** The proposed method seems to be technically sound. The experiments are thoroughly conducted and support the claims well. Keeping the number of constrains fixed at 6000 seemed a bit strange to me, but the authors have one result (figure 2) showing the affect of varying the number of constraints. \n\n**Significance:** Overall, the paper seems to be an important contribution to the constrained clustering research.\n\nThe fact the that mixture probability distribution $\\pi$ is fixed at $1/K$ limits, in my opinion, the usability of the proposed method. The datasets used in the experiments section all have more or less equal cluster probabilities, but in real cases there might be a strong imbalance.\n\nAlthough, as the authors describe, the size of the supervision is only a small fraction of all possible pairs, it is still significant both practically (manually labeling 6000 pairs) and theoretically - for example, if the pairwise constraints define chains of samples that must reside in the same clusters (e.g. $(x_1, x_2), (x_2, x_3), ...$), then $N$ constrains are sufficient to completely define the clusters. A large number of pairwise constrains, however, seems to be the typical case in the *constrained clustering* literature. \n\nMinor comments:\n- It took me a while to understand why the cluster assignments random variable $c$ in the PGM in figure 1 is not part of the plate (I think because individual assignments are not iid). A short explanation can help.\n\n\n[1] Multi-Modal Deep Clustering: Unsupervised Partitioning of Images, Shiran and Weinshall 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the main limitations of their method. I'm not aware of any potential negative social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviews have many positive comments about the proposed approach to incorporating pairwise constraints into prior work on VAE-based clustering, which allows for fairly simple inference but nevertheless has good empirical performance.  There are some concerns about significance of the results, but agreement that technical details are novel.  Some reviewers raised questions about conceptual comparisons to related work, that were reasonably addressed by the author response; please be sure to include these comparisons in future revisions. 
Decision: Accept (Poster)